ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation

# ACAT: A Collaborative Platform for Efficient Aspect-Based Sentiment Dataset Annotation
Source: [https://arxiv.org/html/2606.04189](https://arxiv.org/html/2606.04189)
11institutetext:1National University of Science and Technology POLITEHNICA Bucharest, Romania
2Academy of Romanian Scientists, Ilfov 3, Bucharest, 050044, Romania11email:\{ana\_maria\.mogoase, ciprian\.truica, elena\.apostol\}@upb\.ro
###### Abstract

Aspect\-Based Sentiment Analysis \(ABSA\) requires high\-quality datasets to train reliable models\. However, existing annotation tools treat output as flat files, leaving researchers to manually consolidate multi\-annotator data, reconstruct relational structures, and compute reliability metrics through custom scripts\. This paper introduces ACAT \(Aspect\-based sentiment analysisCollaborativeAnnotationTool\), a web\-based platform natively supporting four ABSA workflows: \(1\) Aspect\-Category Sentiment Analysis, \(2\) Clause\-Level Segmentation, \(3\) Aspect\-Term Sentiment Analysis with character\-level position tracking, and \(4\) Aspect Sentiment Triplet Extraction with dual span offset preservation\. Its core contribution is an automated Extract, Transform, Load \(ETL\) pipeline that aligns collaborative annotations and computes Inter\-Annotator Agreement \(IAA\) metrics directly at export, yielding training\-ready datasets\. In a preliminary validation on1,0021\{,\}002restaurant reviews with two annotators of differing expertise, ACAT achieves a median annotation time of31\.5831\.58seconds and a raw IAA ranging from0\.780\.78to0\.860\.86across all tasks\.

## 1Introduction

Sentiment Analysis has evolved from document\-level polarity classification to Aspect\-Based Sentiment Analysis \(ABSA\)\[[14](https://arxiv.org/html/2606.04189#bib.bib14),[2](https://arxiv.org/html/2606.04189#bib.bib2)\], encompassing multiple advanced tasks\. Although Large Language Models \(LLMs\) have improved, human\-annotated*gold standard*datasets remain essential for reliable evaluation\[[1](https://arxiv.org/html/2606.04189#bib.bib1)\]\. Yet the data management ecosystem for ABSA datasets remains fragmented: generic tools treat annotation as flat files, leaving data integration, multi\-annotator merging, and IAA computation as the researcher’s burden\.

This paper introducesACAT, addressing the following research questions:\(RQ1\)How can the platform support ABSA knowledge extraction without complex pre\-configurations?\(RQ2\)What architectural solutions eliminate the data consolidation bottleneck in collaborative environments?\(RQ3\)How can the annotation schema capture implicit or latent semantics usually lost in standard tools?\(RQ4\)How can IAA be validated automatically without external post\-processing scripts?\(RQ5\)Why are direct one\-to\-one metric comparisons between existing solutions unreliable?

ACAT is deployed as a Docker container \(PostgreSQL, Python Flask, Vanilla JavaScript\)\. We validate it on1,0021\{,\}002restaurant reviews, achieving a median annotation time of 31\.58 seconds and raw IAA up to 0\.86\. The contributions are: \(C1\) Unified Multi\-Task Knowledge Extraction\.Native support for four ABSA architectures without pre\-annotation configuration: ACSA\[[14](https://arxiv.org/html/2606.04189#bib.bib14)\], Clause\-Level\[[8](https://arxiv.org/html/2606.04189#bib.bib8)\], ATSA\[[14](https://arxiv.org/html/2606.04189#bib.bib14)\], and ASTE\[[11](https://arxiv.org/html/2606.04189#bib.bib11)\]\. \(C2\) Automated ETL and Data Consolidation\.Row\-level alignment of multi\-user data with natively computed IAA metrics \(Cohen’s Kappa\[[3](https://arxiv.org/html/2606.04189#bib.bib3)\], Fleiss’ Kappa\[[5](https://arxiv.org/html/2606.04189#bib.bib5)\], Macro F1\[[19](https://arxiv.org/html/2606.04189#bib.bib19)\]\)\. \(C3\) Implicit Semantic Modeling\.An Implicit Toggle captures latent semantics when explicit target terms are absent\[[17](https://arxiv.org/html/2606.04189#bib.bib17)\]\. \(C4\) Data Governance\.A high\-speed double\-click interaction model with built\-in time tracking and hierarchical supervision\.

## 2Related Work

While ABSA algorithms advance rapidly, annotation infrastructure remains underdeveloped\[[4](https://arxiv.org/html/2606.04189#bib.bib4)\]\. General\-purpose tools such as Doccano\[[9](https://arxiv.org/html/2606.04189#bib.bib9)\], LightTag\[[12](https://arxiv.org/html/2606.04189#bib.bib12)\], and Label Studio\[[18](https://arxiv.org/html/2606.04189#bib.bib18)\]handle flat sequence labeling but export nested JSONs requiring custom scripts for ABSA task reconstruction and multi\-annotator consolidation\. Heavyweight environments like BRAT\[[16](https://arxiv.org/html/2606.04189#bib.bib16)\]support relational tasks and offer built\-in curation modules, but impose click\-heavy graph\-drawing that introduce cognitive load\. Offline tools such as YEDDA\[[20](https://arxiv.org/html/2606.04189#bib.bib20)\]and ASQE\-DPT\[[6](https://arxiv.org/html/2606.04189#bib.bib6)\]optimise local speed but lack centralized collaboration architectures, leading many ABSA datasets to rely on bespoke single\-use pipelines\[[15](https://arxiv.org/html/2606.04189#bib.bib15)\]\. Finally, capturing implicit aspects remains challenging\[[7](https://arxiv.org/html/2606.04189#bib.bib7),[17](https://arxiv.org/html/2606.04189#bib.bib17)\]: while neural models encode implicit meaning\[[7](https://arxiv.org/html/2606.04189#bib.bib7)\], traditional tools restrict annotations to visible text spans\. ACAT addresses all these gaps with native ABSA support, automated ETL, an implicit toggle, and a streamlined double\-click interaction\.

## 3Annotation Structure and Export

ACAT supports four annotation granularity levels, each serialised as coordinate\-based strings for storage and transformed into CSV, JSON, and XML at export\. We employ a discrete 3\-class polarity systemP=\{Positive,Negative,Neutral\}P=\\\{\\textsc\{Positive\},\\textsc\{Negative\},\\allowbreak\\textsc\{Neutral\}\\\}\[[10](https://arxiv.org/html/2606.04189#bib.bib10),[13](https://arxiv.org/html/2606.04189#bib.bib13)\]\. Figure[1](https://arxiv.org/html/2606.04189#S3.F1)illustrates all tasks using our running example\.111The export example is available at[https://ronlp\-clarin\.upb\.ro/acat\.html](https://ronlp-clarin.upb.ro/acat.html)\.

The export logic composes four linguistic variables, Aspect Category \(aa\), Sentiment Polarity \(ss\), Aspect Term \(tt\), and Opinion Word \(oo\), into task\-specific outputs:ACSA & Clause\-Level \(a\+sa\{\+\}s\)maps latent categories to polarity labels;ATSA \(t\+st\{\+\}s\)combines explicit terms with character\-level offsets;ASTE \(t\+o\+st\{\+\}o\{\+\}s\)links aspect and opinion spans, preserving dual offset pairs\.

ACSAserialises annotations ascategory:polarity:implicit\. TheImplicit Toggle\(C3\) extends the category\-polarity pair to a formal triplet\(c,p,i\)\(c,p,i\),i∈\{0,1\}i\\in\\\{0,1\\\}, flagging cases where a category is discussed without an explicit surface term\[[14](https://arxiv.org/html/2606.04189#bib.bib14)\]\. ACAT supports an open\-world paradigm: annotators can dynamically extend the aspect taxonomy during active sessions\.

Clause\-Level Sentiment Analysissegments complex sentences into Elementary Discourse Units \(EDUs\), minimal, clause\-level text spans each expressing a single sentiment, via an interactive splitter, resolving the mixed\-sentiment problem\. Clauses are delimited by\|\|in CSV; JSON uses aclausearray pairing each segment with atagsarray\.

ATSAcaptures character\-level offsets from the annotator’s text selection, e\.g\.burgers\[4,11\]:positive;fries\[32,37\]:neutral\[[2](https://arxiv.org/html/2606.04189#bib.bib2)\]\.

ASTEextracts the complete relational triplet\(t,o,s\)\(t,o,s\)\[[11](https://arxiv.org/html/2606.04189#bib.bib11)\], serialising dual offset pairs, e\.g\.burgers\[4,11\]:top\-notch\[21,30\]:positive;fries\[32,37\] :ok\[43,45\]:neutral\. Duplicate entries are automatically detected and rejected\.

![Refer to caption](https://arxiv.org/html/2606.04189v1/x1.png)Figure 1:ACAT export architecture\. Linguistic variables\(a,s,t,o\)\(a,s,t,o\)are composed into four tasks of increasing granularity \(top\), with their CSV, JSON, and XML serialisations shown below, including character\-level offsets and the implicit flag\.Collaborative ETL Pipeline\.During export, ACAT performs automatic row\-level alignment, dynamically joining dataset records with user\-specific annotation layers and resolving sparse matrices where annotators were assigned disjoint subsets\. LetR=\{r1,…,rn\}R=\\\{r\_\{1\},\\ldots,r\_\{n\}\\\}be the review instances andU=\{u1,…,um\}U=\\\{u\_\{1\},\\ldots,u\_\{m\}\\\}the annotators\. The pipeline generates a unified feature matrixMMwhereMi,jM\_\{i,j\}contains the annotation byuju\_\{j\}forrir\_\{i\}\. Three IAA metrics are computed via scikit\-learn and statsmodels and embedded in all export formats \(answering RQ4\)\. Cohen’s Kappa\[[3](https://arxiv.org/html/2606.04189#bib.bib3)\]is applied for two\-annotator datasets; Fleiss’ Kappa\[[5](https://arxiv.org/html/2606.04189#bib.bib5)\]is automatically selected for three or more annotators; Macro F1\[[19](https://arxiv.org/html/2606.04189#bib.bib19)\]accounts for class imbalance across all sub\-tasks\.

## 4Analysis and Discussions

Two annotators each processedN=100N\{=\}100reviews per task, sampled from a corpus of1,0021\{,\}002restaurant reviews\. ACAT’s dataset\-sharing ensured identical review sets per task\. One annotator had prior ABSA experience \(A1\), while the other was a novice \(A2\), a deliberate choice to assess platform intuitiveness across expertise levels\.

Table[1](https://arxiv.org/html/2606.04189#S4.T1)summarises temporal and agreement results\. The overall median annotation time is 31\.58 seconds \(mean 37\.77 s\)\. A1 completed 58\-67% of reviews within 30 seconds; A2 completed 32\-37%\. Overall, 49% of annotations were completed within 30 seconds, suggesting that the double\-click interaction model reduces friction for typical inputs\. The long tail \(maximum 164 s, standard deviation up to 30\.35 s\) reflects the complexity of dense reviews requiring open\-world aspect extension\.

Table 1:Temporal performance \(seconds\) and inter\-annotator agreement metrics\.Agr\.is Raw Agreement \(% of identical labels between annotators\)\.Eff\.is efficiency \(% of reviews annotated in≤30\\leq 30seconds\)\.Raw agreement ranges from 0\.78 \(ASTE\) to 0\.86 \(ACSA\), reflecting task complexity\. Cohen’s Kappa indicates moderate\-to\-substantial agreement, peaking atκ=0\.65\\kappa\{=\}0\.65for Clause\-Level, where A2 had adapted after completing ACSA first\. ASTE yields the lowest agreement \(κ=0\.52\\kappa\{=\}0\.52, F1=0\.50\\,\{=\}\\,0\.50\), reflecting the inherent difficulty of extracting complete triplets\.

Discussion\.Direct performance comparisons between ACAT and generic platforms are complicated by hidden data engineering costs \(answering RQ5\)\. Benchmarking raw annotation speed on a generic tool ignores the effort of writing custom ETL pipelines, reconstructing ASTE triplets from flat exports, and computing IAA externally\. While ACAT achieves these results out of the box, equivalent figures on a generic platform would require post\-hoc engineering\.

Limitations\.The evaluation uses two annotators on a single domain \(restaurant reviews\), limiting generalisation across annotator counts, text types, and domains\. Fleiss’ Kappa, implemented for≥3\{\\geq\}3annotators, was not exercised\. Scalability and concurrent\-load benchmarks, as well as formal usability studies, remain as future work\. The platform does not support all ABSA tasks\. The interaction model is optimised for desktop\.

## 5Conclusion

We introduced ACAT, an end\-to\-end data curation platform for ABSA\. A preliminary experiment on1,0021\{,\}002restaurant reviews demonstrates efficient annotation \(median 31\.58 s\) with moderate\-to\-substantial IAA\. Revisiting our research questions:\(RQ1\)ACAT natively integrates four ABSA workflows without pre\-annotation configuration\.\(RQ2\)An automated ETL pipeline performs row\-level alignment at export\.\(RQ3\)The Implicit Toggle captures unstated aspects and sentiments\.\(RQ4\)IAA metrics are computed natively during export\.\(RQ5\)Direct comparisons with generic tools are unreliable because they ignore hidden ETL costs; ACAT handles the end\-to\-end pipeline natively\.

Future workincludes the official platform release, LLM\-based pre\-annotations to reduce time and cost, support for additional ABSA sub\-tasks, cross\-domain validation with more annotators, scalability benchmarks, and a real\-time analytics dashboard\.

Acknowledgments:The research presented in this paper was supported in part by \(1\) The Academy of Romanian Scientists, through the funding of the project “NetGuardAI: Intelligent system for harmful content detection and immunization on social networks” \(AOȘR\-TEAMS\-IV\); and \(2\) the National University of Science and Technology, POLITEHNICA Bucharest, through the PubArt program\.

## References

- \[1\]Aldeen, M\., Luo, J\., Lian, A\., Zheng, V\., Hong, A\., Yetukuri, P\., Cheng, L\.: Chatgpt vs\. human annotators: A comprehensive analysis of chatgpt for text annotation\. In: International Conference on Machine Learning and Applications\. pp\. 602–609\. IEEE \(2023\)\. https://doi\.org/10\.1109/ICMLA58977\.2023\.00089
- \[2\]Apostol, E\.S\., Pisică, A\.G\., Truică, C\.O\.: ATESA\-BÆRT: A heterogeneous ensemble learning model for aspect\-based sentiment analysis\. Knowledge\-Based Systems326, 113987 \(2025\)\. https://doi\.org/10\.1016/j\.knosys\.2025\.113987
- \[3\]Cohen, J\.: A coefficient of agreement for nominal scales\. Educational and psychological measurement20\(1\), 37–46 \(1960\)\. https://doi\.org/10\.1177/001316446002000104
- \[4\]Colucci Cante, L\., D’Angelo, S\., Di Martino, B\., Graziano, M\.: Text annotation tools: A comprehensive review and comparative analysis\. In: International Conference on Complex, Intelligent, and Software Intensive Systems\. pp\. 353–362\. Springer \(2024\)\. https://doi\.org/10\.1007/978\-3\-031\-70011\-8\_33
- \[5\]Fleiss, J\.L\.: Measuring nominal scale agreement among many raters\. Psychological bulletin76\(5\), 378 \(1971\)\. https://doi\.org/10\.1037/h0031619
- \[6\]Hua, Y\.C\., Denny, P\., Wicker, J\., Taskova, K\.: EduRABSA: An Education Review Dataset for Aspect\-based Sentiment Analysis Tasks\. arXiv preprint arXiv:2508\.17008 \(2025\)\. https://doi\.org/10\.48550/arXiv\.2508\.17008
- \[7\]Li, B\.Z\., Nye, M\., Andreas, J\.: Implicit representations of meaning in neural language models\. In: Annual Meeting of the ACL and IJCNLP\. pp\. 1813–1827 \(2021\)\. https://doi\.org/10\.18653/v1/2021\.acl\-long\.143
- \[8\]Na, J\.C\., Kyaing, W\.: Sentiment analysis of user\-generated content on drug review websites\. Journal of Information Science Theory and Practice3, 6–23 \(2015\)\. https://doi\.org/10\.1633/JISTaP\.2015\.3\.1\.1
- \[9\]Nakayama, H\., Kubo, T\., Kamura, J\., Taniguchi, Y\., Liang, X\.: doccano: Text annotation tool for human \(2018\),[https://github\.com/doccano/doccano](https://github.com/doccano/doccano)
- \[10\]Pang, B\., Lee, L\.: Opinion mining and sentiment analysis\. Comput\. Linguist35\(2\), 311–312 \(2009\)\. https://doi\.org/10\.1561/1500000011
- \[11\]Peng, H\., et al\.: Knowing what, how and why: A near complete solution for aspect\-based sentiment analysis\. In: AAAI conference on artificial intelligence\. pp\. 8600–8607 \(2020\)\. https://doi\.org/10\.1609/aaai\.v34i05\.6383
- \[12\]Perry, T\.: LightTag: Text Annotation Platform\. In: Conference on Empirical Methods in Natural Language Processing\. pp\. 20–27 \(2021\)\. https://doi\.org/10\.18653/v1/2021\.emnlp\-demo\.3
- \[13\]Petrescu, A\., Truică, C\.O\., Apostol, E\.S\., Paschke, A\.: EDSA\-Ensemble: An Event Detection Sentiment Analysis Ensemble Architecture\. IEEE Transactions on Affective Computing16\(2\), 555–572 \(2025\)\. https://doi\.org/10\.1109/taffc\.2024\.3434355
- \[14\]Pontiki, M\., et al\.: SemEval\-2014 task 4: Aspect based sentiment analysis\. In: International Workshop on Semantic Evaluation\. pp\. 27–35 \(2014\)\. https://doi\.org/10\.3115/v1/S14\-2004
- \[15\]Saeidi, M\., Bouchard, G\., Liakata, M\., Riedel, S\.: SentiHood: Targeted Aspect Based Sentiment Analysis Dataset for Urban Neighbourhoods\. In: International Conference on Computational Linguistics\. pp\. 1546–1556 \(2016\),[https://aclanthology\.org/C16\-1146/](https://aclanthology.org/C16-1146/)
- \[16\]Stenetorp, P\., Pyysalo, S\., Topić, G\., Ohta, T\., Ananiadou, S\., Tsujii, J\.: brat: a Web\-based Tool for NLP\-Assisted Text Annotation\. In: European Chapter of the ACL\. pp\. 102–107 \(2012\),[https://aclanthology\.org/E12\-2021/](https://aclanthology.org/E12-2021/)
- \[17\]Sun, Y\., Huang, Q\., Tung, A\.K\., Yu, J\.: Text embeddings should capture implicit semantics, not just surface meaning\. arXiv preprint arXiv:2506\.08354 \(2025\)\. https://doi\.org/10\.48550/arXiv\.2506\.08354
- \[18\]Tkachenko, M\., Malyuk, M\., Holmanyuk, A\., Liubimov, N\.: Label Studio: Data labeling software \(2020\-2025\),[https://github\.com/HumanSignal/label\-studio](https://github.com/HumanSignal/label-studio)
- \[19\]Truică, C\.O\., Leordeanu, C\.A\.: Classification of an imbalanced data set using decision tree algorithms\. UPB Scientific Bulletin \- Series C79\(4\), 69–84 \(2017\)
- \[20\]Yang, J\., Zhang, Y\., Li, L\., Li, X\.: YEDDA: A Lightweight Collaborative Text Span Annotation Tool\. In: ACL 2018, System Demonstrations\. pp\. 31–36 \(2018\)\. https://doi\.org/10\.18653/v1/P18\-4006