NEPSEE：用于尼泊尔语情感分析的双重标注基准

# NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis
Source: [https://arxiv.org/html/2610.19190](https://arxiv.org/html/2610.19190)
Mimansak NepalAffiliation:Tribhuvan University Kirtipur, Kathmandu, Nepal mimansaknepal@gmail\.comGrishma BasnetAffiliation:Subhakamana English School Urlabari, Morang, Nepal grishmabsnt@proton\.me

###### Abstract

We present NEPSEE, a benchmark for Nepali sentiment analysis with 3,189 Devanagari sentences from tweets of Nepali public figures\. Two annotators labelled each sentence blind on a five\-point scale, and we keep the sentences where they agree on polarity\. We evaluate 12 fine\-tuned encoder configurations, TF\-IDF baselines and five zero\-shot large language models \(LLMs\), three commercial and two with open weights\. On three classes, NepBERTa with a fixed tokenizer gets a macro\-F1 of 0\.877, IndicBERTv2 0\.867 and the best TF\-IDF model 0\.766\. Three commercial LLMs get 0\.882 to 0\.896 with no training data, but they are not significantly better than the best encoders\. On five points, Claude Opus 5\.5 gets 0\.838 and beats every encoder by at least 0\.10, because the encoders see few extreme examples\. A tokenizer audit adds a second result\. The default tokenizers of three Nepali BERT models delete the non\-spacing Devanagari vowel signs and the virama\. When we restore the marks, macro\-F1 increases by 0\.084 for NepBERTa but decreases by 0\.095 for a model pretrained on stripped text\. A masked\-language\-model probe that needs no labels predicts the direction of each effect\.

###### Index Terms:

Nepali, sentiment analysis, double annotation, tokenization, large language models

## IIntroduction

Nepali sentiment classification has no shared, fine\-grained benchmark\. Published Nepali sentiment datasets label whole tweets, comments or posts with two or three classes\[[1](https://arxiv.org/html/2610.19190#bib.bib1),[2](https://arxiv.org/html/2610.19190#bib.bib2),[3](https://arxiv.org/html/2610.19190#bib.bib3),[4](https://arxiv.org/html/2610.19190#bib.bib4)\]\. The Nepali language\-understanding benchmarks have no sentiment task\[[5](https://arxiv.org/html/2610.19190#bib.bib5)\]or a binary task built partly from translated English data\[[6](https://arxiv.org/html/2610.19190#bib.bib6)\]\. Their label sets, domains and metrics differ, so their scores do not compare\.

A benchmark score also depends on who made the labels\. Models learn the conventions of the annotators of their training set\[[7](https://arxiv.org/html/2610.19190#bib.bib7)\]\. Thus a high score can show agreement with one person and not with other readers\[[8](https://arxiv.org/html/2610.19190#bib.bib8)\]\. To prevent this problem, two annotators labelled our sentences blind, and we keep only the labels that they agree on\. Their agreement is low on the five\-point scale \(unweightedκ=0\.289\\kappa=0\.289\), but the two persons give opposite polarity on only 2\.4% of sentences\.

A second problem is in the tokenizers\. Devanagari writes most vowels as combining marks attached to consonants, and it joins consonants with a virama\. The three Nepali BERT checkpoints on the Hugging Face Hub have no tokenizer configuration file\. Thus the library uses default BERT settings, which delete every Unicode non\-spacing mark\. These marks include the vowel signs,,,and the virama\.

We use a new dataset111[https://github\.com/Nyvora\-Vision\-Labs/NEPSEE\-A\-Nepali\-Sentiment\-Emotion\-and\-Explanation\-Dataset](https://github.com/Nyvora-Vision-Labs/NEPSEE-A-Nepali-Sentiment-Emotion-and-Explanation-Dataset)and make four contributions:

- •a benchmark of 3,189 sentences with the agreed polarity of two blind annotators;
- •a five\-point task with two acceptable labels for each sentence where the annotators differ only in intensity;
- •results for 12 encoder configurations, TF\-IDF baselines, and three commercial and two open\-weight LLMs on the same test sentences;
- •an audit of the Nepali BERT tokenizers, with a masked\-language\-model \(MLM\) probe that predicts the effect of the fix on each checkpoint\.

## IIRelated Work

Nepali sentiment dataare small and coarse\. NepCOV19Tweets has about 33,247 COVID\-19 tweets in three classes\[[1](https://arxiv.org/html/2610.19190#bib.bib1)\], and later work used hybrid features and CNNs on it\[[9](https://arxiv.org/html/2610.19190#bib.bib9),[10](https://arxiv.org/html/2610.19190#bib.bib10)\]\. NepSA labels 3,068 YouTube comments with binary polarity for six aspects\[[2](https://arxiv.org/html/2610.19190#bib.bib2)\], and earlier work used classical classifiers and lexicons\[[11](https://arxiv.org/html/2610.19190#bib.bib11),[12](https://arxiv.org/html/2610.19190#bib.bib12)\]\. NAET labels 4,445 election tweets in three classes, and its best sentiment baseline gets a macro\-F1 of 0\.467\[[3](https://arxiv.org/html/2610.19190#bib.bib3)\]\. NepEMO labels 4,462 Reddit posts\[[4](https://arxiv.org/html/2610.19190#bib.bib4)\], and a 2026 shared task covers Nepali memes\[[13](https://arxiv.org/html/2610.19190#bib.bib13)\]\. Other resources target hate speech and offensive language\[[14](https://arxiv.org/html/2610.19190#bib.bib14),[15](https://arxiv.org/html/2610.19190#bib.bib15),[16](https://arxiv.org/html/2610.19190#bib.bib16)\], and none uses a five\-point scale\.

Nepali encoders\.NepBERTa is a BERT\-base model pretrained on about 0\.8 billion words of Nepali news, and its Nep\-gLUE benchmark has no sentiment task\[[5](https://arxiv.org/html/2610.19190#bib.bib5)\]\. NLUE adds a binary sentiment task\[[6](https://arxiv.org/html/2610.19190#bib.bib6)\], and Maskey et al\. compare several Nepali encoders\[[17](https://arxiv.org/html/2610.19190#bib.bib17)\]\. We found no study of the tokenizer settings of these checkpoints\. Fine\-grained sentiment has a longer history in English\[[18](https://arxiv.org/html/2610.19190#bib.bib18),[19](https://arxiv.org/html/2610.19190#bib.bib19),[20](https://arxiv.org/html/2610.19190#bib.bib20)\]\.

Agreement and label variation\.Krippendorff’sα\\alphaaccepts ordinal scales, andα=0\.667\\alpha=0\.667is the usual threshold for tentative conclusions\[[21](https://arxiv.org/html/2610.19190#bib.bib21),[22](https://arxiv.org/html/2610.19190#bib.bib22)\]\. Human label variation can be signal and not only noise\[[23](https://arxiv.org/html/2610.19190#bib.bib23)\], and soft labels are one way to train with it\[[24](https://arxiv.org/html/2610.19190#bib.bib24)\]\. Zero\-shot LLMs can label text more accurately than crowd workers\[[25](https://arxiv.org/html/2610.19190#bib.bib25)\], but they do not beat the best fine\-tuned models on most classification tasks\[[26](https://arxiv.org/html/2610.19190#bib.bib26)\]\.

## IIIData

### III\-ACorpus

We collected tweets from public accounts of Nepali politicians and journalists\. We kept the tweets that contain Devanagari and removed Latin script, URLs and Latin hashtags\. Then we removed tweets with fewer than four Nepali words\. We split each tweet at the danda \(\) and at line breaks, and we removed sentences shorter than five words\. The dataset has 5,614 sentences from 3,388 tweets of 11 accounts \(9 politicians and 2 journalists\), posted between December 2017 and August 2026\.

### III\-BBlind double annotation

Two annotators, A and B, labelled all 5,614 sentences\. Each sentence gets one of six labels: strongly negative \(SNeg\), negative \(Neg\), neutral \(Neu\), positive \(Pos\), strongly positive \(SPos\) or “don’t know”\. The brief tells annotators to judge the sentiment that the writer expresses, not the event\. Annotators use a mobile web application, which saves each label to a hosted database immediately\. Each annotator has a personal login, and the application shows only the labels of that annotator\. No annotator saw the labels of another person or of an LLM\.

### III\-CAgreement between the annotators

Table[I](https://arxiv.org/html/2610.19190#S3.T1)gives the agreement on the 4,943 sentences where both annotators give a polar label\. They choose the same five\-point label on 49\.5% of sentences, and unweightedκ\\kappais only 0\.289\. But 95\.6% of their label pairs are at most one step apart, and ordinalα\\alphais 0\.625\. They give opposite polarity on only 2\.4% of sentences\.

TABLE I:Agreement between annotators A and B on the 4,943 sentences with two polar labels\. Five points: ordinalα\\alpha; three classes: nominalα\\alpha\. Brackets: 95% bootstrap CI \(1,000 resamples\)\.Most disagreements concern the boundary between Neu and a polar label\. Annotator B uses Neu for 45\.0% of polar pairs, and annotator A for 29\.5%\. Of the 1,934 sentences that annotator A calls Pos, annotator B calls 906 Neu \(47%\)\. Annotator A also uses SPos more often \(13\.1% against 3\.7%\), and annotator B uses SNeg more often \(4\.3% against 0\.7%\)\. Thus the two persons have a different calibration, but they rarely disagree on the direction of the sentiment\.

### III\-DThe agreed benchmark

We removed 671 sentences where one annotator chose “don’t know” and 1,704 sentences with a different polarity\. Most of the 1,704 have Neu from one annotator and a polar label from the other\. Then we removed 50 exact duplicates and kept the first copy\. This leaves 3,189 sentences from 2,381 tweets \(Table[II](https://arxiv.org/html/2610.19190#S3.T2)\)\.

The three\-class label is the agreed polarity\. On the five\-point scale, the two annotators give the same label on 2,423 sentences\. The other 766 sentences have two adjacent labels with the same polarity, Neg and SNeg \(139\) or Pos and SPos \(627\)\. For these sentences, we accept both labels as correct\. We split the data by tweet with a stratified group 20\-fold split \(14, 3 and 3 folds\), so no tweet crosses two splits\.

TABLE II:Labels in each split\. Rows with two labels: the annotators differ only in intensity, and both labels are correct\. The splits keep each tweet together \(1,668, 357 and 356 tweets\)\.

## IVMethod

### IV\-AModels and baselines

We fine\-tune nine public checkpoints\. The multilingual checkpoints are multilingual BERT \(bert\-base\-multilingual\-cased\)\[[27](https://arxiv.org/html/2610.19190#bib.bib27)\], XLM\-R base\[[28](https://arxiv.org/html/2610.19190#bib.bib28)\], MuRIL base\[[29](https://arxiv.org/html/2610.19190#bib.bib29)\]and IndicBERTv2 \(IndicBERTv2\-MLM\-only\)\[[30](https://arxiv.org/html/2610.19190#bib.bib30)\]\. The Nepali checkpoints are NepBERTa\[[5](https://arxiv.org/html/2610.19190#bib.bib5)\], two community Nepali BERTs \(Rajan/NepaliBERTandShushant/nepaliBERT\), a Nepali RoBERTa \(IRIIS\-RESEARCH/RoBERTa\_Nepali\_125M\) and a Nepali DeBERTa \(Sakonii/deberta\-base\-nepali\)\. The original IndicBERT\[[31](https://arxiv.org/html/2610.19190#bib.bib31)\]is gated, so we use IndicBERTv2\. We converted the TensorFlow weights of NepBERTa to PyTorch \(202 of 202 tensors\)\. Each model has the same head: the first\-token hidden state, dropout 0\.1 and a linear layer\.

The baselines are logistic regression and a linear SVM on TF\-IDF features of word 1–2\-grams, character 2–5\-grams or both\. We tune the regularization and class weights on development macro\-F1\. A majority\-class model gives the floor\.

### IV\-BTokenizer audit and MLM probe

The three Nepali BERTs \(NepBERTa, Rajan, Shushant\) have notokenizer\_config\.json\. Thus thetransformerslibrary loads them withdo\_lower\_case=True, which also strips accents\. It decomposes the text and deletes all Unicode non\-spacing marks \(category Mn\)\. The spacing vowel signs such asandstay\. For the sentence\(“Nepal’s politics is extremely disappointing\.”\), the NepBERTa tokenizer gives these subwords:

By default,\(“of Nepal”\) loses its vowel sign, and\(“extremely”\) loses two viramas and a vowel sign\.

We run each Nepali BERT with the default tokenizer and with a fixed tokenizer \(do\_lower\_case=False,strip\_accents=False\), with the same seeds\. For each tokenizer we measure fertility \(subwords per word\), the share of combining marks lost, and the share of words kept intact\. The MLM probe masks 15% of the tokens of 600 benchmark sentences\. It then measures the top\-1 accuracy of the MLM head of the checkpoint under each setting\. A checkpoint should predict masked tokens best with its pretraining normalization\.

### IV\-CTraining with two acceptable labels

We train a separate model for each scale\. On three classes, the target is the agreed polarity\. On five classes, the target is the vote distribution of the two annotators\[[24](https://arxiv.org/html/2610.19190#bib.bib24)\]\. Thus a sentence with Pos and SPos has the target 0\.5 for each label, and a sentence with one label has the target 1\.0\. The TF\-IDF baselines use the same targets: each sentence with two labels gives two training rows with weight 0\.5\.

We fine\-tune with AdamW \(learning rate2×10−52\\times 10^\{\-5\}, weight decay 0\.01, 10% linear warm\-up\), batch size 16, maximum length 128 tokens and gradient clipping at 1\.0\. Training stops after at most six epochs, with early stopping on development macro\-F1 \(patience 2\)\. Each configuration runs with seeds 1, 2 and 3 on one NVIDIA A100 GPU\.

### IV\-DEvaluation

The main metric is macro\-F1\. On five classes, the*lenient*score counts a prediction as correct if it matches either label of the sentence\. We also give a*strict*score on the 375 test sentences where the two annotators chose the same five\-point label\. Other metrics are accuracy, quadratic weighted kappa \(QWK\) and the severe\-error rate\. A severe error has a predicted polarity opposite to the reference polarity\. Paired bootstrap tests resample the test set 2,000 times and use the mean macro\-F1 over three seeds\.

### IV\-EZero\-shot LLMs

We test three commercial LLMs and two LLMs with open weights\. Four LLMs labelled each sentence through OpenRouter on 1 October 2026: Gemini 3\.8 Flash, Gemini 3\.1 Pro \(preview\), and the open\-weight Qwen3\.5\-397B\-A17B and Qwen3\.5\-9B\. The prompt repeats the annotator brief and the six labels, and asks for JSON output at temperature 0, one sentence per call\. The Gemini models used reasoning effort “low”\. The Qwen models used no reasoning, because it gave thousands of tokens per sentence\.

Claude Opus 5\.5 \(claude\-opus\-5\-5\) labelled each sentence on 2 October 2026 through the Claude Code agent\. Twelve parallel sub\-sessions read the sentences in batches of 40 to 60 and wrote one label for each sentence\. Unlike the other LLMs, Claude saw the other sentences of its batch\. No LLM saw a human label\. We count a “don’t know” or missing answer as an error\.

## VResults

TABLE III:Test results on 479 sentences \(encoders: mean±\\pms\.d\. over three seeds, in order of three\-class macro\-F1\)\. Five\-point lenient: a prediction is correct if it matches either annotator\. Strict: the 375 test sentences with one five\-point label\. Severe: predicted polarity opposite to the reference\. For LLMs, QWK and Severe use only the sentences with a polar answer\. Bold: best in each column\.### V\-AThree classes: encoders and LLMs are equal

Fine\-tuned encoders get much higher scores than the TF\-IDF baselines \(Table[III](https://arxiv.org/html/2610.19190#S5.T3)\)\. On three classes, NepBERTa with the fixed tokenizer gets a macro\-F1 of 0\.877 and IndicBERTv2 0\.867, against 0\.766 for the best TF\-IDF model\. IndicBERTv2 has the best development macro\-F1 \(0\.896\)\. The difference between the two encoders is not significant \(\+0\.010, 95% CI \[−0\.012\-0\.012, 0\.032\]\)\.

The three commercial LLMs get 0\.882 to 0\.896 with no training data\. Claude Opus 5\.5 has the highest score, but its gain over NepBERTa is not significant \(\+0\.019, CI \[−0\.015\-0\.015, 0\.051\]\)\. The open\-weight Qwen3\.5\-397B\-A17B \(0\.867\) is equal to IndicBERTv2, and the small Qwen3\.5\-9B \(0\.790\) is near mBERT\. The LLMs answer “don’t know” or give no label on 0\.6% \(Claude\) to 3\.3% \(Qwen3\.5\-397B\-A17B\) of test sentences\. On all 3,189 sentences, the three\-class LLM scores change by at most 0\.026, so the test set is representative\.

### V\-BFive points: LLMs are better at intensity

On the five\-point scale, the order changes\. Claude Opus 5\.5 gets a lenient macro\-F1 of 0\.838, against 0\.721 for NepBERTa and 0\.734 for IndicBERTv2\. Its gain over NepBERTa is significant \(\+0\.116, CI \[0\.036, 0\.187\]\), and so is its gain over IndicBERTv2 \(\+0\.104, CI \[0\.002, 0\.186\]\)\. Claude also makes the fewest severe errors \(0\.4%\)\. Gemini 3\.8 Flash \(0\.786\) is second, but its gain over NepBERTa is not significant \(CI \[−0\.011\-0\.011, 0\.131\]\)\. The strict scores give the same order: 0\.750 for Claude and at most 0\.640 for an encoder\.

The extreme classes limit the encoders\. The training set has 14 sentences with only SNeg and 43 with only SPos\. Over all encoders, the mean F1 on SNeg is 0 to 0\.22, and MuRIL gets an F1 of 0 on SPos\. Development macro\-F1 is not a reliable guide on five points, because the development set has one sentence with only SNeg\. It selects Rajan’s NepaliBERT \(0\.795 on development\), which is ninth on test\. Thus we give all encoders and do not select one\.

### V\-CTokenizer settings must match pretraining

The default tokenizers of the three Nepali BERTs remove 44\.5% to 44\.6% of all Devanagari combining marks\. Only 35\.1% to 36\.8% of words get to the model intact, and fertility increases from 1\.20–1\.27 to 1\.95–2\.04 subwords per word\.

TABLE IV:Default and fixed tokenizer for the three Nepali BERTs\. Fert\.: subwords per word\. Lost: combining marks deleted\. MLM: top\-1 probe accuracy\. F1: test macro\-F1 \(mean of 3 seeds\) on our three classes, NepCOV19Tweets and news topics\. Bold: the better setting where the 95% CI of the difference excludes zero\.Fig\. 1:\(a\) Three\-class test macro\-F1 of each seed with the default \(hollow\) and fixed \(filled\) tokenizer\. The numbers give the mean change\. \(b\) Accuracy of the MLM probe for each checkpoint\. It predicts the direction of each effect\.The fix has a different effect on each checkpoint \(Fig\.[1](https://arxiv.org/html/2610.19190#S5.F1)a\)\. On three classes, it raises the macro\-F1 of NepBERTa by 0\.084 \(95% CI \[0\.057, 0\.115\]\) and of Rajan’s NepaliBERT by 0\.049 \(CI \[0\.019, 0\.079\]\)\. It lowers the macro\-F1 of Shushant’s nepaliBERT by 0\.095 \(CI \[−0\.131\-0\.131,−0\.063\-0\.063\]\)\. All three effects have the same direction on all seeds\. On five points, the changes are\+0\.050\+0\.050,\+0\.036\+0\.036and−0\.089\-0\.089, and all CIs exclude zero\.

The MLM probe predicts each direction \(Fig\.[1](https://arxiv.org/html/2610.19190#S5.F1)b\)\. NepBERTa predicts masked tokens much better with the marks \(top\-1 accuracy 0\.518 against 0\.205\), so its pretraining text had the marks\. The Shushant checkpoint shows the opposite \(0\.650 stripped, 0\.071 kept\), so its pretraining text was stripped\. Thus the fixed tokenizer gives it input that it did not see in pretraining\. The probe needs no labels, so it is a quick check before fine\-tuning\.

On two public datasets, NepCOV19Tweets\[[1](https://arxiv.org/html/2610.19190#bib.bib1)\]and the Nepali news topics of MTEB, the fix has the same effect \(Table[IV](https://arxiv.org/html/2610.19190#S5.T4)\)\. It helps NepBERTa and hurts Shushant on all seeds, and the four CIs exclude zero\. Rajan shows a gain on news but not on NepCOV19Tweets\.

### V\-DLoss, training size and errors

An ordinal loss gives no consistent gain on five points\. Cross\-entropy \(CE\) plus the expected distance∑kpk​\|k−y\|\\sum\_\{k\}p\_\{k\}\\,\|k\-y\|raises the lenient macro\-F1 of NepBERTa from 0\.721 to 0\.746, which is less than one seed standard deviation\. It lowers that of IndicBERTv2 from 0\.734 to 0\.691\. With 10% of the training data \(224 sentences\), NepBERTa gets 0\.833±\\pm\.014on three classes, which is more than TF\-IDF on all data \(0\.766\)\. IndicBERTv2 is unstable with 10% of the data \(0\.481±\\pm\.058\), but it gets 0\.838 with 25%\.

Severe errors are rare and often come from sarcasm\. NepBERTa makes the same severe error on all three seeds for only two test sentences\. One is\(“The victim girls died without justice because of PM Khadga Prasad Oli, so I could not give Children’s Day wishes\. I want to give these wishes to the PM himself, please accept\.”\)\. Both annotators say Neg, but the words of greeting make NepBERTa predict Pos\.

## VIDiscussion and Limitations

Agreed labels make a better reference\.Two blind annotators rarely disagree on the direction of sentiment, but often disagree on Neu against a polar label and on intensity\. Thus we keep only the agreed polarity, and we accept two labels where the annotators differ only in intensity\. We recommend three reports for each new model: three\-class, lenient five\-point and strict five\-point scores\.

LLMs are now competitive\.On three classes, the best commercial LLMs equal the best fine\-tuned encoders with no training data\. On five points, Claude Opus 5\.5 is better than every encoder, because the encoders see few extreme examples\. The open\-weight Qwen3\.5\-397B\-A17B equals a mid\-sized encoder on three classes\.

Check the tokenizer before fine\-tuning\.Make sure that the tokenizer keeps the script, and that it matches the normalization of the checkpoint\. A blind fix would cost the Shushant model 0\.095 macro\-F1, and the default would cost NepBERTa 0\.084\.

Limitations\.The agreed subset is easier than the full dataset, because we removed the 1,704 sentences where the annotators disagree on polarity\. Thus our scores are higher than scores on all sentences would be, and the benchmark does not test the Neu–Pos boundary well\. Only two annotators labelled the benchmark\. The SNeg and SPos classes have few sentences with one label\. We used three seeds, one set of hyperparameters and one zero\-shot prompt for each LLM, and Claude saw its sentences in batches\.

## VIIConclusion

We built a Nepali sentiment benchmark from the agreed labels of two blind annotators\. On three classes, NepBERTa with a fixed tokenizer, IndicBERTv2 and three commercial LLMs score about 0\.87 to 0\.90 macro\-F1\. On five points, Claude Opus 5\.5 is best by a significant margin\. Next, we will add annotators to the sentences with disagreement and release them as a second test set\.

## References

- \[1\]C\. Sitaula, A\. Basnet, A\. Mainali, and T\. B\. Shahi, “Deep learning\-based methods for sentiment analysis on Nepali COVID\-19\-related tweets,”*Computational Intelligence and Neuroscience*, vol\. 2021, no\. 1, p\. 2158184, 2021\.
- \[2\]O\. M\. Singh, S\. Timilsina, B\. K\. Bal, and A\. Joshi, “Aspect based abusive sentiment detection in Nepali social media texts,” in*Proc\. 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining \(ASONAM\)*, 2020, pp\. 301–308\.
- \[3\]K\. Rauniyar, S\. Poudel, S\. Shiwakoti, S\. Thapa, J\. Rashid, J\. Kim, M\. Imran, and U\. Naseem, “Multi\-aspect annotation and analysis of Nepali tweets on anti\-establishment election discourse,”*IEEE Access*, vol\. 11, pp\. 143 092–143 115, 2023\.
- \[4\]S\. Sitoula, T\. B\. Shahi, L\. P\. Bhatt, A\. Pokhrel, and A\. Neupane, “NepEMO: A multi\-label emotion and sentiment analysis on Nepali Reddit with linguistic insights and temporal trends,” arXiv preprint arXiv:2512\.22823, 2025\.
- \[5\]S\. Timilsina, M\. Gautam, and B\. Bhattarai, “NepBERTa: Nepali language model trained in a large corpus,” in*Proc\. 2nd Conf\. Asia\-Pacific Chapter of the ACL and 12th Int\. Joint Conf\. on Natural Language Processing \(Volume 2: Short Papers\)*\. Association for Computational Linguistics, 2022, pp\. 273–284\.
- \[6\]J\. Nyachhyon, M\. Sharma, P\. Thapa, and B\. K\. Bal, “Consolidating and developing benchmarking datasets for the Nepali natural language understanding tasks,” in*Findings of the Association for Computational Linguistics: IJCNLP\-AACL 2025*, Mumbai, India, 2025, pp\. 1906–1925\.
- \[7\]M\. Geva, Y\. Goldberg, and J\. Berant, “Are we modeling the task or the annotator? An investigation of annotator bias in natural language understanding datasets,” in*Proc\. EMNLP\-IJCNLP*, 2019, pp\. 1161–1166\.
- \[8\]S\. Tedeschi, J\. Bos, T\. Declerck*et al\.*, “What’s the meaning of superhuman performance in today’s NLU?” in*Proc\. ACL \(Volume 1: Long Papers\)*, 2023, pp\. 12 471–12 491\.
- \[9\]T\. B\. Shahi, C\. Sitaula, and N\. Paudel, “A hybrid feature extraction method for Nepali COVID\-19\-related tweets classification,”*Computational Intelligence and Neuroscience*, vol\. 2022, p\. 5681574, 2022\.
- \[10\]C\. Sitaula and T\. B\. Shahi, “Multi\-channel CNN to classify Nepali COVID\-19 related tweets using hybrid features,”*Journal of Ambient Intelligence and Humanized Computing*, vol\. 15, no\. 3, pp\. 2047–2056, 2024\.
- \[11\]S\. Tamrakar, B\. K\. Bal, and R\. B\. Thapa, “Aspect based sentiment analysis of Nepali text using support vector machine and naive Bayes,”*Technical Journal*, vol\. 2, no\. 1, pp\. 22–29, 2020\.
- \[12\]R\. Piryani, B\. Piryani, V\. K\. Singh, and D\. Pinto, “Sentiment analysis in Nepali: Exploring machine learning and lexicon\-based approaches,”*Journal of Intelligent & Fuzzy Systems*, vol\. 39, no\. 2, pp\. 2201–2212, 2020\.
- \[13\]A\. Acharya, A\. Khatiwada, R\. Khadka, and P\. Aryal, “TeamHerald@CHIPSAL 2026: Hate speech detection and sentiment analysis of Nepali memes using transformer\-based architectures and ensemble learning,” arXiv preprint arXiv:2606\.08770, 2026\.
- \[14\]S\. Thapa, K\. Rauniyar, S\. Shiwakoti, S\. Poudel, U\. Naseem, and M\. Nasim, “NEHATE: Large\-scale annotated data shedding light on hate speech in Nepali local election discourse,” in*Proc\. ECAI 2023, Frontiers in Artificial Intelligence and Applications*\. IOS Press, 2023\.
- \[15\]N\. B\. Niraula, S\. Dulal, and D\. Koirala, “Offensive language detection in Nepali social media,” in*Proc\. 5th Workshop on Online Abuse and Harms \(WOAH 2021\)*\. Association for Computational Linguistics, 2021, pp\. 67–75\.
- \[16\]S\. Thapa, K\. Rauniyar, F\. A\. Jafri, S\. Adhikari, K\. Sarveswaran, B\. K\. Bal, H\. Veeramani, and U\. Naseem, “Natural language understanding of Devanagari script languages: Language identification, hate speech and its target detection,” in*Proc\. First Workshop on Challenges in Processing South Asian Languages \(CHiPSAL 2025\)*\. Abu Dhabi, UAE: International Committee on Computational Linguistics, 2025, pp\. 71–82\.
- \[17\]U\. Maskey, M\. Bhatta, S\. Bhatt, S\. Dhungel, and B\. K\. Bal, “Nepali encoder transformers: An analysis of auto encoding transformer language models for Nepali text classification,” in*Proc\. 1st Annual Meeting of the ELRA/ISCA Special Interest Group on Under\-Resourced Languages*, 2022, pp\. 106–111\.
- \[18\]R\. Socher, A\. Perelygin, J\. Wu, J\. Chuang, C\. D\. Manning, A\. Ng, and C\. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in*Proc\. 2013 Conf\. Empirical Methods in Natural Language Processing*, 2013, pp\. 1631–1642\.
- \[19\]P\. Nakov, A\. Ritter, S\. Rosenthal, F\. Sebastiani, and V\. Stoyanov, “SemEval\-2016 task 4: Sentiment analysis in Twitter,” in*Proc\. 10th International Workshop on Semantic Evaluation \(SemEval\-2016\)*, 2016, pp\. 1–18\.
- \[20\]S\. Rosenthal, N\. Farra, and P\. Nakov, “SemEval\-2017 task 4: Sentiment analysis in Twitter,” in*Proc\. 11th International Workshop on Semantic Evaluation \(SemEval\-2017\)*, 2017, pp\. 502–518\.
- \[21\]K\. Krippendorff, “Computing Krippendorff’s alpha\-reliability,” Annenberg School for Communication, University of Pennsylvania, Tech\. Rep\., 2011\.
- \[22\]R\. Artstein and M\. Poesio, “Survey article: Inter\-coder agreement for computational linguistics,”*Computational Linguistics*, vol\. 34, no\. 4, pp\. 555–596, 2008\.
- \[23\]B\. Plank, “The “problem” of human label variation: On ground truth in data, modeling and evaluation,” in*Proc\. EMNLP*, 2022, pp\. 10 671–10 682\.
- \[24\]A\. N\. Uma, T\. Fornaciari, D\. Hovy, S\. Paun, B\. Plank, and M\. Poesio, “Learning from disagreement: A survey,”*Journal of Artificial Intelligence Research*, vol\. 72, pp\. 1385–1470, 2021\.
- \[25\]F\. Gilardi, M\. Alizadeh, and M\. Kubli, “ChatGPT outperforms crowd workers for text\-annotation tasks,”*Proceedings of the National Academy of Sciences*, vol\. 120, no\. 30, p\. e2305016120, 2023\.
- \[26\]C\. Ziems, W\. Held, O\. Shaikh, J\. Chen, Z\. Zhang, and D\. Yang, “Can large language models transform computational social science?”*Computational Linguistics*, vol\. 50, no\. 1, pp\. 237–291, 2024\.
- \[27\]J\. Devlin, M\.\-W\. Chang, K\. Lee, and K\. Toutanova, “BERT: Pre\-training of deep bidirectional transformers for language understanding,” in*Proc\. 2019 Conf\. North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, 2019, pp\. 4171–4186\.
- \[28\]A\. Conneau, K\. Khandelwal, N\. Goyal, V\. Chaudhary, G\. Wenzek, F\. Guzmán, E\. Grave, M\. Ott, L\. Zettlemoyer, and V\. Stoyanov, “Unsupervised cross\-lingual representation learning at scale,” in*Proc\. 58th Annual Meeting of the Association for Computational Linguistics*, 2020, pp\. 8440–8451\.
- \[29\]S\. Khanuja, D\. Bansal, S\. Mehtani*et al\.*, “MuRIL: Multilingual representations for Indian languages,” arXiv preprint arXiv:2103\.10730, 2021\.
- \[30\]S\. Doddapaneni, R\. Aralikatte, G\. Ramesh, S\. Goyal, M\. M\. Khapra, A\. Kunchukuttan, and P\. Kumar, “Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages,” in*Proc\. 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2023, pp\. 12 402–12 426\.
- \[31\]D\. Kakwani, A\. Kunchukuttan, S\. Golla, G\. N\.C\., A\. Bhattacharyya, M\. M\. Khapra, and P\. Kumar, “IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre\-trained multilingual language models for Indian languages,” in*Findings of the Association for Computational Linguistics: EMNLP 2020*, 2020, pp\. 4948–4961\.