Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

# What Multilingual Affective GenerationBenchmarks Actually Measure
Source: [https://arxiv.org/html/2609.29445](https://arxiv.org/html/2609.29445)
## Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

Adib Sakhawat11footnotemark:1Affiliation:Department of Computer Science and EngineeringAffiliation:Islamic University of Technology, Dhaka, BangladeshAffiliation:\{sakhadib, fardeensadab\}@gmail\.com

###### Abstract

We audit a multilingual affective generation benchmark—eight instruction\-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements—and find its headline conclusions to be artefacts of the measurement instrument rather than properties of the systems\. Treating annotators as a random rather than a fixed factor, no system differs significantly from any other \(F⁡\(7,14\)=0\.59F\(7,14\)=0\.59,p=0\.76p=0\.76\), although the conventional analysis declares 19 of 28 pairwise differences significant\. Annotator identity explains far more rating variance than system identity, and the winning system changes whenever any single annotator is removed\. The ordering that does emerge tracks output length: mean emoji count explains 78\.7% of between\-system variance, and a within\-item length\-matched comparison over 2,599 pairs reverses the leaderboard\. We further show that cross\-provider anisotropy differences vanish under mean\-centring, that per\-language token costs change sign with the normalising unit, and that multi\-view row\-wise splits inflate macro\-F1 by3\.13\.1points and change the top\-ranked system\. In place of preference scoring we propose*emoji–affect decodability*, a reference\-based probe whose rankings are stable to±0\.003\\pm 0\.003macro\-F1 across seeds\.

## 1Introduction

Affective computing for low\-resource languages is increasingly evaluated by asking large language models to generate something—a label, a rationale, an emoji—and then asking a small panel of native speakers which output they prefer\. The pattern is attractive: it needs no gold annotation in the target language, it produces a clean leaderboard, and it appears to centre the judgements of the speech community rather than those of a distant annotation vendor\.

This paper reports what happened when we subjected one such benchmark to a full measurement audit\. The benchmark is our own: eight instruction\-tuned LLMs generate one to five emojis for each of 17,100 sentences in Bangla, English and Hindi, and three fluent Bangla speakers rate 290 sentences×\\times8 systems on a five\-point scale, yielding a completely crossed design with 6,960 judgements and no missing cells\. Our initial analysis of this benchmark reported a leaderboard, a “consistency paradox” between human preference and emoji–label agreement, provider differences in embedding anisotropy, a per\-language token premium, and near\-certain data leakage under naive splits\.

Every one of those findings survives re\-examination only in a substantially weakened form, and several reverse\. The failures are not exotic\. They are the standard failure modes of small\-panel preference evaluation, applied to a setting—emoji generation in a low\-resource language—where the effect being measured is small, the annotator population is heterogeneous, and there is no external anchor\. We think the resulting picture is more useful than the leaderboard we set out to produce, because the same design is now common across low\-resource affective NLP\.

#### Contributions\.

1. 1\.A generalisability\-theory analysis of a fully crossed290×8×3290\\times 8\\times 3preference study \(§[4](https://arxiv.org/html/2609.29445#S4)\)\. Treating annotators as a random factor, no system differs significantly from any other; the conventional analysis finds 19 of 28 pairwise differences significant\. We give the decision study: no number of items yields a reliable ranking with three annotators\.
2. 2\.An explanation, not merely an observation \(§[5](https://arxiv.org/html/2609.29445#S5)\)\. Output length accounts for 78\.7% of the between\-system preference variance; a within\-item length\-matched comparison over 2,599 pairs reverses the ordering\. The “smaller\-model\-wins” effect reported previously is a verbosity effect\.
3. 3\.*Emoji–affect decodability*\(ead\), a reference\-based, annotator\-free metric \(§[6](https://arxiv.org/html/2609.29445#S6)\), with cross\-lingual transfer results showing that the emoji code models emit is language\-specific: transferring a decoder across languages costs0\.2520\.252macro\-F1 despite\>99\.7%\>99\.7\\%vocabulary overlap\.
4. 4\.Corrections to three widely reported geometric and economic claims \(§[7](https://arxiv.org/html/2609.29445#S7),§[8](https://arxiv.org/html/2609.29445#S8)\): provider anisotropy gaps are a mean\-offset artefact; language separability is not alignment when corpora are not parallel; and per\-language token premia change sign with the choice of normalising unit\.
5. 5\.An empirical, rather than asserted, quantification of multi\-view leakage \(§[9](https://arxiv.org/html/2609.29445#S9)\):\+3\.1\+3\.1macro\-F1 points of optimism and a change in the top\-ranked system\.

## 2Related Work

#### Human evaluation and its reliability\.

Concerns about the statistical treatment of human judgements in NLP are long\-standing\([Clark et al\., 2021](https://arxiv.org/html/2609.29445#bib.bib1);[Card et al\., 2020](https://arxiv.org/html/2609.29445#bib.bib2)\), and the specific error we document—treating annotators as a fixed rather than a random factor—has been raised for machine translation and summarisation\([Graham et al\., 2015](https://arxiv.org/html/2609.29445#bib.bib3);[Mathur et al\., 2020](https://arxiv.org/html/2609.29445#bib.bib4)\)\. Generalisability theory\([Cronbach et al\., 1972](https://arxiv.org/html/2609.29445#bib.bib5);[Brennan, 2001](https://arxiv.org/html/2609.29445#bib.bib6)\)provides the standard apparatus and has been applied only sporadically in NLP\. Product\-moment correlation is still frequently reported as an agreement statistic despite being insensitive to systematic rater bias;[Krippendorff \(2018\)](https://arxiv.org/html/2609.29445#bib.bib7)and[Shrout and Fleiss \(1979\)](https://arxiv.org/html/2609.29445#bib.bib8)give the appropriate coefficients\.

#### Length and verbosity bias\.

Human and model\-based preference judgements are known to favour longer outputs\([Zheng et al\., 2023](https://arxiv.org/html/2609.29445#bib.bib9);[Dubois et al\., 2024](https://arxiv.org/html/2609.29445#bib.bib10);[Singhal et al\., 2024](https://arxiv.org/html/2609.29445#bib.bib11)\)\. Length\-controlled variants of preference metrics have accordingly been proposed for open\-ended text\. We show the same bias operating over a radically shorter output space—one to five emojis—where it accounts for almost all apparent system quality\.

#### Emoji, affect and low\-resource languages\.

Emoji carry affective content that is partly conventional and partly culturally specific\([Kralj Novak et al\., 2015](https://arxiv.org/html/2609.29445#bib.bib12);[Barbieri et al\., 2018](https://arxiv.org/html/2609.29445#bib.bib13);[Shoeb and de Melo, 2020](https://arxiv.org/html/2609.29445#bib.bib14)\)\. Bangla affective NLP has grown rapidly\([Islam et al\., 2022](https://arxiv.org/html/2609.29445#bib.bib15);[Das et al\., 2021](https://arxiv.org/html/2609.29445#bib.bib16)\), and code\-mixed South Asian resources such as EmoMix\-3L\([Raihan et al\., 2024](https://arxiv.org/html/2609.29445#bib.bib17)\)highlight transliteration and script noise as failure sources\. Our contribution is not a new emoji resource but a measurement protocol\.

#### Embedding geometry\.

[Ethayarajh \(2019\)](https://arxiv.org/html/2609.29445#bib.bib18)introduced the expected\-cosine measure of anisotropy;[Gao et al\. \(2021\)](https://arxiv.org/html/2609.29445#bib.bib19);[Li et al\. \(2020\)](https://arxiv.org/html/2609.29445#bib.bib20)link it to representation quality, and[Rudman et al\. \(2022\)](https://arxiv.org/html/2609.29445#bib.bib21)show that several popular anisotropy statistics are confounded with the embedding mean\. Cross\-lingual alignment is conventionally evaluated by parallel\-sentence retrieval\([Artetxe and Schwenk, 2019](https://arxiv.org/html/2609.29445#bib.bib22);[Feng et al\., 2022](https://arxiv.org/html/2609.29445#bib.bib23)\); we argue that alignment claims made without parallel data measure something else entirely\.

#### Contamination and leakage\.

Multi\-view leakage is a special case of the contamination problem\([Sainz et al\., 2023](https://arxiv.org/html/2609.29445#bib.bib24);[Golchin and Surdeanu, 2024](https://arxiv.org/html/2609.29445#bib.bib25)\)\. Grouped splitting is standard practice in clinical and speech corpora but is frequently omitted in multi\-model NLP benchmarks\.

## 3Benchmark and Study Design

#### Corpora\.

We use 17,100 sentences: 5,813 Bangla, 6,547 English and 4,740 Hindi, each carrying an emotion label from its source corpus\. A design constraint that we make explicit, and that our initial analysis did not, is that*the three corpora do not share a label scheme*\. Bangla and English use \{Anger, Fear, Joy, Love, Sadness, Surprise\}; Hindi adds Neutral and Disgust and omits Love\. Bangla is multi\-label \(17\.7% of items carry more than one label\) while English and Hindi are single\-label by construction\. Any cross\-lingual comparison of label statistics therefore measures the annotation guidelines of three different corpora, not the behaviour of a model\. All cross\-lingual analyses below are restricted to the five emotions common to all three corpora and to single\-label items, giving 4,206 Bangla, 5,380 English and 3,518 Hindi sentences\.

#### Generation\.

Eight systems—Claude\-3\-Haiku, DeepSeek\-V3\.2, Gemini\-2\.0\-Flash, Gemma\-3\-27B, Mistral\-Large\-2512, GPT\-4\.1\-nano, Qwen3\-VL\-235B and Grok\-4\-Fast—were queried through a single API gateway with an identical English\-language prompt for all languages,temperature=0\.7=0\.7,max\_tokens=50=50, no system prompt, and up to three retries when the response contained no non\-ASCII character\. The prompt requests one to five emojis and nothing else;*the models are never shown the emotion label and are never asked to predict one*\. We restate this because it delimits what the benchmark can support: it is a study of affective*expression*through a constrained symbolic channel, not of emotion classification\. The full prompt is in Appendix[A](https://arxiv.org/html/2609.29445#A1)\.

#### Human study\.

Three independent fluent Bangla speakers rated a stratified 5% sample of the Bangla data \(290 sentences\) for all eight systems on a 1–5 scale of how well the emojis captured the sentence’s emotion\. The design is completely crossed: every annotator rated every \(sentence, system\) pair, giving290×8×3=6,960290\\times 8\\times 3=6\{,\}960judgements with no missing cells\. This is the strongest possible design for the analyses in §[4](https://arxiv.org/html/2609.29445#S4), and it is what allows us to separate the annotator×\\timessystem interaction from residual noise at all\.

#### Embeddings\.

Each sentence was embedded with five commercial encoders: Gemini\-Embedding\-001 \(3072d\), Mistral\-Embed\-2312 \(1024d\), OpenAI text\-embedding\-3\-large \(3072d\) and \-small \(1536d\), and Qwen3\-Embedding\-8B \(4096d\)\. Geometry analyses use a label\-stratified sample of 1,202 sentences \(400/401 per language\); we report the sample size wherever it constrains a conclusion\.

## 4Preference Scores Do Not Identify a Winner

### 4\.1Agreement

Pairwise Pearson correlations between annotators are0\.170\.17,0\.180\.18and0\.170\.17, values which we initially reported as evidence of “low but positive agreement”\. Pearson correlation is not an agreement coefficient: it is invariant to additive and multiplicative rater bias, and here the raters differ enormously in location, with mean ratings of2\.862\.86,4\.354\.35and3\.313\.31\. Recomputing with coefficients that penalise systematic bias gives Krippendorff’sαordinal=0\.014\\alpha\_\{\\text\{ordinal\}\}=0\.014\(95% CI\[−0\.011,0\.039\]\[\-0\.011,0\.039\], 2,000 unit\-bootstrap replicates\) andICC​\(2,1\)=0\.116\\text\{ICC\}\(2,1\)=0\.116,ICC​\(2,k\)=0\.282\\text\{ICC\}\(2,k\)=0\.282\. For the annotator pair whose Pearson correlation is0\.1660\.166,αordinal=−0\.109\\alpha\_\{\\text\{ordinal\}\}=\-0\.109: they are, after accounting for their different scale usage, in slightly worse than chance agreement\. Individual judgements in this task carry almost no shared signal\.

### 4\.2The system effect disappears under the correct error term

Table 1:Three\-way ANOVA on the crossed290×8×3290\\times 8\\times 3design\. The system effect is significant against the residual and non\-significant against the system×\\timesrater interaction, which is the correct error term when annotators are a sample from a population\. The interaction mean square exceeds the system mean square\.Table[1](https://arxiv.org/html/2609.29445#S4.T1)gives the three\-way ANOVA\. The choice of error term decides the result\. Testing the system effect against the residual—which is what one does implicitly by running a paired test over items on the panel\-averaged score, the near\-universal practice in NLP—givesF⁡\(7,4046\)=18\.62F\(7,4046\)=18\.62,p<10−16p<10^\{\-16\}\. Testing it against the system×\\timesrater interaction, which is the correct error term if the three annotators are a sample from a population of possible annotators rather than the objects of study, givesF⁡\(7,14\)=0\.59F\(7,14\)=0\.59,p=0\.76p=0\.76\. The interaction mean square \(20\.2520\.25\) is larger than the system main effect \(11\.8511\.85\): annotators disagree about systems more than systems differ\.

The same contrast appears in the pairwise comparisons\. Over the 28 system pairs, a paired test over items finds 19 significant atp<\.05p<\.05and 13 after Holm correction; a test that treats the annotator as the unit of replication finds*zero*significant at either level\. Only 5 of 28 pairs have a difference of consistent sign across all three annotators\.

### 4\.3Variance components and the decision study

Figure[1](https://arxiv.org/html/2609.29445#S4.F1)\(a\) decomposes the variance attributable to the random factors\. Annotator identity accounts for 29\.1%, the item×\\timesannotator interaction for 22\.9%, residual noise for 32\.6% and item difficulty for 7\.3%\. The system is a fixed factor and so does not appear in that decomposition; placing it on a common scale via the sums of squares in Table[1](https://arxiv.org/html/2609.29445#S4.T1), the annotator main effect accounts for 22\.1% of the total and the system—the quantity the benchmark exists to measure—for0\.7%\. Which sentence is being rated \(18\.8% of the total sum of squares\) matters more than twenty times as much as which frontier LLM produced the emojis\.

Because the design is fully crossed we can run a decision study\([Brennan, 2001](https://arxiv.org/html/2609.29445#bib.bib6)\)\. The generalisability coefficient for the system mean is

G⁡\(np,nr\)=σs2σs2\+σs​r2nr\+σp​s2np\+σp​s​r2np​nr,G\(n\_\{p\},n\_\{r\}\)=\\frac\{\\sigma^\{2\}\_\{s\}\}\{\\sigma^\{2\}\_\{s\}\+\\frac\{\\sigma^\{2\}\_\{sr\}\}\{n\_\{r\}\}\+\\frac\{\\sigma^\{2\}\_\{ps\}\}\{n\_\{p\}\}\+\\frac\{\\sigma^\{2\}\_\{psr\}\}\{n\_\{p\}n\_\{r\}\}\},wherenpn\_\{p\}is the number of items andnrn\_\{r\}the number of annotators\. The current design attainsG=0\.336G=0\.336\. Reaching the conventional thresholdG≥0\.8G\\geq 0\.8requires 27 annotators at the present 290 items, and 70 annotators forG≥0\.9G\\geq 0\.9\. Figure[1](https://arxiv.org/html/2609.29445#S4.F1)\(c\) makes the important point: becauseσs​r2\\sigma^\{2\}\_\{sr\}is divided bynrn\_\{r\}alone,*the curves saturate innpn\_\{p\}*\. With three annotators, no number of sentences—not 5,000, not 500,000—reachesG=0\.8G=0\.8\. Reliability in this design cannot be bought with more data; it can only be bought with more annotators\.

### 4\.4The winner depends on who is on the panel

Table 2:Leave\-one\-annotator\-out leaderboards\. Removing any single annotator from a three\-person panel can change the winner\. Claude\-3\-Haiku ranges from first to last\.Table[2](https://arxiv.org/html/2609.29445#S4.T2)and Figure[1](https://arxiv.org/html/2609.29445#S4.F1)\(b\) show the practical consequence\. Dropping one annotator changes the top\-ranked system in two of three cases\. Claude\-3\-Haiku is ranked first by one panel and last by another\. Between\-annotator Kendall correlations over the eight system means are−0\.36\-0\.36,0\.070\.07and0\.000\.00: at the level of the leaderboard, the three annotators are not measuring the same thing\.

Figure 1:\(a\) Random\-effect variance components; annotator\-related terms \(red\) dominate\. The system is a fixed factor and accounts for 0\.7% of the total sum of squares against 22\.1% for the annotator main effect\. \(b\) System rank as a function of which annotator is excluded\. \(c\) Decision study:GGsaturates in the number of items, so three annotators cannot reachG=0\.8G=0\.8at any corpus size\.

## 5What the Ratings Actually Track

The obvious next question is*why*the smaller open models outperformed the larger proprietary ones\. The answer is that they emit more emojis\.

Table 3:Systems ordered by rawmos\. Mean emoji count tracksmosalmost perfectly; decodability does not\. winLM\{\}\_\{\\text\{LM\}\}is the within\-item length\-matched win rate, which reorders the table\.p∗⁣∗<\.01\{\}^\{\*\*\}p<\.01\.Table[3](https://arxiv.org/html/2609.29445#S5.T3)and Figure[2](https://arxiv.org/html/2609.29445#S5.F2)\(a\) show that the mean number of emojis a system emits predicts itsmoswithρ=0\.90\\rho=0\.90\(p=0\.002p=0\.002\), accounting for 78\.7% of the between\-system variance\. A judgement\-level mixed model with item and annotator random effects estimatesβ=0\.159\\beta=0\.159mospoints per additional emoji \(SE=0\.015\\text\{SE\}=0\.015,p=1\.6×10−25p=1\.6\\times 10^\{\-25\}\)\. The entire spread between the best and worst of eight frontier systems is0\.3410\.341mospoints—slightly more than two emojis’ worth of output length\.

#### Length\-matched comparison\.

Adjusting for length statistically is not sufficient, because the number of emojis a system emits is a system\-level property\. We therefore run a comparison that conditions on nothing at the system level: for every sentence and every pair of systems that happened to emit*the same number of emojis on that sentence*, we record which was rated higher\. This yields 2,599 matched comparisons\. The resulting ordering \(Table[3](https://arxiv.org/html/2609.29445#S5.T3), Figure[2](https://arxiv.org/html/2609.29445#S5.F2)\(c\)\) correlates with the rawmosordering atρ=0\.12\\rho=0\.12\(p=0\.78p=0\.78\)\. Gemini\-2\.0\-Flash moves from sixth to first \(win rate0\.6030\.603, 95% CI\[0\.540,0\.664\]\[0\.540,0\.664\]\); Grok\-4\-Fast moves from last to fourth\. The reported “efficiency–performance” effect, in which smaller models were preferred, does not survive\.

#### Who is length\-biased?

The per\-annotator slope on emoji count is0\.0850\.085\(p<10−5p<10^\{\-5\}\),−0\.024\-0\.024\(p=0\.14p=0\.14\) and0\.3290\.329\(p<10−48p<10^\{\-48\}\)\. One annotator’s system\-level ranking correlates with output length atρ=0\.95\\rho=0\.95; another’s atρ=−0\.26\\rho=\-0\.26\. The length bias, like everything else in this study, is an annotator property that the panel average silently aggregates\.

Figure 2:\(a\) Mean output length explains 78\.7% of between\-systemmosvariance\. \(b\) Emoji–affect decodability does not correlate withmos\. \(c\) Within\-item length\-matched win rates reorder the leaderboard; error bars are 95% binomial CIs\.

## 6Emoji–Affect Decodability

If preference scores cannot rank systems, something else must\. We propose a reference\-based metric that uses the emotion labels the source corpora already carry, requires no annotators, and is defined without reference to any emoji–affect lexicon, thereby avoiding the unvalidated emoji\-to\-label mapping our initial analysis relied on\.

#### Definition\.

Letggbe a generator and𝒟L=\{\(xi,yi\)\}\\mathcal\{D\}\_\{L\}=\\\{\(x\_\{i\},y\_\{i\}\)\\\}the single\-label items of languageLLover the shared five\-emotion set\. LetEg​\(x\)E\_\{g\}\(x\)be the multiset of emoji graphemesggemits forxx\(ZWJ sequences and skin\-tone modifiers kept intact\)\.ead\(g,L\)\(g,L\)is the 5\-fold cross\-validated macro\-F1 of anℓ2\\ell\_\{2\}\-regularised multinomial logistic regression over sublinear TF\-IDF features ofEg​\(x\)E\_\{g\}\(x\), predictingyy\. It measures how much of the corpus’s affective distinction survives compression intogg’s emoji channel\. A generator that emits the same three emojis for everything scores at the floor regardless of how pleasant those emojis are\.

#### Properties\.

eadis stable: across ten random seeds the largest per\-system standard deviation is0\.00330\.0033macro\-F1, and 2,000\-replicate item bootstraps give CIs of width≈0\.032\\approx 0\.032that separate the systems into three distinct tiers \(Figure[3](https://arxiv.org/html/2609.29445#S7.F3)\(c\)\)\. It is cheap: no annotators, and deterministic given a fixed set of model outputs\. And it disagrees with human preference:ρ=−0\.19\\rho=\-0\.19\(p=0\.65p=0\.65\) withmos\. Per annotator the correlation is\+0\.07\+0\.07,\+0\.76\+0\.76and−0\.40\-0\.40\. Given §[4](https://arxiv.org/html/2609.29445#S4), we do not read this as evidence againstead; a metric cannot be validated against a criterion that does not reliably order the systems\.

#### Results\.

Banglaeadranges from0\.5600\.560\(DeepSeek\) to0\.6570\.657\(Gemini\), against a majority\-class floor of0\.0870\.087; English from0\.5150\.515to0\.5930\.593; Hindi from0\.3760\.376to0\.4580\.458\(Figure[3](https://arxiv.org/html/2609.29445#S7.F3)\(b\)\)\. The union of all eight systems’ emojis reaches0\.6860\.686in Bangla, above any single system, indicating that the generators encode partly complementary information\.

#### The emoji code is language\-specific\.

Training the decoder on one language’s emoji outputs and testing on another costs0\.2520\.252macro\-F1 on average, even though target\-language vocabulary coverage exceeds99\.7%99\.7\\%in every direction: the models emit the*same*emojis across languages but use them to mean different things\. Emoji are frequently described as a language\-neutral affective interface; under a controlled test, at least as produced by these systems, they are not\.

#### Matched comparison with sentence embeddings\.

On identical items \(n≈400n\\approx 400per language\), the best emoji channel reaches0\.6730\.673macro\-F1 in Bangla against0\.5950\.595for the best sentence embedding,0\.6130\.613against0\.7360\.736in English, and0\.4300\.430against0\.3870\.387in Hindi\. It is tempting to conclude that for Bangla a five\-emoji summary carries as much affective information as a 4096\-dimensional commercial encoder\. We report a control that tempers this: the embedding learning curve is still rising steeply atn=400n=400\(0\.428→0\.480→0\.5940\.428\\to 0\.480\\to 0\.594forn=100,200,400n=100,200,400\) whereas the emoji curve is nearly flat \(0\.6140\.614atn=400n=400,0\.6570\.657atn=4,206n=4\{,\}206\)\. The embedding numbers are therefore lower bounds and we claim only that the emoji channel is*competitive*at matched sample size, not that it is superior\.

## 7Geometry: Two Corrections

#### Anisotropy differences are a mean offset\.

Following[Ethayarajh \(2019\)](https://arxiv.org/html/2609.29445#bib.bib18)we computeA⁡\(X\)=𝔼i≠j​\[cos⁡\(xi,xj\)\]A\(X\)=\\mathbb\{E\}\_\{i\\neq j\}\[\\cos\(x\_\{i\},x\_\{j\}\)\], evaluated exactly as\(‖∑iui‖2−n\)/\(n⁡\(n−1\)\)\(\\\|\\sum\_\{i\}u\_\{i\}\\\|^\{2\}\-n\)/\(n\(n\-1\)\)for unit\-normaliseduiu\_\{i\}\. Raw values differ sharply across providers:0\.1490\.149\(OpenAI\-3\-small\),0\.1500\.150\(OpenAI\-3\-large\),0\.3190\.319\(Qwen3\),0\.5740\.574\(Gemini\) and0\.7000\.700\(Mistral\)—a range we initially interpreted as a meaningful quality difference\. After subtracting the global mean, every model lies in\[−0\.0008,0\.0013\]\[\-0\.0008,0\.0013\]\(Figure[3](https://arxiv.org/html/2609.29445#S7.F3)\(a\)\)\. The differences are entirely attributable to a common displacement vector, removable by one centring operation, as[Rudman et al\. \(2022\)](https://arxiv.org/html/2609.29445#bib.bib21)predict\. Moreover, raw anisotropy does not predict downstream utility: its correlation with the pooled emotion\-probe macro\-F1 across the five encoders isr=−0\.48r=\-0\.48\(p=0\.41p=0\.41,n=5n=5\)\. We can find no sense in which the anisotropy gap matters\.

#### Language separability is not cross\-lingual alignment\.

Our initial analysis reported a “cross\-lingual divergence” of0\.090\.09for one provider against0\.720\.72for another and read the lower value as better alignment\. That inference is not available here, because the three corpora are not parallel: no Bangla sentence is a translation of any English sentence\. Any statistic comparing language\-conditional distributions is therefore confounded with topic, register and domain\. What can be measured is*separability*\. A linear language classifier reaches≥0\.993\\geq 0\.993accuracy on four of five encoders and0\.5970\.597on Gemini\-Embedding\-001, whose Fisher separation ratio is correspondingly low \(0\.0090\.009vs\.0\.1050\.105–0\.2800\.280\)\. But low separability is not a virtue: Gemini also has the weakest emotion probe of the five \(0\.2900\.290pooled macro\-F1 vs\.0\.5930\.593for Qwen3\)\. Its space does not so much align languages as fail to represent the distinctions we tested\. Establishing alignment would require parallel data and a retrieval evaluation\([Artetxe and Schwenk, 2019](https://arxiv.org/html/2609.29445#bib.bib22)\), neither of which this corpus supports\.

Figure 3:\(a\) Provider anisotropy differences vanish under mean\-centring\. \(b\) Emoji–affect decodability by system and language\. \(c\) Width of the 95% CI foreadand formosrescaled to a common range;eadis the more precise instrument\.

## 8Cost Accounting Changes Sign with the Unit

Raw token counts are not comparable across providers with different tokenizers, and the choice of normalising unit determines the direction of any per\-language conclusion\. Measured in prompt tokens per source*byte*, Bangla is the*cheapest*of the three languages \(1\.1081\.108vs\.1\.4501\.450for English and1\.5571\.557for Hindi\)\. Measured in prompt tokens per source*character*, Bangla costs2\.932\.93against English’s1\.451\.45—a2\.0×2\.0\\timespremium—because Bangla text averages2\.642\.64UTF\-8 bytes per character\. Both statements are true; the “token premium” reported previously is a statement about Unicode encoding, not about tokenizer fairness, and should be reported per character with the byte ratio disclosed\.

A second confound: raw completion\-token counts conflate emitted output with internal reasoning traces\. Grok\-4\-Fast averages294\.7294\.7completion tokens per Bangla item while emitting2\.602\.60emojis \(124124tokens per emoji\); the other seven systems average1\.41\.4–4\.44\.4\. Comparing systems on billed completion tokens compares reasoning configurations, not task efficiency\.

Finally, no cost measure predicts quality\. Spearman correlations witheadare−0\.14\-0\.14\(total tokens\),−0\.12\-0\.12\(completion tokens\) and−0\.14\-0\.14\(tokens per emoji\), none significant\. The only cost\-like quantity that predictsmosis the number of emojis emitted \(ρ=0\.90\\rho=0\.90\), which §[5](https://arxiv.org/html/2609.29445#S5)shows to be a bias rather than a quality signal\.

## 9Multi\-View Leakage, Measured

A benchmark that storesVVviews of each ofNNsource items \(hereV=8V=8generators, orV=5V=5encoders\) and is then split row\-wise places the same source sentence on both sides of the partition\. The probability that no item is split is\(ρV\+\(1−ρ\)V\)N\(\\rho^\{V\}\+\(1\-\\rho\)^\{V\}\)^\{N\}, which forV≥2V\\geq 2andN≥100N\\geq 100is numerically zero at any usualρ\\rho; reporting it as a percentage close to 100, as we initially did, is uninformative\. The decision\-relevant quantity is the expected fraction of test rows whose source item also appears in training,1−\(1−ρ\)V−11\-\(1\-\\rho\)^\{V\-1\}:99\.998%99\.998\\%atV=8,ρ=0\.8V=8,\\rho=0\.8and99\.84%99\.84\\%atV=5V=5\.

We measure the resulting optimism\. On the Bangla multi\-view table \(4,206 items×\\times8 views=33,648=33\{,\}648rows\), a naive row\-wise 80/20 split gives0\.630±0\.0030\.630\\pm 0\.003macro\-F1 against0\.598±0\.0140\.598\\pm 0\.014for a split grouped by source item:\+3\.1\+3\.1points absolute,\+5\.3%\+5\.3\\%relative \(t=4\.97t=4\.97,p=0\.0006p=0\.0006, 6 replicates\)\. The inflation is stable at1\.71\.7–2\.32\.3points across split ratios from0\.50\.5to0\.90\.9\. More importantly it perturbs the leaderboard: system rankings under leaky and clean protocols correlate atρ=0\.76\\rho=0\.76and the top\-ranked system changes from Mistral\-Large to Gemma\-3\-27B\. Grouped splitting is not a hygiene footnote; it changes which system a benchmark declares best\.

## 10Recommendations

1. 1\.Report agreement with a bias\-sensitive coefficient\.Pearson correlation between annotators is not agreement\. Report Krippendorff’sα\\alphaor an ICC with a bootstrap CI, and report annotator mean and SD separately\.
2. 2\.Treat annotators as a random factor\.If the claim is about systems in general rather than about three specific people, the error term must include the system×\\timesannotator interaction\. Reporting a paired test over items on panel\-averaged scores is not sufficient\.
3. 3\.Run a decision study before scaling up collection\.A small fully crossed pilot is enough to estimateσs​r2\\sigma^\{2\}\_\{sr\}and hence the panel size the intended claim requires\. If more items cannot fix the design, that is worth knowing before 6,960 judgements are collected\.
4. 4\.Length\-match, do not length\-adjust\.Report a within\-item comparison restricted to outputs of equal length alongside the raw score\.
5. 5\.Report a reference\-based metric next to preference\.eador an analogue gives a reproducible number that does not move when the panel changes\.
6. 6\.Centre embeddings before reporting anisotropy; do not describe non\-parallel corpora as evidence of cross\-lingual alignment; normalise token counts by characters and disclose the bytes\-per\-character ratio; split multi\-view tables by source item\.

## Limitations

The human study has three annotators, all of them university\-educated Bangla speakers from a similar age band; the variance components we report are estimates fromnr=3n\_\{r\}=3and are correspondingly imprecise, and the population of annotators they generalise to is narrow\. A larger and demographically broader panel could plausibly show smaller annotator variance, though our decision study indicates the panel would need to be an order of magnitude larger before the current design became reliable\. We report per\-annotator results throughout rather than only the panel average, since the panel average is precisely the quantity our analysis shows to be unstable\.

The human study covers Bangla only, so the reliability findings are demonstrated rather than shown to be universal; we expect them to be worse, not better, for languages with fewer available annotators\. Our three languages are all South Asian or English, and we avoid claims about “multilingual affective AI” in general\.

eadinherits whatever biases the source corpora’s emotion labels carry, and the three corpora differ in label scheme, granularity and provenance; we therefore compare systems only within a language, never across\. The metric also rewards systems whose emoji usage is*consistent*with the corpus, which is not the same as being good, useful or culturally appropriate—a system that consistently uses a culturally inappropriate emoji for an emotion would score well\.eadreplaces the lexicon\-based consistency metric it supersedes, not human judgement properly powered\.

The geometry analyses use 1,202 sentences and five commercial encoders accessed through an API; we do not have access to their training data, and API models can change without notice, so the geometric measurements are properties of the endpoints as we queried them\. The embedding\-versus\-emoji comparison is sample\-limited as described in §[6](https://arxiv.org/html/2609.29445#S6)\.

Finally, this is an audit of a single benchmark that we built ourselves\. We believe the failure modes generalise because the design is conventional, but we have not demonstrated that they do\.

## Ethics Statement

The three annotators are fluent Bangla speakers who worked independently of the research team, participated voluntarily, and were informed of the purpose of the study before beginning\. The sentences derive from publicly available Bangla, English and Hindi corpora and were not filtered for personally identifying content beyond what the source corpora applied, and we work from sentence identifiers and model outputs under the source corpora’s licences\. Emotion attribution to text written by identifiable individuals carries a risk of misrepresentation, and we caution against deploying emoji\-based affect systems in consequential settings, particularly given that our best system recovers only 66% of a five\-way emotion distinction in Bangla\. Model outputs were not filtered for offensive emoji use; we observed no such cases but did not audit exhaustively\.

## References

- Artetxe and Schwenk \(2019\)M\. Artetxe and H\. SchwenkMassively multilingual sentence embeddings for zero\-shot cross\-lingual transfer and beyond\.Transactions of the Association for Computational Linguistics7,pp\. 597–610\.External Links:[Link](https://aclanthology.org/Q19-1038)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2609.29445#S7.SS0.SSS0.Px2.p1.1)\.
- Barbieriet al\.\(2018\)F\. Barbieri, J\. Camacho\-Collados, F\. Ronzano, L\. Espinosa\-Anke, M\. Ballesteros, V\. Basile, V\. Patti, and H\. SaggionSemEval\-2018 task 2: multilingual emoji prediction\.InProceedings of the 12th International Workshop on Semantic Evaluation,pp\. 24–33\.External Links:[Link](https://aclanthology.org/S18-1003)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px3.p1.1)\.
- Brennan \(2001\)R\. L\. BrennanGeneralizability theory\.Springer,New York\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1),[§4\.3](https://arxiv.org/html/2609.29445#S4.SS3.p2.1)\.
- Cardet al\.\(2020\)D\. Card, P\. Henderson, U\. Khandelwal, R\. Jia, K\. Mahowald, and D\. JurafskyWith little power comes great responsibility\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9263–9274\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.745)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Clarket al\.\(2021\)E\. Clark, T\. August, S\. Serrano, N\. Haduong, S\. Gururangan, and N\. A\. SmithAll that’s ‘human’ is not gold: evaluating human evaluation of generated text\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 7282–7296\.External Links:[Link](https://aclanthology.org/2021.acl-long.565)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Cronbachet al\.\(1972\)L\. J\. Cronbach, G\. C\. Gleser, H\. Nanda, and N\. RajaratnamThe dependability of behavioral measurements: theory of generalizability for scores and profiles\.John Wiley,New York\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Daset al\.\(2021\)A\. Das, O\. Sharif, M\. M\. Hoque, and I\. H\. SarkerEmotion classification in a resource constrained language using transformer\-based approach\.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop,pp\. 150–158\.External Links:[Link](https://aclanthology.org/2021.naacl-srw.19)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px3.p1.1)\.
- Duboiset al\.\(2024\)Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. HashimotoLength\-controlled AlpacaEval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px2.p1.1)\.
- Ethayarajh \(2019\)K\. EthayarajhHow contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT\-2 embeddings\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 55–65\.External Links:[Link](https://aclanthology.org/D19-1006)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2609.29445#S7.SS0.SSS0.Px1.p1.1)\.
- Fenget al\.\(2022\)F\. Feng, Y\. Yang, D\. Cer, N\. Arivazhagan, and W\. WangLanguage\-agnostic BERT sentence embedding\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 878–891\.External Links:[Link](https://aclanthology.org/2022.acl-long.62)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px4.p1.1)\.
- Gaoet al\.\(2021\)T\. Gao, X\. Yao, and D\. ChenSimCSE: simple contrastive learning of sentence embeddings\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 6894–6910\.External Links:[Link](https://aclanthology.org/2021.emnlp-main.552)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px4.p1.1)\.
- Golchin and Surdeanu \(2024\)S\. Golchin and M\. SurdeanuTime travel in LLMs: tracing data contamination in large language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px5.p1.1)\.
- Grahamet al\.\(2015\)Y\. Graham, T\. Baldwin, and N\. MathurAccurate evaluation of segment\-level machine translation metrics\.InProceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,pp\. 1183–1191\.External Links:[Link](https://aclanthology.org/N15-1124)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Islamet al\.\(2022\)K\. I\. Islam, T\. Yuvraz, S\. Kar, and M\. S\. HasanEmoNoBa: a dataset for analyzing fine\-grained emotions on noisy Bangla texts\.InProceedings of the 2nd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing \(Volume 2: Short Papers\),pp\. 128–134\.External Links:[Link](https://aclanthology.org/2022.aacl-short.17)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px3.p1.1)\.
- Kralj Novaket al\.\(2015\)P\. Kralj Novak, J\. Smailović, B\. Sluban, and I\. MozetičSentiment of emojis\.PLOS ONE10\(12\),pp\. e0144296\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px3.p1.1)\.
- Krippendorff \(2018\)K\. KrippendorffContent analysis: an introduction to its methodology\.4th edition,SAGE Publications,Thousand Oaks, CA\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2020\)B\. Li, H\. Zhou, J\. He, M\. Wang, Y\. Yang, and L\. LiOn the sentence embeddings from pre\-trained language models\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 9119–9130\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.733)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px4.p1.1)\.
- Mathuret al\.\(2020\)N\. Mathur, T\. Baldwin, and T\. CohnTangled up in BLEU: reevaluating the evaluation of automatic machine translation evaluation metrics\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 4984–4997\.External Links:[Link](https://aclanthology.org/2020.acl-main.448)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Raihanet al\.\(2024\)N\. Raihan, D\. Goswami, A\. Mahmud, A\. Anastasopoulos, and M\. ZampieriEmoMix\-3L: a code\-mixed dataset for Bangla\-English\-Hindi emotion detection\.InProceedings of the 7th Workshop on Indian Language Data: Resources and Evaluation \(WILDRE\-7\) at LREC\-COLING 2024,External Links:[Link](https://aclanthology.org/2024.wildre-1.2)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px3.p1.1)\.
- Rudmanet al\.\(2022\)W\. Rudman, N\. Gillman, T\. Rayne, and C\. EickhoffIsoScore: measuring the uniformity of embedding space utilization\.InFindings of the Association for Computational Linguistics: ACL 2022,pp\. 3325–3339\.External Links:[Link](https://aclanthology.org/2022.findings-acl.262)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2609.29445#S7.SS0.SSS0.Px1.p1.1)\.
- Sainzet al\.\(2023\)O\. Sainz, J\. A\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. AgirreNLP evaluation in trouble: on the need to measure LLM data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10776–10787\.External Links:[Link](https://aclanthology.org/2023.findings-emnlp.722)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px5.p1.1)\.
- Shoeb and de Melo \(2020\)A\. A\. M\. Shoeb and G\. de MeloEmoTag1200: understanding the association between emojis and emotions\.InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 8957–8967\.External Links:[Link](https://aclanthology.org/2020.emnlp-main.720)Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px3.p1.1)\.
- Shrout and Fleiss \(1979\)P\. E\. Shrout and J\. L\. FleissIntraclass correlations: uses in assessing rater reliability\.Psychological Bulletin86\(2\),pp\. 420–428\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px1.p1.1)\.
- Singhalet al\.\(2024\)P\. Singhal, T\. Goyal, J\. Xu, and G\. DurrettA long way to go: investigating length correlations in RLHF\.arXiv preprint arXiv:2310\.03716\.Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems 36 \(NeurIPS 2023\), Datasets and Benchmarks Track,Cited by:[§2](https://arxiv.org/html/2609.29445#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix APrompt

Identical for all systems and all three languages, sent as a single user turn with no system prompt:

> You MUST respond with 1\-5 emojis representing the emotion in this sentence\. CRITICAL: Your response MUST contain emojis\. Empty responses are NOT allowed\. Do NOT include any text, words, or explanations \- ONLY emojis\. Sentence: \{sentence\} Emojis \(REQUIRED\):

Decoding:temperature=0\.7=0\.7,max\_tokens=50=50, provider defaults fortop\_pand stop sequences, no seed \(the provider gateway does not expose one\)\. A response containing no character above U\+007F triggered a retry, up to three attempts; observed retry rates were0\.0010\.001–0\.0640\.064per system\.

## Appendix BAnalysis Settings

Every analysis in this paper runs on CPU from the stored model outputs; no further API access is needed once generation and embedding are complete\. Random seeds are fixed throughout\. Bootstrap replicate counts are 2,000 for agreement coefficients andeadrank distributions and 5,000 formosconfidence intervals\. Probes areℓ2\\ell\_\{2\}\-regularised multinomial logistic regressions \(C=4C=4on sublinear TF\-IDF emoji features,C=1C=1onℓ2\\ell\_\{2\}\-normalised embeddings\) under 5\-fold stratified cross\-validation, with class\-balanced weights throughout\.

Emoji are segmented into graphemes, keeping zero\-width\-joiner sequences and skin\-tone modifiers attached to their base character and discarding the occasional stray ASCII a model emits despite the prompt\. This matters for §[5](https://arxiv.org/html/2609.29445#S5): naive per\-codepoint segmentation inflates the mean emoji count by 15\.0% overall, but unevenly across systems \(from\+5\.4%\+5\.4\\%for Claude\-3\-Haiku to\+31\.5%\+31\.5\\%for DeepSeek\-V3\.2\), so it distorts the between\-system length ordering rather than merely rescaling it\.