The Translation Tax Is Not a Scalar: A Counterfactual Audit of English-Source Cue Inheritance in Chinese Multilingual Benchmarks

# The Translation Tax Is Not a Scalar: A Counterfactual Audit of English-Source Cue Inheritance in Chinese Multilingual Benchmarks
Source: [https://arxiv.org/html/2605.07093](https://arxiv.org/html/2605.07093)
###### Abstract

The Translation Tax is often treated as a scalar: translated benchmarks are assumed to inflate scores by preserving English\-source cues\. We audit this claim in an English\-to\-Chinese setting\. Three proxy estimators disagree: back\-translation gaps are small and parser\-fragile; cue\-score calibration does not predict item\-level gains; and a six\-model native\-control comparison shows model\-family rather than uniform benchmark effects\. We add a same\-item LLM\-naturalization stress test that holds answer, options, and content fixed while rewriting Chinese surface form\. After correcting a prompt\-construction bug, this contrast no longer supports a model\-family interaction, but it preserves a residue dose\-response: high\-residue items benefit while low\-residue items do not\. The result is not a single Translation Tax, but a set of estimator\- and item\-dependent validity risks\. We release per\-cell evidence, the naturalization protocol, human QC, and a reporting checklist for translated multilingual benchmark papers\.

## 1Introduction

The de facto standard for evaluating multilingual capability of LLMs depends on translated benchmarks\. MMMLU is the multilingual translation of MMLU\(Hendrycks et al\.,[2021](https://arxiv.org/html/2605.07093#bib.bib5)\)released by OpenAI in 2024; the released MMMLU test split used here contains14,04214\{,\}042items across5757subjects, and the Chinese \(ZH\_CN\) subset matches this count\.111We confirmed the released item count programmatically against the OpenAI MMMLU dataset card; earlier drafts of this paper reported15,90815\{,\}908items following a citation that conflated the MMLU benchmark size with a related dataset\.Belebele\(Bandarkar et al\.,[2024](https://arxiv.org/html/2605.07093#bib.bib2)\)is a parallel reading comprehension dataset spanning 122 language variants with approximately 900 items per variant\. XCOPA\(Ponti et al\.,[2020](https://arxiv.org/html/2605.07093#bib.bib7)\)covers 11 languages with all non\-English data translated from English COPA\. The translation processes were carefully implemented, but the structural fact that all non\-English items derive from a common English source introduces a systematically underestimated side effect: model performance on non\-English items may reflect not only true language understanding but also identification of English cues preserved through translation\.Artetxe et al\. \([2020](https://arxiv.org/html/2605.07093#bib.bib1)\)demonstrated precisely this mechanism in cross\-lingual transfer, showing that translation introduces subtle artifacts that models exploit as non\-semantic shortcuts\.

The NeurIPS 2025 workshop*Centering Low Resource Languages and Cultures in the Age of Large Language Models*demonstrated rapidly rising community attention to multilingual evaluation\.Sitaram \([2025](https://arxiv.org/html/2605.07093#bib.bib9)\)explicitly identified three core challenges of multilingual benchmark evaluation: coverage, representativeness, and trust/scientific rigor\. Translated English benchmarks were listed alongside US\-centric framing as the two main representativeness deficiencies\.Wu et al\. \([2025](https://arxiv.org/html/2605.07093#bib.bib10)\)sharpened this concern by showing that translated benchmarks align far worse with local human judgments than natively constructed alternatives \(Spearman0\.470\.47vs\.0\.680\.68\)\. Yet rigorous quantification of how much benchmark score is driven by translation cues rather than true language understanding remains scarce\.

#### Contributions\.

The paper makes three contributions\.

1. 1\.Conceptual\.The Translation Tax is defined as the cue\-driven component of translation\-induced score shift, distinguished from semantic translation error\. Each estimator is paired with explicit identification assumptions and named failure modes; a fourth estimator \(E4, matched LLM\-naturalization stress test\) is introduced to hold item content fixed between conditions\.
2. 2\.Empirical\.Three proxy estimators on a 228\-item MMMLU and 100\-item Belebele subset converge on the same diagnosis: small back\-translation gaps that are parser\-fragile, an annotation calibration that does not support item\-level cue exploitation, and a six\-model native\-control comparison in which the largest gaps appear within the Chinese\-optimized subgroup rather than the English\-centric one\. E4 \(after a corrected prompt\-construction bug\) returns a small positive average effect concentrated on items selected ex ante for high translation residue \(Δhigh=\+0\.103\\Delta\_\{\\text\{high\}\}=\+0\.103vs\.Δlow=−0\.015\\Delta\_\{\\text\{low\}\}=\-0\.015excl\. parser outlier\) and no statistically significant model\-family interaction\. The Translation Tax is not a single scalar correction but an estimator\- and item\-dependent set of validity risks\.
3. 3\.Reporting\.A translation\-cue identifiability reporting checklist covers estimator scope, parser fragility, and subgroup contrasts that translated multilingual benchmark papers can adopt as a standard validity dimension at submission time\.

#### Scope\.

Point estimates are small \(11–55percentage points\) and most individual cells do not exclude zero\. The paper reports proxy estimates with their identification assumptions disclosed; the latent Translation Tax is not directly measured by any single estimator\. The annotation estimator is executed only as a 30\-item single\-annotator rubric calibration, treated as calibration evidence rather than a full bilingual\-human estimator \(Section[5\.6](https://arxiv.org/html/2605.07093#S5.SS6)\)\. The study uses English\-to\-Chinese as a single case pair to enable detailed within\-pair analysis rather than thin breadth across many languages\.

## 2Related Work

XNLI\(Conneau et al\.,[2018](https://arxiv.org/html/2605.07093#bib.bib4)\)extending MultiNLI to 14 non\-English languages explicitly discussed translation effects on evaluation validity\. XCOPA\(Ponti et al\.,[2020](https://arxiv.org/html/2605.07093#bib.bib7)\)discussed translation effects in its Limitations section\. Belebele\(Bandarkar et al\.,[2024](https://arxiv.org/html/2605.07093#bib.bib2)\)detailed its FLORES\-200 translation pipeline quality control while acknowledging traces detectable by models\.Artetxe et al\. \([2020](https://arxiv.org/html/2605.07093#bib.bib1)\)systematically analyzed how translation artifacts in cross\-lingual benchmarks create exploitable shortcuts: independent translation of premises and hypotheses reduced lexical overlap relative to English originals, introducing a spurious signal\. Their work is the most direct predecessor to ours, though they focused on diagnosing artifact types rather than quantifying score inflation across benchmarks\.

Wu et al\. \([2025](https://arxiv.org/html/2605.07093#bib.bib10)\), surveying over 2,000 multilingual benchmarks published between 2021 and 2024, provided the most comprehensive recent evidence that translation quality directly undermines the validity of human preference alignment in multilingual evaluation\.Clark et al\. \([2020](https://arxiv.org/html/2605.07093#bib.bib3)\)introduced TyDi QA, designed to avoid translationese artifacts by collecting questions natively\. INCLUDE\(Romanou et al\.,[2025](https://arxiv.org/html/2605.07093#bib.bib8)\)sources items natively from regional exam pools across 44 languages; we use its Chinese subset as the non\-translation control source for E3\.

Prompt sensitivity\(Mizrahi et al\.,[2024](https://arxiv.org/html/2605.07093#bib.bib6); Zhuo et al\.,[2024](https://arxiv.org/html/2605.07093#bib.bib11)\)is complementary to our focus on the linguistic validity of evaluation data\. The work nearest to ours isArtetxe et al\. \([2020](https://arxiv.org/html/2605.07093#bib.bib1)\), which we extend by separating cue inheritance from translation noise and by quantifying score\-level effects rather than diagnosing artifact types\.

## 3Formal Definitions and Identification

#### Translation Tax\.

For a translated benchmarkBBin target languageLLand modelMM, letscoreL​\(M\)\\text\{score\}\_\{L\}\(M\)denote the observed score on the translated benchmark andscoreLnatural​\(M\)\\text\{score\}\_\{L\}^\{\\text\{natural\}\}\(M\)denote the \(unobservable\) score on a hypothetical natural\-LLversion of the same items\. The translation effect decomposes as:

scoreL​\(M\)−scoreLnatural​\(M\)=TT\+E,\\text\{score\}\_\{L\}\(M\)\-\\text\{score\}\_\{L\}^\{\\text\{natural\}\}\(M\)=\\text\{TT\}\+E,whereTTis the Translation Tax \(cue\-driven inflation from residual source\-language structures\) andEEis the Semantic Error Effect \(degradation from meaning distortion\)\. SincescoreLnatural\\text\{score\}\_\{L\}^\{\\text\{natural\}\}is unobservable, TT is not directly identified from a single observable contrast\. We use three complementary proxy estimators, each with its own identification assumptions, and triangulate \(Figure[1](https://arxiv.org/html/2605.07093#S3.F1)\)\.

![Refer to caption](https://arxiv.org/html/2605.07093v1/x1.png)Figure 1:What each estimator varies between conditions\. E1, E2, and E3 each contrast scores across items that differ in content \(translation noise, item identity, or benchmark register\), so none of them holds content fixed\. E4 \(matched naturalization, Section[6](https://arxiv.org/html/2605.07093#S6)\) is the only contrast in the paper that varies surface naturalness with content held fixed\.
### 3\.1Estimator E1 \(Back\-Translation\): Identification and Failure Modes

For an English\-centric modelMENM\_\{\\text\{EN\}\}, letTbackT\_\{\\text\{back\}\}denote the L\-to\-English back\-translation pipeline andqLback=Tback​\(qL\)q\_\{L\}^\{\\text\{back\}\}=T\_\{\\text\{back\}\}\(q\_\{L\}\)\. The back\-translation estimator is:

TTback​\(B,L,MEN\)=scoreENqEN​\(MEN\)−scoreENqLback​\(MEN\)\.\\text\{TT\}\_\{\\text\{back\}\}\(B,L,M\_\{\\text\{EN\}\}\)=\\text\{score\}\_\{\\text\{EN\}\}^\{q\_\{\\text\{EN\}\}\}\(M\_\{\\text\{EN\}\}\)\-\\text\{score\}\_\{\\text\{EN\}\}^\{q\_\{L\}^\{\\text\{back\}\}\}\(M\_\{\\text\{EN\}\}\)\.TTback\{\}\_\{\\text\{back\}\}measures the score gap that an English\-centric model exhibits between original\-English items and items round\-tripped through L\. If back\-translation preserves semantic content, this gap reflects the structural cue residue lost in the round trip\.

#### Failure modes \(where E1 does not identify TT\)\.

\(F1\)*Back\-translation noise\.*Real pipelines introduce semantic noiseEbackE\_\{\\text\{back\}\}, soTTback≈TT\+Eback\\text\{TT\}\_\{\\text\{back\}\}\\approx\\text\{TT\}\+E\_\{\\text\{back\}\}, making the estimator a noisy upper bound\. BLEU/BERTScore QC reduces but does not eliminateEbackE\_\{\\text\{back\}\}\. \(F2\)*Naturalness loss\.*Back\-translation may produce grammatical but unnatural English; the observed gap can include English\-text\-quality penalties unrelated to cue inheritance\. \(F3\)*Asymmetric translation difficulty\.*Some content \(e\.g\., Chinese\-specific cultural items in MMMLU localized subjects\) may be harder to back\-translate than to translate forward, biasingEbackE\_\{\\text\{back\}\}in subject\-correlated ways\. The paper therefore does not interpret a positiveTTback\\text\{TT\}\_\{\\text\{back\}\}as evidence of cue exploitation by itself; it functions as one of three triangulating signals\.

### 3\.2Estimator E2 \(Bilingual Native Annotation\): Calibration Status

For a stratified random sample of items, two or more bilingual native annotators score each item on a 5\-point Likert scale on three dimensions—*cue identifiability*,*cultural residue*, and*syntactic residue*—and Spearman correlation between cue identifiability and item\-level TTback\{\}\_\{\\text\{back\}\}signal is computed by model subgroup\. The full protocol blinds annotators to all model outputs, uses anchor\-calibrated rubric prompts, includes at least one external bilingual annotator, and reports weighted Cohen’sκ\\kappaand Krippendorff’sα\\alphaon the ordinal Likert scale \(target weightedκ\>0\.7\\kappa\>0\.7\)\.

#### What is executed in this study\.

A 30\-item single\-annotator rubric\-calibration run with one LLM\-rubric annotator\. This is calibration evidence on rubric coverage and score distribution, not the bilingual\-human estimator\. It cannot support or refute the cue\-exploitation hypothesis at this scale and is reported as a null calibration result \(Section[5\.6](https://arxiv.org/html/2605.07093#S5.SS6)\)\. Triangulation in the results section therefore rests on E1 and E3; E2 enters as a calibration check only\.

### 3\.3Estimator E3 \(INCLUDE Non\-Translation Control\): Identification Scope

INCLUDE\(Romanou et al\.,[2025](https://arxiv.org/html/2605.07093#bib.bib8)\)sources items natively from Chinese regional exam pools rather than via translation; INCLUDE\-base\-44 Chinese contains 545 four\-choice test items spanning 57 topics\. The paper compares aggregate \(per\-model\) accuracy on translated benchmark Chinese against INCLUDE Chinese\.

#### Identification scope\.

INCLUDE Chinese items are drawn from different content distributions than MMMLU Chinese \(regional exams vs\. academic subjects\)\. The comparison is unmatched at the item level: it does not match items on difficulty, topic, or register, and is therefore an external\-validity comparison rather than an item\-level counterfactual\. Item\-level effects cannot be isolated from content\-distribution effects under this design\.

## 4Experimental Design

### 4\.1Benchmarks and Models

#### Benchmarks\.

\(1\) MMMLU Chinese subset: theZH\_CNtest split released by OpenAI contains 14,042 items across 57 subjects, matching the MMLU test split used by the released MMMLU dataset; we draw a stratified subset of 228 items at 4 per subject \(seed 42\)\. \(2\) Belebele Chinese subset \(zho\_Hans; 900 items\); subset of 100 items \(seed 42\)\. \(3\) INCLUDE Chinese subset as non\-translation control \(545 test items, 4\-choice\)\.

#### Models\.

Nine total in three groups, all accessed via a unified provider gateway:

- •Group A \(English\-centric frontier\): gpt\-4o, gpt\-4o\-mini, gpt\-5\.4\-mini\.
- •Group B \(Chinese\-optimized\): deepseek\-chat, qwen\-max, glm\-4\.5\.
- •Group C \(Open\-source multilingual mid\-sized\): llama\-3\.3\-70b\-instruct, qwen2\.5\-72b\-instruct, glm\-4\-air\.

Anthropic Claude and Google Gemini could not be included because the gateway provider’s channels for both were unavailable during the analysis window\. This narrows Group A’s coverage and is discussed in Limitations\.

### 4\.2Scoring Protocol

#### Scoring protocol\.

Provider\-API access does not uniformly expose log\-probabilities across all nine models, so we use afixed\-prompt, single\-letter\-extraction protocol, not thelm\-evaluation\-harnesslibrary\. Specifically: zero\-shot prompting with the question, four labeled choices, and an instruction to respond with a single letter \(A, B, C, or D\); answer parsing extracts the first matching letter from the response\. We report parser validity rates per model \(the proportion of responses that yielded a parsable letter\); models with low validity rates are flagged\. We use 0\-shot for all benchmarks at this stage; a 5\-shot replication is not part of this study\.

#### Background and rationale\.

The original literature on these benchmarks reports 5\-shot results for MMMLU\. The 0\-shot setup here is therefore not directly comparable to published benchmark numbers in absolute terms; it is internally consistent across our three quantities \(qZHq\_\{\\text\{ZH\}\},qENq\_\{\\text\{EN\}\},qbackq\_\{\\text\{back\}\}\) for the same model, which is what TTback\{\}\_\{\\text\{back\}\}requires\.

#### Validity statistics\.

Across the MMMLU 228\-item sample, eight of nine models achieve parser validity≥0\.978\\geq 0\.978\. The exception is gpt\-5\.4\-mini \(0\.8250\.825\), which sometimes returns multi\-token reasoning before producing a letter; we report results for this model separately and exclude itsqZHq\_\{\\text\{ZH\}\}items with parser failure from per\-model accuracy estimates\.

### 4\.3Back\-Translation Pipeline

We use a commercial LLM \(deepseek\-chat\) as the primary back\-translation pipeline because \(a\) it produced higher BLEU scores than NLLB\-200 distilled in preliminary tests, \(b\) the local NLLB\-200 1\.3B model exceeded our sandbox memory budget\. Note that this conflicts with the methodological preference for an open reproducible pipeline; an NLLB\-200 distilled\-600M comparison is not part of this study\.

#### Quality control\.

For each item, we compute BLEU betweenqENq\_\{\\text\{EN\}\}\(the original English item\) andqLbackq^\{\\text\{back\}\}\_\{L\}\(the back\-translated item\)\. We attempted BERTScore F1 withroberta\-large; the model download exceeded our sandbox disk budget during this analysis, so BLEU is the sole QC criterion\. Items with BLEU<0\.30<0\.30are flaggedexcluded\_qcand strictly excluded from the TTback\{\}\_\{\\text\{back\}\}paired analysis\.

#### QC pass rates\.

MMMLU sample: 213/228 items pass \(93\.4%93\.4\\%, mean BLEU0\.5680\.568\)\. Belebele sample: 87/100 items pass \(87\.0%87\.0\\%, mean BLEU0\.4040\.404\)\.

#### Source alignment audit\.

A separate concern is whether theqENq\_\{\\text\{EN\}\},qZHq\_\{\\text\{ZH\}\}, andqbackq^\{\\text\{back\}\}records correspond to the same underlying source item\. Misaligned English references would mechanically bias TTback\{\}\_\{\\text\{back\}\}\. We ran a triple\-alignment audit against the MMMLU 228\-item sample: all 228 IDs align across the three sources, and all 228 items have matching choice\-counts \(4 each\)\. We then checked whether the gold\-answer index is consistent betweenqENq\_\{\\text\{EN\}\}andqZHq\_\{\\text\{ZH\}\}as a content\-level alignment proof, and found 2 mismatches \(items05954,06001, both inhigh\_school\_world\_history\): the Chinese item and English item are entirely different questions, indicating a source\-side alignment error in MMMLUZH\_CNfor those positions\. Both flagged items are already excluded by the BLEU<0\.30<0\.30QC filter \(BLEU=0\.041=0\.041and0\.0250\.025\), so the strict\-QC subset below is unaffected\. The audit script and detailed results are released asrun\_alignment\_audit\.pyandresults/alignment\_audit\.json\.

## 5Results

We reportstrict\-QCTTback\{\}\_\{\\text\{back\}\}estimates: paired comparisons restricted to items where \(a\) the back\-translation passed BLEU QC, \(b\) the model produced parsable answers on bothqENq\_\{\\text\{EN\}\}andqLbackq^\{\\text\{back\}\}\_\{L\}\. This corrects an issue in the initial earlier scoring run where some QC\-failed items were included\.

### 5\.1TTback\{\}\_\{\\text\{back\}\}Point Estimates and Confidence Intervals

Table[1](https://arxiv.org/html/2605.07093#S5.T1)reports the strict\-QC TTback\{\}\_\{\\text\{back\}\}paired bootstrap estimates \(10,000 resamples, seed 42\) for the three Group\-A models on the two benchmarks\.

Table 1:Strict\-QC TTback\{\}\_\{\\text\{back\}\}estimates \(paired bootstrap, 10,000 resamples\)\. Strict QC restricts to items with BLEU≥0\.30\\geq 0\.30*and*valid model predictions on bothqENq\_\{\\text\{EN\}\}andqLbackq^\{\\text\{back\}\}\_\{L\}\. Discordant counts are paired items where one set is correct and the other incorrect; ties are paired items where both are correct or both are incorrect\.#### Empirical pattern\.

Five of six cells produce positive point estimates and one is exactly zero; no cell is negative\. One cell, MMMLU/gpt\-5\.4\-mini, has a bootstrap CI that narrowly excludes zero \(\[\+0\.006,\+0\.095\]\[\+0\.006,\+0\.095\]\), but this is also the model with the lowest parser validity \(Section[4](https://arxiv.org/html/2605.07093#S4)\) and does not pass the exact sign\-test criterion \(1212/44discordant, sign\-testp=0\.077p=0\.077\)\. The cell is therefore a fragile positive rather than a robust significant result\. The other five cells have bootstrap CIs that include zero, with the Belebele/gpt\-4o lower bound touching zero\. Figure[2](https://arxiv.org/html/2605.07093#S5.F2)visualizes the same numbers\.

![Refer to caption](https://arxiv.org/html/2605.07093v1/x2.png)Figure 2:Strict\-QC TTback\{\}\_\{\\text\{back\}\}across six \(model×\\timesbenchmark\) cells\. Circles: MMMLU; squares: Belebele\. Point estimates with 95% paired\-bootstrap CIs\. Five of six cells are positive and one is exactly zero; most individual CIs contain zero\. Effect sizes are small in magnitude \(range0\.0000\.000–0\.0470\.047\)\. The largest cell \(MMMLU×\\timesgpt\-5\.4\-mini\) is parser\-fragile\.

### 5\.2Cross\-Cell Pattern and Parser Sensitivity

All non\-zero point estimates have the same sign \(five positive, one exactly zero\); the cells share pipeline, items, and scoring and are not independent\. The pattern is descriptive directional evidence; the item\-clustered bootstrap of Section[5\.3](https://arxiv.org/html/2605.07093#S5.SS3)is the inferential summary of record\.222Treating the five non\-zero cells as independent draws under a no\-effect null gives an exact sign\-testp=2−5=0\.0313p=2^\{\-5\}=0\.0313one\-sided\. The independence assumption is not supported by the design and the binomial\-tail interpretation does not apply\.The largest single cell estimate \(\+0\.047\+0\.047on MMMLU/gpt\-5\.4\-mini\) comes from the model with lowest parser validity \(0\.8250\.825onqZHq\_\{\\text\{ZH\}\}\); excluding gpt\-5\.4\-mini leaves three positive cells and one zero\. The qualitative finding survives this exclusion; the≥0\.95\\geq 0\.95parser\-validity subset reduces to gpt\-4o and gpt\-4o\-mini cells\.

### 5\.3Item\-Clustered Bootstrap \(Primary Inference\)

Because the six cells share items, scoring, and pipeline, a cross\-cell binomial test overstates evidence by treating correlated observations as independent\. We therefore report an item\-level cluster bootstrap as the primary inferential object: each cluster is one\(benchmark,item\)\(\\text\{benchmark\},\\text\{item\}\)pair containing up to three Group\-A model observations ofaccEN−accback\\text\{acc\}\_\{\\text\{EN\}\}\-\\text\{acc\}\_\{\\text\{back\}\}\. Resamplingnclusters=296n\_\{\\text\{clusters\}\}=296clusters with replacement \(B=10,000B=10\{,\}000, seed4242\) gives

TT^cluster=\+0\.022,95%​CI=\[\+0\.001,\+0\.044\]\.\\widehat\{\\text\{TT\}\}\_\{\\text\{cluster\}\}=\+0\.022,\\quad 95\\%\\text\{ CI\}=\[\+0\.001,\+0\.044\]\.The CI excludes zero, providing the paper’s strongest single piece of evidence for a small positive translation tax averaged across items and Group\-A models, after item correlation is accounted for\. Two sensitivity checks: excluding gpt\-5\.4\-mini \(the parser\-fragile model\) gives\+0\.015​\[−0\.008,\+0\.039\]\+0\.015\\,\[\-0\.008,\+0\.039\], with the CI now including zero; restricting to MMMLU items only gives\+0\.026​\[−0\.002,\+0\.054\]\+0\.026\\,\[\-0\.002,\+0\.054\], with the lower bound just below zero\. The item\-clustered estimate is consistent with the cell\-level pattern but is more conservative than a cross\-cell sign test because it does not assume independence across cells; we treat it, rather than the binomial, as the primary quantitative summary\.

### 5\.4Group\-Level Accuracy onqZHq\_\{\\text\{ZH\}\}\(MMMLU\)

We tested whether English\-centric models \(Group A\) outperform Chinese\-optimized \(Group B\) or open\-source multilingual \(Group C\) on translated Chinese MMMLU\. Group means:Aen=0\.705\\mathrm\{A\_\{en\}\}=0\.705,Bzh=0\.694\\mathrm\{B\_\{zh\}\}=0\.694,Copen=0\.750\\mathrm\{C\_\{open\}\}=0\.750\. Group A and Group B are within0\.0110\.011of each other; Group C \(with qwen2\.5\-72b at0\.8070\.807\) leads\.This does not supportthe hypothesis that English\-centric models systematically exploit cues to outperform Chinese\-optimized models on the translated benchmark\. The cue exploitation hypothesis, if it holds, must operate at item\-level variation rather than aggregate accuracy\.

### 5\.5INCLUDE Non\-Translation Control \(E3\)

INCLUDE Chinese \(545 four\-choice items\) was scored on six models: three Group\-A and three Group\-B\. Aggregate accuracies \(Wilson 95% CI\) and bootstrap gap MMMLU−\-INCLUDE:

- •Group A \(English\-centric\):gpt\-4o0\.7840\.784vs0\.7820\.782, gap\+0\.002\+0\.002\[−0\.064,\+0\.065\]\[\-0\.064,\+0\.065\]; gpt\-4o\-mini0\.6740\.674vs0\.6450\.645, gap\+0\.029\+0\.029\[−0\.043,\+0\.101\]\[\-0\.043,\+0\.101\]; gpt\-5\.4\-mini0\.8030\.803vs0\.7910\.791, gap\+0\.012\+0\.012\[−0\.057,\+0\.076\]\[\-0\.057,\+0\.076\]\. All three CIs include zero\.
- •Group B \(Chinese\-optimized\):deepseek\-chat0\.7670\.767vs0\.8450\.845, gap−0\.078\\mathbf\{\-0\.078\}\[−0\.143,−0\.015\]\\mathbf\{\[\-0\.143,\-0\.015\]\}; qwen\-max0\.7270\.727vs0\.8510\.851, gap−0\.125\\mathbf\{\-0\.125\}\[−0\.190,−0\.059\]\\mathbf\{\[\-0\.190,\-0\.059\]\}; glm\-4\.50\.6120\.612vs0\.5720\.572, gap\+0\.040\+0\.040\[−0\.037,\+0\.115\]\[\-0\.037,\+0\.115\]\. Two of three Group\-B CIs exclude zero in the negative direction; glm\-4\.5’s CI includes zero and is positive in sign\.

#### Subgroup pattern\.

Two of three Chinese\-optimized models \(deepseek\-chat−7\.8\-7\.8pp; qwen\-max−12\.5\-12\.5pp\) score substantially lower on translated MMMLU Chinese than on natively\-sourced INCLUDE Chinese, with CIs excluding zero\. The third Group\-B model, glm\-4\.5, runs the other way \(\+4\.0\+4\.0pp, CI including zero\) and has substantially lower overall accuracy, making the gap estimate noisier\. The two\-of\-three pattern is compatible with a subgroup reading in which stronger native Chinese training advantages models on native Chinese text, but is not a universal Group\-B effect\. INCLUDE and MMMLU differ in domain, register, and construction, so a Group\-B model’s higher INCLUDE score could also reflect distributional differences rather than translation\-cue asymmetry; full identification would require item\-matched native\-vs\-translated pairs that INCLUDE does not provide\.

Group A models show small positive gaps consistent with zero; the Group A E3 estimates \(\+0\.002\+0\.002,\+0\.029\+0\.029,\+0\.012\+0\.012\) are similar in scale to E1 TTback\{\}\_\{\\text\{back\}\}estimates \(\+0\.029\+0\.029,\+0\.005\+0\.005,\+0\.047\+0\.047\)\.

#### Three readings of the Group A near\-zero pattern\.

\(a\) Cue exploitation is small or absent for frontier English\-centric models on this language pair; \(b\) MMMLU \(academic\) and INCLUDE \(regional\-exam\) content distributions differ enough that aggregate comparison is uninformative; \(c\) the two benchmarks coincide in difficulty for these models\. The Group\-B pattern suggests a possible model\-family interaction, but it does not rule out group\-specific domain fit to INCLUDE\-style regional exam items: Chinese\-optimized models may be more familiar with Chinese regional\-exam content and register, independent of any translation\-cue effect on MMMLU\. The six\-model result is exploratory; three models \(all Group C\) remain unscored in this report\.

### 5\.6Annotation Calibration \(E2\)

A 30\-item annotation calibration was run with one LLM\-rubric annotator using a structured rubric covering three dimensions on a 5\-point Likert scale\. 26 items received complete annotations \(4 fell on QC\-excluded back\-translations\)\. Annotation distribution: cue score mean3\.083\.08\(sd0\.550\.55\), cultural residue mean2\.582\.58\(sd1\.181\.18, range 1–5 with high\_school\_us\_history correctly identified at55and abstract\_algebra at11\), syntactic residue mean2\.582\.58\(sd0\.570\.57\)\.

#### Calibration result\.

Spearman correlations between cue score and item\-level TT signal: gpt\-4oρ=−0\.34\\rho=\-0\.34\(p=0\.072p=0\.072\); gpt\-4o\-miniρ=0\.00\\rho=0\.00\(all 26 paired items tied\); gpt\-5\.4\-miniρ=\+0\.05\\rho=\+0\.05\(p=0\.82p=0\.82\)\. The gpt\-4o coefficient runs opposite the cue\-exploitation hypothesis\. Two factors block interpretation: of 26 annotated items, 25 produce TT signal=0=0\(only one discordant pair,96%96\\%ties\); the calibration uses a single LLM\-rubric annotator, not two bilingual humans\. The calibration is reported as a null result on rubric\-based item\-level cue exploitation under this protocol; it is not informative about the bilingual\-human estimator at scale\.

### 5\.7Parser Validity Audit

We report parser validity per model\-set as a check on whether extraction failures bias accuracy estimates\. 8 of 9 models on MMMLU achieve validity≥0\.978\\geq 0\.978; the exception is gpt\-5\.4\-mini at0\.8250\.825onqZHq\_\{\\text\{ZH\}\},0\.8640\.864onqENq\_\{\\text\{EN\}\},0\.8700\.870onqbackq^\{\\text\{back\}\}\. Inspection of parser failures reveals empty raw responses, consistent with token\-budget truncation on an extended\-reasoning model\. We report gpt\-5\.4\-mini results on validity\-restricted subsets; the strict\-QC analysis in Section[5](https://arxiv.org/html/2605.07093#S5)already excludes parser\-failed items\.

## 6Matched Naturalization Counterfactual \(E4\)

The three estimators in Section[5](https://arxiv.org/html/2605.07093#S5)are proxies: they do not vary item content between conditions\. This section reports a fourth estimator, E4, that holds item content fixed and varies only surface naturalness through a paired counterfactual rewrite of each translated Chinese item\.

#### LLM\-augmented protocol \(disclosure\)\.

The naturalization rewrites were produced by a large language model \(deepseek\-chat\) under a structured prompt that requires preservation of meaning, answer key, choice ordering, difficulty, and technical terminology, and a second large language model \(gpt\-4o\) was used as a verifier scoring six dimensions per rewrite \(semantic preservation, difficulty preservation, answer\-key preservation, option\-order preservation, cue removal, excessive\-rewrite risk\)\. This is an*LLM\-naturalization stress test*, not a bilingual\-human naturalization counterfactual; reviewers should read the E4 treatment as “LLM\-rewritten Chinese with LLM verifier QC,” not “human\-naturalized text\.” The prompts, raw rewrites, and verifier scores are released in the analysis package\.

#### Sample and quality\.

A 120\-item stratified sample was drawn from the 228\-item MMMLU pilot: 30 high\-residue items \(Group A all correct onqENq\_\{\\text\{EN\}\}but at least one wrong onqZHq\_\{\\text\{ZH\}\}\), 60 low\-residue items \(Group A all correct on both versions\), and 21 disagreement items \(Group A and Group B majority disagree\)\. 54 distinct subjects, capped at four items per subject\. One item failed the naturalization API;n=119n=119retained\. LLM\-verifier strict\-QC pass rate was119/119119/119\(mean semantic preservation4\.984\.98, mean difficulty preservation5\.005\.00, answer\-key and option\-order preserved on all items, mean cue removal4\.974\.97, mean excessive\-rewrite risk1\.061\.06\)\. These are LLM\-verifier scores, not human QC\.

#### Human QC and human two\-coder validation \(n=50n=50\)\.

A bilingual blind QC was performed by the paper author on a stratified 50\-item subset \(13 high\-residue, 9 disagreement, 28 low\-residue\), blind to per\-model scoring records\. Strict pass requires semantic and difficulty≥4\\geq 4, answer\-key and option\-order pass, and excessive\-rewrite risk≤2\\leq 2\. The author strict\-pass rate is49/5049/50; the only failure \(mmmlu\_zh\_00924\) dropped the I/II/III statement list, making the item unanswerable\. On the author strict\-pass subset, pooledΔ\\Deltais\+0\.022​\[−0\.030,\+0\.078\]\+0\.022\\,\[\-0\.030,\+0\.078\]excluding glm\-4\.5, with high\-residueΔ=\+0\.133\\Delta=\+0\.133and low\-residueΔ=−0\.022\\Delta=\-0\.022\. We also include a second human QC of the same sheet by an independent human coder\. This is human two\-coder validation\. The second human coder is stricter:46/5046/50strict pass, strict\-pass agreement94%94\\%with the first author QC, and binaryκ=0\.38\\kappa=0\.38\(low because pass/fail is highly imbalanced\)\. It flags three additional semantic or rewrite\-risk concerns \(state/nation, half\-sister/ step\-sister, and doctrinal\-expansion cueing\)\. On the intersection strict\-pass subset \(n=46n=46\), pooledΔ\\Deltaexcluding glm\-4\.5 weakens to\+0\.009​\[−0\.042,\+0\.068\]\+0\.009\\,\[\-0\.042,\+0\.068\], while the high\-residue stratum remains directionally positive \(\+0\.125​\[0\.000,\+0\.281\]\+0\.125\\,\[0\.000,\+0\.281\]\) and low\-residue remains near zero \(−0\.023​\[−0\.062,\+0\.008\]\-0\.023\\,\[\-0\.062,\+0\.008\]\)\. The released supplement contains the first\-coder QC sheet, the second\-coder QC sheet, and the intersection sensitivity ine4\_naturalization/human\_qc/ande4\_naturalization/second\_coder\_qc/\.

#### Scoring protocol\.

Six models were scored: three Group A \(gpt\-4o, gpt\-4o\-mini, gpt\-5\.4\-mini\) and three Group B \(deepseek\-chat, qwen\-max, glm\-4\.5\)\. A prompt\-construction audit found that an earlier naturalized\- condition scoring run duplicated choice labels: the naturalized choices retained an in\-text leading “A\./B\./C\./D\.” label from the rewriter while the prompt builder also prepended canonical labels, producing options of the form “A\. A\. text”\. This affected the naturalized condition only; the translated condition uses the original answer\-only prompt\. The corrected protocol strips any leading\[A\[A–D\]D\]label from each naturalized choice before prompt construction and uses the same prompt builder as the translated condition\. All E4 numbers reported below use the corrected scoring protocol \(release identifierscoring\_v2/\)\. All translated\-condition scores are unchanged\. Parser\-validity rates after correction: gpt\-4o, gpt\-4o\-mini, qwen\-max1\.001\.00; deepseek\-chat0\.990\.99; gpt\-5\.4\-mini0\.910\.91; glm\-4\.50\.340\.34\. The naturalized prompt removes a format crutch glm\-4\.5 had relied on; the model is treated as a parser\-failure outlier and excluded from the main analysis, included only in sensitivity\. Per\-item paired analysis below uses cells where both translated and naturalized scoring produced a valid letter\.

![Refer to caption](https://arxiv.org/html/2605.07093v1/x3.png)Figure 3:After correcting the prompt\-construction bug, E4 no longer supports a stable model\-family interaction\. The remaining signal is item\-level: high\-residue items benefit from naturalization while low\-residue items do not; glm\-4\.5 is treated as a parser\-failure outlier\.
#### Item\-clustered primary inference \(E4 v2, main: 5 models, glm\-4\.5 sensitivity only\)\.

Item\-clustered bootstrap withB=2,000B=2\{,\}000:

- •Pooled \(5 non\-outlier models\):Δ=\+0\.025​\[−0\.017,\+0\.071\]\\Delta=\+0\.025\\,\[\-0\.017,\+0\.071\]; Group A:\+0\.024​\[−0\.009,\+0\.059\]\+0\.024\\,\[\-0\.009,\+0\.059\]; Group B excl\. glm\-4\.5:\+0\.025​\[−0\.017,\+0\.071\]\+0\.025\\,\[\-0\.017,\+0\.071\]; DiD \(A−\-B excl\. glm\-4\.5\):−0\.001​\[−0\.053,\+0\.058\]\-0\.001\\,\[\-0\.053,\+0\.058\]\.
- •Pooled \(all 6, glm\-4\.5 included\):Δ=\+0\.039​\[\+0\.010,\+0\.070\]\\Delta=\+0\.039\\,\[\+0\.010,\+0\.070\]; Group B \(incl\. glm\-4\.5 on its 41\-item parser\-valid subset\):\+0\.058​\[\+0\.014,\+0\.104\]\+0\.058\\,\[\+0\.014,\+0\.104\]; DiD:−0\.034​\[−0\.087,\+0\.023\]\-0\.034\\,\[\-0\.087,\+0\.023\]\(CI crosses zero\)\.

The model\-family interaction reported in earlier drafts was driven by glm\-4\.5 in combination with the bug\-affected prompt format\. After the fix, E4 returns no statistically significant model\-family interaction; the interaction observed in v1 is reclassified as an artifact of the combination of \(i\) double\-labeled options and \(ii\) glm\-4\.5’s degraded parser behaviour without that format crutch\.

#### Stratum and leave\-one\-out sensitivity \(Figure[3](https://arxiv.org/html/2605.07093#S6.F3)B–C\)\.

The dose\-response pattern survives the bug fix: high\-residueΔ=\+0\.103\\Delta=\+0\.103versus−0\.015\-0\.015on low\-residue items \(excl\. glm\-4\.5\);\+0\.112\+0\.112versus−0\.006\-0\.006with glm\-4\.5 included\. The high\-to\-low ratio is roughly7×7\\timesin the main analysis \(compared to the artefactual∼\\sim90×90\\timesin v1\)\. PooledΔ\\Deltain leave\-one\-out configurations ranges\[\+0\.025,\+0\.045\]\[\+0\.025,\+0\.045\]; the only excluded model that materially shifts the pooled estimate is glm\-4\.5\. Full sensitivity tables are inregression\_results\_v2\.json\.

#### Interpretation and triangulation\.

When item content is held fixed correctly, E4 delivers a positive point estimate on high\-residue items and a near\-zero effect on low\-residue items; the model\-family interaction in v1 was a prompt\-format artefact\. The stricter second human QC weakens pooled E4 estimates, so E4 is treated as a diagnostic stress test rather than confirmatory evidence\. The four estimators thus converge on one finding—residue\-sensitive items are more fragile—and disagree on magnitude: E1 and E4 give small positive contrasts, while E2/E3 give null or model\-family\-internal patterns\. No single estimator identifies a Translation Tax; the four together describe estimator\- and item\-dependent validity risks rather than a scalar correction\. The supplement \([https://github\.com/chi\-mi\-rvard/translation\-tax\-supplement](https://github.com/chi-mi-rvard/translation-tax-supplement)\) recomputes all estimates including the corrected E4 protocol and QC sensitivities\.

#### Limitations\.

E4 uses an LLM rewriter and LLM verifier; the human QC subset uses two human coders\. The second human QC provides human two\-coder validation\. A human\-naturalized subset \(translatedqZHq\_\{\\text\{ZH\}\}vs\. human\-rewrittenqZHq\_\{\\text\{ZH\}\}on the same items\) is not included in this submission\. E2 is a 30\-item single\-annotator calibration, not a full bilingual\-human estimator\. glm\-4\.5’s 0\.34 parser\-validity rate under the corrected naturalized prompt is reported as prompt sensitivity in that model rather than as a feature of translation residue\. The raw outputs, bug\-fix re\-scoring, human QC, second human QC, and strict\-subset sensitivities are released\.

## References

- Artetxe et al\. \[2020\]Mikel Artetxe, Gorka Labaka, and Eneko Agirre\.Translation artifacts in cross\-lingual transfer learning\.In*EMNLP*, 2020\.
- Bandarkar et al\. \[2024\]Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa\.The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants\.In*ACL*, 2024\.
- Clark et al\. \[2020\]Jonathan H\. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki\.TyDi QA: A benchmark for information\-seeking question answering in typologically diverse languages\.*TACL*, 8:454–470, 2020\.
- Conneau et al\. \[2018\]Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov\.XNLI: Evaluating cross\-lingual sentence representations\.In*EMNLP*, 2018\.
- Hendrycks et al\. \[2021\]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt\.Measuring massive multitask language understanding\.In*ICLR*, 2021\.
- Mizrahi et al\. \[2024\]Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky\.State of what art? a call for multi\-prompt LLM evaluation\.*TACL*, 12:933–949, 2024\.
- Ponti et al\. \[2020\]Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen\.XCOPA: A multilingual dataset for causal commonsense reasoning\.In*EMNLP*, 2020\.
- Romanou et al\. \[2025\]Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Tanneru, Zeming Chen, Antoine Bosselut, and Syrielle Montariol\.INCLUDE: Evaluating multilingual language understanding with regional knowledge\.In*ICLR*, 2025\.
- Sitaram \[2025\]Sunayana Sitaram\.Coverage, representativeness, trust and scientific rigor in multilingual benchmark evaluation\.Invited talk, NeurIPS 2025 Workshop on Centering Low Resource Languages and Cultures\.
- Wu et al\. \[2025\]Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang\.The bitter lesson learned from 2,000\+ multilingual benchmarks\.arXiv:2504\.15521, 2025\.
- Zhuo et al\. \[2024\]Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen\.ProSA: Assessing and understanding the prompt sensitivity of LLMs\.In*EMNLP Findings*, 2024\.