Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge

# Does the Judge Prefer English? Evaluating Language-Switching Invariance in LLM-as-a-Judge
Source: [https://arxiv.org/html/2606.14278](https://arxiv.org/html/2606.14278)
Shaojie Yin Shanghai Artificial Intelligence Laboratory Shanghai, China Misaka10086@sjtu\.edu\.cn [ORCID: 0009\-0008\-1526\-1824](https://orcid.org/0009-0008-1526-1824)

###### Abstract

Large language models \(LLMs\) are now widely used as automatic judges for open\-ended instruction\-following evaluation\. This practice is convenient, scalable, and often more semantically aware than reference\-based metrics, but it also introduces a new reliability question: does a judge evaluate the quality of an answer, or does it also react to the language in which the comparison is presented? We propose Judge\-LS, a lightweight meta\-evaluation protocol that transforms LLMBar response\-pair items into English, Chinese, and Chinese\-English language\-switched variants\. A reliable judge should preserve its preference under label\-preserving language transformations and should not prefer a language when two answers are translation\-equivalent\. We evaluate four API\-accessible judges on the full 419\-item LLMBar benchmark, producing 13,408 successful pairwise judgments\. Across models, Chinese and language\-switched presentations induce 10\.7–14\.4% preference flips relative to English, and all judges achieve their highest accuracy in English\. However, translation\-equivalent tie probes do not reveal a systematic English preference: most probes are judged as ties, and non\-tie decisions more often favor Chinese\. We add confidence intervals, paired significance tests, and an automatic transformation audit with a sensitivity analysis that excludes mechanically flagged high\-risk variants\. The experiment requires no model training, uses only API calls, and is feasible on modest local hardware\.

## 1Introduction

Automatic evaluation has become one of the practical bottlenecks in building and comparing LLM systems\. Human preference annotation is expensive, slow, and difficult to repeat whenever a model, prompt, or decoding strategy changes\. As a result, LLM\-as\-a\-Judge has become a common replacement or pre\-filter for human evaluation\. A judge model is given an instruction and one or more candidate answers, and is asked to produce a preference, score, or explanation\. This setting has been used for chat benchmarks, instruction\-following comparisons, and increasingly for multimodal or domain\-specific evaluation\.

The appeal of LLM judges is clear: they can read open\-ended outputs, reason over task requirements, and produce structured decisions at large scale\. The risk is also clear: the judge itself is another model with its own biases\. Prior work has documented position bias, verbosity bias, self\-preference, prompt sensitivity, and limitations in judging instruction\-following behavior\[[2](https://arxiv.org/html/2606.14278#bib.bib2),[3](https://arxiv.org/html/2606.14278#bib.bib3),[4](https://arxiv.org/html/2606.14278#bib.bib4),[10](https://arxiv.org/html/2606.14278#bib.bib10),[12](https://arxiv.org/html/2606.14278#bib.bib12),[13](https://arxiv.org/html/2606.14278#bib.bib13)\]\. These biases matter because evaluation results influence model selection, leaderboard rankings, and research conclusions\.

This paper studies a smaller but important reliability question:*does a judge preserve its preference when the same pairwise comparison is presented in English, Chinese, or Chinese\-English language\-switching?*If two answer pairs differ only by a label\-preserving language transformation, a judge should ideally make the same choice\. If the best answer is translated into Chinese and compared against the same English answer, the judge should usually call a tie rather than reward a particular language\. Violations of these expectations imply that the evaluation pipeline is sensitive to surface presentation, not only response quality\.

We focus on English–Chinese because Chinese is widely used in real multilingual LLM deployments and because language\-switching is common in technical, educational, and online communication\. The study is designed for low\-resource experimental settings: no model is trained, no local large model inference is required, and a modest local GPU is not a bottleneck\. All generation and judging are done through API calls, while the local machine performs orchestration, parsing, plotting, and LaTeX compilation\.

Our contributions are:

- •We introduceJudge\-LS, a simple protocol for testing language\-switching invariance in pairwise LLM\-as\-a\-Judge evaluation\.
- •We transform the full 419\-item LLMBar benchmark into English, Chinese, and Chinese\-English language\-switched variants, preserving the original gold preference labels\.
- •We evaluate four API\-accessible judges using 13,408 successful judgments, including answer\-order swaps and translation\-equivalence tie probes\.
- •We report not only accuracy, but also language\-invariance flip rate, gold\-correctness flip rate, position inconsistency, source\-level robustness, language preference, uncertainty estimates, paired significance tests, and usage cost\.

![Refer to caption](https://arxiv.org/html/2606.14278v1/x1.png)Figure 1:Overview of Judge\-LS\. The full LLMBar response\-pair benchmark is converted into English, Chinese, and Chinese\-English language\-switched variants\. Four API judges are evaluated with answer\-order swaps and translation\-equivalent tie probes, then summarized by accuracy, judgment\-flip rate, correctness\-flip rate, position inconsistency, language preference, and transformation\-audit diagnostics\.
## 2Related Work

#### Automatic text evaluation\.

Early automatic evaluation metrics made large\-scale model comparison practical by replacing or reducing human annotation\. BLEU and ROUGE use lexical overlap for machine translation and summarization evaluation\[[5](https://arxiv.org/html/2606.14278#bib.bib5),[6](https://arxiv.org/html/2606.14278#bib.bib6)\]\. Later learned or embedding\-based metrics such as BERTScore and COMET improve semantic matching and multilingual machine\-translation quality estimation\[[7](https://arxiv.org/html/2606.14278#bib.bib7),[8](https://arxiv.org/html/2606.14278#bib.bib8)\]\. LLM\-as\-a\-Judge continues this trajectory toward semantic evaluation, but it also inherits a harder reliability problem: the evaluator is a generative model whose decisions can change with prompt form, answer order, or language presentation\.

#### LLM\-as\-a\-Judge\.

G\-Eval showed that GPT\-4\-style evaluators can align better with human ratings than many traditional natural language generation metrics when prompted with explicit criteria\[[2](https://arxiv.org/html/2606.14278#bib.bib2)\]\. MT\-Bench and Chatbot Arena popularized LLM judges for pairwise conversational evaluation and analyzed biases such as position and verbosity effects\[[4](https://arxiv.org/html/2606.14278#bib.bib4)\]\. AlpacaFarm and AlpacaEval use low\-cost pairwise feedback or auto\-annotation to compare instruction\-following models, while length\-controlled AlpacaEval explicitly debiases verbosity effects\[[9](https://arxiv.org/html/2606.14278#bib.bib9),[10](https://arxiv.org/html/2606.14278#bib.bib10)\]\. Prometheus studies open evaluator models trained for fine\-grained rubric\-based assessment\[[11](https://arxiv.org/html/2606.14278#bib.bib11)\]\. LLMBar targets a stricter meta\-evaluation setting: an evaluator must identify which of two outputs objectively follows an instruction better\[[3](https://arxiv.org/html/2606.14278#bib.bib3)\]\. We use LLMBar because its gold labels make it possible to measure when a judge’s decision changes under a language transformation\.

#### Judge bias and calibration\.

FairEval shows that simply swapping candidate order can alter LLM\-judge rankings and proposes position\-balancing calibration\[[12](https://arxiv.org/html/2606.14278#bib.bib12)\]\. Other work shows that automatic evaluators may favor longer answers or even their own generations\[[10](https://arxiv.org/html/2606.14278#bib.bib10),[13](https://arxiv.org/html/2606.14278#bib.bib13)\]\. These studies motivate our answer\-order swaps and our decision to report invariance metrics rather than only aggregate accuracy\. Judge\-LS adds a language\-surface intervention: the answer content is intended to remain label\-preserving, while the presentation language changes\.

#### Multilingual judge reliability\.

Multilingual LLM evaluation is not just English evaluation translated into other languages\. A judge may have different competence, calibration, and cultural assumptions across languages\. Broad multilingual benchmarks such as XTREME, MEGA, and multilingual ChatGPT evaluations show that model behavior varies substantially across languages and resource levels\[[14](https://arxiv.org/html/2606.14278#bib.bib14),[15](https://arxiv.org/html/2606.14278#bib.bib15),[16](https://arxiv.org/html/2606.14278#bib.bib16),[17](https://arxiv.org/html/2606.14278#bib.bib17)\]\. Recent work on multilingual LLM\-as\-a\-Judge reliability reports low agreement in many non\-English settings\[[1](https://arxiv.org/html/2606.14278#bib.bib1)\]\. Judge\-LS is complementary: instead of surveying many languages, it isolates a controlled English–Chinese–language\-switching intervention on the same pairwise comparisons\. This lets us ask whether the judge is invariant to language presentation when the intended quality relation is fixed\.

#### Evaluation benchmarks and domain robustness\.

Judge\-LS is also motivated by broader evaluation work from trustworthy AI and multimodal benchmarking\. AIBench studies trustworthy evaluation principles under the 45∘law, while a recent large multimodal model survey summarizes evaluation dimensions for modern multimodal systems\[[18](https://arxiv.org/html/2606.14278#bib.bib18),[19](https://arxiv.org/html/2606.14278#bib.bib19)\]\. Related first\-author work by Tian studies video understanding, action recognition, video compression, and representation learning benchmarks, including early ASD detection, event\-adaptive action recognition, low\-bitrate video understanding, semantic video compression, video rescaling, and self\-supervised motion representation\[[20](https://arxiv.org/html/2606.14278#bib.bib20),[21](https://arxiv.org/html/2606.14278#bib.bib21),[22](https://arxiv.org/html/2606.14278#bib.bib22),[23](https://arxiv.org/html/2606.14278#bib.bib23),[24](https://arxiv.org/html/2606.14278#bib.bib24),[25](https://arxiv.org/html/2606.14278#bib.bib25),[26](https://arxiv.org/html/2606.14278#bib.bib26),[27](https://arxiv.org/html/2606.14278#bib.bib27)\]\. These efforts motivate our emphasis on evaluator robustness rather than treating benchmark scores as transparent measurements\.

## 3Judge\-LS Protocol

### 3\.1Base Data

We use LLMBar, a 419\-item benchmark for evaluating whether an LLM evaluator can judge instruction\-following responses\[[3](https://arxiv.org/html/2606.14278#bib.bib3)\]\. Each item contains an instruction, two candidate outputs, and an objective gold label indicating whether output 1 or output 2 better follows the instruction\. LLMBar contains one natural subset and four adversarial subsets\. In our full experiment, the source distribution is:

Using all LLMBar items avoids the sampling uncertainty of the earlier pilot run\. Because the benchmark is pairwise and label\-bearing, it supports both standard accuracy and invariance diagnostics\.

### 3\.2Language Transformations

For every LLMBar item we keep the original English version and create two additional versions:

- •ZH: the instruction and both candidate outputs are translated into natural Simplified Chinese\.
- •LS: the item is converted into natural Chinese\-English language\-switching, preserving English task terms, named entities, or technical terms when appropriate\.

We usegpt\-4\.1\-minias the transformation model\. The prompt explicitly asks the model to preserve every factual, mathematical, formatting, and instruction\-following error\. This is important because a translation that silently repairs the weaker answer would invalidate the original gold label\. One item repeatedly produced invalid JSON because of nested quotation marks in a vocabulary quiz; we manually repaired that item while preserving the same response\-quality relation\.

### 3\.3Transformation Audit

Because label preservation is a central assumption, we add an automatic audit before interpreting judge behavior\. The audit verifies that every unique item has complete English, Chinese, and language\-switched instruction and response fields, then conservatively flags variants that may need human review because of empty fields, severe length shrinkage, or large numeric\-token count changes\. These checks are mechanical rather than semantic: for example, a numeric\-token mismatch can be caused by a valid Chinese unit conversion\. We therefore keep the full 419\-item set as the primary experiment and report a sensitivity analysis that excludes the 19 flagged high\-risk items\.

### 3\.4Judgment Construction

For each item and each language condition, the judge receives the instruction and two responses labeled A and B\. We evaluate both original and swapped response order\. After the model chooses A, B, or Tie, we map the output back to the original output IDs\. This produces six quality judgments per item per judge:

3​language conditions×2​answer orders\.3\\text\{ language conditions\}\\times 2\\text\{ answer orders\}\.
We also create two tie probes per item\. The gold answer in English is compared with its Chinese translation in both orders\. In this setting, the intended answer isTie\. We use the gold answer because it is the response most likely to be semantically complete and least likely to contain fragile errors whose translation could create a new quality difference\. A weaker\-answer tie probe would be useful, but it can confound language preference with error\-preservation drift\. A judge that chooses English or Chinese in many gold\-answer probes is therefore exhibiting language preference rather than quality sensitivity under a conservative high\-quality equivalence condition\. Each item yields eight judgments per model\.

## 4Metrics

#### Tie\-half accuracy\.

Lety^j\\hat\{y\}\_\{j\}be the normalized judgment for quality comparisonjjandyjy\_\{j\}be the LLMBar gold label\. We report strict accuracy and tie\-half accuracy:

Acc1/2=1N​∑j\{1y^j=yj,0\.5y^j=Tie,0otherwise\.\\mathrm\{Acc\}\_\{1/2\}=\\frac\{1\}\{N\}\\sum\_\{j\}\\begin\{cases\}1&\\hat\{y\}\_\{j\}=y\_\{j\},\\\\ 0\.5&\\hat\{y\}\_\{j\}=\\textsc\{Tie\},\\\\ 0&\\text\{otherwise\}\.\\end\{cases\}
Tie\-half accuracy is useful because some model outputs genuinely report uncertainty\. Strict accuracy counts all ties as incorrect\. We report both metrics in the paper because they answer different questions: strict accuracy measures decisive agreement with LLMBar, while tie\-half accuracy avoids treating uncertainty as fully wrong\.

#### Language\-invariance flip rate\.

For target languagel∈\{ZH,LS\}l\\in\\\{\\mathrm\{ZH\},\\mathrm\{LS\}\\\}, we compare the normalized judgment in English with the judgment under the transformed presentation:

Flip​\(l\)=1N​∑i,o𝟙​\[y^i,EN,o≠y^i,l,o\],\\mathrm\{Flip\}\(l\)=\\frac\{1\}\{N\}\\sum\_\{i,o\}\\mathbb\{1\}\[\\hat\{y\}\_\{i,\\mathrm\{EN\},o\}\\neq\\hat\{y\}\_\{i,l,o\}\],
whereiiindexes items andooindexes answer order\. We also report gold\-correctness flip rate: whether the judge changes from correct to incorrect, or from incorrect to correct, after the language transformation\.

#### Position inconsistency\.

For each item and language condition, we compare the normalized decision before and after swapping response order\. If the normalized winner changes, the judge is position\-inconsistent\. This metric is especially important because language effects and answer\-order effects can interact\.

#### Tie\-probe language preference\.

For translation\-equivalent probes, we count English wins, Chinese wins, and ties\. We also record the English share among non\-tie decisions\. A high English win rate would support the hypothesis that the judge prefers English; a high flip rate without English wins instead suggests instability rather than simple English preference\.

#### Uncertainty and paired tests\.

For binomial rates such as flip rate, position inconsistency, and tie\-probe outcomes, we report Wilson 95% confidence intervals\. For tie\-half accuracy, which is an average over\{0,0\.5,1\}\\\{0,0\.5,1\\\}scores, we report normal 95% intervals over per\-judgment scores\. To test whether language changes alter strict gold correctness, we use an exact two\-sided McNemar/binomial test on paired English\-vs\.\-transformed judgments with the same item and answer order\.

## 5Experimental Setup

We evaluate four API\-accessible judges:

- •GPT\-4\.1 Mini:gpt\-4\.1\-mini
- •Claude Haiku 4\.5:claude\-haiku\-4\-5\-20251001
- •Gemini 2\.5 Flash:gemini\-2\.5\-flash
- •DeepSeek V4 Flash:deepseek\-v4\-flash

All calls use temperature 0 and a JSON\-only prompt with allowed winnersA,B, orTie\. The prompt asks for a self\-reported confidence value, but we treat confidence only as logged metadata because it is not calibrated across model families\. The experiment contains 419 items, four judges, and eight judgments per item per judge, yielding 13,408 successful unique judgments\. The full prompt templates, parsing rules, retry logic, and analysis code are inscripts/judge\_cs\.py\. The local machine only runs Python orchestration and plotting\. A local conda environment avoids package conflicts, and API credentials are supplied through environment variables rather than written to project files\.

The DeepSeek endpoint often emitted long reasoning content before the final JSON answer\. We therefore used a higher output cap for DeepSeek and retried rare empty\-content failures\. This engineering detail matters for reproducibility: if a model spends all allowed tokens on hidden or exposed reasoning, a judge pipeline may falsely record a parser failure rather than a model decision\.

## 6Main Results

Table 1:Quality\-judgment accuracy by language condition on LLMBar\. Strict accuracy counts ties as incorrect; tie\-half gives ties half credit\. Position\-A is choosing the displayed first answer before order normalization; all rates are percentages\.Table 2:Language\-invariance flip rates relative to English\. Judgment flip counts any changed normalized preference; correctness flip counts a change between strict gold\-correct and not strict gold\-correct\. All rates and intervals are percentages\.Table 3:Language preference in translation\-equivalent tie probes\. EN/ZH win rates indicate which language version is preferred; EN share is computed over non\-tie decisions\. All rates are percentages\.Table 4:Observed or approximated API usage\. Costs depend on the endpoint’s billing plan\.![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/accuracy_by_language.png)

![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/language_invariance_flip_rate.png)

![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/tie_probe_language_preference.png)

Figure 2:Main results on the 419\-item LLMBar experiment\. Left: tie\-half accuracy by language condition, with LS denoting Chinese\-English language\-switching\. Middle: judgment\-flip rate versus EN for EN–ZH and EN–LS\. Right: outcomes for EN\-vs\.\-ZH translation\-equivalent tie probes\.#### Accuracy is highest in English\.

All four judges perform best in English \(Table[1](https://arxiv.org/html/2606.14278#S6.T1)\)\. GPT\-4\.1 Mini drops from 78\.8% tie\-half accuracy in English to 72\.8% in Chinese and 73\.0% in language\-switched form\. Claude Haiku drops from 78\.9% to roughly 74\.3–74\.4%\. Gemini drops from 87\.8% to 83\.7% in Chinese and 82\.9% in language\-switched form\. DeepSeek is strongest overall, but still drops from 90\.5% English accuracy to 87\.8% in Chinese and 88\.9% in language\-switched form\. The strict\-accuracy column shows the same qualitative pattern, so the English advantage is not an artifact of giving half credit to ties\.

#### Language transformations cause substantial preference flips\.

Language\-invariance flip rates range from 10\.7% to 14\.4% \(Table[2](https://arxiv.org/html/2606.14278#S6.T2)\)\. GPT\-4\.1 Mini and Claude Haiku are most sensitive to Chinese transformations, with English\-vs\.\-Chinese flip rates of 14\.4% and 14\.2%, respectively\. DeepSeek and Gemini have lower English\-vs\.\-Chinese flip rates at 11\.1%, but still change about one in nine judgments\. Language\-switched comparisons are also unstable, with flip rates from 10\.7% to 12\.9%\. The Wilson intervals remain well above zero, with lower bounds from 8\.8% to 12\.2%\.

#### Correctness changes nearly as often as preferences\.

The gold\-correctness flip rates are close to the raw preference flip rates\. This means that flips are not just harmless swaps between a correct answer and a tie\. A language transformation often changes whether the judge agrees with the objective LLMBar label\. Exact paired tests on strict correctness reject symmetry for every model\-language comparison \(p≤0\.011p\\leq 0\.011; Table[8](https://arxiv.org/html/2606.14278#S7.T8)\)\. For users relying on automatic judges to compare multilingual outputs, this is the more consequential failure mode\.

#### No simple English preference in tie probes\.

The title asks whether the judge prefers English\. The tie\-probe results suggest a more nuanced answer: not systematically\. Most translation\-equivalent comparisons are judged as ties\. DeepSeek reports ties 97\.5% of the time, GPT\-4\.1 Mini 95\.6%, Gemini 92\.6%, and Claude 82\.3%\. When judges do not tie, Chinese wins more often than English for every model\. The problem is therefore not a universal English\-favoring prior; it is instability under language presentation\.

## 7Extended Analysis

Table 5:Subset\-level averages across the four judges\. Accuracy uses tie\-half scoring; judgment flip compares transformed language conditions with English\. All rates are percentages\.Table 6:Position inconsistency by model and language condition\. A case is inconsistent when the normalized winner changes after swapping answer order; all rates are percentages\.Table 7:Automatic transformation audit\. High\-risk flags are conservative mechanical warnings based on empty fields, severe length shrinkage, or large numeric\-token count changes; they are not treated as semantic labels\.Table 8:Sensitivity and paired\-significance checks\. Filtered judgment flip excludes mechanically flagged high\-risk transformation items\. The p\-value is an exact two\-sided McNemar/binomial test on strict gold\-correctness changes between English and the target language\.![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/accuracy_drop_relative_to_english.png)

![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/position_inconsistency_by_language.png)

Figure 3:Aggregate diagnostics\. Left: tie\-half accuracy loss relative to EN for ZH and LS\. Right: answer\-order inconsistency by language condition; a change after swapping A/B indicates position sensitivity\.![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/source_language_accuracy_heatmap.png)

![Refer to caption](https://arxiv.org/html/2606.14278v1/figures/source_flip_heatmap.png)

Figure 4:Subset\-level behavior on LLMBar\. Left: mean tie\-half accuracy by source subset and language condition\. Right: mean judgment\-flip rate by source subset and judge, averaged over EN–ZH and EN–LS\.#### Transformation audit supports a sensitivity check\.

Table[7](https://arxiv.org/html/2606.14278#S7.T7)shows that all 419 unique items have complete English, Chinese, and language\-switched fields, with one manual JSON repair\. The conservative mechanical audit flags 19 items \(4\.5%\) as high\-risk because they contain severe length shrinkage or large numeric\-token discrepancies\. Excluding these items leaves 400 items and 12,800 quality judgments\. The filtered flip rates in Table[8](https://arxiv.org/html/2606.14278#S7.T8)remain substantial, ranging from 9\.1% to 13\.8%\. This does not prove that every transformation is semantically perfect, but it reduces the chance that the main conclusion is driven only by a small set of visibly risky variants\.

#### Subset averages reveal two distinct effects\.

Table[5](https://arxiv.org/html/2606.14278#S7.T5)separates accuracy loss from decision instability\. The Natural subset has the highest English accuracy at 92\.7%, but its English–Chinese flip rate is still 12\.9%\. This means that high average accuracy does not guarantee stable judgments\. Adv\-GPTOut is the hardest subset, with mean English accuracy of 74\.5% and Chinese accuracy of 68\.9%, and it also has the largest average flip rates\. The subset contains adversarially generated outputs that are often plausible but subtly wrong, so a translation or language\-switched presentation can change which details the judge attends to\.

#### Chinese and language\-switching are not identical perturbations\.

Chinese translation and Chinese\-English language\-switching both reduce accuracy relative to English, but their failure patterns differ\. Chinese creates larger drops on Adv\-Manual and Adv\-GPTOut, while language\-switching creates the largest increase in position inconsistency\. This difference is useful diagnostically\. A pure translation condition tests whether semantic content survives a language change; a language\-switched condition additionally tests whether the judge can handle mixed linguistic cues, partially translated task terms, and local changes in discourse structure\. In real technical usage, language\-switching is common: users may write the task in Chinese while keeping API names, mathematical expressions, software identifiers, or domain terms in English\.

#### Language\-switching amplifies position sensitivity\.

Figure[3](https://arxiv.org/html/2606.14278#S7.F3)and Table[6](https://arxiv.org/html/2606.14278#S7.T6)show that language\-switched inputs often increase answer\-order sensitivity\. GPT\-4\.1 Mini has 15\.0% position inconsistency in English, rising to 19\.1% in language\-switched form\. Claude rises from 13\.6% to 18\.4%, and Gemini rises from 10\.3% to 22\.2%\. DeepSeek is more stable overall, but still rises from 9\.3% to 12\.7%\. Averaged across models, position inconsistency is 12\.1% in English, 14\.9% in Chinese, and 18\.1% in language\-switched form\. This suggests that language\-switching is not merely a language translation issue; it interacts with how pairwise prompts are parsed and compared\.

#### Adversarial subsets are harder and less invariant\.

Figure[4](https://arxiv.org/html/2606.14278#S7.F4)shows that the Natural subset is easiest, while adversarial subsets are more difficult and less stable\. Adv\-GPTOut is particularly challenging: it has lower mean accuracy and higher language\-flip rates across several models\. This is consistent with LLMBar’s goal of stressing evaluator reasoning\. The language transformation appears to add another layer of difficulty on top of adversarial response construction\.

#### Model families differ in failure mode\.

DeepSeek achieves the best average accuracy and lowest position inconsistency, but it also has high output\-token usage because of reasoning content\. Gemini achieves strong English accuracy but has relatively high language\-switched position inconsistency and a high position\-A rate\. GPT\-4\.1 Mini and Claude Haiku are more sensitive to Chinese transformations\. These differences indicate that a multilingual judge benchmark should report multiple diagnostics rather than only one aggregate accuracy\.

#### The tie probe separates bias from instability\.

If a judge had a simple English preference, we would expect English to win many translation\-equivalent probes\. Instead, the non\-tie decisions usually favor Chinese\. This matters conceptually: language\-related judge failures can occur even without a clear preference for a high\-resource language\. The judge may be inconsistent because translation changes perceived specificity, fluency, reasoning traces, or formatting rather than because it blindly rewards English\.

#### Qualitative failure modes\.

Manual inspection of sampled disagreements suggests three recurring patterns\. First, some judges overweight fluency after translation and become less strict about whether the response follows a hidden constraint in the instruction\. Second, some language\-switched prompts make an otherwise minor formatting mismatch more salient, especially when one answer keeps English terminology and the other is more fully localized\. Third, adversarial examples often contain a strong but wrong answer; when the surface language changes, the judge may shift from checking the constraint to rewarding the more coherent response\. These observations are not used as labels in the quantitative analysis, but they help explain why language flips are distributed across both correct\-to\-incorrect and incorrect\-to\-correct changes\.

#### Reproducibility checks\.

The pipeline records every raw model response, parsed winner, confidence value, language condition, item ID, answer order, and token count when available\. The analysis script deduplicates retries by stable judgment keys and reports only successful unique judgments\. We also keep failed responses in the raw log during execution so that parser or endpoint problems can be separated from model behavior\. This is important for API\-based experiments because transient endpoint failures, model\-specific formatting habits, and output\-token limits can otherwise masquerade as evaluation results\.

## 8Recommended Workflow

Judge\-LS is meant to be small enough for routine use\. A practical evaluation workflow can be implemented in five stages\. First, run the normal English judge benchmark with swapped answer order and report both strict and tie\-half accuracy\. Second, transform each item into the target deployment language and at least one realistic mixed\-language condition\. Third, audit the transformed items and run a sensitivity analysis that excludes flagged variants\. Fourth, compute language\-invariance flips against the English baseline, not only accuracy in each language, and attach confidence intervals or paired tests\. Fifth, add translation\-equivalent tie probes to distinguish language preference from general instability\. If a model has high accuracy but high flip rate, it may still be unsuitable as the only judge for multilingual model selection\.

This workflow does not require a new benchmark for every project\. Any pairwise evaluation set with stable labels can be reused, provided that the transformation process preserves the response\-quality relation\. For deployment studies, the same idea can be applied to domain\-specific tasks: customer\-support answers, educational feedback, coding explanations, or safety\-policy judgments\. The key design principle is to perturb the language surface while keeping the intended preference label fixed\.

#### Cost remains low\.

The expanded 419\-item run required 13,408 unique judgments plus 419 language\-variant generations\. The observed or approximated usage is reported in Table[4](https://arxiv.org/html/2606.14278#S6.T4)\. Under reference public prices embedded in the script, the end\-to\-end run remains inexpensive for a small research project\. The proxy endpoint may bill differently, so we report token counts separately from estimated cost\.

## 9Discussion

#### Implication for multilingual evaluation\.

A multilingual LLM benchmark often assumes that the judge’s decision is comparable across language conditions\. Judge\-LS shows that this assumption should be tested\. A 10–14% flip rate is large enough to alter model rankings when systems have close scores\. If a leaderboard uses an English judge prompt but evaluates Chinese or language\-switched answers, the reported performance may mix answer quality with judge\-language sensitivity\.

#### Implication for prompt design\.

The prompts in this experiment explicitly tell the judge not to favor language, length, politeness, or formatting unless directly relevant\. The observed instability therefore does not disappear with a simple instruction\. More robust designs may need bilingual rubrics, explicit translation\-equivalence checks, self\-consistency over language variants, or cascades where a second judge adjudicates language\-sensitive cases\.

#### Relation to robustness evaluation\.

Judge\-LS treats the judge as a system under test rather than as a transparent measuring device\. This is closer to robustness evaluation than to ordinary scoring: the intended quality relation is held fixed, while the surface form is perturbed\. A robust judge should preserve the core preference under such label\-preserving transformations, just as robust classifiers should preserve decisions under irrelevant input perturbations\. The language\-flip and position\-inconsistency metrics therefore measure whether the evaluator itself is stable enough to support multilingual model comparison\.

#### Practical recommendation\.

For multilingual LLM\-as\-a\-Judge studies, we recommend reporting at least five checks: answer\-order swaps, translation or paraphrase invariance, tie probes for semantically equivalent answers, transformation\-quality audits, and per\-subset breakdowns with confidence intervals or paired tests\. These checks are inexpensive compared with human annotation and can reveal whether an evaluation pipeline is brittle before it is used for model selection\.

## 10Limitations

First, the language variants are generated by an LLM\. The automatic audit and filtered sensitivity analysis reduce obvious mechanical risks, but they do not replace bilingual human semantic validation; 19 items are flagged as high\-risk and should be manually checked before using the variants as a release benchmark\. Second, the tie probes use the gold answer because it gives a conservative high\-quality equivalence condition; weaker\-answer tie probes would further test error\-preservation but require additional controls\. Third, the study focuses on Chinese and Chinese\-English language\-switching\. Other language pairs, scripts, and mixed\-language communities may show different behavior\. Fourth, only four API judge models are tested, and API versions can change\. Fifth, LLMBar is pairwise and instruction\-following oriented; tasks involving long\-form reasoning, safety, medical advice, or multimodal grounding may require additional protocols\.

## 11Conclusion

We introduced Judge\-LS, a lightweight protocol for evaluating whether LLM judges remain stable under English, Chinese, and Chinese\-English language\-switched presentations\. On the full 419\-item LLMBar benchmark, four API judges produced 13,408 successful judgments\. All judges performed best in English, and language transformations caused 10\.7–14\.4% preference flips\. The result remains visible after excluding mechanically flagged high\-risk transformations, and paired tests show significant strict\-correctness changes\. Yet translation\-equivalent tie probes did not show a systematic English preference\. The central risk is therefore not simply that judges prefer English; rather, judges can be sensitive to language presentation in ways that change correctness and interact with position bias\. Judge\-LS is inexpensive, training\-free, and easy to extend, making it a practical diagnostic for multilingual evaluation pipelines\.

## References

- \[1\]Xiyan Fu and Wei Liu\.2025\.How Reliable is Multilingual LLM\-as\-a\-Judge?In*Findings of the Association for Computational Linguistics: EMNLP 2025*, pages 11040–11053\.
- \[2\]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu\.2023\.G\-Eval: NLG Evaluation using GPT\-4 with Better Human Alignment\.In*Proceedings of EMNLP 2023*, pages 2511–2522\.
- \[3\]Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen\.2024\.Evaluating Large Language Models at Evaluating Instruction Following\.In*International Conference on Learning Representations*\.
- \[4\]Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P\. Xing, Hao Zhang, Joseph E\. Gonzalez, and Ion Stoica\.2023\.Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.In*Advances in Neural Information Processing Systems 36*\.
- \[5\]Kishore Papineni, Salim Roukos, Todd Ward, and Wei\-Jing Zhu\.2002\.BLEU: A Method for Automatic Evaluation of Machine Translation\.In*Proceedings of ACL 2002*, pages 311–318\.
- \[6\]Chin\-Yew Lin\.2004\.ROUGE: A Package for Automatic Evaluation of Summaries\.In*Text Summarization Branches Out*, pages 74–81\.
- \[7\]Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q\. Weinberger, and Yoav Artzi\.2020\.BERTScore: Evaluating Text Generation with BERT\.In*International Conference on Learning Representations*\.
- \[8\]Ricardo Rei, Craig Stewart, Ana C\. Farinha, and Alon Lavie\.2020\.COMET: A Neural Framework for MT Evaluation\.In*Proceedings of EMNLP 2020*, pages 2685–2702\.
- \[9\]Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B\. Hashimoto\.2023\.AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback\.arXiv:2305\.14387\.
- \[10\]Yann Dubois, Balazs Galambosi, Percy Liang, and Tatsunori B\. Hashimoto\.2024\.Length\-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators\.arXiv:2404\.04475\.
- \[11\]Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seonghyeon Shin, Sungdong Kim, James Thorne, and Minjoon Seo\.2024\.Prometheus: Inducing Fine\-grained Evaluation Capability in Language Models\.In*International Conference on Learning Representations*\.
- \[12\]Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui\.2024\.Large Language Models are not Fair Evaluators\.In*Findings of the Association for Computational Linguistics: ACL 2024*\.
- \[13\]Nina Panickssery, Samuel R\. Bowman, and Shi Feng\.2024\.LLM Evaluators Recognize and Favor Their Own Generations\.In*Advances in Neural Information Processing Systems 37*\.
- \[14\]Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson\.2020\.XTREME: A Massively Multilingual Multi\-task Benchmark for Evaluating Cross\-lingual Generalization\.In*International Conference on Machine Learning*, pages 4411–4421\.
- \[15\]Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al\.2023\.A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity\.In*Proceedings of IJCNLP\-AACL 2023*, pages 675–718\.
- \[16\]Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen\.2023\.ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 13171–13189\.
- \[17\]Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram\.2023\.MEGA: Multilingual Evaluation of Generative AI\.In*Proceedings of EMNLP 2023*, pages 4232–4267\.
- \[18\]Zicheng Zhang, Junying Wang, Yijin Guo, Farong Wen, Zijian Chen, Hanqing Wang, Wenzhe Li, Lu Sun, Yingjie Zhou, Jianbo Zhang, Bowen Yan, Ziheng Jia, Jiahao Xiao, Yuan Tian, Xiangyang Zhu, Kaiwei Zhang, Chunyi Li, Xiaohong Liu, Xiongkuo Min, Qi Jia, and Guangtao Zhai\.2025\.AIBench: Towards trustworthy evaluation under the 45∘law\.*Displays*, page 103255\.doi:[10\.1016/j\.displa\.2025\.103255](https://doi.org/10.1016/j.displa.2025.103255)\.
- \[19\]Zicheng Zhang, Junying Wang, Farong Wen, Yijin Guo, Xiangyu Zhao, Xinyu Fang, Shengyuan Ding, Ziheng Jia, Jiahao Xiao, Ye Shen, Yushuo Zheng, Xiaorong Zhu, Yalun Wu, Ziheng Jiao, Wei Sun, Zijian Chen, Kaiwei Zhang, Kang Fu, Yuqin Cao, Ming Hu, Yue Zhou, Xuemei Zhou, Juntai Cao, Wei Zhou, Jinyu Cao, Ronghui Li, Donghao Zhou, Yuan Tian, Xiangyang Zhu, Chunyi Li, Haoning Wu, Xiaohong Liu, Junjun He, Yu Zhou, Hui Liu, Lin Zhang, Zesheng Wang, Huiyu Duan, Yingjie Zhou, Xiongkuo Min, Qi Jia, Dongzhan Zhou, Wenlong Zhang, Jiezhang Cao, Xue Yang, Junzhi Yu, Songyang Zhang, Haodong Duan, and Guangtao Zhai\.2025\.Large Multimodal Models Evaluation: A Survey\.*SCIENCE CHINA Information Sciences*, 68\(12\):221301–221369\.doi:[10\.1007/s11432\-025\-4676\-4](https://doi.org/10.1007/s11432-025-4676-4)\.
- \[20\]Yuan Tian, Xiongkuo Min, Guangtao Zhai, and Z\. Gao\.2019\.Video\-based early ASD detection via temporal pyramid networks\.In*IEEE International Conference on Multimedia and Expo*, pages 272–277\.
- \[21\]Yuan Tian, Yichao Yan, Guangtao Zhai, G\. Guo, and Z\. Gao\.2022\.EAN: Event adaptive network for enhanced action recognition\.*International Journal of Computer Vision*, 130\(10\):2453–2471\.
- \[22\]Yuan Tian, Guo Lu, Yichao Yan, Guangtao Zhai, L\. Chen, and Z\. Gao\.2024\.A Coding Framework and Benchmark towards Low\-Bitrate Video Understanding\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*\.
- \[23\]Yuan Tian, Guo Lu, Guangtao Zhai, and Z\. Gao\.2023\.Non\-semantics suppressed mask learning for unsupervised video semantic compression\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 13610–13622\.
- \[24\]Yuan Tian, Yichao Yan, Guangtao Zhai, L\. Chen, and Z\. Gao\.2023\.CLSA: A contrastive learning framework with selective aggregation for video rescaling\.*IEEE Transactions on Image Processing*, 32:1300–1314\.
- \[25\]Yuan Tian, Guo Lu, Xiongkuo Min, Z\. Che, Guangtao Zhai, G\. Guo, and Z\. Gao\.2021\.Self\-conditioned probabilistic learning of video rescaling\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 4490–4499\.
- \[26\]Yuan Tian, Zhaohui Che, Wenbo Bao, Guangtao Zhai, and Z\. Gao\.2020\.Self\-supervised Motion Representation via Scattering Local Motion Cues\.In*European Conference on Computer Vision*, pages 71–89\.
- \[27\]Yuan Tian, Xiaoyue Ling, Cong Geng, Q\. Hu, Guo Lu, and Guangtao Zhai\.2025\.SMC\+\+: Masked learning of unsupervised video semantic compression\.*IEEE Transactions on Pattern Analysis and Machine Intelligence*\.