When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

# When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Source: [https://arxiv.org/html/2607.08535](https://arxiv.org/html/2607.08535)
Zongyou Yang,Yinghan HouDepartment of Electrical and Electronic Engineering, Imperial College LondonLondonUnited Kingdom[yh24@ic\.ac\.uk](https://arxiv.org/html/2607.08535v1/mailto:[email protected])andXiaokun YangSchool of Electronic Information, Nanchang Institute of TechnologyNanchangChina[yangxk@bupt\.cn](https://arxiv.org/html/2607.08535v1/mailto:[email protected])

###### Abstract\.

An LLM\-as\-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed\. We treat this evaluator\-replacement ambiguity as a measurement\-validity problem\. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1\.7B to 32B parameters and moving across MiniMax M2–M2\.7 released APIs\. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1\.7B→\\rightarrow4B gives a robust adjacent gain, while MiniMax adjacent releases do not\. Stronger judges reduce but do not remove position and verbosity bias\. Repeated\-sample juries add little when errors are correlated\. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation\. We argue that LLM\-as\-judge reports should include dataset slices, bias probes, error\-dependence estimates, and protocol audit trails\.

LLM\-as\-judge, automatic evaluation, model scaling, evaluation reliability, bias, jury aggregation

††copyright:none††conference:The 10th International Conference on Advances in Artificial Intelligence; September 21–24, 2026; London, United Kingdom††journalyear:2026††ccs:Computing methodologies Natural language processing††ccs:Computing methodologies Machine learning## 1\.Introduction

LLM\-as\-judge evaluation is now often used as a measuring instrument for model quality\. The difficulty is that the instrument is itself a model\. When a system’s score changes after replacing the evaluator, the change is ambiguous: it may indicate that the new judge is more capable, that it is biased differently, that it fails on a different slice of the benchmark, or that the evaluation pipeline parsed and aggregated its outputs differently\. This ambiguity is not a minor implementation detail\. It determines whether an LLM\-as\-judge score can be interpreted as evidence about the candidate systems at all\.

We call this the*evaluator\-replacement ambiguity*: when the measured preference outcome changes after replacing the judge, the source of the change is not identifiable from accuracy alone\. This paper studies the ambiguity through two observable interventions available in evaluation practice\. The first is a parameter\-scaling decision, represented by Qwen3 dense judges from 1\.7B to 32B parameters\(Yang and others,[2025](https://arxiv.org/html/2607.08535#bib.bib33)\)\. The second is a released\-model upgrade path, represented by MiniMax M2–M2\.7 APIs evaluated as released\(MiniMax,[2026](https://arxiv.org/html/2607.08535#bib.bib35)\)\. These axes are evidence sources rather than the paper’s object of explanation\. The MiniMax M2 report documents the released series and its agent\-oriented training pipeline, but it does not make our API sequence a controlled ablation; consequently, this paper does not make a causal claim about MiniMax internals\.

This formulation leads to three research questions\. RQ1 asks whether judge reliability improves similarly along a parameter axis and a released\-model upgrade path\. If evaluator capability is the dominant factor, both axes should show consistently positive adjacent gains across datasets; if reliability is workload\-dependent, gains should vary by dataset and significance should not transfer uniformly\. RQ2 asks whether higher aggregate accuracy also reduces known judge biases\. Higher\-capability judges should be less bias\-sensitive, but nonzero flip rates would show that capability does not eliminate measurement artifacts\. RQ3 asks whether protocol\-level upgrades, such as juries and debate, change reliability beyond single\-judge scaling\. Jury gains should be limited when error correlation is high, while debate shifts should depend on capability asymmetry and protocol auditability\.

The central thesis is that evaluator capability is important but incomplete: the same judge can look different across workloads, bias probes, correlated votes, and protocol implementations\.

This paper contributes an auditable measurement framework for LLM\-as\-judge reliability, separating model capability from workload dependence, bias behavior, correlated voting, and measurement artifacts\. First, it presents a two\-axis evaluator study that compares parameter scaling with released\-model generation while explicitly avoiding causal claims about MiniMax internals\. Second, it decomposes reliability into single\-judge accuracy, bias probes, correlated jury errors, and debate auditability\. Third, it distills a reporting standard: benchmark slices, parse\-shared tests, A/B randomization,ρ\\rhoestimates, and protocol logs\.

The study yields three main results\. First, judge upgrades are not interchangeable: parameter scaling produces one robust low\-end gain, while adjacent MiniMax releases do not\. Second, higher accuracy reduces but does not remove bias\. Third, aggregation protocols are interpretable only when their assumptions are auditable: jury voting is bounded by correlated errors, and debate shifts cannot be attributed to deliberation without parser logs\.

![Refer to caption](https://arxiv.org/html/2607.08535v1/x1.png)Figure 1\.LLM\-as\-judge as a measurement pipeline\. A reported score depends not only on candidate responses, but also on the benchmark slice, judge model, prompt and decoding choices, parser, and any aggregation or debate protocol\.A pipeline diagram showing candidate responses, benchmark slice, judge model, prompt and decoding, parser, aggregation or debate, and reported score, with risk labels for validity, bias, dependence, and parser artifacts\.
## 2\.Related Work

LLM judges and preference evaluation\.MT\-Bench, Chatbot Arena, PandaLM, JudgeLM, G\-Eval, Auto\-J, Prometheus, and RewardBench have helped establish LLM judges as practical evaluators for open\-ended generation\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib1); Wanget al\.,[2023c](https://arxiv.org/html/2607.08535#bib.bib2); Zhuet al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib3); Liuet al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib5); Liet al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib20); Kimet al\.,[2024a](https://arxiv.org/html/2607.08535#bib.bib18),[b](https://arxiv.org/html/2607.08535#bib.bib19); Lambertet al\.,[2024](https://arxiv.org/html/2607.08535#bib.bib21)\)\. These studies show that LLM judges can correlate with human preferences, but also that correlations depend on task, prompt, scoring scale, and model family\. Our work follows this empirical tradition but treats the judge itself as the object of scaling analysis\. The gap is that prior reports often present evaluator performance as a one\-dimensional model ordering, leaving unclear whether gains arise from capability, benchmark slice, bias reduction, or protocol design\.

Judge bias and protocol effects\.Prior work documents position bias, verbosity bias, sensitivity to rating format, and reliability changes caused by evaluation design choices\(Duboiset al\.,[2024](https://arxiv.org/html/2607.08535#bib.bib7); Saitoet al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib8); Wanget al\.,[2023a](https://arxiv.org/html/2607.08535#bib.bib17); Yamauchiet al\.,[2025](https://arxiv.org/html/2607.08535#bib.bib41)\)\. These effects matter because they can change rankings without changing candidate responses\. What remains under\-specified is how bias changes jointly with evaluator scaling rather than under isolated prompt perturbations\. Unlike a full prompt\-factorial study, our experiments use a fixed main prompt and treat prompt sensitivity as a robustness check; the main focus is how model capability, bias, aggregation, and debate interact under a controlled reporting protocol\.

Aggregation and debate\.Classical Condorcet\-style results predict gains from independent voters, while work on correlated voters shows that dependence can sharply reduce those gains\(de Condorcet,[1785](https://arxiv.org/html/2607.08535#bib.bib9); Ladha,[1992](https://arxiv.org/html/2607.08535#bib.bib10)\)\. In LLM evaluation, self\-consistency, multi\-agent debate, and ChatEval\-style protocols offer analogous routes to aggregation or deliberation\(Wanget al\.,[2023b](https://arxiv.org/html/2607.08535#bib.bib24); Duet al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib12); Lianget al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib23); Chanet al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib13); Khanet al\.,[2024](https://arxiv.org/html/2607.08535#bib.bib22)\)\. Existing studies often emphasize final accuracy; our focus is whether the aggregation assumptions required for such gains are empirically satisfied\. Our jury experiment directly estimates error correlation, and our debate experiment is reported as an auditability case study\.

## 3\.Experimental Design

### 3\.1\.Operationalizing Judge Reliability

We treat LLM\-as\-judge reliability as a multi\-dimensional measurement property rather than a single accuracy score\. In this study, reliability consists of four observable components: judgment validity, bias robustness, aggregation independence, and protocol auditability\. Table[1](https://arxiv.org/html/2607.08535#S3.T1)maps each construct to an operational measure and experiment\.

Table 1\.Operationalizing LLM\-as\-judge reliability\.
### 3\.2\.Models and Datasets

The main panel contains eight evaluated judges: Qwen3\-1\.7B, Qwen3\-4B, Qwen3\-14B, Qwen3\-32B, MiniMax\-M2, MiniMax\-M2\.1, MiniMax\-M2\.5, and MiniMax\-M2\.7\. This panel separates an approximate parameter intervention from an observed released\-model intervention\. GLM\-5\.1 and mimo\-v2\-pro are used as cross\-family reference judges in selected experiments, but are not members of either axis\.

Four datasets cover complementary judgment settings so that reliability is not inferred from a single task slice\. LLMBar is an adversarial pairwise benchmark with 419 examples\(Zenget al\.,[2024](https://arxiv.org/html/2607.08535#bib.bib4)\)\. PandaLM testset\-v1 provides broad\-domain pairwise judgments; after removing tie\-majority examples, 894 valid pairwise examples remain\(Wanget al\.,[2023c](https://arxiv.org/html/2607.08535#bib.bib2)\)\. A 2,000\-example seed\-42 sample from Chatbot Arena provides broad human preference comparisons, with 1,997 valid pairwise examples\(Zhenget al\.,[2023](https://arxiv.org/html/2607.08535#bib.bib1)\)\. Our Judge’s Verdict slice uses 200 TechQA\-derived examples with three\-level pointwise labels, supporting exact\-match and rank\-based evaluation\(Hanet al\.,[2025](https://arxiv.org/html/2607.08535#bib.bib36); Castelliet al\.,[2019](https://arxiv.org/html/2607.08535#bib.bib14)\)\.

### 3\.3\.Metrics and Protocols

Pairwise experiments report accuracy and Cohen’sκ\\kappaon parseable verdicts; Judge’s Verdict additionally reports Spearman rank correlation\. Main single\-judge runs use near\-greedy decoding \(T=0\.1T=0\.1\) to reduce sampling noise when estimating model\-level reliability\. Homogeneous juries sample one judge multiple times atT=0\.7T=0\.7because repeated low\-temperature calls would understate the dependence structure among stochastic jurors; heterogeneous juries combine different judges atT=0\.1T=0\.1to isolate model composition from sampling variance\. Adjacent model comparisons use exact two\-sided McNemar tests on parse\-shared examples, so significance reflects paired judgment changes rather than differences in parser coverage\. Holm correction controls the family of 18 adjacent tests\.

The experiments instantiate the constructs in Table[1](https://arxiv.org/html/2607.08535#S3.T1)\. Experiment 1 measures single\-judge validity along the two axes\. Experiment 2 measures LLMBar position, verbosity, and granularity robustness\. Experiment 3 evaluates majority\-vote juries under measured error correlation\. Experiment 4 evaluates a structured debate protocol as an auditability case study\. Four robustness checks cover human\-ceiling calibration, corrected adjacent significance, limited prompt sensitivity, and Arena seed stability\.

![Refer to caption](https://arxiv.org/html/2607.08535v1/x2.png)Figure 2\.Study design\. The paper uses a Qwen3 parameter axis and a MiniMax release\-generation axis as observable evaluator\-replacement interventions, then tests whether single\-judge scaling, bias probes, jury aggregation, and structured debate answer the same reliability question\.A schematic of the two\-axis LLM\-as\-judge testbed with model axes, datasets, experiments, and robustness checks\.

## 4\.Results

Table 2\.Main answers by research question\.### 4\.1\.Finding 1: Judge Upgrades Are Not Interchangeable

A capability\-only account would predict similar adjacent gains along both upgrade paths; the paired tests do not show that pattern\. Table[3](https://arxiv.org/html/2607.08535#S4.T3)summarizes the main single\-judge pattern\. Qwen3 improves sharply from 1\.7B to 4B on LLMBar \(0\.463 to 0\.617\) and more modestly on Arena, but later parameter steps are smaller and not uniformly monotone\. MiniMax release generations show no reliable adjacent improvement in this panel: adjacent pairwise differences are at most 0\.022 accuracy, and none of the nine MiniMax adjacent tests reaches uncorrectedp<0\.05p<0\.05\. After Holm correction across all 18 adjacent tests, only the Qwen3 1\.7B to 4B steps on LLMBar and Arena remain significant\.

Table 3\.Single\-judge aggregate results\. Values are accuracies except Judge’s Verdict, where exact match is reported for comparability; brackets give Wilson 95% intervals\.The relevant conclusion is not that one family is universally stronger\. No judge wins all datasets: GLM\-5\.1 leads LLMBar, MiniMax\-M2\.7 leads PandaLM, and mimo\-v2\-pro leads the sampled Arena slice\. This shifts the design question from one\-dimensional evaluator ordering to slice\-specific measurement validity\.

![Refer to caption](https://arxiv.org/html/2607.08535v1/x3.png)Figure 3\.Adjacent\-pair McNemar tests for the two axes\. After Holm correction over 18 adjacent tests, only the Qwen3 1\.7B→\\rightarrow4B steps on LLMBar and Arena remain significant; no MiniMax adjacent release step is significant in this panel\.A heat map of adjacent model significance values for Qwen3 and MiniMax on multiple datasets\.
### 4\.2\.Finding 2: Stronger Judges Are Less Biased, Not Unbiased

On LLMBar, the bias probes move in the same direction as capability, but they do not disappear\. The position\-flip rate falls from 0\.320 for Qwen3\-1\.7B to 0\.117–0\.147 for MiniMax releases\. Verbosity bias under a fixed generic and innocuous padding string falls from 0\.547 for Qwen3\-1\.7B to roughly 0\.13 for MiniMax\. Granularity sensitivity is also smaller for the stronger judges\.

This pattern is best interpreted as a capability–fairness association rather than a mechanism\. Across the eight evaluated judges, the correlation between LLMBar accuracy and position\-flip rate is strong \(Pearsonr=−0\.957r=\-0\.957\), but it is measured on one dataset and one model panel\. Stronger judges are also not unbiased: MiniMax\-M2\.7 still changes 14\.7% of verdicts under A/B reversal\. Thus position randomization and slice\-level bias reporting remain necessary even when using high\-accuracy judges\.

![Refer to caption](https://arxiv.org/html/2607.08535v1/x4.png)Figure 4\.Capability–fairness association on LLMBar\. Higher single\-judge accuracy co\-varies with lower position\-flip rate across the eight evaluated judges, but the non\-zero flip rates show that stronger judges still require A/B randomization and bias reporting\.A scatter plot of LLMBar accuracy versus position bias flip rate, with a negative linear trend\.
### 4\.3\.Finding 3: Jury Size Matters Less Than Error Dependence

The jury experiment fails for a familiar statistical reason: the votes are not independent\. Majority voting is therefore not a generic reliability amplifier for LLM judges\. For homogeneous juries, we estimate the intra\-class error correlationρ\\rhofrom the vote matrix and compare three quantities: the independence prediction, aρ\\rho\-corrected beta\-binomial prediction, and observed jury accuracy\. Letppbe the single\-judge baseline accuracy,q∼Beta​\(α,β\)q\\sim\\mathrm\{Beta\}\(\\alpha,\\beta\),α=p​s\\alpha=ps,β=\(1−p\)​s\\beta=\(1\-p\)s, ands=1/ρ−1s=1/\\rho\-1\. The jury accuracy is then the beta\-binomial probability that a majority ofKKvotes are correct\.

The correction is empirically necessary\. Across homogeneous juries, the independence prediction has median absolute error 0\.078 on LLMBar and 0\.093 on PandaLM\. Theρ\\rho\-corrected prediction reduces these errors to 0\.008 and 0\.004, with maximum error below 0\.02\. Estimated correlations are high: Qwen3 homogeneous juries haveρ=0\.944\\rho=0\.944–0\.972 on LLMBar, while MiniMax juries are lower but still correlated atρ=0\.664\\rho=0\.664–0\.706\. As a result, increasing jury size barely moves accuracy; for Qwen3\-1\.7B on LLMBar,K=1,3,5K=1,3,5yields 0\.463, 0\.475, and 0\.482\.

Heterogeneous juries also underperform Poisson\-binomial independence predictions\. Family mixing alone therefore does not restore independent errors under a shared prompt\. This shifts the relevant design question from “how many jurors should be sampled?” to “how independent are their errors?”

![Refer to caption](https://arxiv.org/html/2607.08535v1/x5.png)Figure 5\.Jury behavior on LLMBar\. Majority voting provides little gain because sampled jurors make correlated errors; theρ\\rho\-corrected beta\-binomial model tracks observed jury accuracy much better than the independence assumption\.A multi\-panel plot comparing independent, rho\-corrected, and observed jury accuracy, plus jury\-size and error\-correlation panels\.
### 4\.4\.Finding 4: Debate Shifts Require Audit Trails

The debate runs produce the largest protocol\-level shifts in the paper, but they are also the least auditable\. We include debate not as evidence for a deliberation effect, but as a stress test for protocol auditability\. The structured\-debate experiment pairs two judges for up to three rebuttal rounds on LLMBar\. Cross\-capability pairs show the largest final\-vs\-round\-1 accuracy shifts: Qwen3\-1\.7B paired with GLM\-5\.1, MiniMax\-M2\.7, or mimo\-v2\-pro changes final decisions toward higher measured accuracy by\+0\.317\+0\.317,\+0\.305\+0\.305, and\+0\.289\+0\.289; Qwen3\-4B paired with GLM\-5\.1 shifts decisions by\+0\.243\+0\.243and reaches 0\.897\. Same\-family and top\-vs\-top shifts are much smaller, ranging from\+0\.005\+0\.005to\+0\.050\+0\.050\.

These changes show that protocol\-level interventions can alter judge outcomes substantially; the missing parser audit trail shows why such shifts should not be accepted without measurement logs\. The implementation records round verdicts and final verdicts, but not raw responses or parse\-success flags\. Round\-1 parse failures fall back to A, and later parse failures retain the previous verdict\. Since fallback rates cannot be audited without rerunning the experiment, the debate results are best read as a suggestive protocol pattern and a reporting requirement, not as a clean estimate of deliberation\.

![Refer to caption](https://arxiv.org/html/2607.08535v1/x6.png)Figure 6\.Structured\-debate accuracy shift versus single\-judge capability gap on LLMBar\. Cross\-capability pairs produce the largest shifts, but the debate implementation’s parse\-fallback behavior makes the result suggestive rather than causal\.A scatter plot showing debate accuracy shift increasing with single\-judge capability gap\.
### 4\.5\.Robustness Checks

The robustness checks support the direction of the main findings while keeping their scope narrow\. Human\-ceiling calibration shows that PandaLM retains headroom: the best judge has matched leave\-one\-annotator\-outκ=0\.753\\kappa=0\.753versus a human ceiling of 0\.920\. Judge’s Verdict is closer to its noisy ceiling, with best\-judgeκ=0\.620\\kappa=0\.620versus humanκ=0\.562\\kappa=0\.562\. Prompt\-sensitivity checks on LLMBar preserve the cross\-family ordering and show greater cross\-prompt stability for stronger judges\. Arena seed perturbations produce 0\.004–0\.032 accuracy spread, smaller than the robust Qwen3 1\.7B to 4B Arena gap\. These checks do not establish cross\-family generality, but they reduce several obvious alternative explanations\.

## 5\.Discussion

Principle\.A judge score is interpretable only when the evaluated slice, perturbation sensitivity, error dependence, and protocol state are jointly reported\. The practical recommendation is therefore simple: an LLM\-as\-judge result should not be accepted as reliable unless it reports what slice was judged, how biased the judge was under simple perturbations, how correlated repeated judgments were, and whether protocol\-level decisions were auditable\.

Table 4\.A minimal audit trail for LLM\-as\-judge reports\.The results suggest two methodological lessons for LLM\-as\-judge evaluation\. First, judge scaling should be reported as a design space rather than a single evaluator ordering\. In this panel, the clearest parameter\-axis gain occurs at the low end, while the MiniMax release\-generation axis changes little in the measured adjacent comparisons\. This should not be read as a claim that all release generations lack meaningful gains, or that the MiniMax releases isolate one training stage\. The supported conclusion is narrower: under the same datasets and prompts, the two observed upgrade paths have different adjacent\-pair patterns\.

Second, reliability reporting has to go beyond single\-number accuracy and name the unit of analysis\. Bias probes show that stronger judges can be less brittle, but they also show that bias remains measurable\. Jury experiments show that aggregation gains depend on error correlation\. Debate experiments show that protocol changes can be large enough to rival scaling effects, but only if the implementation preserves an audit trail for parse failures, tie\-breaking, and abstentions\. For pairwise datasets, this means parseable\-subset definitions, parse\-shared significance tests, and A/B randomization\. For juries, it means reportingρ\\rhoin addition toKK\. For debate or multi\-agent protocols, it means storing raw responses, parser status, and every intermediate vote before interpreting final\-score shifts\. These requirements are operationally simple, but without them the apparent reliability of a judge can be an artifact of the evaluation pipeline\.

## 6\.Threats to Validity

Construct validity\.Reliability is operationalized through selected proxies: accuracy and agreement, bias probes, error correlation, and auditability logs\. These do not exhaust all reliability dimensions, such as calibration, long\-form rationale quality, or domain\-specific consistency\.

Internal validity\.The Qwen3 axis is closer to a parameter sweep, whereas the MiniMax axis is an observed release sequence rather than a controlled ablation\. The debate experiment is especially limited by missing raw outputs and parse\-success logs\.

Statistical conclusion validity\.McNemar tests are adjacent\-pair tests and do not constitute a formal between\-axis test\. Multiple comparisons are controlled through Holm correction, but sample sizes and parser coverage differ across datasets\.

External validity\.Results are conditioned on two model families, two reference judges, four datasets, a small number of prompts, and one primary decoding regime per experiment\. Additional domains, languages, and instruction distributions are needed before turning these observations into general judge\-selection rules\.

## 7\.Conclusion

This paper presents a two\-axis empirical study of LLM\-as\-judge reliability\. The results show that evaluator capability is important but incomplete: reliability is also shaped by dataset slice, bias behavior, correlated errors, and protocol implementation\. In the tested panel, the strongest robust adjacent effect is the Qwen3 1\.7B to 4B transition; no MiniMax adjacent release step is significant; majority\-vote juries are limited by high error correlation; and debate with a strong reference changes outcomes substantially but lacks sufficient evidence for causal interpretation under the current logs\. These findings recast LLM\-as\-judge reliability as a measurement\-validity problem rather than a model\-selection problem alone\.

###### Acknowledgements\.

This work was supported by the Open Research Project of the State Key Laboratory of Industrial Control Technology, China \(Grant No\. ICT2025B70\); the Jiangxi Provincial Natural Science Foundation \(Grant Nos\. 20242BAB20041 and 20232BAB212006\); the Hubei Provincial Natural Science Foundation of China \(Grant Nos\. 2023AFB474 and 2024AFB881\); the Anhui Provincial Special Project for Special Needs in Humanities and Social Sciences \(Grant No\. 2025AHGXSK50067\); and the Postgraduate Quality Engineering Project of Anhui Province \(Grant No\. 2024jyjxggyjY232\)\.

## References

- V\. Castelli, R\. Chakravarti, S\. Dana, A\. Ferritto, R\. Florian, M\. Franz, D\. Garg, D\. Khandelwal, S\. McCarley, M\. McCawley, M\. Nasr, L\. Pan, C\. Pendus, J\. Pitrelli, S\. Pujar, S\. Roukos, A\. Sakrajda, A\. Sil, R\. Uceda\-Sosa, T\. Ward, and R\. Zhang \(2019\)The TechQA dataset\.arXiv preprint arXiv:1911\.02984\.External Links:1911\.02984Cited by:[§3\.2](https://arxiv.org/html/2607.08535#S3.SS2.p2.1)\.
- C\. Chan, W\. Chen, Y\. Su, J\. Yu, W\. Xue, S\. Zhang, J\. Fu, and Z\. Liu \(2023\)ChatEval: towards better LLM\-based evaluators through multi\-agent debate\.arXiv preprint arXiv:2308\.07201\.External Links:2308\.07201Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- M\. J\. A\. N\. C\. de Condorcet \(1785\)Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix\.Imprimerie Royale, Paris\.Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- Y\. Du, S\. Li, A\. Torralba, J\. B\. Tenenbaum, and I\. Mordatch \(2023\)Improving factuality and reasoning in language models through multiagent debate\.InInternational Conference on Machine Learning \(ICML\),External Links:2305\.14325Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto \(2024\)Length\-controlled AlpacaEval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.External Links:2404\.04475Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p2.1)\.
- S\. Han, G\. Titericz Junior, T\. Balough, and W\. Zhou \(2025\)Judge’s Verdict: a comprehensive analysis of LLM judge capability through human agreement\.arXiv preprint arXiv:2510\.09738\.External Links:2510\.09738Cited by:[§3\.2](https://arxiv.org/html/2607.08535#S3.SS2.p2.1)\.
- A\. Khan, J\. Hughes, D\. Valentine, L\. Ruis, K\. Sachan, A\. Radhakrishnan, E\. Grefenstette, S\. R\. Bowman, T\. Rocktäschel, and E\. Perez \(2024\)Debating with more persuasive LLMs leads to more truthful answers\.International Conference on Machine Learning \(ICML\)\.External Links:2402\.06782Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- S\. Kim, J\. Shin, Y\. Cho, J\. Jang, S\. Longpre, H\. Lee, S\. Yun, S\. Shin, S\. Kim, J\. Thorne, and M\. Seo \(2024a\)Prometheus: inducing fine\-grained evaluation capability in language models\.International Conference on Learning Representations \(ICLR\)\.External Links:2310\.08491Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1)\.
- S\. Kim, J\. Suk, S\. Longpre, B\. Y\. Lin, J\. Shin, S\. Welleck, G\. Neubig, M\. Lee, K\. Lee, and M\. Seo \(2024b\)Prometheus 2: an open source language model specialized in evaluating other language models\.arXiv preprint arXiv:2405\.01535\.External Links:2405\.01535Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1)\.
- K\. K\. Ladha \(1992\)The Condorcet jury theorem, free speech, and correlated votes\.American Journal of Political Science36\(3\),pp\. 617–634\.External Links:[Document](https://dx.doi.org/10.2307/2111584)Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- N\. Lambert, V\. Pyatkin, J\. Morrison, L\. Miranda, B\. Y\. Lin, K\. Chandu, N\. Dziri, S\. Kumar, T\. Zick, Y\. Choi, N\. A\. Smith, and H\. Hajishirzi \(2024\)RewardBench: evaluating reward models for language modeling\.arXiv preprint arXiv:2403\.13787\.External Links:2403\.13787Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1)\.
- J\. Li, S\. Sun, W\. Yuan, R\. Fan, H\. Zhao, and P\. Liu \(2023\)Generative judge for evaluating alignment\.arXiv preprint arXiv:2310\.05470\.External Links:2310\.05470Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1)\.
- T\. Liang, Z\. He, W\. Jiao, X\. Wang, Y\. Wang, R\. Wang, Y\. Yang, S\. Shi, and Z\. Tu \(2023\)Encouraging divergent thinking in large language models through multi\-agent debate\.arXiv preprint arXiv:2305\.19118\.External Links:2305\.19118Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. Zhu \(2023\)G\-Eval: NLG evaluation using GPT\-4 with better human alignment\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:2303\.16634Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1)\.
- MiniMax \(2026\)The MiniMax\-M2 series: mini activations unleashing max real\-world intelligence\.arXiv preprint arXiv:2605\.26494\.External Links:2605\.26494Cited by:[§1](https://arxiv.org/html/2607.08535#S1.p2.1)\.
- K\. Saito, A\. Wachi, K\. Wataoka, and Y\. Akimoto \(2023\)Verbosity bias in preference labeling by large language models\.arXiv preprint arXiv:2310\.10076\.External Links:2310\.10076Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p2.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui \(2023a\)Large language models are not fair evaluators\.arXiv preprint arXiv:2305\.17926\.External Links:2305\.17926Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p2.1)\.
- X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou \(2023b\)Self\-consistency improves chain of thought reasoning in language models\.International Conference on Learning Representations \(ICLR\)\.External Links:2203\.11171Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p3.1)\.
- Y\. Wang, Z\. Yu, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. Zhang \(2023c\)PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization\.arXiv preprint arXiv:2306\.05087\.External Links:2306\.05087Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.08535#S3.SS2.p2.1)\.
- Y\. Yamauchi, T\. Yano, and M\. Oyamada \(2025\)An empirical study of LLM\-as\-a\-judge: how design choices impact evaluation reliability\.arXiv preprint arXiv:2506\.13639\.External Links:2506\.13639,[Document](https://dx.doi.org/10.48550/arXiv.2506.13639)Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p2.1)\.
- A\. Yanget al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.External Links:2505\.09388Cited by:[§1](https://arxiv.org/html/2607.08535#S1.p2.1)\.
- Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. Chen \(2024\)Evaluating large language models at evaluating instruction following\.InInternational Conference on Learning Representations \(ICLR\),External Links:2310\.07641Cited by:[§3\.2](https://arxiv.org/html/2607.08535#S3.SS2.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-Judge with MT\-Bench and Chatbot Arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,External Links:2306\.05685Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.08535#S3.SS2.p2.1)\.
- L\. Zhu, X\. Wang, and X\. Wang \(2023\)JudgeLM: fine\-tuned large language models are scalable judges\.arXiv preprint arXiv:2310\.17631\.External Links:2310\.17631Cited by:[§2](https://arxiv.org/html/2607.08535#S2.p1.1)\.