JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation

# JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
Source: [https://arxiv.org/html/2608.02620](https://arxiv.org/html/2608.02620)
Erlis Lushtaku University of Freiburg &Bora Kargi ELLIS Institute Tübingen &Ali Elganzory University of Freiburg &Fabio Ferreira Microsoft AI &Alejandro R\. Salamanca Cohere Labs &Julia Kreutzer Cohere Labs &David Salinas11footnotemark:122footnotemark:2 ELLIS Institute Tübingen, Prior Labs Equal contribution\.Corresponding authors:erlislushtaku@gmail\.com,david\.salinas\.pro@gmail\.comWork done while at University of Freiburg\.

###### Abstract

LLM\-as\-a\-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed\-model judge, and support a single evaluation protocol\. This fragmentation makes it difficult to study how design choices—the benchmark, the judge model, the prompt, the inference backend—affect the conclusions we draw about model quality\. We introduceJudgeArena, an open\-source framework that unifies major LLM\-judge benchmarks \(AlpacaEval, Arena\-Hard, MT\-Bench, and m\-Arena\-Hard\) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility\. It enables systematic studies of judge choices, as any model accessible via vLLM, llama\.cpp, or OpenRouter can serve as both candidate and judge\. Furthermore,JudgeArenaships with tuned judge configurations for open models that match or outperform closed\-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models\. Finally, by combining existing human annotations with LLM\-judge evaluations of a target model,JudgeArenacan simulate LMArena Elo scores with high accuracy offering a practical, open, and low\-cost alternative to large\-scale human annotation campaigns\.

## 1Introduction

Evaluating instruction\-tuning for Large Language Models \(LLMs\) is challenging because generative models output free\-form text in response to questions that are often open or ambiguous\. The gold standard would be human annotations, including the so\-called crowd\-sourcing*arenas*where users annotate their preferred output to a prompt given two modelsChianget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib42)\); Termignonet al\.\([2026](https://arxiv.org/html/2608.02620#bib.bib62)\)\. Since annotation costs are high, and turnaround is slow, this approach is less practical in early LLM development stages or projects with large scale evaluation needs \(e\.g\. tracking quality across multiple languages and tasks\)\. The alternative is to deploy an LLM judge as a human proxy\(Zhenget al\.,[2023](https://arxiv.org/html/2608.02620#bib.bib16)\)\. A key advantage is that it enables automatic and cheaper benchmarks which can be run at will\.

Many benchmarks relying on LLM\-judges have been proposed\. Among the most prominent, AlpacaEvalLiet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib13)\); Duboiset al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib12)\)and Arena\-HardLiet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib2)\), propose to annotate a candidate model on a fixed set of instructions\. The answer is then compared to a baseline like a recent GPT model, and the win rates are used to construct a leaderboard\. Different setups have been studied, for instance, considering multi\-turn instead of single\-turn in MT\-BenchZhenget al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib16)\)or considering multiple languages in m\-Arena\-Hard by automatically translating instructions of Arena\-Hard from English to 23 languagesDanget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib61)\)\.

There is currently a clear lack of unification and reproducibility across LLM\-judge benchmarks as shown in[Table˜1](https://arxiv.org/html/2608.02620#S1.T1)\. For example, each benchmark ships its own independent code\-base or rarely logs sufficient metadata for reproducibility\. Moreover, they largely default to closed\-weight judges such as GPT\-4\. While LLM judges exhibit self\-preference bias regardless of opennessPanicksseryet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib37)\); Wataokaet al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib7)\), closed\-weight judges make this bias hard to audit and leave benchmarks vulnerable to silent updates and deprecations, rendering many evaluations stale and irreproducible111For instance, GPT\-4o is the model that was used in Alpaca\-Eval, Arena\-Hard, m\-Arena\-Hard and is being retired[https://openai\.com/index/retiring\-gpt\-4o\-and\-older\-models/](https://openai.com/index/retiring-gpt-4o-and-older-models/)\. As a result, researchers use benchmarks coming from different packages \(or write their own\) which makes results sometimes incomparable\(Laskaret al\.,[2024](https://arxiv.org/html/2608.02620#bib.bib5)\)and strongly resembles the status of*static*evaluations before the introduction of packages such as Eval HarnessGaoet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib66)\)or LightevalHabibet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib67)\)\. This paper describesJudgeArena, an open\-source library that addresses these issues\. In particular, it makes the following contributions \(see[Table˜1](https://arxiv.org/html/2608.02620#S1.T1)\):

- •Standardized access to the most common LLM\-as\-a\-judge benchmarks under a unified API, with swappable judges and broad support for inference backends\.
- •Tuned open\-weight judge configurations that match or outperform closed\-model judges while being cheaper and more transparent\.
- •Comprehensive metadata logging that ensures full reproducibility of every run\.
- •Extensive meta\-evaluations supported by the library covering both English and 33 languages\.
- •A new LLM benchmark for estimating Elo ratings with small error across holdout models—enabling model performance estimation without participating in online arenas

Overall,JudgeArenaunifies access to major benchmarks while exposing each meta\-evaluation component for controlled study and continuous improvement\.

Table 1:Comparison of libraries supporting LLM\-judges\. Most support only a single benchmark, except Lighteval and Evalchemy\. “vLLM” indicates built\-in, orchestrated support for in\-process local vLLM inference for both candidates and judges\.FrameworkMT\-BenchAlpacaEvalArena\-Hardm\-Arena\-HardTuned judgevLLMMetadataFastChat✓×\\times×\\times×\\times×\\times×\\times×\\timesAlpacaEval×\\times✓×\\times×\\times×\\times×\\times×\\timesArena\-Hard×\\times×\\times✓×\\times×\\times×\\times×\\timesLighteval✓×\\times×\\times×\\times×\\times✓✓Evalchemy✓✓×\\times×\\times×\\times×\\times✓JudgeArena✓✓✓✓✓✓✓
## 2Related Work

#### LLM\-as\-a\-Judge Benchmarks\.

Pairwise LLM\-as\-a\-judge benchmarks have become the default cheap surrogate for human preference rankings, an alternative to crowd\-sourced platformsChianget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib42)\)and to static benchmark mixturesNiet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib52)\)\. The pairwise paradigm was popularized by MT\-Bench and Chatbot ArenaZhenget al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib16)\); Chianget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib42)\), where strong LLMs were shown to agree well with crowd preferences when used as judges, despite known biases such as self\-preferencePanicksseryet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib37)\)and verbosity\. AlpacaEvalLiet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib13)\)pairs805805single\-turn instructions with a single GPT\-4 judge scored against a fixed reference; its length\-controlled variantDuboiset al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib12)\)debiases that same judge along the verbosity axis\. Arena\-HardLiet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib2)\)curates500500harder English prompts auto\-mined from Chatbot Arena conversations\. Multilingual coverage is mostly bolted on through translation: m\-Arena\-HardDanget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib61)\); Khairiet al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib11)\)automatically translates Arena\-Hard into 23 languages and again leverages a single closed judge, while native\-language preference data beyond English remain rareTermignonet al\.\([2026](https://arxiv.org/html/2608.02620#bib.bib62)\)and translation\-based auto\-eval is known to require careful meta\-evaluationKreutzeret al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib63)\); Fu and Liu \([2025](https://arxiv.org/html/2608.02620#bib.bib10)\)\.

#### LLM Judge Frameworks\.

Running LLM\-judge evaluations at scale poses non\-trivial engineering challenges, such as managing API calls, parsing judge outputs, and logging runs, which led benchmarks like FastChatZhenget al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib16)\), AlpacaEval, and Arena\-Hard to provide benchmark\-specific implementations tightly coupled to proprietary APIs\. While users can indirectly circumvent this by manually routing requests to local OpenAI\-compatible servers \(e\.g\., standalone vLLM processes\), this detached setup is error\-prone and fails to capture vital inference metadata, compromising reproducibility\. This architectural lock\-in historically encourages reliance on closed\-weight models, which are subject to silent updates and exhibit systematic self\-preferencePanicksseryet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib37)\)\. Furthermore, evaluating models across different benchmarks requires managing disparate codebases with incompatible interfaces\.

Generalized frameworks such as Eval HarnessGaoet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib66)\)and LightevalHabibet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib67)\)unify diverse tasks but are optimized for static evaluation \(e\.g\., the Open LLM LeaderboardFourrieret al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib70)\)\), requiring custom logic to support dynamic pairwise judges\. EvalchemyRaoofet al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib69)\)wraps existing codebases and supports vLLM for candidate generation, but its judge pipeline still relies on indirect API calls, leaving it focused on fixed proprietary judges and lacking the abstractions needed to integrate easily additional benchmarks such as Arena\-Hard\.

As summarized in[Table˜1](https://arxiv.org/html/2608.02620#S1.T1), this fragmented ecosystem hinders reproducibility\.JudgeArenaresolves these limitations by decoupling instruction sets from evaluator configurations\. By providing a unified API with native local inference for both candidates and judges, optimized prompt templates, and rigorous metadata tracking for every generated sample,JudgeArenaseamlessly enables sovereign, open\-weight evaluations with end\-to\-end reproducibility\.

#### Meta\-Evaluation and Tuning of LLM Judges\.

Meta\-evaluation assesses judge reliability to determine when and for which instruction types LLM judges are suitable surrogates for human preferences\. Meta\-evaluation often considers two categories of metrics:*global*metrics measuring ranking correlation—Spearman rank correlation between judge rankings and Elo ratings from human arenasLiet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib13),[2024](https://arxiv.org/html/2608.02620#bib.bib2)\); Niet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib52)\)—and*local*instruction\-level agreement metrics such as accuracy or Cohen\-KappaZhenget al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib16)\); Wanget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib38)\); Zenget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib39)\); Fu and Liu \([2025](https://arxiv.org/html/2608.02620#bib.bib10)\)\. Recent work further probes judges along specific axes: hard coding and math instructionsTanet al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib30)\); Laiet al\.\([2026](https://arxiv.org/html/2608.02620#bib.bib25)\), positional and verbosity biasesLaiet al\.\([2026](https://arxiv.org/html/2608.02620#bib.bib25)\), or multilingual capabilitiesKreutzeret al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib63)\); Fu and Liu \([2025](https://arxiv.org/html/2608.02620#bib.bib10)\)\. Being able to measure the quality of LLM judges motivate tuning via prompt optimizationFernandoet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib15)\)or systematic hyperparameter search over judge design choicesSalinaset al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib65)\)\. Despite this activity, tuned open\-weight judges are rarely bundled with evaluation benchmarks;JudgeArenadirectly addresses this gap by integrating meta\-evaluation routines and competitive benchmarks in a single compatible framework\.

## 3JudgeArenaFramework

### 3\.1Architecture and Design

JudgeArenais exposed through a single command\-line interface, see[Fig\.˜1](https://arxiv.org/html/2608.02620#S3.F1)\. The benchmark and the judge are selected through CLI options rather than being baked into benchmark\-specific scripts\. Every other aspect of a run like the truncation budgets, the chat template and the inference\-engine options, is controlled through the same interface\. The metadata containing all information required for reproducibility is stored in a JSON similarly to Eval HarnessGaoet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib66)\)\.

To perform an evaluation, instructions from the selected benchmark are loaded through a common interface and optionally subsampled to a specified number of instructions\. For each evaluated model,JudgeArenareuses pre\-shipped completions bundled by the benchmark itself when available, and otherwise generates them, persisting fresh completions through a deterministic on\-disk cache\. The two completions and the judge prompt are then fed to the judge, its output is parsed into a numerical preference in\[0,1\]\[0,1\], and a per\-run folder of annotations, aggregate win/loss summary and metadata descriptor is written for downstream analysis\.

judgearena\-\-taskalpaca\-eval\-\-model\_ALlamaCpp/\./path/llama\-3\.2\-3b\-q8\_0\.gguf\-\-model\_BVLLM/utter\-project/EuroLLM\-9B\-\-judge\_modelOpenRouter/google/gemma\-3\-4b\-it\-\-n\_instructions10run1\.Pull tasks2\.Generate completions3\.Generate judge annotations4\.Store metadata and results

Figure 1:Anatomy of aJudgeArenarun\. A single CLI invocation \(left\) selects the task, the two models and the judge as orthogonal flags; the tool pulls the task, generates or reuses cached completions, collects judge annotations, and writes the shown result \(right\) together with a versioned metadata bundle \(package version, results JSON, CLI arguments, run date\) for post\-hoc reproducibility\.
### 3\.2Inference Backends

Model inference is factored behind a uniform backend abstraction that routes each evaluated model and each judge to the appropriate runtime\. Backends are addressed through a unified\{Backend\}/\{ModelPath\}naming convention; local backends run inference in\-process while hosted ones are dispatched to the corresponding provider APIs via LangChain\. Coverage spans vLLM \(local GPU\), llama\.cpp \(CPU and Apple Silicon\), and hosted providers such as OpenRouter or Together AI\.JudgeArenahandles both chat\-templated instruction models and plain\-completion pretrained base models\. Optional reasoning\-token budgets are honored for thinking judges where the backend supports them, and per\-request token usage is recorded across all backends\. Engine\-level options such as tensor parallelism or quantization can be configured independently for the battle models and for the judge, so that a large judge can run at a different scale or precision than the evaluated models on the same GPU\.

### 3\.3Supported Benchmarks and Arenas

Every benchmark follows a common instruction interface\. This shared schema lets the rest of the pipeline remain agnostic to the underlying benchmark, so that adding a new benchmark only requires expressing it in the common interface rather than extending a new pipeline\.

#### Benchmarks\.

The current release supports four pairwise instruction benchmarks with fixed baselines\.222The choice of baselines or anchors is by itself subject to research studies\(Don\-Yehiyaet al\.,[2026](https://arxiv.org/html/2608.02620#bib.bib26)\)—which we aimJudgeArenato foster\.AlpacaEval and Arena\-Hard exposes their standard instructions together with pre\-shipped completions for common baselines such asgpt\-4\-1106\-preview\. Arena\-Hard is supported in both its v0\.1 and v2\.0 releases\. m\-Arena\-Hard brings multilingual coverage across 23 languages in two releases: v0\.1 \(500 prompts\)\(Danget al\.,[2024](https://arxiv.org/html/2608.02620#bib.bib61)\)and v2\.0 \(498 prompts\)\(Khairiet al\.,[2025](https://arxiv.org/html/2608.02620#bib.bib11)\)\. MT\-Bench follows the FastChat protocol of 80 multi\-turn questions across eight categories\.

JudgeArenapreserves the original instructions of each benchmark\. In the runs reported here, AlpacaEval, Arena\-Hard, m\-Arena\-Hard, and MT\-Bench all use the tuneddefaultpreset \([Section˜3\.4](https://arxiv.org/html/2608.02620#S3.SS4)\), which asks the judge for two numerical scores rather than a verdict label, so that the judge remains the sole axis of variation across these benchmarks\. As a consequence, absolute scores produced byJudgeArenaare not directly comparable to each benchmark’s published leaderboard, but every run is fully reproducible from the shipped metadata\. For completeness,[Appendix˜A](https://arxiv.org/html/2608.02620#A1)summarizes the original baselines, default judges and judge protocols used by the original benchmarks\.

#### Arenas available\.

In addition to the benchmarking available described in[Table˜1](https://arxiv.org/html/2608.02620#S1.T1),JudgeArenaincludes also several dataset of battles from crowd source platforms which can be used for meta\-analysis or to evaluate a new model\. We support ComparIATermignonet al\.\([2026](https://arxiv.org/html/2608.02620#bib.bib62)\)as well as the three splits of LMArena splits which were previously releasedChianget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib42)\)and show some of their statistics in[Fig\.˜2](https://arxiv.org/html/2608.02620#S3.F2)\. The model intersection is shown in the appendix, in particular a single model appears in all arenas which we use as a common anchor to compute Elo\-ratings so that results are comparable\. The datasets have different trade\-offs, LMArena\-100K has weaker models than LMArena\-140K which has only recent high\-performing models whereas ComparIA and LMArena\-55K covers all the quality spectrum\.

![Refer to caption](https://arxiv.org/html/2608.02620v1/x1.png)

![Refer to caption](https://arxiv.org/html/2608.02620v1/x2.png)

Figure 2:Elo\-ratings distribution of models on different arenas \(left\) and number of battles per language \(right\)\.The number of battles per languages is also shown in[Fig\.˜2](https://arxiv.org/html/2608.02620#S3.F2)right where we show all languages having at least 100 battles\. The number of battles per language varies widely for different arenas\. ComparIA has a much larger proportion of French and multilingual data than LMArena splits which all have at least half of their data as English\. The combination of all those datasets is important to cover sufficiently many languages with 100 instructions which is the minimum number we set to perform meta\-evaluations\.

### 3\.4Tuned Judge Configurations

JudgeArenaoperationalizes the judge\-design study ofSalinaset al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib65)\), in which multi\-objective multi\-fidelity optimization was used to search over the main axes of a pairwise LLM judge: the choice of judge base model, the prompt template, the output format and scoring scheme, the decoding temperature, the use of chain\-of\-thought \(CoT\) explanations and the use of position swaps\. Rather than re\-running this search for every new benchmark,JudgeArenaships the resulting design as a default pipeline and exposes the main operational choices, including the judge model, prompt preset, explanation prompting, swap mode, and judge sampling and backend options, so that users can reuse the tuned configuration out of the box or run controlled ablations over supported axes\.

Thedefaultpreset implements the tuned protocol: the judge receives a short bias\-mitigating system prompt together with a user template that asks for a structured response with two numerical scoressA,sB∈\[0,10\]s\_\{A\},s\_\{B\}\\in\[0,10\], one per answer\. These scoressA,sBs\_\{A\},s\_\{B\}are then converted into a winning probability

ℙ​\[A≺B\]=σ​\(β​\(sB−sA\)\)=1/\(1\+e−β​\(sB−sA\)\)\\mathbb\{P\}\[A\\prec B\]=\\sigma\(\\beta\(s\_\{B\}\-s\_\{A\}\)\)=1/\\left\(1\+e^\{\-\\beta\(s\_\{B\}\-s\_\{A\}\)\}\\right\)\(1\)with a sharpness factorβ=0\.3\\beta=0\.3\. Equal scores map to a preference probability of0\.50\.5, while larger score differences smoothly shift the preference toward the higher\-scored answer\.

The remaining design axes are controlled through orthogonal configuration options\. Chain\-of\-thought to solicit explanation can be toggled on or off, swapping the user template for a variant that requests an explanation before the scores\. Swap evaluation runs bothAA–BBandBB–AAorderings so that each instruction contributes two annotations rather than one, preserving per\-swap information for downstream analysis; a cheaper single\-pass mode is retained for large\-scale sweeps\. Sampling parameters and backend\-specific options are configured independently for the judge, so that the judge can run at a different scale or precision than the battle models\. Reasoning\-trace management can cap the number of reasoning tokens that thinking models emit for supported backends, and visible reasoning traces can optionally be stripped from battle completions before they are sent to the judge\.

### 3\.5Reproducibility Infrastructure

Every evaluation run is treated as a self\-describing artifact, following the reproducibility lessons distilled byBidermanet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib64)\)\. Metadata is stored containing key information such as the entry point and CLI command that produced the run, the complete run configuration \(judge prompt preset, position\-swap setting, truncation budgets\) or the software version of main dependencies\. The exact schema is detailed in[Appendix˜E](https://arxiv.org/html/2608.02620#A5)\.

In addition, a run writes a small bundle of files into a single directory whose name encodes the dataset, the two battled model sides, the judge, the position\-swap setting and a timestamp: a snapshot of the full run configuration, a per\-battle annotation table, an aggregate results file and a versioned metadata descriptor\. This bundle is the unit of record for each evaluation run, so that any annotation produced byJudgeArenacan be re\-analyzed from its artifacts alone\.

## 4Experiments

We first meta\-evaluate judge quality in English and multilingual settings, then show how Elo ratings can be estimated from arenas, and finally present a case study on open\-weight models across all benchmarks in the library\. All scripts, completions, and annotations will be released to facilitate future research\. Throughout, we report 95% confidence intervals via the 2\.5th and 97\.5th percentiles over 1,000 bootstrap resamples: over instructions for instruction\-level metrics \(Cohen\-Kappa\) and over models for model\-level metrics \(Spearman and Elo ratings\)\.

### 4\.1Meta\-evaluations

#### English meta\-evaluation\.

In[Table˜2](https://arxiv.org/html/2608.02620#S4.T2), we evaluate several judges available inJudgeArena\. We consider prompted judges with both open\-weights options and close\-weights options with our tuned configuration \(top\), Alpaca\-Eval and Arena\-Hard prompted judges \(middle\) and the best fine\-tuned judge reported fromTanet al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib30)\)\. We report Cohen\-Kappa, Spearman correlation and cost per annotation when selecting 100 random English instructions of LMArena\-140K for the top 20 models in battle counts\.

Cohen\-Kappa measures the match between two annotators \(higher is better\) and is normalized so that values close to 0 indicates random\-performance\. The values of Cohen\-Kappa are competitive and outperform the Cohen\-Kappa obtained by ourselves \(0\.2\) when re\-annotating manually 200 instructions in French and English, they are far from the optimal value of1\.01\.0due to the difficulty of the annotation and the noise and subjectivity inherent to the process\. Looking at the Qwen3\.5 series, we can see that this metric monotonically increases with the model size\.

On the other hand, Spearman rank correlation metrics are noisy and provide large confidence\-intervals and much lower signal\-to\-noise ratio compared to Cohen\-Kappa instruction level metrics as noted inSalinaset al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib65)\)\. The Spearman rank correlation is also typically lower than observed inLiet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib13),[2024](https://arxiv.org/html/2608.02620#bib.bib2)\)as the models selected are much closer and stronger as shown in[Fig\.˜2](https://arxiv.org/html/2608.02620#S3.F2)which is emphasized by the fact that we picked the top 20 models\.

Importantly, we see that the open\-weights configurations offered inJudgeArenaare competitive for both accuracy and cost as they were tuned to minimize both objectives\. In particular, the tuned prompt combined with gemma\-4\-31b\-it provides the best Cohen\-Kappa agreement while being the cheapest option\.

Table 2:Meta\-evaluations for Cohen\-Kappa, Spearman rank correlation and Cost \(expressed in dollar\-cost per 1k annotations\) when evaluating English instructions\. We report results with respectively the default configuration with open\-weights models and proprietary models \(top\), the configuration from Arena\-Hard and Alpaca\-Eval \(medium\) and fine\-tuned judges \(bottom\)\.JudgeCohen\-Kappa↑\\uparrowSpearman↑\\uparrowCost↓\\downarrowqwen3\.5\-122b\-a10b0\.150\.120\.180\.15\_\{0\.12\}^\{0\.18\}0\.690\.290\.920\.69\_\{0\.29\}^\{0\.92\}4\.16qwen3\.5\-27b0\.140\.110\.170\.14\_\{0\.11\}^\{0\.17\}0\.670\.280\.870\.67\_\{0\.28\}^\{0\.87\}4\.07qwen3\.5\-9b0\.120\.090\.150\.12\_\{0\.09\}^\{0\.15\}0\.750\.410\.930\.75\_\{0\.41\}^\{0\.93\}0\.43deepseek\-v3\.20\.130\.100\.160\.13\_\{0\.10\}^\{0\.16\}0\.780\.470\.92\\text\{\{0\.78\}\}\_\{0\.47\}^\{0\.92\}0\.44gemma\-4\-31b\-it0\.170\.140\.20\\text\{\{0\.17\}\}\_\{0\.14\}^\{0\.20\}0\.720\.360\.910\.72\_\{0\.36\}^\{0\.91\}0\.25gpt\-5\.4\-nano0\.110\.080\.140\.11\_\{0\.08\}^\{0\.14\}0\.750\.390\.910\.75\_\{0\.39\}^\{0\.91\}0\.36gemini\-3\.1\-flash0\.170\.130\.20\\text\{\{\{0\.17\}\}\}\_\{0\.13\}^\{0\.20\}0\.730\.370\.910\.73\_\{0\.37\}^\{0\.91\}0\.47Arena\-Hard \+ gemini\-3\.1\-flash0\.160\.120\.190\.16\_\{0\.12\}^\{0\.19\}0\.750\.410\.930\.75\_\{0\.41\}^\{0\.93\}1\.14Alpaca\-Eval \+ gpt\-5\.4\-nano0\.110\.080\.140\.11\_\{0\.08\}^\{0\.14\}0\.730\.370\.910\.73\_\{0\.37\}^\{0\.91\}0\.38Skywork\-Critic\-70B0\.080\.050\.110\.08\_\{0\.05\}^\{0\.11\}0\.480\.030\.800\.48\_\{0\.03\}^\{0\.80\}0\.74
#### Multilingual meta\-evaluation or assessing which language can be judged\.

To use an LLM\-judge, we need to make sure that its performance is good enough for the targeted use\-case\. While general instructions or coding in English is well investigated, other use\-cases such as multilingual instructions typically follow much less verification, which can make their results unreliable for model development and comparisons\. Suboptimal choices can lead to misleading conclusions about model capabilities, as reported inDanget al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib61)\), where results for GPT4\-o vs GPT4\-o\-mini diverged notably\. As there are large divergences in support across languages for different models, it becomes even more challenging to select a suitable judge for all\.

Any LLM judge selected to perform this task should at least clearly outperform random predictionsKreutzeret al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib63)\)\. Before evaluating and reporting results with LLM\-judges on any language, we therefore evaluate whether LLM\-judges outperform random predictions on all languages with more than 100 instructions in all arenas supported\. In[Fig\.˜3](https://arxiv.org/html/2608.02620#S4.F3), we evaluate Cohen\-Kappa agreement\-rate for multiple languages for three strong open\-weight judge configurations\. Interestingly, gemma\-4\-31b outperforms qwen\-3\.5\-27b for most languages\. For 3 languages, the Cohen\-Kappa confidence interval overlaps zero, not discarding random\-chance performance, we therefore exclude all such languages by default\. We note that having higher agreement scores between LLM and human annotators for some languages does not mean that those languages are inherently easier to judge as there are other confounding effects such as annotators of a certain language looking at answer into more details or asking easier questions\. However, a high agreement rate indicates that LLM judges are reliable on the given distribution\.

![Refer to caption](https://arxiv.org/html/2608.02620v1/x3.png)Figure 3:Cohen\-Kappa per language for three different open\-weight judges\. We use up to 200 random battles per language and select only languages with at least 100 battles\. The number of battles is indicated in parentheses for languages with fewer than 200 battles\.

### 4\.2Elo Approximation via Hybrid Human and LLM Annotations

Evaluating against a single fixed baseline can cause multiple issues including non transitivityXuet al\.\([2025](https://arxiv.org/html/2608.02620#bib.bib68)\)of results or self\-preferencePanicksseryet al\.\([2024](https://arxiv.org/html/2608.02620#bib.bib37)\)\. To avoid this issue,JudgeArenaproposes to evaluate Elo ratings from battles previously made available in an crowd\-source arena\.

Elo ratings are computed given a set of battles in crowd\-sourced arenas which we denote as\(pi,oAi,oBi,ΦHuman​\(pi,oAi,oBi\)\)i=1N\(p^\{i\},o^\{i\}\_\{A\},o^\{i\}\_\{B\},\\Phi^\{\\text\{Human\}\}\(p^\{i\},o^\{i\}\_\{A\},o^\{i\}\_\{B\}\)\)\_\{i=1\}^\{N\}whereoAi,oBio^\{i\}\_\{A\},o^\{i\}\_\{B\}denote the completions of two modelsAAandBBgiven the promptpip^\{i\}andΦHuman​\(pi,oAi,oBi\)∈\{−1,1\}\\Phi^\{\\text\{Human\}\}\(p^\{i\},o^\{i\}\_\{A\},o^\{i\}\_\{B\}\)\\in\\\{\-1,1\\\}denotes the completion chosen, with\+1\+1indicating thatBBis preferred overAA\. For a given set of modelsℳ\\mathcal\{M\}\{\}, Elo ratingsβ∈ℝ\|ℳ\|\\beta\\in\\mathbb\{R\}^\{\|\\mathcal\{M\}\|\}are estimated via the Bradley–Terry \(BT\) modelBradley and Terry \([1952](https://arxiv.org/html/2608.02620#bib.bib71)\)such that the marginal probability that modelAAis preferred over modelBBon a random prompt is given by:

ℙ​\[ΦHuman​\(p,oA,oB\)=1\]=σ​\(βB−βA\)\.\\mathbb\{P\}\\\!\\left\[\\Phi^\{\\text\{Human\}\}\(p,o\_\{A\},o\_\{B\}\)=1\\right\]=\\sigma\(\\beta\_\{B\}\-\\beta\_\{A\}\)\.\(2\)Given the set of battles, the strengthsβ\\betaare estimated by maximising the log\-likelihood of the observed preferences on[Eq\.˜2](https://arxiv.org/html/2608.02620#S4.E2), with one model strength fixed to a constant such as10001000\. Elo scores are then obtained asElo​\(m\)=βm⋅400/log⁡10\\text\{Elo\}\(m\{\}\)=\\beta\_\{m\{\}\}\\cdot 400/\\\!\\log 10\.

#### Estimating Elo ratings\.

To approximate Elo ratings without human annotators, we annotate 2 000 battles from LMArena\-140K, 100 instructions from 20 models, with an LLM judge instead of human annotators\. That is, we replaceΦHuman\\Phi^\{\\text\{Human\}\}byΦLLM\\Phi^\{\\text\{LLM\}\}in[Eq\.˜2](https://arxiv.org/html/2608.02620#S4.E2)\. In[Fig\.˜4](https://arxiv.org/html/2608.02620#S4.F4), we show a scatter plot of Elo ratings estimated with Human and LLM annotators for each of the 20 models for four different judges\. We report the average Mean Absolute Error \(MAE\) over models between the Elo ratings estimated by human and LLM judge\. For all judges, we usegemma\-4\-31b\-itas the base model to avoid confounding factors\. For Alpaca\-Eval and Arena\-Hard, the judge produces only a hard verdict, so we fit BT with hard labels in\{−1,1\}\\\{\-1,1\\\}for the losing/winning side\.

Judges inJudgeArenainstead emit a continuous probability \([Eq\.˜1](https://arxiv.org/html/2608.02620#S3.E1)\) which captures judge confidence: model A is more likely to beat B when scored1010versus11than when scored1010versus99\. We leverage this confidence by replacing the hard target with the predicted probability, so the BT cross\-entropy is computed against continuous labels rather than discrete ones, related to richer preference targets used in reward modeling\(Liuet al\.,[2024a](https://arxiv.org/html/2608.02620#bib.bib72),[b](https://arxiv.org/html/2608.02620#bib.bib74); Afsharradet al\.,[2026](https://arxiv.org/html/2608.02620#bib.bib73)\)\. This up\-weights battles with larger score difference \(lower uncertainty\) and down\-weights more uncertain battles\. We also report results when using only hard\-labels \("Ours w/o prob\."\) to ablate the benefit from modeling the uncertainty obtained from LLM judgements\. Accounting for uncertainty significantly lowers the MAE error\. The default judge inJudgeArenaprovides a faithful approximation as the MAE error on Elo ratings is5454\{\}on average per model\. We believe this approximation can be extremely useful for model developers who cannot easily access live arenas for cost reasons\.

![Refer to caption](https://arxiv.org/html/2608.02620v1/x4.png)Figure 4:Elo ratings estimated by LLM vs\. human annotators for four judges\.

## 5Case study

### 5\.1Evaluating Open Models Across Standard Benchmarks

Having determined the quality of different judges and which languages can be judged, we conduct a case study benchmarking several open models on popular benchmarks available inJudgeArena\. We evaluate AlpacaEval, Arena\-Hard \(v0\.1 and v2\.0\), MT\-Bench and m\-Arena\-Hard v2\.0 \(Arabic, Polish, Ukrainian, Chinese\) for the following open models: Olmo\-3\-7B, Olmo\-3\-7B\-Think, Tiny\-Aya\-Global, Apertus\-8B, SmolLM3\-3B, EuroLLM\-1\.7B and 9B, and Qwen3\.5\-9B, considering fully open models, except for Qwen3\.5 and Tiny\-Aya\-Global which are only open\-weights\. The judge is set to Gemma\-4\-31B\-IT using the tuneddefaultscore\-style preset across all benchmarks\. We additionally estimate each model’s LMArena Elo by combining 1,000 Gemma\-4\-judged target\-model battles from a shared seeded random LMArena\-140K sample\.

Table 3:Performance of eight open\-weight models onJudgeArenabenchmarks\.Results are reported in[Table˜3](https://arxiv.org/html/2608.02620#S5.T3)\. Qwen3\.5\-9B is the clear outlier in this case study, ranking first on every benchmark in the table as well as on LMArena Elo, which yields an average rank of 1\.000\. The next\-best models are Olmo\-3\-7B\-Think \(2\.222\) and Olmo\-3\-7B \(3\.056\), and the gap to the rest of the table is much larger than the per\-cell bootstrap intervals, so the ordering of the strongest models is stable at the instruction\-sampling level\. Arena\-Hard v2\.0 is the most difficult English benchmark for every evaluated model, while the translated m\-Arena\-Hard cells show a clear drop relative to AlpacaEval, Arena\-Hard v0\.1, and MT\-Bench\. The EuroLLM results on long\-prompt cells should be interpreted with additional caution because several inputs were truncated to fit the models’ 4096\-token positional cap, as marked by†; the corresponding truncation statistics are reported in[Appendix˜C](https://arxiv.org/html/2608.02620#A3)\.

In[Fig\.˜5](https://arxiv.org/html/2608.02620#S5.F5), we study whether benchmarks show monotonic performance when scaling model parameters\. We evaluate the Qwen3\.5 series \(0\.8B–35B\-A3B\) with Gemma\-4\-31B\-IT as judge; the 27B and 35B\-A3B points use released FP8 variants\. Except for MT\-Bench and m\-AH, all tasks show non\-overlapping confidence intervals with monotonically improving performance from 0\.8B to 27B, providing a sanity check on the discriminative power of the benchmarks inJudgeArena\.

![Refer to caption](https://arxiv.org/html/2608.02620v1/x5.png)Figure 5:Qwen3\.5 performance scales monotonically with model size for most benchmarks\.

## 6Limitations

#### Opening the blackbox\.

While we allow the user to change many aspects about the judge— the model, the temperature, the prompt—this could also lead to an undesirable situation where the judge configuration is tuned to make a particular method shine or could lead to user confusion\. We believe this is alleviated by shipping a default judge configuration for every benchmark and also by reporting all metadata corresponding to a run so that a user can share the JSON corresponding to their result so that the community can reproduce their result\.

#### Cross\-version comparability\.

It should be clear that numbers between different versions will most likely not be comparable\. This is expected and similar to Eval\-Harness where tasks, data from evaluations, code for generation are changed continuously, meaning any benchmark needs to be run on all methods and numbers can never be pulled from a table \(unless the exact software versions are matched\)\.JudgeArenabehaves similarly: whenever reporting a comparison such as[Table˜3](https://arxiv.org/html/2608.02620#S5.T3), the JSON of results should be shared to enable reproducibility\.

#### Limit of reproducibility\.

While we log all metadata such as software version, command\-line arguments, or dataset hashes, and seed all stochastic components, some aspects are fundamentally irreproducible—among them CUDA non\-deterministic execution or the use of hosted closed models whose behavior can change over timeChenet al\.\([2023](https://arxiv.org/html/2608.02620#bib.bib3)\)\. We acknowledge this limitation but still believe that having the exact code and software versions is a step up when sharing results\.

#### Contamination\.

All benchmarks available except for ComparIA are static and can be contaminated\. We do not address this, as it remains an active area of research\. We note that ComparIA could be used in the future to do evaluations with forward\-looking data as the dataset is continuously updated\.

## 7Conclusion

We presentedJudgeArena, which lets users perform LLM\-judge evaluations on multiple benchmarks including Alpaca\-Eval, Arena\-Hard, m\-Arena\-Hard, MT\-Bench, ComparIA and LMArena variants\. In addition to benchmarking, the library supports performing meta\-evaluations and includes anopen\-weightdefault judge that was tuned for accuracy and cost and will be continuously updated as newer models are released\. The library supports the evaluation across 33 languages which have been selected after careful meta\-evaluations\. It also supports estimating Elo\-ratings with small errors, which we believe will be useful for model developers who want to avoid the issue of benchmarks with fixed baselines\. We hope the library will help to study, consolidate and improve LLM\-judge benchmarks\.

We believe the work has positive societal impact as it contributes to more transparent and reproducible LLM evaluation practices together with thorough meta\-evaluation\. The multilingual scope helps ensure evaluation quality is not limited to English\. We encourage users to report full metadata alongside results to enable community\-level scrutiny of benchmark conclusions\.

In future work, we will continue improving the accuracy of the default judge we provide\. We are planning to have an automatic leaderboard to track the progress in Arenas to report simulated Elo\-ratings for both English and multilingual models with publicly available completions and judgements\. Finally, we are working on including rubrics which provide additional decomposition of model performance which we hope can be useful for practitioners\.

## Acknowledgments and Disclosure of Funding

This research was partially supported by the following sources: EC under the grant No\. 101195233 \(OpenEuroLLM\) and 101198470 \(LLMs4EU\)\. Views and opinions expressed are however those of the author\(s\) only and do not necessarily reflect those of the European Union\. Neither the European Union nor the granting authority can be held responsible for them\.

![[Uncaptioned image]](https://arxiv.org/html/2608.02620v1/figures/openeurollm-logo.png)

## References

- \[1\]\(2026\)Beyond binary preferences: a principled framework for reward modeling with ordinal feedback\.arXiv preprint arXiv:2603\.02232\.External Links:[Link](https://arxiv.org/abs/2603.02232)Cited by:[§4\.2](https://arxiv.org/html/2608.02620#S4.SS2.SSS0.Px1.p2.5)\.
- \[2\]S\. Biderman, H\. Schoelkopf, L\. Sutawika, L\. Gao, J\. Tow, B\. Abbasi, A\. F\. Aji, P\. S\. Ammanamanchi, S\. Black, J\. Clive,et al\.\(2024\)Lessons from the trenches on reproducible evaluation of language models\.arXiv preprint arXiv:2405\.14782\.Cited by:[§3\.5](https://arxiv.org/html/2608.02620#S3.SS5.p1.1)\.
- \[3\]R\. A\. Bradley and M\. E\. Terry\(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.External Links:ISSN 00063444, 14643510,[Link](http://www.jstor.org/stable/2334029)Cited by:[§4\.2](https://arxiv.org/html/2608.02620#S4.SS2.p2.13)\.
- \[4\]L\. Chen, M\. Zaharia, and J\. Zou\(2023\)How is chatgpt’s behavior changing over time?\.arXiv preprint arXiv:2307\.09009\.Cited by:[§6](https://arxiv.org/html/2608.02620#S6.SS0.SSS0.Px3.p1.1)\.
- \[5\]W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, H\. Zhang, B\. Zhu, M\. Jordan, J\. E\. Gonzalez, and I\. Stoica\(2024\)Chatbot arena: an open platform for evaluating llms by human preference\.External Links:2403\.04132,[Link](https://arxiv.org/abs/2403.04132)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p1.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§3\.3](https://arxiv.org/html/2608.02620#S3.SS3.SSS0.Px2.p1.1)\.
- \[6\]J\. Dang, S\. Singh, D\. D’souza, A\. Ahmadian, A\. Salamanca, M\. Smith, A\. Peppin, S\. Hong, M\. Govindassamy, T\. Zhao, S\. Kublik, M\. Amer, V\. Aryabumi, J\. A\. Campos, Y\. Tan, T\. Kocmi, F\. Strub, N\. Grinsztajn, Y\. Flet\-Berliac, A\. Locatelli, H\. Lin, D\. Talupuru, B\. Venkitesh, D\. Cairuz, B\. Yang, T\. Chung, W\. Ko, S\. S\. Shi, A\. Shukayev, S\. Bae, A\. Piktus, R\. Castagné, F\. Cruz\-Salinas, E\. Kim, L\. Crawhall\-Stein, A\. Morisot, S\. Roy, P\. Blunsom, I\. Zhang, A\. Gomez, N\. Frosst, M\. Fadaee, B\. Ermis, A\. Üstün, and S\. Hooker\(2024\)Aya expanse: combining research breakthroughs for a new multilingual frontier\.External Links:2412\.04261,[Link](https://arxiv.org/abs/2412.04261)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p2.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§3\.3](https://arxiv.org/html/2608.02620#S3.SS3.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2608.02620#S4.SS1.SSS0.Px2.p1.1)\.
- \[7\]S\. Don\-Yehiya, A\. Yehudai, L\. Choshen, and O\. Abend\(2026\)Mediocrity is the key for llm as a judge anchor selection\.arXiv preprint arXiv:2603\.16848\.Cited by:[footnote 2](https://arxiv.org/html/2608.02620#footnote2)\.
- \[8\]Y\. Dubois, B\. Galambosi, P\. Liang, and T\. B\. Hashimoto\(2024\)Length\-controlled alpacaeval: a simple way to debias automatic evaluators\.arXiv preprint arXiv:2404\.04475\.Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p2.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2)\.
- \[9\]C\. Fernando, D\. Banarse, H\. Michalewski, S\. Osindero, and T\. Rocktäschel\(2023\)Promptbreeder: self\-referential self\-improvement via prompt evolution\.arXiv preprint arXiv:2309\.16797\.Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.
- \[10\]C\. Fourrier, N\. Habib, A\. Lozovskaya, K\. Szafer, and T\. Wolf\(2024\)Open llm leaderboard v2\.Hugging Face\.External Links:[Link](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px2.p2.1)\.
- \[11\]X\. Fu and W\. Liu\(2025\-11\)How reliable is multilingual LLM\-as\-a\-judge?\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 11040–11053\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.587/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.587),ISBN 979\-8\-89176\-335\-7Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.
- \[12\]L\. Gao, J\. Tow, B\. Abbasi, S\. Biderman, S\. Black, A\. DiPofi, C\. Foster, L\. Golding, J\. Hsu, A\. Le Noac’h, H\. Li, K\. McDonell, N\. Muennighoff, C\. Ociepa, J\. Phang, L\. Reynolds, H\. Schoelkopf, A\. Skowron, L\. Sutawika, E\. Tang, A\. Thite, B\. Wang, K\. Wang, and A\. Zou\(2024\-07\)The language model evaluation harness\.Zenodo\.External Links:[Document](https://dx.doi.org/10.5281/zenodo.12608602),[Link](https://zenodo.org/records/12608602)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p3.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px2.p2.1),[§3\.1](https://arxiv.org/html/2608.02620#S3.SS1.p1.1)\.
- \[13\]N\. Habib, C\. Fourrier, H\. Kydlíček, T\. Wolf, and L\. Tunstall\(2023\)LightEval: a lightweight framework for llm evaluation\.External Links:[Link](https://github.com/huggingface/lighteval)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p3.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px2.p2.1)\.
- \[14\]A\. Khairi, D\. D’souza, Y\. Shen, J\. Kreutzer, and S\. Hooker\(2025\-11\)When life gives you samples: the benefits of scaling up inference compute for multilingual LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 27559–27583\.External Links:[Link](https://aclanthology.org/2025.emnlp-main.1402/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.1402),ISBN 979\-8\-89176\-332\-6Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§3\.3](https://arxiv.org/html/2608.02620#S3.SS3.SSS0.Px1.p1.1)\.
- \[15\]J\. Kreutzer, E\. Briakou, S\. Agrawal, M\. Fadaee, and K\. Tom\(2025\)Déjà vu: multilingual llm evaluation through the lens of machine translation evaluation\.External Links:2504\.11829,[Link](https://arxiv.org/abs/2504.11829)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.02620#S4.SS1.SSS0.Px2.p2.1)\.
- \[16\]P\. Lai, Z\. Ou, Y\. Wang, L\. Wang, J\. Yang, Y\. Chen, and G\. Chen\(2026\)BiasScope: towards automated detection of bias in LLM\-as\-a\-judge evaluation\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QGOw6AU8Lp)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.
- \[17\]M\. T\. R\. Laskar, S\. Alqahtani, M\. S\. Bari, M\. Rahman, M\. A\. M\. Khan, H\. Khan, I\. Jahan, A\. Bhuiyan, C\. W\. Tan, M\. R\. Parvez, E\. Hoque, S\. Joty, and J\. X\. Huang\(2024\-11\)A systematic survey and critical review on evaluating large language models: challenges, limitations, and recommendations\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 13785–13816\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.764/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.764)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p3.1)\.
- \[18\]T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica\(2024\)From crowdsourced data to high\-quality benchmarks: arena\-hard and benchbuilder pipeline\.arXiv preprint arXiv:2406\.11939\.Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p2.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.02620#S4.SS1.SSS0.Px1.p3.1)\.
- \[19\]X\. Li, T\. Zhang, Y\. Dubois, R\. Taori, I\. Gulrajani, C\. Guestrin, P\. Liang, and T\. B\. Hashimoto\(2023\)Alpacaeval: an automatic evaluator of instruction\-following models\.Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p2.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.02620#S4.SS1.SSS0.Px1.p3.1)\.
- \[20\]J\. Liu, D\. Ge, and R\. Zhu\(2024\)Reward learning from preference with ties\.arXiv preprint arXiv:2410\.05328\.External Links:[Link](https://arxiv.org/abs/2410.05328)Cited by:[§4\.2](https://arxiv.org/html/2608.02620#S4.SS2.SSS0.Px1.p2.5)\.
- \[21\]S\. Liu, Y\. Pan, G\. Chen, and X\. Li\(2024\)Reward modeling with ordinal feedback: wisdom of the crowd\.arXiv preprint arXiv:2411\.12843\.Cited by:[§4\.2](https://arxiv.org/html/2608.02620#S4.SS2.SSS0.Px1.p2.5)\.
- \[22\]J\. Ni, F\. Xue, X\. Yue, Y\. Deng, M\. Shah, K\. Jain, G\. Neubig, and Y\. You\(2024\)MixEval: deriving wisdom of the crowd from llm benchmark mixtures\.External Links:2406\.06565,[Link](https://arxiv.org/abs/2406.06565)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.
- \[23\]A\. Panickssery, S\. R\. Bowman, and S\. Feng\(2024\)LLM evaluators recognize and favor their own generations\.External Links:2404\.13076,[Link](https://arxiv.org/abs/2404.13076)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p3.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2608.02620#S4.SS2.p1.1)\.
- \[24\]N\. Raoof, E\. K\. Guha, R\. Marten, J\. Mercat, E\. Frankel, S\. Keh, H\. Bansal, G\. Smyrnis, M\. Nezhurina, T\. Vu, Z\. R\. Sprague, M\. A\. Merrill, L\. Chen, C\. Choi, Z\. Khan, S\. Grover, B\. Feuer, A\. Suvarna, S\. Su, W\. Zhao, K\. Sharma, C\. C\. Ji, K\. Arora, J\. Li, A\. Gokaslan, S\. M\. Pratt, N\. Muennighoff, J\. Saad\-Falcon, J\. Yang, A\. Aali, S\. Pimpalgaonkar, A\. Albalak, A\. Dave, H\. Pouransari, G\. Durrett, S\. Oh, T\. Hashimoto, V\. Shankar, Y\. Choi, M\. Bansal, C\. Hegde, R\. Heckel, J\. Jitsev, M\. Sathiamoorthy, A\. Dimakis, and L\. Schmidt\(2025\-06\)Evalchemy: automatic evals for llms\.External Links:[Link](https://github.com/mlfoundations/evalchemy)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px2.p2.1)\.
- \[25\]D\. Salinas, O\. Swelam, and F\. Hutter\(2025\)Tuning LLM judge design decisions for 1/1000 of the cost\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=cve4NOiyVp)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1),[§3\.4](https://arxiv.org/html/2608.02620#S3.SS4.p1.1),[§4\.1](https://arxiv.org/html/2608.02620#S4.SS1.SSS0.Px1.p3.1)\.
- \[26\]S\. Tan, S\. Zhuang, K\. Montgomery, W\. Y\. Tang, A\. Cuadron, C\. Wang, R\. Popa, and I\. Stoica\(2025\)JudgeBench: a benchmark for evaluating LLM\-based judges\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=G0dksFayVq)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2608.02620#S4.SS1.SSS0.Px1.p1.1)\.
- \[27\]L\. Termignon, S\. Zilinskas, H\. Pélissier, A\. Barrot, N\. Chesnais, and E\. Gavoty\(2026\)Compar:ia: the french government’s llm arena to collect french\-language human prompts and preference data\.External Links:2602\.06669,[Link](https://arxiv.org/abs/2602.06669)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p1.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§3\.3](https://arxiv.org/html/2608.02620#S3.SS3.SSS0.Px2.p1.1)\.
- \[28\]Y\. Wang, Z\. Yu, W\. Yao, Z\. Zeng, L\. Yang, C\. Wang, H\. Chen, C\. Jiang, R\. Xie, J\. Wang, X\. Xie, W\. Ye, S\. Zhang, and Y\. Zhang\(2024\)PandaLM: an automatic evaluation benchmark for LLM instruction tuning optimization\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5Nn2BLV7SB)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.
- \[29\]K\. Wataoka, T\. Takahashi, and R\. Ri\(2025\)Self\-preference bias in LLM\-as\-a\-judge\.External Links:[Link](https://openreview.net/forum?id=Ns8zGZ0lmM)Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p3.1)\.
- \[30\]Y\. Xu, L\. Ruis, T\. Rocktäschel, and R\. Kirk\(2025\)Investigating non\-transitivity in LLM\-as\-a\-judge\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=clJIQ4TKR0)Cited by:[§4\.2](https://arxiv.org/html/2608.02620#S4.SS2.p1.1)\.
- \[31\]Z\. Zeng, J\. Yu, T\. Gao, Y\. Meng, T\. Goyal, and D\. Chen\(2024\)Evaluating large language models at evaluating instruction following\.External Links:2310\.07641,[Link](https://arxiv.org/abs/2310.07641)Cited by:[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.
- \[32\]L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in Neural Information Processing Systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2608.02620#S1.p1.1),[§1](https://arxiv.org/html/2608.02620#S1.p2.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px1.p1.2),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.02620#S2.SS0.SSS0.Px3.p1.1)\.

## Appendix ABenchmark Reproduction Fidelity

We show a summary of the differences to the original benchmarks in[Table˜4](https://arxiv.org/html/2608.02620#A1.T4)\.

Table 4:Reproduction fidelity of the benchmarks shipped withJudgeArena\. Instructions are loaded as released with each benchmark\. The “original default judge” column reports the judge model used by the upstream benchmark or paper\. The runs reported here useJudgeArena’s tuneddefaultpreset \([Section˜3\.4](https://arxiv.org/html/2608.02620#S3.SS4)\), which scores both answers on\[0,10\]\[0,10\]and aggregates them via a softmax withβ=0\.3\\beta=0\.3with a selected judge\. Absolute numbers are therefore not directly comparable to each benchmark’s published leaderboard, while every run remains reproducible from the shipped metadata\.
## Appendix BInference and Compute Details

All inference except for the case study battle models in[Section˜5\.1](https://arxiv.org/html/2608.02620#S5.SS1)and Skywork\-Critic\-Llama\-3\.1\-70B in[Section˜4\.1](https://arxiv.org/html/2608.02620#S4.SS1)was obtained through OpenRouter\. For the case study battle models and Skywork\-Critic\-Llama\-3\.1\-70B in meta\-evaluations, model completions and judgements were generated on an L40 GPU \(48 GB VRAM\) using vLLM, with the judge \(Gemma\-4\-31B\-IT\) served via OpenRouter\.

In the meta\-evaluations results[Table˜2](https://arxiv.org/html/2608.02620#S4.T2)cost estimates are derived from OpenRouter pricing\. For Skywork\-Critic\-Llama\-3\.1\-70B which is not available on OpenRouter, costs are estimated using the closest equivalent in this case using the Llama\-3\.1\-70B\-Instruct price\.

## Appendix CCase\-study Run Configuration and Truncation Statistics

This appendix records the configuration and observed truncation statistics for the Gemma\-4\-31B\-IT case study in[Section˜5\.1](https://arxiv.org/html/2608.02620#S5.SS1)\.

Table 5:Run configuration for the Gemma\-4\-31B\-IT case study\. Judge\-side rows are Phase B settings; battle\-model rows are Phase A cache settings that produced the completions later judged in Phase B\. TheStatistic columnfield points to the corresponding observed\-statistics columns in[Tables˜6](https://arxiv.org/html/2608.02620#A3.T6)and[7](https://arxiv.org/html/2608.02620#A3.T7)that each cap can influence\.SettingValueStatistic columnShort descriptionJudge\-side configuration \(Phase B\)JUDGE\_MODELOpenRouter/google/gemma\-4\-31b\-itall judge\-side columnsOpenRouter Gemma\-4 judge route\.JUDGE\_PROMPT\_PRESET"default"leaderboard win\-ratesScore\-stylescore\_A/score\_Bparser\.SWAP\_MODE"fixed"judged battlesOne judge call per battle; no A/B swap\.PROVIDE\_EXPLANATIONFalsejudge output cap hitsScore lines only, no rationale\.TRUNCATE\_JUDGE\_INPUT\_CHARS200000judge input char\-cap hitsJudge\-prompt character cap for the unified score\-style judge path\.STRIP\_THINKING\_BEFORE\_JUDGINGTruethinking traces strippedRemoves candidate<think\>blocks before judging\.MAX\_JUDGE\_MODEL\_LEN200000judge prompt length\-check hitsToken\-length check for rendered judge prompt\.MAX\_OUT\_TOKENS\_JUDGE16384judge output cap hitsDefault judgemax\_tokensbudget\.MT\_BENCH\_MAX\_OUT\_TOKENS\_JUDGE16384judge output cap hitsOptional MT\-Bench\-specific override; set equal to the default budget in this run\.JUDGE\_MAX\_CONCURRENCY32noneAsync judge\-call concurrency cap\.Battle\-model generation configuration \(Phase A cache\)MAX\_MODEL\_LEN57344raw/legit output capDefault vLLM context window\.MAX\_OUT\_TOKENS\_MODELS49152raw/legit output capDefault generation output budget\.TRUNCATE\_ALL\_INPUT\_CHARS30000prompt input trunc\.Default battle\-model prompt character cap\.BATTLE\_THINKING\_TOKEN\_BUDGET32768raw/legit thinking capThinking\-token budget for reasoning models\.EUROLLM\_MAX\_MODEL\_LEN4096raw/legit output capEuroLLM architectural context window\.EUROLLM\_MAX\_OUT\_TOKENS\_MODELS4096raw/legit output capEuroLLM generation output budget\.EUROLLM\_LONG\_PROMPT\_TRUNCATION3500prompt input trunc\.EuroLLM char cap on long\-prompt datasets\.EUROLLM\_SHORT\_PROMPT\_TRUNCATION8000prompt input trunc\.EuroLLM char cap on short\-prompt datasets\.Table 6:Observed battle\-model generation truncation and cap statistics from the Phase A cache used by the Gemma\-4\-31B\-IT judging run, aggregated by evaluated model across all eight benchmark cells\.Prompt input trunc\.is the share of candidate prompts clipped byTRUNCATE\_ALL\_INPUT\_CHARSbefore local vLLM generation\.Rawcap rates count every generation that ended exactly at the configured output or thinking\-token budget\. Qualitative inspection suggested that many capped generations were degenerate or low\-signal rather than clearly useful continuations\. The corresponding cap settings are listed in[Table˜5](https://arxiv.org/html/2608.02620#A3.T5)\.Table 7:Observed judge\-side truncation, output\-cap, and content\-modification events in the 64\-cell Gemma\-4\-31B\-IT case\-study runs, aggregated by evaluated model\.Judge input char\-cap hitscounts score\-style judge prompts where the candidate A or B answer was clipped before it was sent to the judge\. Thinking\-trace stripping is reported separately because it is a deliberate content normalization step, not a cap hit\. It counts stripped candidate answer fields before judging, so MT\-Bench 2\-turn prompts can yield more stripping events than judged battles\. The zero values in the output\-cap and prompt length\-check columns document that neither the judge generation budget nor the rendered\-prompt length check was reached\.
## Appendix DLocalized m\-Arena\-Hard Prompt Ablation

This appendix reports the 32 m\-Arena\-Hard v2\.0 case\-study battles from[Section˜5\.1](https://arxiv.org/html/2608.02620#S5.SS1)when the Gemma\-4\-31B\-IT judge uses translated language\-matched score prompts for Arabic, Polish, Ukrainian, and Chinese\. The evaluated model completions and all other judge settings are unchanged from the main case\-study runs; only the judge prompt text differs\.

Table 8:Localized judge\-prompt ablation on m\-Arena\-Hard v2\.0\. The evaluated model completions and Gemma\-4\-31B\-IT judge route match[Table˜3](https://arxiv.org/html/2608.02620#S5.T3), but each language uses a translated score\-style judge prompt instead of the Englishdefaultprompt\.Avg Rankreports the mean rank over AR/PL/UK/ZH, with lower being better\.![Refer to caption](https://arxiv.org/html/2608.02620v1/x6.png)Figure 6:Model intersection counts across arenas
## Appendix EMetadata

EveryJudgeArenarun writes a versioned metadata file \(run\-metadata\.\{version\}\.json\) to the output directory alongside the evaluation artifacts, see[Fig\.˜7](https://arxiv.org/html/2608.02620#A5.F7)for an example\. The version token is incremented on schema changes so that older files remain parseable\.

The metadata descriptor is the central reproducibility anchor\. It records the entry point that produced the run, UTC timestamps and duration, the complete run configuration \(judge prompt preset, position\-swap setting, truncation budgets, generation budgets and engine options\), a compacted version of the results in which the per\-sample preference list is replaced by its count, and counts for the logged input payloads\. It also captures the execution environment: the Python version, the platform string, the installed versions of declared dependencies and the git commit hash of theJudgeArenacheckout\. When available, runtime accounting records prompt and completion token counts, while limit\-event summaries report truncations and generation or reasoning caps that were hit\. To make reruns comparable without duplicating the inputs, the metadata stores cryptographic hashes of the normalized set of instruction indices and of the judge system prompt and user prompt template, so that a difference in inputs or in the judge prompt is detectable at a glance\. A recursive listing of all other artifacts produced in the output directory, together with their byte sizes, closes the descriptor\.

### Field reference

#### Schema version \(schema\_version\)\.

Identifies the file layout; downstream tools use it to select the correct parser as the format evolves\.

#### Timestamps \(timestamps\)\.

UTC start/end times and elapsed duration in seconds, enabling ordering of concurrent experiments and correlation with scheduler logs\.

#### Entrypoint \(entrypoint\)\.

Fully\-qualified Python callable used to launch the run \(e\.g\.judgearena\.generate\_and\_evaluate\.main\)\. Together withgit\.commitit uniquely identifies the executed code path\.

#### Command \(command\)\.

The exact command\-line call, including all arguments, as it was run\.

#### Run configuration \(run\)\.

Key arguments and their resolved values at run time, including any defaults, removing ambiguity when defaults change in later versions\.

#### Results \(results\)\.

Aggregate statistics: win rate, win/loss/tie/missing counts\. Per\-sample annotations are saved separately as a CSV and listed inartifacts\.

#### Inputs \(inputs\)\.

Item counts per split \(instruction indices, completions from each model\)\. A mismatch with the annotation CSV row count indicates an interrupted or filtered run\.

#### Environment \(environment\)\.

Python version and OS platform string, sufficient to flag compatibility issues across runs\.

#### Dependencies \(dependencies\)\.

Pinned versions of all direct dependencies frompyproject\.toml; uninstalled optional extras are recorded asnull\.

#### Git \(git\)\.

Branch name and SHA\-1 commit hash ofHEADat run time; omitted when run outside a git repository\.

#### Prompt hashes \(judge\_system\_prompt\_sha256,judge\_user\_prompt\_template\_sha256\)\.

SHA\-256 digests of the judge system prompt and user prompt template\. Any change to the prompt — enabling explanation mode, switching templates, etc\. — produces a different hash, making prompt variations detectable across runs\.

#### Instruction indices hash \(instruction\_indices\_sha256\)\.

Order\-insensitive SHA\-256 digest of the sampled instruction indices, confirming that two runs covered the same subset of prompts\.

#### Artifacts \(artifacts\)\.

List of all output files with relative paths and byte sizes, enabling detection of incomplete runs\.

### Reproducibility

The metadata captures five elements needed to reproduce a run: \(1\)git\.commitidentifies the source code; \(2\)dependenciesrecords the software versions; \(3\)runrecords all arguments including resolved defaults; \(4\)commandrecords the exact command as typed; \(5\) prompt hashes confirm the judge prompts were unchanged\. Together, these are sufficient to fully reconstruct any evaluation\.

\{

"schema\_version":"judgearena\-run\-metadata/v1",

"timestamps":\{

"started\_at\_utc":"2026\-02\-27T12:51:24\+00:00",

"finished\_at\_utc":"2026\-02\-27T12:56:50\+00:00",

"duration\_sec":325\.7

\},

"entrypoint":"judgearena\.generate\_and\_evaluate\.main",

"command":\{

"argv":\["judgearena/generate\_and\_evaluate\.py",

"\-\-dataset","alpaca\-eval",

"\-\-model\_A","VLLM/org/ModelA\-Small",

"\-\-model\_B","VLLM/org/ModelB\-Large",

"\-\-judge\_model","VLLM/org/Judge\-8B",

"\-\-n\_instructions","30","\-\-swap\_mode","fixed"\]

\},

"run":\{

"dataset":"alpaca\-eval",

"model\_A":"VLLM/org/ModelA\-Small",

"model\_B":"VLLM/org/ModelB\-Large",

"judge\_model":"VLLM/org/Judge\-8B",

"swap\_mode":"fixed",

"n\_instructions":30

\},

"results":\{

"num\_battles":30,

"winrate":0\.143,

"num\_wins":3,"num\_losses":23,

"num\_ties":2,"num\_missing":2

\},

"inputs":\{

"instruction\_index":\{"count":30\},

"completions\_A":\{"count":30\},

"completions\_B":\{"count":30\}

\},

"environment":\{

"python\_version":"3\.12\.12",

"platform":"Linux\-x86\_64\-with\-glibc2\.28"

\},

"dependencies":\{

"datasets":"4\.5\.0","pandas":"2\.3\.3",

"vllm":"0\.10\.2","transformers":"4\.57\.6"

\},

"git":\{

"branch":"feat/versioned\-metadata",

"commit":"08a04362\.\.\."

\},

"instruction\_indices\_sha256":"ea94c75c\.\.\.0cd51",

"judge\_system\_prompt\_sha256":"0912ced0\.\.\.239c",

"judge\_user\_prompt\_template\_sha256":"ca30b326\.\.\.ffa1",

"artifacts":\[

\{"path":"annotations\.csv","size\_bytes":155010\},

\{"path":"args\.json","size\_bytes":483\},

\{"path":"results\.json","size\_bytes":1041\}

\]

\}

Figure 7:Anonymised and abridgedrun\-metadata\.v1\.jsonfrom a realJudgeArenarun \(30 instructions, AlpacaEval dataset\)\. Model identifiers replaced with placeholders; commit hash truncated\.

## Appendix FAsset Licenses

Benchmarks\.AlpacaEval, Arena\-Hard, FastChat \(MT\-Bench\), and m\-ArenaHard \(CohereLabs\) are released under the Apache 2\.0 License\. The Aya Expanse baseline completions for m\-arena\-hard\-v0\.1 are imported separately fromCohereLabs/deja\-vu\-pairwise\-evalsreleased under CC BY\-NC\-SA 4\.0\.

Human preference datasets\.The LMArena human preference datasets used for meta\-evaluation are released under CC BY 4\.0 for user prompts; model outputs are governed by the respective model providers’ terms of use\. ComparIA conversations are released under the Etalab Open License 2\.0; the ComparIA codebase is released under Apache 2\.0\.

Judge and candidate models\.Gemma 4 31B\-IT is released under Apache 2\.0\. Skywork\-Critic\-Llama\-3\.1\-70B is released under the Skywork Community License\. Candidate models evaluated in the case study—Qwen3\.5\-9B, OLMo\-3\-7B, OLMo\-3\-7B\-Think, SmolLM3\-3B, Apertus\-8B, EuroLLM\-9B, and EuroLLM\-1\.7B—are released under Apache 2\.0 while Tiny\-Aya\-Global is released under CC BY\-NC 4\.0\. All assets are used in accordance with their respective licenses for non\-commercial research purposes\.