DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

# DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
Source: [https://arxiv.org/html/2606.26429](https://arxiv.org/html/2606.26429)
Aaron J\. Li1,Hao Huang1,Youngmin Park1,Yitong Ma1, Wei\-Lin Chiang2,Li Chen2,Cho\-Jui Hsieh2,3,Bin Yu1,Ion Stoica1,2 1University of California, Berkeley2Arena3University of California, Los Angeles [https://dualeval\.github\.io/](https://dualeval.github.io/) [https://github\.com/aaron\-jx\-li/DualEval](https://github.com/aaron-jx-li/DualEval)

###### Abstract

Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena\-style preference data that better reflect open\-ended user interactions\. We introduceDualEval, a latent model\-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness\. We applyDualEvalacross four domains: coding, math, miscellaneous domain\-knowledge tasks, and generic everyday user queries\. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward\-model scores validated against held\-out human preferences for open\-ended model responses\. Empirically, our framework produces reliable and balanced model rankings, and its learned item\-level profiles support downstream applications such as benchmark compression for sample\-efficient evaluation and anomaly detection for contamination or outlier analysis\. Overall,DualEvalunifies static and arena\-style evaluation through joint model\-item calibration, producing model rankings and item\-level diagnostics that support more sample\-efficient, interpretable, and auditable evaluation pipelines\.

DualEval: Joint Model\-Item Calibration for Unified LLM Evaluation

Aaron J\. Li1, Hao Huang1, Youngmin Park1, Yitong Ma1,Wei\-Lin Chiang2,Li Chen2,Cho\-Jui Hsieh2,3,Bin Yu1,Ion Stoica1,21University of California, Berkeley2Arena3University of California, Los Angeles[https://dualeval\.github\.io/](https://dualeval.github.io/)[https://github\.com/aaron\-jx\-li/DualEval](https://github.com/aaron-jx-li/DualEval)

††Correspondence:aaronjli@berkeley\.edu## 1Introduction

LLM evaluation increasingly draws on two complementary signals: static benchmark correctness and human preferences in open\-ended arenas\. Static benchmarks provide ground\-truth labels and standardized grading protocols\(Hendryckset al\.,[2020](https://arxiv.org/html/2606.26429#bib.bib9); Chenet al\.,[2021](https://arxiv.org/html/2606.26429#bib.bib27); Austinet al\.,[2021](https://arxiv.org/html/2606.26429#bib.bib28)\), but can become saturated or contaminated as models improve\(Ottet al\.,[2022](https://arxiv.org/html/2606.26429#bib.bib3); Magar and Schwartz,[2022](https://arxiv.org/html/2606.26429#bib.bib2)\)\. Arena\-style evaluations, such as Chatbot Arena\(Chianget al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib1)\), better reflect open\-ended user interactions through Bradley–Terry\-style rankings\(Bradley and Terry,[1952](https://arxiv.org/html/2606.26429#bib.bib5)\), but open\-ended preference judgments can involve annotator disagreement, subjectivity, and systematic evaluator biases, making them difficult to interpret without additional structure\(Van Der Leeet al\.,[2019](https://arxiv.org/html/2606.26429#bib.bib36); Basileet al\.,[2021](https://arxiv.org/html/2606.26429#bib.bib37); Zhenget al\.,[2023](https://arxiv.org/html/2606.26429#bib.bib18)\)\. Recent work has begun to connect benchmark and preference\-based evaluation\(Liet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib4); Niet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib25)\), but the two settings still provide weakly integrated views of model capability\.

Furthermore, both settings are largely model\-centric: they aggregate item\-level outcomes into overall model scores, treating all items as equally informative\. This misses a key insight: ranking signal depends on model\-item interaction\. Items that are nearly always solved or nearly always failed provide limited separation among the current model pool; the most informative items are those for which small ability differences among evaluated models translate into detectable differences in correctness or preference outcomes\. Modeling this item structure can make evaluation more stable, interpretable, and cost\-efficient, while also identifying which items are saturated, discriminative, or future\-facing\.

Toward this end, we proposeDualEval, a unified evaluation framework that jointly estimates model abilities and item properties from both static benchmark correctness and arena\-style preference signals\. Inspired by Item Response Theory \(IRT\)\(Ackermanet al\.,[2003](https://arxiv.org/html/2606.26429#bib.bib7); Lord and Novick,[2008](https://arxiv.org/html/2606.26429#bib.bib6)\),DualEvalplaces model abilities and item difficulties on a shared latent scale\. For static benchmarks, we use a two\-parameter logistic IRT model over binary correctness labels\. For arena\-style data, we use reward\-model scores to construct soft pairwise preference targets and fit them through the same latent ability–difficulty structure\. This novel joint formulation allows correctness and preference supervision to update shared model and item parameters, yielding not only model rankings but also item\-level diagnostic information\.

This joint formulation has two key benefits\. First, it bridges static and arena evaluation: correctness labels and preference signals can update the same latent model parameters, allowing each source to compensate for the other’s weaknesses\. Static benchmarks provide clean, controlled supervision, while arena prompts provide open\-ended signals closer to realistic user interactions\. Second,DualEvalturns evaluation data itself into an object of analysis\. The learned item parameters characterize each question’s difficulty and sharpness, identify saturated or low\-signal items, and support low\-cost subsets that preserve the full\-data ranking\.

We applyDualEvalto four domains: coding, math, miscellaneous domain\-knowledge tasks, and generic everyday user queries\. The first three domains combine static benchmark labels with reward\-model scores for open\-ended arena responses, while the generic domain uses arena data only\. Our experiments cover 18 frontier LLMs from multiple providers\. Arena supervision uses our proprietary scalar reward model, developed internally at LM Arena and validated against held\-out human preferences; we also verify robustness using the publicSkywork\-Reward\-V2\-Qwen3\-8Bmodel\. Across domains,DualEvalreconstructs both evaluation signals, achieving 88–92% static\-label accuracy in the three static\-anchored domains and 68–81% decisive\-pair agreement on arena comparisons\. Compared with static\-only and arena\-only baselines, the joint model yields more balanced rankings across evaluation sources and remains robust under reward\-model substitution\. Its learned item profiles further support diagnostic applications: selecting the top 10% of items by Fisher information recovers the full\-data ranking with Spearmanρ≥0\.95\\rho\\geq 0\.95across all static\-anchored domains, while residual\-based anomaly detection recovers injected contamination with AUROC≥0\.995\\geq 0\.995for static\-label targets and AUROC0\.930\.93–0\.980\.98for arena\-reward targets under a clean\-fit reference\.

Overall, we reframe LLM evaluation as joint model\-item calibration, and our contributions can be summarized as follows:

- •We proposeDualEval, a novel evaluation paradigm that unifies static benchmark correctness and open\-ended preference signals on a shared latent model\-item scale, enabling mutually informed frontier model evaluation and item diagnostics\.
- •We evaluateDualEvalon 18 frontier LLMs across coding, math, miscellaneous domain knowledge, and generic everyday user queries, showing that it produces balanced rankings across signal sources, stable bootstrap estimates, and robustness to reward\-model choice in static\-anchored domains\.
- •We demonstrate two diagnostic applications: Fisher\-information\-based benchmark compression, which recovers full\-data rankings from small high\-signal subsets, and residual\-based anomaly detection, which flags \(model, item\) pairs whose observed outcomes deviate from the fitted model\-item expectations\.

## 2Related Work

#### Static and open\-ended LLM evaluation\.

Static benchmarks cover NLI and reasoning\(Bowmanet al\.,[2015](https://arxiv.org/html/2606.26429#bib.bib15); Williamset al\.,[2018](https://arxiv.org/html/2606.26429#bib.bib16); Wanget al\.,[2018](https://arxiv.org/html/2606.26429#bib.bib17)\), QA\(Rajpurkaret al\.,[2016](https://arxiv.org/html/2606.26429#bib.bib12); Yanget al\.,[2018](https://arxiv.org/html/2606.26429#bib.bib13); Kwiatkowskiet al\.,[2019](https://arxiv.org/html/2606.26429#bib.bib11); Lianget al\.,[2022](https://arxiv.org/html/2606.26429#bib.bib19)\), and math\(Hendryckset al\.,[2020](https://arxiv.org/html/2606.26429#bib.bib9); Cobbeet al\.,[2021](https://arxiv.org/html/2606.26429#bib.bib14); Hendryckset al\.,[2021](https://arxiv.org/html/2606.26429#bib.bib10)\), but vary substantially in difficulty and discriminability\(Zhouet al\.,[2026](https://arxiv.org/html/2606.26429#bib.bib29); Castlemanet al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib33); Wanget al\.,[2026](https://arxiv.org/html/2606.26429#bib.bib31)\)\. Open\-ended preference evaluations such as Chatbot Arena\(Chianget al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib1)\)and MT\-Bench\(Zhenget al\.,[2023](https://arxiv.org/html/2606.26429#bib.bib18)\)capture user\-facing quality through aggregate Bradley\-Terry rankings without explicit item modeling\. Several efforts integrate the two settings\(Liet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib4); Niet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib25)\)\.

#### IRT and efficient evaluation\.

Item Response Theory\(Embretson and Reise,[2013](https://arxiv.org/html/2606.26429#bib.bib21)\)has long been used in educational testing and was applied early to NLP for instance\-level analysis\(Laloret al\.,[2019](https://arxiv.org/html/2606.26429#bib.bib22); Martínez\-Plumedet al\.,[2019](https://arxiv.org/html/2606.26429#bib.bib23)\)\. Recent work uses IRT\-style modeling for benchmark diagnostics\(Zhouet al\.,[2026](https://arxiv.org/html/2606.26429#bib.bib29); Castlemanet al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib33)\), label\-free evaluation\(Robertson,[2025](https://arxiv.org/html/2606.26429#bib.bib26)\), data\-efficient evaluation\(Liaoet al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib24); Wanget al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib32)\), and adaptive testing\(Liet al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib30)\)\. These methods generally operate within a single evaluation source, most often objective correctness labels\.DualEvalinstead couples static correctness with reward\-distilled open\-ended preferences in a shared latent model\-item space\.

## 3Method

In ourDualEvalframework, each modeliiis assigned an ability parameterθi∈ℝ\\theta\_\{i\}\\in\\mathbb\{R\}, and each taskqqis assigned a difficulty parameterbq∈ℝb\_\{q\}\\in\\mathbb\{R\}and a sharpness parameteraq\>0a\_\{q\}\>0, parameterized asaq=exp⁡\(kq\)a\_\{q\}=\\exp\(k\_\{q\}\)withkq∈ℝk\_\{q\}\\in\\mathbb\{R\}\. Intuitively,θi−bq\\theta\_\{i\}\-b\_\{q\}measures how capable modeliiis relative to taskqq, whileaqa\_\{q\}is a sharpness parameter that controls how rapidly the predicted success probability changes as model ability moves relative to item difficulty\.

#### Static IRT\.

For static benchmarks with binary correctness labels, we use a standard two\-parameter logistic IRT model:

pi,q=P​\(yi,q=1\)=σ​\(aq​\(θi−bq\)\),p\_\{i,q\}=P\(y\_\{i,q\}=1\)=\\sigma\(a\_\{q\}\(\\theta\_\{i\}\-b\_\{q\}\)\),whereσ​\(⋅\)\\sigma\(\\cdot\)is the sigmoid function\. Let𝒮\\mathcal\{S\}denote the set of observed static labels\. We define the static loss as:

ℒstatic=𝔼\(i,q\)∈𝒮​BCE​\(pi,q,yi,q\)\.\\mathcal\{L\}\_\{\\mathrm\{static\}\}=\\mathbb\{E\}\_\{\(i,q\)\\in\\mathcal\{S\}\}\\mathrm\{BCE\}\\\!\\left\(p\_\{i,q\},y\_\{i,q\}\\right\)\.

#### Reward\-Distilled Pairwise IRT\.

For open\-ended evaluations, a reward model assigns each model response a scalar scoreri,qr\_\{i,q\}\. Our framework accepts any scalar response\-quality scorer; the joint formulation provides robustness to RM noise when static correctness labels are available \(Appendix §[F](https://arxiv.org/html/2606.26429#A6)\)\. We first standardize rewards globally byzi,q=ri,q−r¯σrz\_\{i,q\}=\\frac\{r\_\{i,q\}\-\\bar\{r\}\}\{\\sigma\_\{r\}\}, wherer¯\\bar\{r\}andσr\\sigma\_\{r\}are the mean and standard deviation over all reward observations\. For each questionqqand model pair\(i,j\)\(i,j\), we define a soft pairwise preference target:

pi​j​q∗=σ​\(zi,q−zj,q\),p^\{\*\}\_\{ijq\}=\\sigma\(z\_\{i,q\}\-z\_\{j,q\}\),which converts reward gaps into calibrated preference strengths: small gaps yield targets near0\.50\.5, while large gaps yield more confident preferences\.

Given the shared IRT parameters, we define the latent success probability of modeliion questionqqas:

pi,q=σ​\(aq​\(θi−bq\)\)\.p\_\{i,q\}=\\sigma\\\!\\left\(a\_\{q\}\(\\theta\_\{i\}\-b\_\{q\}\)\\right\)\.The predicted preference probability is then:

P^​\(i≻j∣q\)=σ​\(γ​\(pi,q−pj,q\)\),\\hat\{P\}\(i\\succ j\\mid q\)=\\sigma\\\!\\left\(\\gamma\(p\_\{i,q\}\-p\_\{j,q\}\)\\right\),whereγ\>0\\gamma\>0is a learned arena\-temperature parameter\.

#### Tie and Both\-Bad Filtering\.

Not all reward\-derived pairs provide useful relative preference signal\. We identify ties using the empirical reward\-gap distribution: a pair is marked as a tie if\|zi,q−zj,q\|<δtie\|z\_\{i,q\}\-z\_\{j,q\}\|<\\delta\_\{\\mathrm\{tie\}\}, whereδtie\\delta\_\{\\mathrm\{tie\}\}is chosen as the configured percentile of pairwise absolute reward gaps\. Ties are excluded from the arena loss\.

We also identify “both\-bad” pairs, where both models receive low standardized rewards withmax⁡\(zi,q,zj,q\)<τbb\\max\(z\_\{i,q\},z\_\{j,q\}\)<\\tau\_\{\\mathrm\{bb\}\}, whereτbb\\tau\_\{\\mathrm\{bb\}\}is chosen as the configured percentile of pairwise maximum rewards\. Let𝒜dec\\mathcal\{A\}\_\{\\mathrm\{dec\}\}denote the decisive arena pairs, defined as non\-tie and non\-both\-bad pairs, and let𝒜bb\\mathcal\{A\}\_\{\\mathrm\{bb\}\}denote the both\-bad, non\-tie pairs\. We define the full arena loss as:

ℒarena=λpref​𝔼\(i,j,q\)∈𝒜dec​BCE​\(P^​\(i≻j∣q\),pi​j​q∗\)\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{arena\}\}=\\lambda\_\{\\mathrm\{pref\}\}\\mathbb\{E\}\_\{\(i,j,q\)\\in\\mathcal\{A\}\_\{\\mathrm\{dec\}\}\}\\mathrm\{BCE\}\\\!\\left\(\\hat\{P\}\(i\\succ j\\mid q\),p^\{\*\}\_\{ijq\}\\right\)−λbb​𝔼\(i,j,q\)∈𝒜bb​\[log⁡\(1−pi,q\)\+log⁡\(1−pj,q\)\]\.\\displaystyle\-\\lambda\_\{\\mathrm\{bb\}\}\\mathbb\{E\}\_\{\(i,j,q\)\\in\\mathcal\{A\}\_\{\\mathrm\{bb\}\}\}\\left\[\\log\(1\-p\_\{i,q\}\)\+\\log\(1\-p\_\{j,q\}\)\\right\]\.The first term fits relative preference evidence from decisive pairs\. The second term treats both\-bad pairs as absolute failure evidence, pushing both models toward low success probability on that question and avoiding conflicting gradients between “both models failed” and “one model was slightly better”\.

#### Joint Objective\.

The static and arena components share the same latent parameters\(θ,b,k\)\(\\theta,b,k\)\. The full optimization objective is:

ℒ=λstatic​ℒstatic\+ℒarena\\displaystyle\\mathcal\{L\}=\\lambda\_\{\\mathrm\{static\}\}\\mathcal\{L\}\_\{\\mathrm\{static\}\}\+\\mathcal\{L\}\_\{\\mathrm\{arena\}\}\+λreg​\(‖θ‖22\+‖b‖22\+‖k‖22\+\(log⁡γ~\)2\)\.\\displaystyle\+\\lambda\_\{\\mathrm\{reg\}\}\\left\(\\\|\\theta\\\|\_\{2\}^\{2\}\+\\\|b\\\|\_\{2\}^\{2\}\+\\\|k\\\|\_\{2\}^\{2\}\+\(\\log\\tilde\{\\gamma\}\)^\{2\}\\right\)\.
Letγ0=4\\gamma\_\{0\}=4denote the default arena\-temperature scale\. We regularize the relative scaleγ~=γ/γ0\\tilde\{\\gamma\}=\\gamma/\\gamma\_\{0\}toward one\.

For identifiability, after each optimization step we center model abilities to have zero mean and absorb the same shift into item difficulties\. This fixes the translation invariance of the latent scale and keeps reported abilities comparable across fits\. Final model rankings are obtained by sorting the learned abilitiesθi\\theta\_\{i\}\.

## 4Experiments

In this section, we first validateDualEval’s joint fit and ranking behavior \(§4\.1\), then study two practical applications enabled by the learned item profiles: sample\-efficient benchmark compression \(§4\.2\) and residual\-based anomaly detection \(§4\.3\)\.

#### Experimental Setup\.

We evaluateDualEvalon four settings: coding, math, miscellaneous domain\-knowledge tasks, and generic everyday user queries\. The first three combine challenging static benchmarks with reward\-scored open\-ended responses; the generic setting uses arena\-style data only, serving as an arena\-only stress test without static anchors\. We select evaluation\-relevant arena prompts using LLM judges, with filtering details in Appendix §[G](https://arxiv.org/html/2606.26429#A7)\. Static benchmarks provide binary correctness labels, while open\-ended responses are scored by our proprietary scalar reward model \(RM\), developed internally at LM Arena and validated against held\-out human preferences \(Appendix §[C](https://arxiv.org/html/2606.26429#A3)\)\. The primary RM uses aQwen\-3\-32Bbackbone with a reward head and was trained on over five million human\-preference pairs using a Bradley–Terry objective\. To test robustness to reward\-model choice, we also repeat the main static\-anchored analyses with the publicSkywork\-Reward\-V2\-Qwen3\-8BRM, which has lower agreement with held\-out human preferences \(Appendix §[F](https://arxiv.org/html/2606.26429#A6)\)\. For each model\-question pair, we generate one response under a consistent inference setup \(Appendix §[G](https://arxiv.org/html/2606.26429#A7)\)\. Table[1](https://arxiv.org/html/2606.26429#S4.T1)lists the benchmark composition and evaluated models\. For agentic coding tasks that require a fixed execution harness, we use the same harness for all models: Terminus2 for TerminalBench\(Merrillet al\.,[2026](https://arxiv.org/html/2606.26429#bib.bib34)\)and mini\-SWE\-agent v2 for SWE\-Bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib35)\)\.

\(a\) Evaluation data composition— four domains, each question answered by all 18 models below\.DomainStatic benchmarks \(\#Q each\)Arena \#QTotalCodingLiveCodeBench v6 \(120\), MBPP\-Plus \(80\), SWE\-Bench\-lite \(60\), TerminalBench\-2\.0 \(40\)300600MathAIME 2025 \(30\), AIME 2026 \(30\), HLE\-Math \(60\), Olympiad\-Math \(80\)200400MiscHLE: Bio\.\+Med\. \(52\), Eng\. \(25\), Hum\.\+Soc\.\+Sci\. \(90\), Other \(83\); SimpleQA \(50\)300600Generic—500500
\(b\) Frontier LLMs evaluated— all 18 models applied to every \(domain, question\) pair\.OpenAIGPT\-5\.5AnthropicClaude Opus 4\.7GoogleGemini 3\.1 ProOpenAIGPT\-5\.4AnthropicClaude Opus 4\.6GoogleGemini 2\.5 ProOpenAIGPT\-5\.4\-miniAnthropicClaude Sonnet 4\.6GoogleGemini 2\.5 FlashOpenAIGPT\-5\-miniAnthropicClaude Haiku 4\.5xAIGrok\-4OpenAIGPT\-4\.1DeepSeekDeepSeek\-V3\.2QwenQwen3\-Max\-ThinkingOpenAIGPT\-4\.1\-miniMistralMistral Large 3MetaLLaMA\-4 Maverick Instruct

Table 1:Evaluation data composition \(top\) and frontier LLMs evaluated \(bottom\)\. Static benchmarks provide binary correctness labels; arena prompts are filtered from LMArena via LLM judges\. The generic domain uses arena prompts only\. All 18 models respond to every question, producing one response per \(model, question\) pair\. Benchmark source citations and sampling details are provided in Appendix §[G](https://arxiv.org/html/2606.26429#A7)\.
### 4\.1DualEvalResults

Figures[1](https://arxiv.org/html/2606.26429#S4.F1),[4](https://arxiv.org/html/2606.26429#A1.F4),[5](https://arxiv.org/html/2606.26429#A1.F5), and[6](https://arxiv.org/html/2606.26429#A1.F6)show the learned model abilities and item\-difficulty distributions across the four settings, with math, miscellaneous, and generic visualizations provided in Appendix §[A](https://arxiv.org/html/2606.26429#A1)\.

#### Static and Arena Fit\.

DualEvalreconstructs both signal types: 88–92% binary accuracy on static labels across the three static\-anchored settings, and 68–81% decisive\-pair agreement on arena comparisons across all four settings\. This arena agreement is comparable to the proprietary reward model’s agreement with held\-out human preferences, suggesting that the joint latent structure captures a substantial share of the preference signal rather than merely fitting noise\. Reliability diagrams in Appendix §[E](https://arxiv.org/html/2606.26429#A5)show approximate calibration for both static success probabilities and arena preference probabilities, with ECE0\.040\.04–0\.070\.07on static cells and0\.010\.01–0\.030\.03on arena soft pairwise targets, suggesting no systematic biases of over\- or under\-confidence across domains\.

#### Difficulty Structure\.

The learned item profiles reveal substantial heterogeneity in difficulty and sharpness\. Many items lie far from the main ability cluster, indicating saturation or difficulty beyond the evaluated model pool\. We also observe a modality difference: static items show near\-zero correlation between difficultybqb\_\{q\}and log\-sharpnesslog⁡aq\\log a\_\{q\}, whereas arena items show a stronger positive association \(ρ≈0\.6\\rho\\approx 0\.6across domains; Appendix §[H](https://arxiv.org/html/2606.26429#A8)\)\. This suggests that difficulty and sharpness contribute differently across evaluation sources, motivating the Fisher\-information score in §[4\.2](https://arxiv.org/html/2606.26429#S4.SS2), which combines both quantities rather than selecting items by difficulty or sharpness alone\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x1.png)Figure 1:Rankings of model abilities and model\-centric question difficulties on thecodingdomain, using our proprietary reward model\.
#### Static–Arena Integration\.

We next test whether joint calibration can integrate static correctness and arena preference signals without collapsing to either source alone\. For each static\-anchored setting, we compareDualEvalwith three baselines: a static\-only 2PL IRT model trained on benchmark correctness, an arena\-only Bradley–Terry model trained on reward\-derived pairwise comparisons, and an arena\-only average\-reward ranking\. We average results over three random question splits\. Each method is evaluated by held\-out arena decisive\-pair accuracy and Spearman correlation with two full\-data references: static\-only IRT and arena\-only Bradley–Terry rankings\.

Table[2](https://arxiv.org/html/2606.26429#S4.T2)shows the expected source\-specialization tradeoff\. The static\-only model aligns best with the static reference but transfers weakly to the arena reference\. Arena\-only baselines achieve the highest arena pair accuracy and strongest agreement with the arena reference, but align less well with the static reference\.DualEvalis designed to balance these sources: it preserves nearly the same static\-reference agreement as static\-only training while improving arena\-reference agreement over the static\-only baseline\. This supports the role of the joint model as an integration mechanism rather than a single\-source optimizer\.

We also illustrate the stabilizing role ofDualEvalwhen arena supervision is noisier\. Replacing our proprietary RM with the public Skywork RM substantially changes arena\-only rankings: Arena BT’s agreement with the public Arena sub\-leaderboards drops toρ=0\.51/0\.54/0\.25\\rho=0\.51/0\.54/0\.25for coding, math, and miscellaneous, respectively\. In contrast,DualEvalwith the public RM preserves high agreement with the static reference ranking, with Staticρ=0\.976/0\.984/0\.963\\rho=0\.976/0\.984/0\.963, and remains closer to the proprietary\-RMDualEvalranking than arena\-only training does\. A direct cross\-RM comparison yields Spearman correlations of0\.9630\.963,0\.9920\.992, and0\.8350\.835for coding, math, and miscellaneous, respectively \(Appendix §[F](https://arxiv.org/html/2606.26429#A6)\)\. The arena\-only generic setting provides the no\-static\-anchor counterfactual: without static labels, proprietary\- and public\-RM rankings diverge sharply \(ρ=0\.25\\rho=0\.25\)\. Together, these results suggest that static anchoring stabilizes the shared latent scale when preference supervision is noisy\.

CodingMathMiscMethodPair Acc\.Staticρ\\rhoArenaρ\\rhoPublicρ\\rhoPair Acc\.Staticρ\\rhoArenaρ\\rhoPublicρ\\rhoPair Acc\.Staticρ\\rhoArenaρ\\rhoPublicρ\\rhoStatic 2PL0\.7311\.0000\.7730\.7630\.6411\.0000\.5710\.6820\.7011\.0000\.6950\.701Arena BT \(prop\.\)0\.8090\.7731\.0000\.8510\.7300\.5711\.0000\.8600\.8060\.6951\.0000\.874Arena BT \(pub\.\)0\.6800\.2550\.4160\.5110\.6670\.4570\.6240\.5360\.7790\.2200\.3500\.247Arena Avg Reward \(prop\.\)0\.8050\.7790\.9940\.8510\.7300\.5711\.0000\.8600\.8060\.6951\.0000\.874Arena Avg Reward \(pub\.\)0\.6730\.1970\.3110\.4320\.6610\.4450\.6390\.5030\.7770\.2280\.3540\.222DUALEVAL \(prop\.\)0\.7780\.9590\.8780\.8450\.6820\.9900\.7340\.7030\.7800\.8930\.9400\.833DUALEVAL \(pub\.\)0\.5910\.9760\.8050\.7890\.6000\.9840\.6250\.6920\.6120\.9630\.7240\.657Table 2:Method comparison across domains and reward models, averaged over three random question splits with a test fraction of 0\.2\. Pair Acc\. is held\-out arena decisive\-pair accuracy evaluated against soft preference targets derived from the proprietary RM for all methods, including rows trained with the public Skywork RM\. All Spearman correlations are computed over the same 18 evaluated models\. Staticρ\\rhocompares the learned model ranking with the full\-data static\-only IRT ranking; Arenaρ\\rhocompares it with the full\-data Arena BT ranking under the proprietary RM;publicρ\\rhocompares it with the matched public LMArena sub\-leaderboard over the 18 overlapping models\. Coding uses Text\-Coding, Math uses Text\-Math, and Misc uses Text\-Expert\. prop\./pub\. denote the proprietary RM andSkywork\-Reward\-V2\-Qwen3\-8B, respectively\.
#### Ablations\.

A pair\-treatment ablation \(Appendix §[I](https://arxiv.org/html/2606.26429#A9), Table[5](https://arxiv.org/html/2606.26429#A9.T5)\) confirms the importance of both\-bad anchoring: removing the loss component drops both\-bad AUC from 0\.903 to 0\.622 while leaving held\-out arena pair accuracy near 0\.78\. Tie filtering has a smaller effect on pair accuracy but slightly improves both\-bad calibration\. The full configuration is used in all subsequent experiments\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x2.png)Figure 2:Subset\-recovery agreement \(Spearmanρ\\rhovs\. the full\-dataDualEvalranking\) as a function of the fraction of items retained, across coding, math, and miscellaneous domains\. Items are selected by Top Fisher info \(DualEval\), the sharpness\-only and difficulty\-matched ablations, and two random baselines\. Top Fisher info recovers the full ranking atρ≥0\.95\\rho\\geq 0\.95with as little as 10% of items in all three domains, indicating that ranking signal in current benchmarks is concentrated in a small subset of high\-informativeness items\.

### 4\.2Item Informativeness

A central diagnostic question for any benchmark is which items carry the bulk of the ranking signal between models, and which are redundant\. The joint formulation produces item\-level parameters that let us answer this question directly: we score each item by its Fisher information about model abilities under the fittedDualEvalobjective, then quantify how much of the full\-data ranking can be recovered from the most informative items alone\.

#### Quantifying Informativeness via Fisher Score\.

For a static item,pi,q=σ​\(aq​\(θi−bq\)\)p\_\{i,q\}=\\sigma\(a\_\{q\}\(\\theta\_\{i\}\-b\_\{q\}\)\), whereaqa\_\{q\},bqb\_\{q\}, andθi\\theta\_\{i\}are the learned item sharpness, difficulty, and model ability\. The Fisher information for modeliion itemqqisℐi,qstatic=aq2​pi,q​\(1−pi,q\)\\mathcal\{I\}^\{\\mathrm\{static\}\}\_\{i,q\}=a\_\{q\}^\{2\}\\,p\_\{i,q\}\(1\-p\_\{i,q\}\), which we average over models and weight by the static loss coefficient:

Sqstatic=λstatic​1M​∑i=1Maq2​pi,q​\(1−pi,q\)\.S\_\{q\}^\{\\mathrm\{static\}\}=\\lambda\_\{\\mathrm\{static\}\}\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}a\_\{q\}^\{2\}\\,p\_\{i,q\}\(1\-p\_\{i,q\}\)\.
For an arena item,DualEvalcompares responses throughμi​j​q=σ​\(γ​\(pi,q−pj,q\)\)\\mu\_\{ijq\}=\\sigma\(\\gamma\(p\_\{i,q\}\-p\_\{j,q\}\)\)\. We score each pair by the trace of the Fisher information with respect to the two involved ability parameters\(θi,θj\)\(\\theta\_\{i\},\\theta\_\{j\}\), which measures local sensitivity of the predicted preference probability to either model’s ability:

ℐi​j​qarena=μi​j​q​\(1−μi​j​q\)​γ2\\displaystyle\\mathcal\{I\}^\{\\mathrm\{arena\}\}\_\{ijq\}=\\mu\_\{ijq\}\(1\-\\mu\_\{ijq\}\)\\gamma^\{2\}⋅\[\(aq​pi,q​\(1−pi,q\)\)2\+\(aq​pj,q​\(1−pj,q\)\)2\]\.\\displaystyle\\quad\\cdot\\Bigl\[\\bigl\(a\_\{q\}p\_\{i,q\}\(1\-p\_\{i,q\}\)\\bigr\)^\{2\}\+\\bigl\(a\_\{q\}p\_\{j,q\}\(1\-p\_\{j,q\}\)\\bigr\)^\{2\}\\Bigr\]\.We average this quantity over decisive arena pairs for itemqq:

Sqarena=λpref​1\|𝒫q\|​∑\(i,j\)∈𝒫qℐi​j​qarena,S\_\{q\}^\{\\mathrm\{arena\}\}=\\lambda\_\{\\mathrm\{pref\}\}\\frac\{1\}\{\|\\mathcal\{P\}\_\{q\}\|\}\\sum\_\{\(i,j\)\\in\\mathcal\{P\}\_\{q\}\}\\mathcal\{I\}^\{\\mathrm\{arena\}\}\_\{ijq\},where𝒫q\\mathcal\{P\}\_\{q\}denotes the set of decisive, i\.e\., non\-tie and non\-both\-bad, model pairs for itemqq\.

Static and arena items are scored in the same units: expected information about the model ability vector\. Averaging over arena pairs prevents items from receiving an artificial advantage fromO​\(M2\)O\(M^\{2\}\)pairwise comparisons\.

#### Subset\-Recovery Experiment\.

For each domain, we select a fraction of items, refit ability parameters on the subset \(item parameters held fixed at full\-data values\), and measure Spearman correlation between the resulting ranking and the full\-dataDualEvalranking\. We compare five subset\-selection strategies, including ablations that isolate the contribution of each component ofSqS\_\{q\}: \(i\)Top Fisher info\(DualEval\), selecting items by descendingSqS\_\{q\}; \(ii\)Top sharpnessaqa\_\{q\}, ablating thep​\(1−p\)p\(1\-p\)frontier weight; \(iii\)Difficulty\-matched, selecting by smallest\|bq−median​\(θ\)\|\|b\_\{q\}\-\\mathrm\{median\}\(\\theta\)\|, isolating frontier\-matched difficulty alone; \(iv\)Random\(DualEval\), a uniform random subset; \(v\)Random\(2PL\+BT\), a uniform random subset refit with a single\-source 2PL\+BT baseline that lacks item parameters for arena data\.

#### Ranking Signal Is Concentrated in a Small Subset\.

Figure[2](https://arxiv.org/html/2606.26429#S4.F2)reveals substantial item\-level redundancy in current benchmarks\. Across all three static\-anchored domains, the top 10% of items by Fisher information recover the full\-data ranking at Spearmanρ≥0\.95\\rho\\geq 0\.95, and the top 30% recover it atρ≥0\.99\\rho\\geq 0\.99\. The full Fisher score is close to the sharpness\-only ablation but is the most consistent at small fractions, suggesting that sharpness explains much of the signal while local non\-saturation still improves item selection\. Difficulty matching alone is less reliable, confirming that proximity to the model\-ability range is not sufficient without sensitivity to model differences\. The 2PL\+BT baseline trails theDualEvalvariants throughout, indicating that item parameters fit jointly across static and arena data carry information missing from disjoint\-source rankings\.

#### Implications\.

This redundancy structure has two practical consequences\. First, leaderboard refreshes against established benchmarks can target the high\-Fisher subset for substantial reductions in evaluation cost while preserving ranking confidence\. Second, the items with lowest Fisher information can be retired without affecting ranking quality, providing a principled diagnostic for identifying saturated or non\-discriminative items\. Because informativeness is relative to the evaluated model pool, high\-signal subsets should be recalibrated as model abilities shift\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x3.png)Figure 3:Synthetic contamination detection across coding, math, and miscellaneous domains, separately for static\-label and arena\-reward injections\. Each panel plots item\-level AUROC/AUPRC versus injection rate; the positive class is the set of synthetically contaminated cells\. Residual scores from the clean\-reference detector are strongest across domains and injection rates, while the contaminated\-fit detector remains above the random baseline\.

### 4\.3Residual\-Based Anomaly Detection

A second diagnostic question is whether the joint fit can identify suspicious\(model,item\)\(\\text\{model\},\\text\{item\}\)pairs whose observed outcomes are inconsistent with whatDualEvalexpects given the model’s ability and the item’s difficulty\. Such residual anomalies are candidates for contamination, leakage, or data\-quality issues, and the joint model’s per\-pair expectations make them directly computable\.

#### Per\-Pair Residual Scores\.

For a static\(i,q\)\(i,q\)pair we define the standardized residual

ri,qstatic=yi,q−pi,qpi,q​\(1−pi,q\)\+ε,r^\{\\mathrm\{static\}\}\_\{i,q\}=\\frac\{y\_\{i,q\}\-p\_\{i,q\}\}\{\\sqrt\{p\_\{i,q\}\(1\-p\_\{i,q\}\)\+\\varepsilon\}\},whereyi,q∈\{0,1\}y\_\{i,q\}\\in\\\{0,1\\\}is the observed correctness label andpi,qp\_\{i,q\}is aDualEvalsuccess probability\. Large positive residuals indicate unexpected successes; large negative residuals, unexpected failures\. For an arena\(i,q\)\(i,q\)pair, we average the target–prediction gap across decisive opponents of modeliion itemqq:

ri,qarena=1\|𝒫q\(i\)\|​∑j∈𝒫q\(i\)\(σ​\(zi,q−zj,q\)−μi​j​q\),r^\{\\mathrm\{arena\}\}\_\{i,q\}=\\frac\{1\}\{\|\\mathcal\{P\}\_\{q\}^\{\(i\)\}\|\}\\sum\_\{j\\in\\mathcal\{P\}\_\{q\}^\{\(i\)\}\}\\bigl\(\\sigma\(z\_\{i,q\}\-z\_\{j,q\}\)\-\\mu\_\{ijq\}\\bigr\),wherezi,qz\_\{i,q\}is the standardized reward,μi​j​q=σ​\(γ​\(pi,q−pj,q\)\)\\mu\_\{ijq\}=\\sigma\(\\gamma\(p\_\{i,q\}\-p\_\{j,q\}\)\)is theDualEval\-predicted pairwise probability, and𝒫q\(i\)\\mathcal\{P\}\_\{q\}^\{\(i\)\}is the set of decisive opponents of modeliion itemqq\.

#### Synthetic Contamination Protocol\.

We inject synthetic contamination into existing evaluation artifacts\. For each static\-anchored domain, we sample candidate cells from low\-probability success regions\. For static labels, the candidate pool consists of failed cells\(yi,q=0\)\(y\_\{i,q\}=0\)with the lowestaccimodel×accqitem\\mathrm\{acc\}^\{\\mathrm\{model\}\}\_\{i\}\\times\\mathrm\{acc\}^\{\\mathrm\{item\}\}\_\{q\}, whereaccimodel\\mathrm\{acc\}^\{\\mathrm\{model\}\}\_\{i\}is modelii’s average static accuracy in the domain andaccqitem\\mathrm\{acc\}^\{\\mathrm\{item\}\}\_\{q\}is the fraction of models that solve itemqq\. Flipping these cells creates unexpected successes by weak models on hard items\. For arena responses, the candidate pool is the bottom30%30\\%of cells by standardized rewardzi,qz\_\{i,q\}\. We sample cells uniformly from each pool at rates\{0\.5%,1%,2%,5%\}\\\{0\.5\\%,1\\%,2\\%,5\\%\\\}, with five random seeds per rate\. Static contamination flips the correctness label from0to11; arena contamination adds2\.02\.0to the standardized rewardzi,qz\_\{i,q\}before pairwise targets are rebuilt\.

We evaluate two detectors\. The contaminated\-fit detector scores each cell by its signed positive residual under aDualEvalmodel refit on the contaminated data\. The clean\-reference detector computes the same scores using aDualEvalmodel fit once on the original clean data, while evaluating the contaminated labels or rewards against clean\-fit expectations\. Injected cells are positives and all other cells are negatives for cell\-level AUROC and AUPRC; Appendix §[K](https://arxiv.org/html/2606.26429#A11)details how these detection metrics are computed\.

#### Calibrated Residuals Recover Injected Contamination\.

Figure[3](https://arxiv.org/html/2606.26429#S4.F3)reports AUROC and AUPRC across injection rates and domains\. The clean\-reference detector recovers injected static\-label contamination nearly perfectly, with AUROC≥0\.995\\geq 0\.995and AUPRC0\.800\.80–0\.990\.99, and remains strong on arena\-reward contamination, with AUROC0\.930\.93–0\.980\.98and AUPRC0\.280\.28–0\.830\.83\. This setting is operationally realistic when a benchmark already has a vetted historical calibration and new submissions or data refreshes are screened against it\. The contaminated\-fit detector, which uses only the corrupted snapshot, remains above the random baseline across all conditions but is less sensitive when flipped labels are absorbed into the refitted latent parameters\. The gap between the two detectors highlights the value of preserving clean benchmark snapshots and calibrated historical fits for ongoing leaderboard auditing\.

## 5Discussion

DualEvalsuggests a shift from treating evaluation items as interchangeable test cases to treating them as calibrated measurement instruments\. Once items have learned difficulty, sharpness, informativeness, and residual profiles, benchmark maintenance becomes a model\-item problem: evaluators can identify saturated items, preserve high\-signal subsets, and audit suspicious outcomes rather than relying only on aggregate leaderboard scores\.

This perspective also points to richer extensions\. A scalar ability parameter is useful for interpretability and ranking, but many evaluation domains contain multiple skills; multidimensional IRT provides a natural path toward modeling specialization across reasoning, coding, factuality, and interaction quality\. More broadly, calibrated item profiles could support adaptive evaluation protocols that choose questions based on the current uncertainty about a model, while retaining item\-level evidence for why a ranking changed\.

## 6Conclusion

We introducedDualEval, a framework that jointly models static benchmark correctness and arena\-style preference signals on a shared latent scale\. Across four domains,DualEvalproduces interpretable model rankings while estimating item difficulty, sharpness, and informativeness\. Baseline comparisons show that combining static and arena supervision yields a more balanced signal than either source alone, while item\-level analyses support sample\-efficient evaluation and diagnostic benchmark auditing\. Together, these results point toward evaluation pipelines that jointly rank models and calibrate benchmark items, making leaderboards more efficient, interpretable, and grounded in item\-level evidence\.

## Limitations

Our study has several limitations\. First, the primary arena supervision relies on our internal reward model, which limits direct reproducibility even though we describe its architecture, training objective, and training\-data scale\. We address this by validating the RM against held\-out human preferences \(Appendix §[C](https://arxiv.org/html/2606.26429#A3)\) and replicating the main analyses with the publicly available Skywork\-Reward\-V2\-Qwen3\-8B model \(Appendix §[F](https://arxiv.org/html/2606.26429#A6)\), but broader validation across multiple public RMs and human preference annotations remains important\.

Second, each model\-question pair is evaluated with a single generated response\. This matches many leaderboard settings but does not capture within\-model sampling variability, sensitivity to decoding parameters, or multi\-attempt performance\. ExtendingDualEvalto repeated responses would allow item sharpness and residual anomalies to separate model capability from response\-level stochasticity\.

Finally, the current formulation uses a scalar ability parameter within each domain\. This keeps rankings interpretable, but future evaluation systems should move toward multidimensional model profiles that capture specialization across skills, task formats, and interaction settings\. Such extensions could make item selection adaptive not only to overall model strength, but also to the specific capabilities an evaluator wants to measure\.

## References

- Using multidimensional item response theory to evaluate educational and psychological tests\.Educational Measurement: Issues and Practice22\(3\),pp\. 37–51\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p3.1)\.
- J\. Austin, A\. Odena, M\. Nye, M\. Bosma, H\. Michalewski, D\. Dohan, E\. Jiang, C\. Cai, M\. Terry, Q\. Le,et al\.\(2021\)Program synthesis with large language models\.arXiv preprint arXiv:2108\.07732\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- V\. Basile, M\. Fell, T\. Fornaciari, D\. Hovy, S\. Paun, B\. Plank, M\. Poesio, and A\. Uma \(2021\)We need to consider disagreement in evaluation\.InProceedings of the 1st workshop on benchmarking: past, present and future,pp\. 15–21\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- S\. Bowman, G\. Angeli, C\. Potts, and C\. D\. Manning \(2015\)A large annotated corpus for learning natural language inference\.InProceedings of the 2015 conference on empirical methods in natural language processing,pp\. 632–642\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- R\. A\. Bradley and M\. E\. Terry \(1952\)Rank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39\(3/4\),pp\. 324–345\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- J\. Castleman, N\. Nadeem, T\. Namjoshi, and L\. T\. Liu \(2025\)Rethinking math benchmarks for llms using irt\.Proceedings of Machine Learning Research273,pp\. 66–82\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- W\. Chiang, L\. Zheng, Y\. Sheng, A\. N\. Angelopoulos, T\. Li, D\. Li, B\. Zhu, H\. Zhang, M\. Jordan, J\. E\. Gonzalez,et al\.\(2024\)Chatbot arena: an open platform for evaluating llms by human preference\.InForty\-first International Conference on Machine Learning,Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2606.26429#S1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- S\. E\. Embretson and S\. P\. Reise \(2013\)Item response theory for psychologists\.Psychology Press\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang,et al\.\(2024\)Olympiadbench: a challenging benchmark for promoting agi with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Basart, A\. Zou, M\. Mazeika, D\. Song, and J\. Steinhardt \(2020\)Measuring massive multitask language understanding\.arXiv preprint arXiv:2009\.03300\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. Steinhardt \(2021\)Measuring mathematical problem solving with the math dataset\.arXiv preprint arXiv:2103\.03874\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Jain, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. Stoica \(2025\)Livecodebench: holistic and contamination free evaluation of large language models for code\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 58791–58831\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)Swe\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 54107–54157\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.26429#S4.SS0.SSS0.Px1.p1.1)\.
- T\. Kwiatkowski, J\. Palomaki, O\. Redfield, M\. Collins, A\. Parikh, C\. Alberti, D\. Epstein, I\. Polosukhin, J\. Devlin, K\. Lee,et al\.\(2019\)Natural questions: a benchmark for question answering research\.Transactions of the Association for Computational Linguistics7,pp\. 453–466\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- J\. P\. Lalor, H\. Wu, and H\. Yu \(2019\)Learning latent parameters without human response patterns: item response theory with artificial crowds\.InProceedings of the Conference on Empirical Methods in Natural Language Processing\. Conference on Empirical Methods in Natural Language Processing,Vol\.2019,pp\. 4240\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- P\. Li, X\. Tang, S\. Chen, Y\. Cheng, R\. Metoyer, T\. Hua, and N\. V\. Chawla \(2025\)Adaptive testing for llm evaluation: a psychometric alternative to static benchmarks\.arXiv preprint arXiv:2511\.04689\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- T\. Li, W\. Chiang, E\. Frick, L\. Dunlap, T\. Wu, B\. Zhu, J\. E\. Gonzalez, and I\. Stoica \(2024\)From crowdsourced data to high\-quality benchmarks: arena\-hard and benchbuilder pipeline\.arXiv preprint arXiv:2406\.11939\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- P\. Liang, R\. Bommasani, T\. Lee, D\. Tsipras, D\. Soylu, M\. Yasunaga, Y\. Zhang, D\. Narayanan, Y\. Wu, A\. Kumar,et al\.\(2022\)Holistic evaluation of language models\.arXiv preprint arXiv:2211\.09110\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Liao, Q\. Zhang, R\. Wu, and G\. Fang \(2025\)Toward a unified framework for data\-efficient evaluation of large language models\.arXiv preprint arXiv:2510\.04051\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- J\. Liu, C\. S\. Xia, Y\. Wang, and L\. Zhang \(2023\)Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation\.Advances in neural information processing systems36,pp\. 21558–21572\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1)\.
- F\. M\. Lord and M\. R\. Novick \(2008\)Statistical theories of mental test scores\.IAP\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p3.1)\.
- I\. Magar and R\. Schwartz \(2022\)Data contamination: from memorization to exploitation\.arXiv preprint arXiv:2203\.08242\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- F\. Martínez\-Plumed, R\. B\. Prudêncio, A\. Martínez\-Usó, and J\. Hernández\-Orallo \(2019\)Item response theory in ai: analysing machine learning classifiers at the instance level\.Artificial intelligence271,pp\. 18–42\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- Mathematical Association of America \(2026\)American invitational mathematics examination \(aime\)\.Note:MAA Invitational CompetitionsCited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1)\.
- M\. A\. Merrill, A\. G\. Shaw, N\. Carlini, B\. Li, H\. Raj, I\. Bercovich, L\. Shi, J\. Y\. Shin, T\. Walshe, E\. K\. Buchanan,et al\.\(2026\)Terminal\-bench: benchmarking agents on hard, realistic tasks in command line interfaces\.arXiv preprint arXiv:2601\.11868\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2606.26429#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Ni, F\. Xue, X\. Yue, Y\. Deng, M\. Shah, K\. Jain, G\. Neubig, and Y\. You \(2024\)Mixeval: deriving wisdom of the crowd from llm benchmark mixtures\.Advances in Neural Information Processing Systems37,pp\. 98180–98212\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Ott, A\. Barbosa\-Silva, K\. Blagec, J\. Brauner, and M\. Samwald \(2022\)Mapping global dynamics of benchmark creation and saturation in artificial intelligence\.Nature Communications13\(1\),pp\. 6793\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- L\. Phan, A\. Gatti, Z\. Han, N\. Li, J\. Hu, H\. Zhang, C\. B\. C\. Zhang, M\. Shaaban, J\. Ling, S\. Shi,et al\.\(2025\)Humanity’s last exam\.arXiv preprint arXiv:2501\.14249\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)Squad: 100,000\+ questions for machine comprehension of text\.arXiv preprint arXiv:1606\.05250\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Robertson \(2025\)Identity\-link irt for label\-free llm evaluation: preserving additivity in tvd\-mi scores\.arXiv preprint arXiv:2510\.14966\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Van Der Lee, A\. Gatt, E\. Van Miltenburg, S\. Wubben, and E\. Krahmer \(2019\)Best practices for the human evaluation of automatically generated text\.InProceedings of the 12th international conference on natural language generation,pp\. 355–368\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1)\.
- A\. Wang, A\. Singh, J\. Michael, F\. Hill, O\. Levy, and S\. Bowman \(2018\)GLUE: a multi\-task benchmark and analysis platform for natural language understanding\.InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP,pp\. 353–355\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Wang, C\. Wang, W\. Fu, Y\. Min, M\. Feng, I\. Guan, X\. Hu, C\. He, C\. Wang, K\. Yang,et al\.\(2025\)Rethinking llm evaluation: can we evaluate llms with 200x less data?\.arXiv preprint arXiv:2510\.10457\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Wang, W\. Wu, G\. Wang, G\. Ye, and Z\. Cheng \(2026\)MetaEval: measuring the discrimination of benchmarks for efficient llm evaluation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 33773–33781\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. Fedus \(2024\)Measuring short\-form factuality in large language models\.arXiv preprint arXiv:2411\.04368\.Cited by:[Appendix G](https://arxiv.org/html/2606.26429#A7.SS0.SSS0.Px1.p1.1)\.
- A\. Williams, N\. Nangia, and S\. Bowman \(2018\)A broad\-coverage challenge corpus for sentence understanding through inference\.InProceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 \(long papers\),pp\. 1112–1122\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: a dataset for diverse, explainable multi\-hop question answering\.InProceedings of the 2018 conference on empirical methods in natural language processing,pp\. 2369–2380\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§1](https://arxiv.org/html/2606.26429#S1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Zhou, H\. Huang, Z\. Zhao, L\. Han, H\. Wang, K\. Chen, M\. Yang, W\. Bao, J\. Dong, B\. Xu,et al\.\(2026\)Lost in benchmarks? rethinking large language model benchmarking with item response theory\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 35085–35093\.Cited by:[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.26429#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AAdditionalDualEvalRanking Visualizations

We provide additional examples ofDualEvalrankings of math, miscellaneous, and generic domains in Figures[4](https://arxiv.org/html/2606.26429#A1.F4),[5](https://arxiv.org/html/2606.26429#A1.F5), and[6](https://arxiv.org/html/2606.26429#A1.F6)\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x4.png)Figure 4:Learned model abilities and question difficulties onmath\.![Refer to caption](https://arxiv.org/html/2606.26429v1/x5.png)Figure 5:Learned model abilities and question difficulties onmiscellaneous\.![Refer to caption](https://arxiv.org/html/2606.26429v1/x6.png)Figure 6:Learned model abilities and question difficulties ongeneric\.![Refer to caption](https://arxiv.org/html/2606.26429v1/x7.png)Figure 7:Rankings of model abilities and model\-centric question difficulties on thecodingdomain, using public reward modelSkywork\-Reward\-V2\-Qwen3\-8B\.
## Appendix BDualEvalHyperparameters and Configurations

AllDualEvalexperiments in this paper use the same hyperparameters unless otherwise noted: Adam optimizer with learning rate0\.020\.02,20002000epochs, loss weightsλstatic=λpref=1\.0\\lambda\_\{\\text\{static\}\}=\\lambda\_\{\\text\{pref\}\}=1\.0andλbb=0\.2\\lambda\_\{\\text\{bb\}\}=0\.2, L2 regularization coefficientλreg=0\.01\\lambda\_\{\\text\{reg\}\}=0\.01applied toθ\\theta,bb,kk, andlog⁡γ~\\log\\tilde\{\\gamma\}, and pair\-treatment thresholds derived as the 15thpercentile ofmax⁡\(zi,zj\)\\max\(z\_\{i\},z\_\{j\}\)for both\-bad flagging and the 15thpercentile of\|zi−zj\|\|z\_\{i\}\-z\_\{j\}\|for tie filtering, applied per\-domain\. The arena temperature is parameterized asγ=γ0⋅γ~\\gamma=\\gamma\_\{0\}\\cdot\\tilde\{\\gamma\}withγ0=4\\gamma\_\{0\}=4, andlog⁡γ~\\log\\tilde\{\\gamma\}is L2\-regularized toward zero\. After each optimizer step, model abilities are zero\-mean centered with the same shift absorbed into question difficulties, preserving all differencesθi−bq\\theta\_\{i\}\-b\_\{q\}\. All fits use single\-precision floating point on CPU; a single domain’s joint fit completes in 10–25 seconds\. Arena\-only fits on the generic domain use the same configuration with the static loss disabled\.

## Appendix CValidation of Reward Model with Real Human Preference

This appendix supports the robustness claims in Section[4](https://arxiv.org/html/2606.26429#S4)by \(i\) quantifying how well each reward model agrees with real human preference on the prompts actually used by our pipeline, \(ii\) visualizing cross\-RM agreement ofDualEval’s learned model abilities in the static\-anchored domains, and \(iii\) documenting the arena\-only generic domain as a no\-anchor reference\.

#### Reward\-model alignment with human preference\.

For each domain we identify a subset of arena prompts for which human preference labels are available, and that have a decisive \(non\-tie\) human winner\. Sample sizes per domain are reported in Table[3](https://arxiv.org/html/2606.26429#A3.T3)\. Both reward models are scored on the same pairs from this filtered subset\. Table[3](https://arxiv.org/html/2606.26429#A3.T3)reports the per\-domain pair count and the RM’s accuracy at predicting the human winner\. Our proprietary RM is substantially more human\-aligned than the public RM in every domain\.

DomainNProprietary RMPublic RMCoding1950\.7390\.573Math1200\.8750\.708Misc2000\.7150\.655Generic3760\.7420\.596Table 3:Pairwise accuracy of each reward model at predicting the human winner\. Restricted to pairs with a decisive \(non\-tie\) human\-preference winner\.

## Appendix DStability of DualEval Rankings

We assess the stability ofDualEvalrankings using a question\-level cluster bootstrap\. For each domain we drawB=100B=100bootstrap replicates, resampling questions with replacement stratified by source \(each benchmark source plus arena\), so every replicate has the same per\-sourceNNas the original\. Drawn copies of a question receive unique IDs so the IRT fit treats them as independent items–the standard cluster\-bootstrap behavior\. Rewardzz\-scores are renormalized per replicate\. For each replicate we refitDualEvalwith the published hyperparameters and record per\-modelθ^i\\hat\{\\theta\}\_\{i\}\. We use the same procedure to bootstrap Static 2PL and Arena BT for direct comparison \(Static 2PL is omitted for the arena\-only generic domain\)\.

Figure[8](https://arxiv.org/html/2606.26429#A4.F8)shows per\-modelθ^i\\hat\{\\theta\}\_\{i\}with symmetric 95% CIs \(θ^i±1\.96,σ^boot\\hat\{\\theta\}\_\{i\}\\pm 1\.96,\\hat\{\\sigma\}\_\{\\text\{boot\}\}\) for all three methods in each domain\. Top\-3 and bottom\-3 positions are stable to±1\\pm 1rank across replicates in every domain\. Middle\-band positions span 4–6 ranks of 95% CI in the three static\-anchored domains, indicating that not all adjacent leaderboard positions are statistically distinguishable at the current evaluation size\. Generic–being arena\-only with 500 prompts contributing\(182\)\\binom\{18\}\{2\}pairwise comparisons per item–produces the tightest CIs across the four domains\.

Table[4](https://arxiv.org/html/2606.26429#A4.T4)reports per\-\(domain, method\) summary statistics\. On equivalent data \(generic, arena\-only\),DualEvalmatches or slightly improves over Arena BT \(medianσ^θ=0\.016\\hat\{\\sigma\}\_\{\\theta\}=0\.016vs0\.0190\.019\), confirming that the IRT structure adds discriminative information without inflating uncertainty\. Against Static 2PL on the static\-anchored domains, the joint formulation reduces ranking variance by 30–60%\. Against Arena BT on those domains,DualEvalcarries a modest variance premium that reflects the additional cross\-source information being integrated, in exchange for substantially better Static\-ρ\\rho\(Table[2](https://arxiv.org/html/2606.26429#S4.T2)\)\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x8.png)Figure 8:Per\-model ability with 95% bootstrap CIs \(question\-level cluster bootstrap,B=100B=100, stratified by source\)\.DomainMethodMedianσ^θ\\hat\{\\sigma\}\_\{\\theta\}Median rank\-CICodingStatic 2PL0\.0653\.0Arena BT0\.0281\.0DualEval0\.0402\.0MathStatic 2PL0\.0743\.0Arena BT0\.0392\.0DualEval0\.0553\.0MiscStatic 2PL0\.0824\.5Arena BT0\.0261\.0DualEval0\.0433\.0GenericArena BT0\.0191\.0DualEval0\.0161\.0Table 4:Bootstrap uncertainty per \(domain, method\) under a question\-level cluster bootstrap \(B=100, stratified by source\)\.σ^θ\\hat\{\\sigma\}\_\{\\theta\}is the bootstrap standard deviation of model ability; median rank\-CI is the median 95% bootstrap rank interval width over the 18 models\. Static 2PL is omitted for Generic \(arena\-only\)\. On equivalent data \(Generic\), DualEval matches or improves over Arena BT; against Static 2PL the joint formulation reduces ranking variance by 30–60%\.
## Appendix EHeld\-out calibration

We evaluate calibration of the joint fit’s per\-pair predictions on a cell\-level 80/20 random holdout \(averaged over three random seeds; same fit hyperparameters as §[4](https://arxiv.org/html/2606.26429#S4)\)\. On held\-out static cells, predicted success probabilitiespi,qp\_\{i,q\}match observed binary outcomes with expected calibration error \(ECE\) of0\.040\.04–0\.070\.07across coding/math/misc domains\. On held\-out arena pairs, predicted preference probabilitiesμi​j​q\\mu\_\{ijq\}match the soft pairwise targetsσ​\(zi,q−zj,q\)\\sigma\(z\_\{i,q\}\-z\_\{j,q\}\)with ECE of0\.010\.01–0\.030\.03, with reliability diagrams nearly diagonal across the predicted\-probability range \(Figure[9](https://arxiv.org/html/2606.26429#A5.F9)\)\. For transparency we also report ECE against the binarised pairwise outcome𝟏​\[σ​\(zi,q−zj,q\)≥0\.5\]\\mathbf\{1\}\[\\,\\sigma\(z\_\{i,q\}\-z\_\{j,q\}\)\\geq 0\.5\\,\], which is higher \(0\.070\.07–0\.160\.16\) due to expected under\-confidence when soft pairwise targets near0\.50\.5are discretised to hard win/loss labels; this binarisation does not reflect a miscalibration of the underlying joint fit\. These calibration results justify treatingpi,qp\_\{i,q\}andμi​j​q\\mu\_\{ijq\}as meaningful probability estimates in the downstream Fisher information \(§[4\.2](https://arxiv.org/html/2606.26429#S4.SS2)\) and residual diagnostics \(§[4\.3](https://arxiv.org/html/2606.26429#S4.SS3)\)\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x9.png)Figure 9:Held\-out calibration of DualEval predictions\. Rows: static cells \(pi,qp\_\{i,q\}against binary correctness\) and arena pairs \(μi​j​q\\mu\_\{ijq\}against soft pairwise targetsσ​\(zi−zj\)\\sigma\(z\_\{i\}\-z\_\{j\}\)\)\. Columns: coding, math, miscellaneous domains\. Each point is a decile of the predicted\-probability axis, with size∝n\\propto\\sqrt\{n\}\. Dashed line marks perfect calibration\. ECE \(mean±\\pmSD over three seeds\) annotated per panel\.
## Appendix FCross\-RM Ranking Agreement

Figure[10](https://arxiv.org/html/2606.26429#A6.F10)plotsDualEval’s learned model abilities under the proprietary and public RMs against each other, one panel per static\-anchored domain\. Points fall close to the diagonal: Spearman correlations are0\.9630\.963\(coding\),0\.9920\.992\(math\), and0\.8350\.835\(misc\), consistent with the Staticρ≥0\.96\\rho\\geq 0\.96entries in Table[2](https://arxiv.org/html/2606.26429#S4.T2)and indicating that static anchoring stabilizes the ranking against RM\-induced noise\. The arena\-only generic domain provides the no\-anchor counterfactual: with no static labels to pin the latent scale, the agreement between the proprietary and public RMs drops toρ=0\.25\\rho=0\.25, empirically isolating static anchoring as the mechanism behind the joint framework’s cross\-RM stability\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x10.png)Figure 10:Per\-domain scatter ofDualEvalmodel abilities under the proprietary RM \(xx\-axis\) and the public RM \(yy\-axis\), three panels for coding/math/misc\. Points are individual frontier LLMs\. Static anchoring keeps the cross\-RM rankings tightly aligned in static\-anchored domains\.
## Appendix GEvaluation Details

#### Benchmark sources\.

Our static benchmark pool draws from LiveCodeBench\(Jainet al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib38)\), MBPP\-Plus from EvalPlus\(Liuet al\.,[2023](https://arxiv.org/html/2606.26429#bib.bib42)\), SWE\-Bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib35)\), TerminalBench\(Merrillet al\.,[2026](https://arxiv.org/html/2606.26429#bib.bib34)\), AIME 2025/2026\(Mathematical Association of America,[2026](https://arxiv.org/html/2606.26429#bib.bib43)\), Humanity’s Last Exam \(HLE\)\(Phanet al\.,[2025](https://arxiv.org/html/2606.26429#bib.bib39)\), OlympiadBench/Olympiad\-Math\(Heet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib40)\), and SimpleQA\(Weiet al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib41)\)\. Arena prompts are drawn from publicly available LMArena\(Chianget al\.,[2024](https://arxiv.org/html/2606.26429#bib.bib1)\)logs and filtered by domain as described below\.

#### Prompt sampling and filtering\.

Static items are drawn from each source benchmark by seeded stratified sampling, with strata defined by the natural axes of each source: difficulty tier for LiveCodeBench\-v6, problem type for AIME and Olympiad\-Math, HLE level for HLE\-Math, and field for HLE\-Misc \(Bio\./Med\., Engineering, Humanities/Social Science, Other\); SimpleQA is stratified by topic\. MBPP\-Plus, SWE\-Bench\-lite, and TerminalBench\-2\.0 are taken in full up to the per\-source quotas in Table[1](https://arxiv.org/html/2606.26429#S4.T1), after dropping items shorter than 150 characters\. Arena prompts are sampled from LMArena releases:arena\-expert\-5kfor coding, math, and miscellaneous, andarena\-human\-preference\-140kfor generic\. We first apply a coarse domain pre\-filter using the dataset\-provided occupational and category tags, then run an LLM judge \(gpt\-5\.4\-mini\) with a domain\-specific rubric that returns an inclusion label and a confidence score: coding requires the answer to depend on runnable code, math requires the core task to be mathematical reasoning or calculation, miscellaneous requires expert domain knowledge outside code/math, and generic requires a non\-technical everyday task with no code or math content\. Candidates with confidence below the per\-domain threshold \(0\.50\.5for coding/math/misc,0\.80\.8for generic\) are dropped, and the top\-NNsurviving prompts by judge confidence are retained to match the per\-domain arena pool sizes in Table[1](https://arxiv.org/html/2606.26429#S4.T1)\(coding300300, math200200, miscellaneous300300, generic500500\)\.

#### Evaluation setup\.

For each \(model, question\) pair we generate a single response with default sampling settings for each provider’s API as of submission \(temperature and top\-ppare model\-default; no system prompt is prepended for arena prompts; a standard “provide a complete, correct solution” instruction is appended for coding/math/misc\)\. Model identifiers are reported in Table[1](https://arxiv.org/html/2606.26429#S4.T1)\. For agentic SWE\-Bench tasks we route every model throughmini\-SWE\-agent\-v2with a single attempt and the harness’s default rollout budget; for TerminalBench\-2\.0 we useTerminus2under identical configuration for all models\.

#### Static grading\.

Coding tasks are graded by their reference execution harness \(test\-suite pass for MBPP\-Plus and LiveCodeBench; resolved\-instance flag for SWE\-Bench; harness pass for TerminalBench\)\. Math and miscellaneous items use exact\-answer matching after a normalization pass on the final\-answer line; ambiguous cases are resolved by an LLM judge with chain\-of\-thought disabled\.

#### Arena prompts\.

For coding, math, and miscellaneous we filter LMArena prompts to the target domain via LLM judges using a fixed rubric\. The generic domain uses LM Arena’s category\-mix without domain filtering\. Per\-domain arena pool sizes also appear in Table[1](https://arxiv.org/html/2606.26429#S4.T1)\.

#### Reward scoring\.

Each model response is scored by a single forward pass of the relevant reward model \(proprietary orSkywork\-Reward\-V2\-Qwen3\-8Bfor the replication\)\. Rewards are global\-standardised per \(domain, RM\) before pairwise targets are constructed\.

## Appendix HDifficulty and Sharpness

Figure[11](https://arxiv.org/html/2606.26429#A8.F11)plots learned item difficultybqb\_\{q\}against log\-sharpnesslog⁡aq\\log a\_\{q\}for every domain, with points colored by benchmark source\. Static items show near\-zero or weakly negative correlation in every static\-anchored domain \(ρ=−0\.00,\+0\.13,−0\.36\\rho=\-0\.00,\+0\.13,\-0\.36for coding, math, misc\), confirming that difficulty and sharpness are largely independent properties of a static item: saturated items can be sharp, hard items can be flat, and Fisher information cannot be inferred from difficulty alone\. Arena items, by contrast, show a robust positive coupling \(ρ∈\[\+0\.58,\+0\.73\]\\rho\\in\[\+0\.58,\+0\.73\]across all four domains\), reflecting how continuous reward signals propagate both location and spread into the IRT parameters\. The jointDualEvalformulation accommodates both regimes without imposing a single structural assumption on either, and the contrast itself confirms that the framework’s learned parameters reflect the structure of the underlying outcome modality\.

![Refer to caption](https://arxiv.org/html/2606.26429v1/x11.png)Figure 11:Learned item difficulty versus sharpness across domains\. Static items show weak difficulty–sharpness coupling, while arena items exhibit a stronger positive association, highlighting modality\-specific item structure captured byDualEval\.
## Appendix IPair\-Treatment Ablation

We ablate the two pair\-treatment components introduced in §[3](https://arxiv.org/html/2606.26429#S3): both\-bad \(BB\) anchoring and tie filtering\. All variants are evaluated on the coding domain using the same 20% held\-out pair set constructed with the full configuration thresholds, so metrics are directly comparable\. The full model uses BB flagging withλb​b=0\.2\\lambda\_\{bb\}=0\.2and tie filtering\. Arena pair accuracy is measured on held\-out decisive pairs; BB\-AUC measures whether1−min⁡\(pi,pj\)1\-\\min\(p\_\{i\},p\_\{j\}\)ranks BB pairs above non\-BB pairs\.

VariantBB LossTie FilterArena Pair Acc\.Both\-Bad AUCFullDualEval✓✓0\.7780\.903No both\-bad loss–✓0\.7710\.622No both\-bad filtering–✓0\.7740\.627No tie filter✓–0\.7770\.899No both\-bad / no tie––0\.7740\.628Table 5:Coding\-domain ablation of pair treatment for arena comparisons\. Arena pair accuracy is measured on held\-out hard pairs\. Both\-bad AUC measures whether the fitted model truly identifies pairs where both responses receive low rewards\.Table[5](https://arxiv.org/html/2606.26429#A9.T5)shows that BB anchoring is the dominant choice: removing the BB loss drops BB\-AUC from0\.9030\.903to0\.6220\.622while leaving arena pair accuracy near0\.780\.78, and removing BB flagging entirely gives essentially the same degradation\. Tie filtering has a smaller, complementary effect — it slightly improves BB calibration when combined with the full model \(0\.903 vs 0\.899\) but removing it alone barely changes either metric\. The full configuration is used in all main\-text experiments\.

## Appendix JPublic Leaderboard Rankings

Table[6](https://arxiv.org/html/2606.26429#A10.T6)lists the public LMArena Text\-leaderboard ranks for the 18 models we evaluate, taken from the matched public Arena Text sub\-leaderboards used in Table 2 \(Coding, Math, Text\-Expert, Text\-Overall\)\. These are the external reference rankings against which Table[2](https://arxiv.org/html/2606.26429#S4.T2)’sPublicρ\\rhocolumn is computed\. Lower rank is better\. Public ranks were retrieved on the same date the paper’s arena prompts were sampled\.

Paper modelText\-CodingText\-MathText\-ExpertText\-Overallclaude\-opus\-4\-64523claude\-opus\-4\-72934gemini\-3\.1\-pro10786gpt\-5\.5294911claude\-sonnet\-4\-611281222gpt\-5\.421331524gpt\-5\.4\-mini38532739gemini\-2\.5\-pro81465649qwen3\-max\-thinking60525062deepseek\-v3\.276637176mistral\-large\-37511010593gpt\-4\.19714611894gemini\-2\.5\-flash1409910098claude\-haiku\-4\-5641256999grok\-41216892100gpt\-5\-mini134101122124gpt\-4\.1\-mini127163143135llama\-4\-maverick\-instruct186180181197Table 6:Public Arena ranks for the 18 models used in our external validation analysis\. Ranks are taken from the matched public Arena Text subleaderboards used in Table[2](https://arxiv.org/html/2606.26429#S4.T2): Text\-Coding, Text\-Math, Text\-Expert, and Text\-Overall\. Lower rank is better\.
## Appendix KContamination\-Detection Metric Details

For each domain, modality, injection rate, and random seed, the synthetic contamination protocol defines a binary maskmi,q∈\{0,1\}m\_\{i,q\}\\in\\\{0,1\\\}over model–item cells, wheremi,q=1m\_\{i,q\}=1denotes an injected cell\. Each detector assigns a scalar anomaly scoresi,qs\_\{i,q\}to every cell using the residual scores defined in §[4\.3](https://arxiv.org/html/2606.26429#S4.SS3)\. We then evaluate whether these scores rank injected cells above non\-injected cells\.

AUROC is computed as the probability that a randomly selected injected cell receives a higher score than a randomly selected non\-injected cell, with ties counted halfway:

AUROC=Pr⁡\(s\+\>s−\)\+12​Pr⁡\(s\+=s−\),\\mathrm\{AUROC\}=\\Pr\(s^\{\+\}\>s^\{\-\}\)\+\\tfrac\{1\}\{2\}\\Pr\(s^\{\+\}=s^\{\-\}\),wheres\+s^\{\+\}ands−s^\{\-\}denote scores from injected and non\-injected cells, respectively\.

AUPRC is the area under the precision–recall curve obtained by thresholding the anomaly scores\. For a thresholdtt, precision and recall are

Prec​\(t\)=∑i,q𝟏​\[si,q≥t\]​mi,q∑i,q𝟏​\[si,q≥t\],\\mathrm\{Prec\}\(t\)=\\frac\{\\sum\_\{i,q\}\\mathbf\{1\}\[s\_\{i,q\}\\geq t\]m\_\{i,q\}\}\{\\sum\_\{i,q\}\\mathbf\{1\}\[s\_\{i,q\}\\geq t\]\},Rec​\(t\)=∑i,q𝟏​\[si,q≥t\]​mi,q∑i,qmi,q\.\\mathrm\{Rec\}\(t\)=\\frac\{\\sum\_\{i,q\}\\mathbf\{1\}\[s\_\{i,q\}\\geq t\]m\_\{i,q\}\}\{\\sum\_\{i,q\}m\_\{i,q\}\}\.We compute the precision–recall curve over all score thresholds and report the area under this curve\. Both AUROC and AUPRC are computed at the cell level and averaged over random injection seeds for each injection rate and modality\.

## Appendix LExamples

#### Item\-Profile Gallery\.

Table[7](https://arxiv.org/html/2606.26429#A12.T7)walks the four corners of the \(difficultybqb\_\{q\}, sharpnessaqa\_\{q\}\) plane on coding\-domain static items\. The example items are drawn directly from the full\-dataDualEvalfit; pass counts are observed across the 18 evaluated models\. This gallery illustrates qualitatively what the Fisher subset experiment in §4\.2 measures quantitatively: ranking signal lives in items where bothaqa\_\{q\}is high and the pass rate is between the extremes, and items in the saturated and uniformly hard corners contribute essentially no ranking information\.

ItemPass /aqa\_\{q\}A\. Frontier\-discriminative\.LCB\-v6 “Stone XOR”\.
“There areNNbags labeled1,…,N1,\\ldots,N\. BagiicontainsAiA\_\{i\}stones\. Choose two bagsAA,BB, and move all stones fromAAtoBB…”
*Hard but very sharp \(b=0\.36b\\\!=\\\!0\.36,aq=72\.7a\_\{q\}\\\!=\\\!72\.7\)\. Only 5 of 18 models pass; the joint fit identifies them as systematically the strongest — large Fisher information for ranking the frontier\.*5/185/18aq=72\.7a\_\{q\}\\\!=\\\!72\.7B\. Uniformly hard\.LCB\-v6 “Min of Restricted Sum”\.
“You are given integersNN,MMand three integer sequences of lengthMM…”
*Hardest tail of the difficulty distribution \(b=1\.57b\\\!=\\\!1\.57\); 0 of 18 models pass\. The item provides no ranking signal — no variation in outcomes to attribute to ability differences\.*0/180/18aq=4\.4a\_\{q\}\\\!=\\\!4\.4C\. Borderline\-discriminative\.LCB\-v6 “Unique 3\-Digit Even Numbers”\.
“You are given an array of digits\. Determine the number of distinct three\-digit even numbers that can be formed using these digits…”
*Easy \(b=−0\.48b\\\!=\\\!\-0\.48\) but sharp \(aq=31\.2a\_\{q\}\\\!=\\\!31\.2\)\. 16 of 18 models pass; the 2 misses are the two weakest\. Informative for separating trailing models, not the frontier\.*16/1816/18aq=31\.2a\_\{q\}\\\!=\\\!31\.2D\. Saturated\.MBPP\-Plus “left insertion point”\.
“Write a Python function to locate the left insertion point for a specified value in sorted order…”
*Trivially easy \(b=−1\.44b\\\!=\\\!\-1\.44\), 18 of 18 models pass\. Contributes nothing to ranking; a candidate for retirement under benchmark refresh\.*18/1818/18aq=5\.0a\_\{q\}\\\!=\\\!5\.0Table 7:Item\-profile gallery \(coding domain\) walking the four corners of the \(difficultybqb\_\{q\}, discriminationaqa\_\{q\}\) plane\. Pass counts are out of the 18 evaluated models\. Frontier\-discriminative items \(A\) carry the bulk of ranking signal; uniformly hard \(B\) and saturated \(D\) items contribute none and are candidates for retirement; borderline\-discriminative items \(C\) inform only the low\-ability tail\. The Fisher\-information subset experiment \(Fig\.[2](https://arxiv.org/html/2606.26429#S4.F2)\) operationalizes this picture quantitatively\.
#### Anomaly Candidates\.

Table[8](https://arxiv.org/html/2606.26429#A12.T8)lists representative anomaly candidates surfaced by the standardizedDualEvalresidual diagnostic of §[4\.3](https://arxiv.org/html/2606.26429#S4.SS3), restricted to natural \(un\-perturbed\) responses with\|ri,q\|\>3\.5​σ\|r\_\{i,q\}\|\>3\.5\\sigma\. These are candidates, not verified contamination: large residuals can also reflect MC guessing on items with few options, judge artifacts, or genuine but narrow capability not captured by the joint fit\. Semantic\-preserving perturbation audits are the appropriate follow\-up to distinguish memorization from robust capability and are left to future work\. The patterns observed across the three picks \(number\-sequence puzzle archives, specialized MC questions, named MBPP function signatures\) are nevertheless consistent with prior exposure as a plausible alternative hypothesis\.

Anomaly candidatep^/r\\hat\{p\}\\,/\\,rHLE\-Math/gpt\-4\.1\(rank 16/18\)\.
“2, 11, 23, 51, 119, \( \) A\. 291 B\. 285 C\. 171 D\. 167\.”
*MC sequence\-completion item; the pattern is widely archived \(puzzle banks/OEIS\) and a weak math model recovering the correct option is consistent with prior exposure\.*0\.040\.04\+4\.81\+4\.81HLE\-Bio/Med/llama\-4\-maverick\-instruct\(18/18\)\.
“Which genetic disorder caused by mutations on chromosome 2 leads to the greatest increases to patients’ basal metabolic rate? A\. Alström B\. Menkes…”
*Specialized medical MC\. The lowest\-ranked model passing where frontier models fail is\>5​σ\>\\\!5\\sigma; consistent with recall from medical\-Q&A corpora\.*0\.030\.03\+5\.36\+5\.36MBPP\-Plus/llama\-4\-maverick\-instruct\(18/18\)\.
“Write a python function to find the sum of the per\-digit difference between two integers\. Your function must be nameddigit\_distance\_nums\.”
*MBPP is among the most\-trained\-on coding benchmarks; the named signature suggests recall rather than synthesis from the spec\.*0\.070\.07\+3\.57\+3\.57Table 8:Anomaly candidates surfaced by standardizedDualEvalresidualsri,q=\(yi,q−p^i,q\)/p^i,q​\(1−p^i,q\)r\_\{i,q\}=\(y\_\{i,q\}\-\\hat\{p\}\_\{i,q\}\)/\\sqrt\{\\hat\{p\}\_\{i,q\}\(1\-\\hat\{p\}\_\{i,q\}\)\}on un\-perturbed responses \(yi,q=1y\_\{i,q\}=1for all rows\)\.