More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges

# Self-Play Reward Hacking of Reference-Free LLM Judges
Source: [https://arxiv.org/html/2607.05904](https://arxiv.org/html/2607.05904)
## More Convincing, Not More Correct: Self\-Play Reward Hacking of Reference\-Free LLM Judges

###### Abstract

Training a language model against its own reference\-free judgments—the premise of self\-rewarding, self\-play, and LLM\-as\-a\-judge pipelines for label\-free self\-improvement—assumes a model’s verdict on a shown answer is a usable proxy for correctness\. We show the premise fails structurally: conditioned on a candidate, a reference\-free judge scores*plausibility*, not correctness—a*verification asymmetry*that leaves*false\-positive basins*of plausible\-but\-wrong answers a policy learns to exploit\. We measure the failure with a*hidden\-anchor audit*: a held\-out, cross\-source exact\-match check the judge never sees\. On GSM8K with Qwen3 policies in a reasoning\-suppressed regime, self\-play drives the judge’s pass rate from0\.720\.72to0\.940\.94while true accuracy stays at0\.200\.20—a0\.740\.74judge–truth gap \(three seeds\)\. This reward hacking is not white\-box gaming: the manufactured errors transfer across judge families \(Qwen, Llama, Gemma\) and scales \(to 14B\), a strict three\-family ensemble still accepts55%55\\%of them, and recompute prompts, stronger judges, and training directly against the ensemble reward all fail to close the basin\. The decisive variable is not whether the judge sees the candidate but whether it commits an answer of its own first: the recompute prompt leaves the false\-positive rate on wrong answers at0\.7190\.719, committing first—candidate still in view—drops it to0\.0120\.012, and blind solving lifts discrimination from near chance to0\.960\.96with no reference answer\. Used as the training reward, the de\-anchored channel keeps the false\-positive rate at zero across self\-play, preventing the basin rather than only detecting it\. A falsifiable bound explains which regimes are exposed—the gap is at most1−accuracy1\-\\text\{accuracy\}—and the de\-anchored reward obeys the analogous judge\-side bound\. The full arc replicates without training under best\-of\-NNselection in code and competition math, and in the full loop with a Gemma policy\.

## 1Introduction

Reinforcement learning from a model’s own evaluations has become a central recipe for improving language models without human labels\. Reference\-free “LLM\-as\-a\-judge” rewards, self\-reward, and self\-play schemes share one premise: a model’s judgment of an answer is a usable proxy for its correctness, and a growing line of methods builds on it as a default assumption\(Bai et al\.,[2022](https://arxiv.org/html/2607.05904#bib.bib1); Lee et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib9); Yuan et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib18); Chen et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib2); Simonds et al\.,[2025](https://arxiv.org/html/2607.05904#bib.bib14); Huang et al\.,[2026](https://arxiv.org/html/2607.05904#bib.bib7)\)\.

The premise has a structural flaw\. A reference\-free judge has no access to ground truth; it can only assess whether an answer*looks*correct\. For tasks where verifying an answer is harder than recognizing a plausible one—most of reasoning—this is a*verification asymmetry*: the judge scores plausibility, not correctness, leaving a*false\-positive basin*of plausible\-but\-wrong answers it accepts\. Optimizing a policy against such a judge does not merely risk noise; it actively rewards finding the basin\.

We measure it with a*hidden anchor*: a held\-out, cross\-source exact\-match check on the final answer that the judge never sees and is never trained against\. On GSM8K with Qwen3 policies trained by self\-play, the audit reveals a large divergence: on the full test set, in a reasoning\-suppressed regime that holds accuracy low, the judge’s pass rate climbs from≈0\.72\{\\approx\}0\.72to0\.940\.94while anchor\-verified accuracy stays flat at≈0\.20\{\\approx\}0\.20—a0\.740\.74judge–truth gap \(three seeds; §[5](https://arxiv.org/html/2607.05904#S5)\); Figure[1](https://arxiv.org/html/2607.05904#S5.F1)a shows the five\-iteration trajectory\. Self\-play does not make the model more correct; it makes the model’s errors more convincing\.

This is not a quirk of one judge that a stronger or more diverse judge would catch\. Re\-scoring the hacked answers with independent judges from other families \(Llama, Gemma\) and larger scales \(up to 14B\) shows the same inflation; a three\-family ensemble that accepts only when all judges agree still passes55%55\\%of the hacked wrong answers, its discrimination collapsing from0\.310\.31to0\.090\.09, and training the policy directly against the ensemble reward does not help\. The errors are a*shared blind spot*of reference\-free judging: self\-play acts as an adversarial\-example generator for the basin\. What restores oversight is breaking the judge’s*anchoring*to the candidate: the same model that rubber\-stamps these errors when scoring a shown answer rejects almost all of them once made to commit an independent answer first, even with the candidate still in view\. The decisive variable is the judge’s independence from the candidate, not its capability and not whether the candidate is visible\.

Two further results characterize the phenomenon: the judge–truth gap is falsifiably bounded by1−accuracy1\-\\text\{accuracy\}, so low\-accuracy regimes are exposed while high\-accuracy ones show little exploitable gap; and an oracle control that rewards exact match instead of the judge shows no inflation, attributing the failure to the judge\-reward rather than to preference optimization\.

Our contributions are:

- •A shared blind spot across reference\-free judges\(§[5](https://arxiv.org/html/2607.05904#S5)\): self\-play manufactures wrong answers that transfer across judge families and scales, defeat scoring\-time and train\-time ensembles, and reappear without any training under best\-of\-NNselection \(§[5\.3](https://arxiv.org/html/2607.05904#S5.SS3)\) and with a Gemma policy in the full loop \(§[5\.4](https://arxiv.org/html/2607.05904#S5.SS4)\)\.
- •The hidden\-anchor audit\(§[3](https://arxiv.org/html/2607.05904#S3)\): a held\-out cross\-source exact\-match probe that quantifies a judge’s over\-reporting under optimization, with gap, drift, discrimination, and risk\-score diagnostics\.
- •A de\-anchoring fix\(§[5](https://arxiv.org/html/2607.05904#S5)\): requiring the judge to commit an answer of its own before using the candidate collapses the false\-positive rate \(0\.719→0\.0120\.719\\\!\\to\\\!0\.012\) and, used as the training reward, prevents the basin\.
- •A predictive account\(§[4](https://arxiv.org/html/2607.05904#S4)\): a falsifiable boundVA\-Gap≤1−EM\\textsc\{VA\-Gap\}\\leq 1\-\\text\{EM\}verified across formats, tasks, and optimization modes; an independence bound \(Prop\.[1](https://arxiv.org/html/2607.05904#Thmproposition1)\) whose measured excess detects anchoring and quantifies it in bits \(Cors\.[1](https://arxiv.org/html/2607.05904#Thmcorollary1)–[2](https://arxiv.org/html/2607.05904#Thmcorollary2)\); and a no\-escape result for monotone aggregation \(Prop\.[2](https://arxiv.org/html/2607.05904#Thmproposition2)\)\.

## 2Related Work

#### LLM\-as\-a\-judge and its biases\.

Using a strong LLM to judge open\-ended responses is standard practice\(Zheng et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib22)\), and its biases are documented: judges favor longer\(Singhal et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib15)\)and more agreeable answers, andSharma et al\. \([2023](https://arxiv.org/html/2607.05904#bib.bib13)\)show that both humans and preference models prefer convincingly written responses over correct ones\. We give this preference\-for\-plausibility a structural account and measure its consequence under self\-play optimization\.

#### Reference\-free self\-improvement\.

Training against AI\-generated feedback\(Bai et al\.,[2022](https://arxiv.org/html/2607.05904#bib.bib1); Lee et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib9)\)underlies a growing family of self\-rewarded\(Yuan et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib18)\)and self\-play\(Chen et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib2)\)methods\.Simonds et al\. \([2025](https://arxiv.org/html/2607.05904#bib.bib14)\)report that reinforcement learning from a model’s self\-assigned reward improves performance without references; our audit identifies the regimes where this signal is corrupted\.Huang et al\. \([2026](https://arxiv.org/html/2607.05904#bib.bib7)\)propose a co\-evolutionary scheme that routes around proxy\-judge reward hacking; our results empirically confirm the threat that motivates that design\.Lu et al\. \([2025](https://arxiv.org/html/2607.05904#bib.bib10)\)find cross\-family verification more reliable than self\-verification; we ask the complementary question of whether a reference\-free verdict is independent of the candidate it scores\.

#### Fooling LLM judges\.

Zhao et al\. \([2025b](https://arxiv.org/html/2607.05904#bib.bib21)\)show that trivial surface tokens elicit false positives from strong judges, whichZhao et al\. \([2025a](https://arxiv.org/html/2607.05904#bib.bib20)\)hardens against\. These are static, surface attacks; the errors self\-play produces are semantic and dynamic—optimized into the judge’s basin—an orthogonal failure mode that recompute\-style prompting does not remove \(§[5](https://arxiv.org/html/2607.05904#S5)\)\.Pan et al\. \([2024](https://arxiv.org/html/2607.05904#bib.bib11)\)observe judge\-score inflation when one model generates and judges in self\-refinement; our errors transfer across families, obey a falsifiable bound, and are preventable by de\-anchoring\.

#### Scalable oversight and reward over\-optimization\.

Kenton et al\. \([2024](https://arxiv.org/html/2607.05904#bib.bib8)\)find debate outperforms consultancy for weak judges supervising stronger models, andWen et al\. \([2025](https://arxiv.org/html/2607.05904#bib.bib16)\)show RLHF teaches models to mislead*human*evaluators; we study the reference\-free LLM\-judge analogue with a structural account, a diagnostic, and a fix\. Reward over\-optimization with a same\-source proxy is well documented\(Gao et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib6); Rafailov et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib12)\), and reward\-model ensembles mitigate but do not eliminate hacking\(Coste et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib3); Eisenstein et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib4)\)\. We add a cross\-family*transfer*axis with a held\-out anchor, and show that scoring\-level ensembles do not survive adversarial self\-play\.

## 3The Hidden\-Anchor Audit

#### Setup\.

A policyπ\\pianswers a questionqqwith a final answeraaand a reasoning trace\. A reference\-free judgeJJ—an LLM givenqqandaabut*no*reference answer—returns a score that we threshold into accept/reject\. The policy is optimized to increaseJJ’s acceptance: in our case study by self\-play, where one model generates candidates, judges them, and is preference\-optimized \(DPO\) on the resulting pairs\. We attach to every question a*hidden anchor*AA: a held\-out cross\-source exact\-match check on the final answer, drawn from an independent set of ground\-truth solutions\. The anchor is never shown toJJ, never in any prompt, never a training signal: it exists only to audit\.

#### Metrics\.

Letpt=Pr⁡\[J​accepts\]p\_\{t\}=\\Pr\[J\\text\{ accepts\}\]be the judge’s pass rate at optimization stepttandEMt=Pr⁡\[A​correct\]\\mathrm\{EM\}\_\{t\}=\\Pr\[A\\text\{ correct\}\]the anchor accuracy\. The primary quantity is the*judge–truth gap*, orVA\-Gap,pt−EMt\\,p\_\{t\}\-\\mathrm\{EM\}\_\{t\}\. BecauseAAis held out and cross\-source, any rise inptp\_\{t\}unmatched by a rise inEMt\\mathrm\{EM\}\_\{t\}is unambiguous over\-reporting rather than a measurement artifact—this is what makes the audit falsifiable\. We further track the false\-positive rate on wrong answersFPR=Pr⁡\[J​accepts∣A​wrong\]\\mathrm\{FPR\}=\\Pr\[J\\text\{ accepts\}\\mid A\\text\{ wrong\}\]and its drift across steps, the true\-positive rateTPR=Pr⁡\[J​accepts∣A​correct\]\\mathrm\{TPR\}=\\Pr\[J\\text\{ accepts\}\\mid A\\text\{ correct\}\], and the judge’s*discrimination*TPR−FPR\\mathrm\{TPR\}\-\\mathrm\{FPR\}\. For a panel ofNNjudges we report the strictest aggregation,Min\(all must accept\)\. An ordinal*risk score*FPRbase⋅\(1−EM\)\\mathrm\{FPR\}\_\{\\text\{base\}\}\\cdot\(1\-\\mathrm\{EM\}\)rank\-orders settings by vulnerability to reward hacking before any optimization is run\.

## 4Verification Asymmetry and False\-Positive Basins

We model the judgeJJas a noisy binary classifier of correctness with true\-positive rateTPR=1−FNR\\mathrm\{TPR\}=1\-\\mathrm\{FNR\}\(accept∣\\midcorrect\) and false\-positive rateFPR\\mathrm\{FPR\}\(accept∣\\midwrong\)\. Its pass rate decomposes asp=EM​\(1−FNR\)\+\(1−EM\)​FPRp=\\mathrm\{EM\}\\,\(1\-\\mathrm\{FNR\}\)\+\(1\-\\mathrm\{EM\}\)\\,\\mathrm\{FPR\}, so the judge–truth gap is exactly

VA\-Gap≡p−EM=\(1−EM\)​FPR−EM​FNR\.\\textsc\{VA\-Gap\}\\;\\equiv\\;p\-\\mathrm\{EM\}\\;=\\;\(1\-\\mathrm\{EM\}\)\\,\\mathrm\{FPR\}\\;\-\\;\\mathrm\{EM\}\\,\\mathrm\{FNR\}\.\(1\)Self\-play optimizes wrong answers to be accepted: it pushesFPR\\mathrm\{FPR\}upward—the false\-positive drift measured in §[5](https://arxiv.org/html/2607.05904#S5)—while leaving true accuracyEM\\mathrm\{EM\}essentially unchanged, soVA\-Gapgrows with\(1−EM\)​FPR\(1\-\\mathrm\{EM\}\)\\,\\mathrm\{FPR\}and, sinceFPR≤1\\mathrm\{FPR\}\\leq 1, obeys a falsifiable*upper bound*at any optimization step,

VA\-Gap≤\(1−EM\)−EM​FNR≤1−EM\.\\textsc\{VA\-Gap\}\\;\\leq\\;\(1\-\\mathrm\{EM\}\)\-\\mathrm\{EM\}\\,\\mathrm\{FNR\}\\;\\leq\\;1\-\\mathrm\{EM\}\.\(2\)Because self\-play preserves accuracy, the operational ceiling is1−EMbase1\-\\mathrm\{EM\}\_\{\\text\{base\}\}: the gap is capped by the policy’s error headroom, explaining the capability dependence we observe\. The bound is structural—a noisier judge, a weaker policy, or a harder task all lowerEM\\mathrm\{EM\}and expose the same basin; reasoning suppression is our controlled instrument to dialEM\\mathrm\{EM\}on a fixed task\. The audit tests the bound’s*tightness*: Table[1](https://arxiv.org/html/2607.05904#S4.T1)shows the gap approaches the ceiling precisely in the exposed regimes and stays far below it elsewhere\.

This bound governs the*plausibility*channel, in which the judge is shown a candidate and scores it\. The same model can instead commit to its own answer independently of the candidate—in the limit, solving blind—which caps its false positives by its own solver error instead\.

###### Proposition 1\(Independence bound\)\.

Suppose the judge produces its own final answer independently of the candidate—e\.g\., by committing to it before the candidate is used—and accepts only on an exact match\. On a wrong candidate it accepts only when it independently produces that same wrong answer, an event no more likely than its own error, soFPR≤1−solve\-acc\\mathrm\{FPR\}\\leq 1\-\\text\{solve\-acc\}, wheresolve\-accis the judge’s accuracy on problems where the candidate is wrong\.

###### Corollary 1\(Anchoring is detectable\)\.

Under exact\-match acceptance, a judge whose measuredFPR\\mathrm\{FPR\}exceeds1−solve\-acc1\-\\text\{solve\-acc\}must violate independence: its verdicts are anchored, and the excess over the bound quantifies the anchoring using only the judge’s own solve accuracy\.

The excess is not only a diagnostic; it is an information measure \(proof in Appendix[B](https://arxiv.org/html/2607.05904#A2)\)\.

###### Corollary 2\(Anchoring in bits\)\.

Under the acceptance model of Proposition[1](https://arxiv.org/html/2607.05904#Thmproposition1), letSSbe the judge’s committed answer andΔ=FPR−\(1−solve\-acc\)\\Delta=\\mathrm\{FPR\}\-\(1\-\\text\{solve\-acc\}\)the measured excess on wrong candidates\. Then the conditional mutual information between the committed answer and the candidate satisfiesI​\(S;A∣Q\)≥2​Δ2I\(S;A\\mid Q\)\\ \\geq\\ 2\\Delta^\{2\}nats\. Insofar as the verify\-prompted judge implements its instructed solve\-then\-compare procedure, itsFPR=0\.719\\mathrm\{FPR\}=0\.719against the0\.070\.07ceiling certifiesI≥0\.84I\\geq 0\.84nats \(≥1\.2\{\\geq\}1\.2bits\) of leakage from the candidate into the judge’s “own” solution\.

The section’s results form a dichotomy for reference\-free rewards\. A verdict either commits an answer independently of the candidate—inheriting the judge\-side ceiling1−solve\-acc1\-\\text\{solve\-acc\}\(Proposition[1](https://arxiv.org/html/2607.05904#Thmproposition1)\)—or it is candidate\-conditioned, in which case nothing protects its false\-positive rate, the judge–truth gap is bounded above by the policy\-side ceiling1−EM1\-\\mathrm\{EM\}\(Eq\.[2](https://arxiv.org/html/2607.05904#S4.E2)\)—a ceiling self\-play measurably approaches \(Table[1](https://arxiv.org/html/2607.05904#S4.T1)\)—and no monotone aggregation escapes the basin \(Proposition[2](https://arxiv.org/html/2607.05904#Thmproposition2)below\)\. The class is decided by the committed answer, not the comparison: a commit\-first verdict may use the candidate to compare, so long as the commitment itself does not\. The ceilings differ by an order of magnitude here \(0\.80\.8versus0\.070\.07\), class membership is measurable \(Corollary[1](https://arxiv.org/html/2607.05904#Thmcorollary1)\), and §[5](https://arxiv.org/html/2607.05904#S5)shows self\-play saturates the anchored channel while the independent one stays intact \(0\.7190\.719versus0\.0120\.012against the0\.070\.07ceiling\)\.

#### Ensembling cannot escape the basin\.

Aggregating several reference\-free judges cannot restore reliability: all of them score how plausible an answer looks, so their verdicts are driven by a common signal\. Writingqi​\(s\)=Pr⁡\[Ji​accepts∣wrong,s\]q\_\{i\}\(s\)=\\Pr\[J\_\{i\}\\text\{ accepts\}\\mid\\text\{wrong\},s\]for a shared plausibility signalssand assuming eachqiq\_\{i\}is non\-decreasing inssand conditionally independent givenss,111Conditional independence is the*best case*for the ensemble: any residual positive association between judges beyond the shared signal only raisesFPRMin\\mathrm\{FPR\}\_\{\\text\{\{Min\}\}\}further, so Eq\. \([3](https://arxiv.org/html/2607.05904#S4.E3)\) is a conservative floor rather than a fragile assumption\.the correlation inequality for monotone functions\(Esary et al\.,[1967](https://arxiv.org/html/2607.05904#bib.bib5)\)gives, for the strictestMinrule,

FPRMin=𝔼s​\[∏iqi​\(s\)\]≥∏i𝔼s​\[qi​\(s\)\]=∏iFPRi\.\\mathrm\{FPR\}\_\{\\text\{\{Min\}\}\}=\\mathbb\{E\}\_\{s\}\\\!\\Big\[\\textstyle\\prod\_\{i\}q\_\{i\}\(s\)\\Big\]\\;\\geq\\;\\prod\_\{i\}\\mathbb\{E\}\_\{s\}\\big\[q\_\{i\}\(s\)\\big\]=\\prod\_\{i\}\\mathrm\{FPR\}\_\{i\}\.\(3\)The ensemble can do no better than the independent product, and strictly worse under positive dependence\. The obstruction is not specific to theMinrule \(proof in Appendix[B](https://arxiv.org/html/2607.05904#A2)\):

###### Proposition 2\(Monotone aggregation shares the basin\)\.

Under the shared\-signal model above, letggbe any non\-decreasing aggregation rule mapping theNNverdicts to accept/reject withg​\(𝟏\)=1g\(\\mathbf\{1\}\)=1\. Then the ensemble’s acceptance probabilityhg​\(s\)h\_\{g\}\(s\)is non\-decreasing in the shared signalss, andhg​\(s\)≥∏iqi​\(s\)h\_\{g\}\(s\)\\geq\\prod\_\{i\}q\_\{i\}\(s\), sohg​\(s\)→1h\_\{g\}\(s\)\\to 1wherever every judge’sqi​\(s\)→1q\_\{i\}\(s\)\\to 1\.

Every monotone rule therefore thresholds the same plausibility axis: an upward \(stochastic\-dominance\) shift of the wrong\-answer signal cannot lower, and generically raises, the false\-positive rate of every monotone aggregator simultaneously, and no monotone rule can reject the high\-ssregion that constitutes the basin\. Self\-play makes the bounds bite: it drives the shared signal upward, concentrating wrong answers in the region every judge jointly accepts—the shared false\-positive basin\. Our measurements confirm the dependent regime directly: the three judges’ acceptances of wrong answers are pairwise positively correlated \(ϕ=0\.29\\phi\{=\}0\.29–0\.380\.38\), and581581wrong answers are accepted unanimously where independence predicts≈497\{\\approx\}497\(Appendix[C\.1](https://arxiv.org/html/2607.05904#A3.SS1)\); training directly against theMinreward moves a held\-out judge*more*\(§[5](https://arxiv.org/html/2607.05904#S5)\), exactly the shared\-axis motion Proposition[2](https://arxiv.org/html/2607.05904#Thmproposition2)describes\. The basin closes only if some judge’s rejections cover the whole reachable plausible\-wrong region—which scoring plausibility cannot do but independent verification can\.

Table 1:The noisy\-judge upper boundVA\-Gap≤1−EMbase\\textsc\{VA\-Gap\}\\leq 1\-\\mathrm\{EM\}\_\{\\text\{base\}\}\(Eq\.[2](https://arxiv.org/html/2607.05904#S4.E2)\) holds and is approached across settings; the last column is the post\-self\-play FPR on wrong answers\. CoT uses the OOD CoT\-GSM\-Plus split; TruthfulQA does not hack, so no post\-FPR is reported\.

## 5Experiments

#### Setup\.

We study GSM8K grade\-school math with Qwen3 policies \(1\.71\.7–1414B\)\. In self\-play, one model generates candidate answers, judges its own answers for correctness without a reference, and is preference\-optimized \(DPO, accept≻\\succreject\) on the result; we run two iterations and three seeds unless noted\.222Headline and cross\-family transfer: fulln=1319n\{=\}1319test set, three seeds \(gap0\.735±0\.0110\.735\{\\pm\}0\.011and every discrimination collapse reproduce in all three\)\. Capability sweep and mechanism analysis: fixedn=128n\{=\}128audit subset, three seeds; five\-iteration, cross\-family\-judge, and train\-time\-ensemble runs likewise\. The OOD CoT self\-play is a single\-seed audit\.Answers are produced in a reasoning\-suppressed JSON format by default\. Reasoning suppression is a*capability\-frontier instrument*: it lowers the policy’s accuracy on a fixed task with the task family and judge held fixed, exposing the error headroom that the bound of §[4](https://arxiv.org/html/2607.05904#S4)predicts governs the gap\. Chain\-of\-thought \(CoT\) generation and the non\-math factual task TruthfulQA serve as high\-accuracy controls\. The hidden anchor is the held\-out exact\-match check on the final answer\. Figure[1](https://arxiv.org/html/2607.05904#S5.F1)summarizes the audit across all settings\.

![Refer to caption](https://arxiv.org/html/2607.05904v1/main_figure.png)Figure 1:\(a\) Five self\-play iterations: the judge’s pass rate climbs to∼0\.9\{\\sim\}0\.9and holds while anchor\-verified accuracy stays statistically flat \(n=128n\{=\}128audit per iteration, three seeds; thin lines seeds, bold mean\)\. \(b\)VA\-Gapversus base accuracy: hacking concentrates at low accuracy\. \(c\)VA\-Gapversus riskFPRbase​\(1−EM\)\\mathrm\{FPR\}\_\{\\text\{base\}\}\(1\-\\mathrm\{EM\}\), an ordinal early\-warning; hollow hexagon==natural\-code*base*point\.
### 5\.1The problem: self\-play manufactures a shared, defense\-resistant blind spot

#### Self\-play inflates the judge, not accuracy\.

On the full GSM8K test set \(n=1319n\{=\}1319, 4B, JSON, mean±\{\\pm\}std over three seeds\), self\-play drives the judge’s pass rate from0\.7160\.716to0\.938±0\.016\\mathbf\{0\.938\{\\pm\}0\.016\}while anchor accuracy stays at0\.209→0\.202±0\.0050\.209\\\!\\to\\\!0\.202\{\\pm\}0\.005, a judge–truth gap of0\.735±0\.011\\mathbf\{0\.735\{\\pm\}0\.011\}: the judge reports94%94\\%correct when20%20\\%are\. Extending to five self\-play iterations across three seeds \(Figure[1](https://arxiv.org/html/2607.05904#S5.F1)a\), the judge holds near0\.90\.9while held\-out accuracy stays flat \(meanΔ​EM=\+0\.024±0\.035\\Delta\\mathrm\{EM\}\{=\}\{\+\}0\.024\{\\pm\}0\.035,n=128n\{=\}128\); the gap is not an artifact of a single update\.

#### Severity is capability\-dependent\.

Across a Qwen3 size sweep \(3 seeds, JSON\), the gap is0\.530\.53–0\.640\.64at base accuracies of0\.180\.18–0\.420\.42\(Table[3](https://arxiv.org/html/2607.05904#A3.T3), Appendix\)\. Under CoT generation \(base accuracy0\.760\.76–0\.880\.88\) the gap falls to0\.0860\.086–0\.1250\.125, and on TruthfulQA \(base truthfulness0\.890\.89\) it stays slightly negative \(−0\.041\-0\.041\)\. On MATH level 4–5 under natural chain\-of\-thought, where Qwen3\-4B sits in a high\-accuracy regime \(EM=0\.637\\mathrm\{EM\}\{=\}0\.637\), the base gap is0\.0860\.086: benchmark difficulty alone is not predictive; the governing variable is the policy’s realized error headroom\. A cross\-policy\-family replication with Llama\-3\.1\-8B as both policy and self\-judge shows the same decoupling \(three seeds atn=128n\{=\}128\), weaker in the direction the risk score predicts: Llama’s strict self\-judge \(baseFPR=0\.13\\mathrm\{FPR\}\{=\}0\.13on its own rollouts, versus Qwen3\-4B’s0\.650\.65\) leaves little false\-positive headroom\. The risk scoreFPRbase​\(1−EM\)\\mathrm\{FPR\}\_\{\\text\{base\}\}\(1\-\\mathrm\{EM\}\)rank\-orders severity across formats, families, and policies, exactly as the bound requires\.

#### The asymmetry persists in organic code generation\.

The capability dependence is not specific to math or format: on natural chain\-of\-thought code generation \(LiveCodeBench, Qwen3\-1\.7B as policy and self\-judge, held\-out unit\-test anchor\) the base gap is0\.275±0\.0300\.275\{\\pm\}0\.030atEM=0\.301±0\.031\\mathrm\{EM\}\{=\}0\.301\{\\pm\}0\.031\(FPR=0\.445±0\.037\\mathrm\{FPR\}\{=\}0\.445\{\\pm\}0\.037, 3 seeds\)—within the bound and in the high\-risk regime the score flags \(≈0\.31\{\\approx\}0\.31\)\. The judge rationalizes structurally clean but wrong programs: it scores plausibility rather than executing the code\.

#### Mechanism: self\-play makes errors more convincing\.

Freezing the base judge and measuring its false\-positive rate on the policy’s*wrong*answers, the rate rises sharply from iter0to iter1\(on then=128n\{=\}128audit subset\) while the number of wrong answers barely changes:0\.65→0\.890\.65\\\!\\to\\\!0\.89\(4B\),0\.44→0\.93\\mathbf\{0\.44\\\!\\to\\\!0\.93\}\(8B\),0\.60→0\.890\.60\\\!\\to\\\!0\.89\(14B\); the strictest base judge is driven to the highest post\-hoc false\-positive rate\. A format\-blind check rules out surface gaming: iter1false positives are*shorter*on average than iter0and structurally clean but arithmetically wrong—the errors are semantic, not surface artifacts\.

#### The inflation is caused by the judge\-reward\.

An oracle control rewards exact match instead of the judge, holding the algorithm and data fixed: judge\-reward DPO inflates the judge \(\+0\.14\+0\.14\) with flat accuracy, while oracle exact\-match DPO leaves the judge statistically flat \(gap shift<0\.01\{<\}0\.01across three seeds\) and raises accuracy \(\+0\.03\+0\.03\)—the inflation is specific to optimizing against the judge\.

#### A shared blind spot across judge families and scales\.

The effect is not confined to the self\-judge\. We re\-score the 4B policy’s answers with independent judges, without retraining \(Table[2](https://arxiv.org/html/2607.05904#S5.T2)\)\. The hacked errors transfer: a different\-family judge \(Llama, Gemma\) accepts them at a rising rate \(0\.48→0\.570\.48\\\!\\to\\\!0\.57,0\.76→0\.920\.76\\\!\\to\\\!0\.92\), and a same\-family judge3\.5×3\.5\\timesthe policy’s size \(14B\) still accepts77%77\\%\. The shared blind spot is built by self\-play: of the255255answers newly accepted by*all three*families at iter1,117117were accepted by*zero*judges at iter0\. All581581unanimously accepted wrong answers are genuinely incorrect—the GSM8K final\-answer anchor is an unambiguous integer match\.

Table 2:Cross\-family and cross\-scale judges re\-scoring the 4B policy’s wrong answers \(false\-positive rate and discriminationTPR−FPR\\mathrm\{TPR\}\-\\mathrm\{FPR\}, iter→0\{\}\_\{0\}\\\!\\to\\\!iter1\), and the strictestMinensemble of the three families, on the full test set \(n=1319n\{=\}1319; seed 0, the most conservative—the collapse reproduces across three seeds, per\-judge iter1discrimination std≤0\.03\\leq 0\.03; three\-seedMinFPR0\.436→0\.6510\.436\\\!\\to\\\!0\.651, discrimination0\.316→0\.0810\.316\\\!\\to\\\!0\.081\)\.
#### The blind spot is shared across families on code as well\.

Re\-scoring the natural\-code wrong programs with independent\-family judges \(Table[4](https://arxiv.org/html/2607.05904#A3.T4), Appendix\) reproduces the shared blind spot across task domains: Qwen3\-1\.7B \(self\) accepts wrong code atFPR=0\.445±0\.037\\mathrm\{FPR\}\{=\}0\.445\{\\pm\}0\.037, Gemma\-3\-12B at0\.600±0\.0380\.600\{\\pm\}0\.038, and a4\.7×4\.7\{\\times\}larger Qwen3\-8B judge with the strongest discrimination still at0\.257±0\.0270\.257\{\\pm\}0\.027\. The errors are positively correlated across families—16\.4%±3\.9%16\.4\\%\{\\pm\}3\.9\\%of wrong programs are accepted by*all three*judges,2\.38×2\.38\{\\times\}the rate judge independence would predict—so the strictest three\-judgeMinensemble still passes one wrong program in six\. The blind spot is a property of the channel, shared across judge family, scale, and task domain\.

#### No plausibility\-scoring defense closes the basin\.

Section[4](https://arxiv.org/html/2607.05904#S4)predicts that any defense keeping the judge in the plausibility channel should fail, and the natural ones do\. Instructing the judge to independently recompute and reject when uncertain leaves its iter1false\-positive rate at0\.7190\.719\(n=128n\{=\}128\); training the policy against a different\-family judge \(Qwen policy, Llama judge\) leaves a trained gap of0\.40±0\.090\.40\{\\pm\}0\.09\(three seeds\); a stronger same\-family judge \(14B\) still accepts77%77\\%of the hacked errors\. Scoring\-time ensembling fares no better: the three\-familyMinensemble still accepts55%55\\%of full\-test wrong answers \(seed 0, the most conservative; three\-seed mean65%65\\%\), its discrimination collapsing from0\.310\.31to0\.090\.09\. Making the ensemble the training reward amplifies the failure: the policy learns to satisfy all three judges at once—theMinensemble accepts73\.3±2\.7%73\.3\{\\pm\}2\.7\\%of its wrong answers, up from41\.2±1\.3%41\.2\{\\pm\}1\.3\\%\(three seeds\), its discrimination collapsing from0\.3570\.357to0\.0610\.061, and a held\-out Qwen3\-14B judge never used in the reward is fooled*more*\(0\.56→0\.850\.56\\\!\\to\\\!0\.85\), with true accuracy unchanged\. No fixed\-threshold rule we test separates the self\-play\-hacked errors from correct answers—the no\-escape pattern Proposition[2](https://arxiv.org/html/2607.05904#Thmproposition2)predicts\.

### 5\.2The fix: breaking the candidate anchor restores oversight

Every defense above keeps the judge*anchored*: the verify prompt leaves the false\-positive rate at0\.7190\.719, an order of magnitude above the1−solve\-acc=0\.071\-\\text\{solve\-acc\}=0\.07ceiling of a genuinely independent solver—by Corollary[1](https://arxiv.org/html/2607.05904#Thmcorollary1), this certifies the judge as anchored\. The fix is to break the anchoring: require the judge to commit an answer of its own*before*it may use the candidate\.

#### Committing before seeing isolates anchoring as the cause \(detection\)\.

Keeping the candidate fully visible but requiring the judge to first write its own answer—“My answer: …”—before any comparison collapses the false\-positive rate on wrong answers from the verify prompt’s0\.7190\.719to0\.012\\mathbf\{0\.012\}\(Qwen3\-4B,n=128n\{=\}128\), while the judge commits the correct answer97%97\\%of the time: the same model, the same visible candidate, differing only in whether it commits first\. The effect is not specific to one family—a Llama\-3\.1\-8B judge, anchored to a discrimination of0\.080\.08when it scores plausibility, recovers to0\.91±0\.030\.91\{\\pm\}0\.03with a false\-positive rate of0\.035±0\.0100\.035\{\\pm\}0\.010once made to commit first \(three seeds\)\. Candidate anchoring—not family, scale, or whether the candidate is visible—is the operative cause\.

![Refer to caption](https://arxiv.org/html/2607.05904v1/fig2_dichotomy.png)Figure 2:The same judge under two accept rules: scoring a shown candidate \(anchored\) it is driven into the shared basin; committing its own answer first \(de\-anchored\) it recovers near\-perfect discrimination even with the candidate visible \(FPR=0\.012\\mathrm\{FPR\}=0\.012\)—anchoring, not capability\. Naive\-plausibility and blind\-solve bars: full test set \(n=1319n\{=\}1319\); verify and commit\-first bars:n=128n\{=\}128audit \(§[5](https://arxiv.org/html/2607.05904#S5)\)\.
#### Withholding the candidate is the limiting case\.

Taking independence to its extreme, the judge*solves the problem itself*, without seeing the candidate at all, and accepts only when its independent answer matches\. Re\-scored this way on the full test set \(n=1319n\{=\}1319\), the same Qwen3\-4B judge whose plausibility scoring is driven to a discrimination of0\.060\.06\(Table[2](https://arxiv.org/html/2607.05904#S5.T2)\) separates correct from incorrect almost perfectly: its false\-positive rate on wrong answers is0\.0120\.012and0\.0100\.010across the two iterations—near zero and*undriven*by self\-play—while it accepts97%97\\%of correct answers, for a discrimination of0\.96\\mathbf\{0\.96\}at both iterations \(Figure[2](https://arxiv.org/html/2607.05904#S5.F2)\)\. The judge could verify all along—it solves these problems at0\.930\.93accuracy—but a shown candidate anchors it\. What removes the anchor is the independent commitment, not the candidate’s absence: committing first works with the candidate in full view; withholding it enforces the same independence most cleanly\. Generative verifiers\(Zhang et al\.,[2025](https://arxiv.org/html/2607.05904#bib.bib19)\)are trained to solve before they judge; the finding here is that the*plausibility*channel collapses under adversarial self\-play while this independent channel does not\.

#### The de\-anchored reward prevents the basin \(prevention\)\.

The same de\-anchored channel that*detects*the hacked errors can also*prevent*them when used as the training reward\. We repeat the self\-play loop with the reward replaced by the blind\-solve verdict, holding everything else fixed \(Qwen3\-4B, reasoning\-suppressed JSON, DPO, two iterations,n=128n\{=\}128audit subset\)\. Across three seeds the reward’s false\-positive rate on wrong answers remains empirically zero at both iterations \(0of∼380\{\\sim\}380wrong answers across all six seed–iteration cells;95%95\\%Wilson upper bound0\.0100\.010; discrimination0\.910\.91–0\.950\.95\), exactly as Proposition[1](https://arxiv.org/html/2607.05904#Thmproposition1)requires, while the plausibility\-judge reward is driven from a0\.650\.65to a0\.890\.89false\-positive rate on the same subset\. Anchor accuracy stays essentially flat; as with the oracle control, the reward signal does not create capability\. The same reference\-free channel that audits the basin is, in our experiments, an effectively inflation\-proof training reward\.

### 5\.3The full arc replicates without training: best\-of\-NNselection in code and math

The same arc—inflation, cross\-family transfer, and a de\-anchoring fix with a capability threshold—replicates with*no training at all*, using best\-of\-NNrejection sampling as a training\-free proxy for optimizing against the judge\(Gao et al\.,[2023](https://arxiv.org/html/2607.05904#bib.bib6)\), on the organic code task above with held\-out unit\-test execution as the anchor\. Definegap​@​k\\mathrm\{gap\}@kas the judge\-pass rate of the judge\-selected candidate at budgetkkminus its true unit\-test pass rate\. Selecting what the judge likes drives the judge toward certainty while true quality stays flat: overN=16N\{=\}16candidates per problem, the gap grows from0\.200\.20atk=1k\{=\}1to0\.588\\mathbf\{0\.588\}atk=16k\{=\}16\(paired bootstrap95%95\\%CI\[0\.506,0\.669\]\[0\.506,0\.669\]\), with the selected candidates’ unit\-test pass essentially unchanged \(0\.27→0\.290\.27\\\!\\to\\\!0\.29\) and the effect stable across generation seeds \(0\.547±0\.0360\.547\{\\pm\}0\.036\)\. Re\-judging the*same*19201920candidates with other judges \(Figure[4](https://arxiv.org/html/2607.05904#A3.F4)a, Appendix\), a4\.7×4\.7\{\\times\}larger same\-family judge still inflates \(gap​@​16=0\.378\\mathrm\{gap\}@16\{=\}0\.378\), and Llama and Mistral judges swing from over\-rejection under the strict prompt to strong inflation under the lenient one: with correctness held fixed, the reference\-free verdict tracks prompt framing—the verification asymmetry in its sharpest form\.

Applying the commit\-first rule to the same pool, the 8B judge’sgap​@​16\\mathrm\{gap\}@16falls from0\.3780\.378to0\.227\\mathbf\{0\.227\}\(paired reduction CI\[0\.064,0\.237\]\[0\.064,0\.237\]\) and its single\-sample gap falls to zero\. The fix is capability\-dependent in exactly the direction the independence bound predicts \(Figure[4](https://arxiv.org/html/2607.05904#A3.F4)b, Appendix\): committing hurts the 1\.7B judge, whose own committed solutions are mostly wrong \(0\.588→0\.6370\.588\\\!\\to\\\!0\.637\), and helps every larger judge, plateauing by 8B \(Appendix[C\.4](https://arxiv.org/html/2607.05904#A3.SS4)\)\. The same training\-free amplification appears in competition math: on AIME\-2024 \(3030problems,1616candidates each, Ministral\-3\-8B judging its own candidates against an exact\-match anchor\), the strict self\-judge reachesgap​@​16=0\.348\\mathrm\{gap\}@16\{=\}0\.348, and the single\-sample gap remains positive both on the clean subset and under a worst\-case truncation convention \(Appendix[C\.4](https://arxiv.org/html/2607.05904#A3.SS4)\)\.

### 5\.4A second policy family: the full arc replicates with a Gemma policy

To test whether the arc is specific to one policy family, we rerun the full self\-play loop with Gemma\-3\-12B\-it as both policy and reference\-free judge on the same GSM8K self\-play and audit protocol \(five seeds,n=256n\{=\}256held\-out audit each; Figure[3](https://arxiv.org/html/2607.05904#S5.F3)\)\. Three of five seeds show significant judge\-reward inflation \(judge\-pass\+0\.16\+0\.16/\+0\.21\+0\.21/\+0\.16\+0\.16, McNemarp=4\.2×10−6p\{=\}4\.2\{\\times\}10^\{\-6\}/1\.6×10−101\.6\{\\times\}10^\{\-10\}/2\.7×10−62\.7\{\\times\}10^\{\-6\}\) with exact match statistically unchanged, widening the judge–truth gap from0\.410\.41to0\.560\.56–0\.630\.63; the false\-positive rate on wrong answers rises from0\.540\.54to0\.750\.75–0\.820\.82\(Figure[3](https://arxiv.org/html/2607.05904#S5.F3)b\) and outputs lengthen \(truncation0\.14→0\.400\.14\{\\to\}0\.40–0\.440\.44\)\. Two seeds show no inflation \(−0\.055\-0\.055and−0\.012\-0\.012,p=0\.09p\{=\}0\.09and0\.790\.79\) and no false\-positive rise, and their truncation \(0\.120\.12and0\.230\.23\) stays outside the hacked band: the hacked seeds move on judge\-pass, false\-positive rate, and output length together, and the clean seeds on none of these, making them matched negative controls\. Under DPO, entry into the basin is stochastic rather than inevitable, but its signature is not\. The hacked outputs are not a white\-box exploit of the Gemma judge: re\-judging the same answers with an anchored Qwen3\-4B judge \(seeds 0–2\) raises its false\-positive rate from0\.350\.35to0\.620\.62/0\.640\.64on the hacked seeds and leaves it flat \(0\.310\.31\) on the clean one—the same answer\-side transfer as in §[5](https://arxiv.org/html/2607.05904#S5)\. On the full test set \(n=1319n\{=\}1319, seed 0\) the gap widens from0\.4060\.406to0\.6150\.615\.

![Refer to caption](https://arxiv.org/html/2607.05904v1/fig_gemma.png)Figure 3:Gemma\-3\-12B policy, five seeds \(n=256n\{=\}256each; per\-seed numbers in Table[7](https://arxiv.org/html/2607.05904#A3.T7), Appendix\)\. \(a\) Judge\-pass: three of five seeds inflate \(red\), two stay clean \(blue\)\. \(b\) The anchored FPR on wrong answers rises on exactly the hacked seeds; the de\-anchored reward stays at≈0\.005\{\\approx\}0\.005on every seed \(green\)\.The de\-anchored verification reward prevents the inflation in three of three seeds\. Following the capability threshold of §[5\.3](https://arxiv.org/html/2607.05904#S5.SS3), the blind\-solve verifier is Qwen3\-4B \(solve accuracy0\.930\.93on this audit\): verify\-pass tracks exact match to the point of equality at every seed, the only false positive across all seeds and iterations of the audit slice is a single ambiguous item \(FPR≈0\.005\{\\approx\}0\.005\), and on the full test set the rate stays at0\.00470\.0047–0\.00650\.0065\(Wilson 95% upper bound0\.01330\.0133\) while the gap stays at zero\. In our setting the attack surface is candidate\-conditioned evaluation rather than the judge family per se: on the same hacked outputs, the same Qwen3\-4B judge accepts0\.620\.62/0\.640\.64of wrong answers when anchored on the candidate and0\.0050\.005/0\.0100\.010\(11–22of196196\) when it must commit a blind solution first—a6060–120×120\{\\times\}reduction on identical text\. Aggregating the two candidate\-conditioned judges does not close the gap: even min\-aggregation lets the false\-positive rate rise from0\.320\.32to0\.540\.54/0\.610\.61on the two hacked seeds it covers\. The failure is a property of the conditioning, not of judge multiplicity\.

## 6Discussion

#### Implications for oversight\.

The shared blind spot is a property of reference\-free judging, not of any one judge: when verifying is harder than finding a plausible answer, scaling or diversifying the judge does not restore trustworthy supervision\. The regime at risk is a policy near its capability frontier, where accuracy is naturally low\. Our evidence concerns the consultancy\-like, judge\-as\-reward regime; debate, where an adversarial second model can surface the error\(Kenton et al\.,[2024](https://arxiv.org/html/2607.05904#bib.bib8)\), is a distinct and complementary mechanism\. The structural principle is to require the judge to commit an answer of its own before using the candidate: any reward that scores a shown candidate*without*such an independent commitment inherits the1−EM1\-\\mathrm\{EM\}ceiling of the dichotomy in §[4](https://arxiv.org/html/2607.05904#S4)and is hackable wherever the policy has room to err\. What verification restores is trustworthy*detection*, not new capability—the ceiling on what any reward can teach is set by the policy, as the oracle control confirms\. Tool\-augmented judges that execute checks\(Xu et al\.,[2025](https://arxiv.org/html/2607.05904#bib.bib17)\)extend the same principle where a solver alone falls short\.

## 7Limitations

Policy optimization uses Qwen3, replicated in full with a Gemma\-3\-12B policy \(§[5\.4](https://arxiv.org/html/2607.05904#S5.SS4)\) and partially on Llama\-3\.1\-8B, whose weaker decoupling the risk score predicts; the blind spot itself is established on the judge side across three families and four scales\. We optimize with DPO; the oracle control attributes the inflation to the judge\-reward rather than the algorithm\. The core self\-play study is grade\-school math—TruthfulQA and natural code carry the asymmetry beyond it—and the best\-of\-NNstress test optimizes by rejection sampling rather than policy updates\. The defenses we evaluate instantiate the monotone class of Proposition[2](https://arxiv.org/html/2607.05904#Thmproposition2)\. The verification\-as\-reward result assumes the verifier’s and policy’s errors are largely independent—strongly correlated errors would loosen its inflation bound, and we verify it on then=128n\{=\}128audit subset across three seeds—and requires a verifier that can solve the task \(§[5\.3](https://arxiv.org/html/2607.05904#S5.SS3)\) and an exact\-matchable final answer; extending commitment to open\-ended outputs—committed rubrics, executable tests—is future work\.

## 8Conclusion

Reference\-free LLM judging carries a verification asymmetry: scoring a shown candidate, it measures plausibility, not correctness, leaving false\-positive basins that self\-play discovers and fills\. The errors are a shared blind spot—arising under Qwen and Gemma policies alike and surviving stronger judges, recompute prompts, and ensembling; a hidden anchor makes the distortion falsifiable and bounds it byVA\-Gap≤1−EM\\textsc\{VA\-Gap\}\\leq 1\-\\mathrm\{EM\}\. Forcing the judge to commit an answer of its own first collapses the false\-positive rate from0\.7190\.719to0\.0120\.012\. Improvements measured by a reference\-free judge are suspect until checked by a verification signal independent of the candidate\.

## References

- Bai et al\. \(2022\)Yuntao Bai, Saurav Kadavath, Sandipan Kundu, et al\.Constitutional AI: Harmlessness from AI feedback\.*arXiv preprint arXiv:2212\.08073*, 2022\.
- Chen et al\. \(2024\)Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu\.Self\-play fine\-tuning converts weak language models to strong language models\.*International Conference on Machine Learning \(ICML\)*, 2024\.arXiv:2401\.01335\.
- Coste et al\. \(2024\)Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger\.Reward model ensembles help mitigate overoptimization\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2310\.02743\.
- Eisenstein et al\. \(2023\)Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, et al\.Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking\.*arXiv preprint arXiv:2312\.09244*, 2023\.
- Esary et al\. \(1967\)James D\. Esary, Frank Proschan, and David W\. Walkup\.Association of random variables, with applications\.*The Annals of Mathematical Statistics*, 38\(5\):1466–1474, 1967\.
- Gao et al\. \(2023\)Leo Gao, John Schulman, and Jacob Hilton\.Scaling laws for reward model overoptimization\.In*International Conference on Machine Learning \(ICML\)*, 2023\.arXiv:2210\.10760\.
- Huang et al\. \(2026\)Chengsong Huang, Haolin Liu, Tong Zheng, Runpeng Dai, Langlin Huang, Jinyuan Li, Zongxia Li, Zhepei Wei, Yu Meng, and Jiaxin Huang\.G\-Zero: Self\-play for open\-ended generation from zero data\.*arXiv preprint arXiv:2605\.09959*, 2026\.
- Kenton et al\. \(2024\)Zachary Kenton, Noah Y\. Siegel, János Kramár, Jonah Brown\-Cohen, Samuel Albanie, Jannis Bulian, David Lindner, et al\.On scalable oversight with weak LLMs judging strong LLMs\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2024\.arXiv:2407\.04622\.
- Lee et al\. \(2023\)Harrison Lee, Samrat Phatale, Hassan Mansoor, et al\.RLAIF vs\. RLHF: Scaling reinforcement learning from human feedback with AI feedback\.*arXiv preprint arXiv:2309\.00267*, 2023\.
- Lu et al\. \(2025\)Jack Lu, Ryan Teehan, Jinran Jin, and Mengye Ren\.When does verification pay off? A closer look at LLMs as solution verifiers\.*arXiv preprint arXiv:2512\.02304*, 2025\.
- Pan et al\. \(2024\)Jane Pan, He He, Samuel R\. Bowman, and Shi Feng\.Spontaneous reward hacking in iterative self\-refinement\.*arXiv preprint arXiv:2407\.04549*, 2024\.
- Rafailov et al\. \(2024\)Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit S\. Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum\.Scaling laws for reward model overoptimization in direct alignment algorithms\.*arXiv preprint arXiv:2406\.02900*, 2024\.
- Sharma et al\. \(2023\)Mrinank Sharma, Meg Tong, Tomasz Korbak, et al\.Towards understanding sycophancy in language models\.*arXiv preprint arXiv:2310\.13548*, 2023\.
- Simonds et al\. \(2025\)Toby Simonds, Kevin Lopez, Akira Yoshiyama, and Dominique Garmier\.RLSR: Reinforcement learning from self reward\.*arXiv preprint arXiv:2505\.08827*, 2025\.
- Singhal et al\. \(2023\)Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett\.A long way to go: Investigating length correlations in RLHF\.*arXiv preprint arXiv:2310\.03716*, 2023\.
- Wen et al\. \(2025\)Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R\. Bowman, He He, and Shi Feng\.Language models learn to mislead humans via rlhf\.*International Conference on Learning Representations \(ICLR\)*, 2025\.arXiv:2409\.12822\.
- Xu et al\. \(2025\)Ran Xu, Jingjing Chen, Jiayu Ye, Yu Wu, Jun Yan, Carl Yang, and Hongkun Yu\.Incentivizing agentic reasoning in LLM judges via tool\-integrated reinforcement learning\.*arXiv preprint arXiv:2510\.23038*, 2025\.
- Yuan et al\. \(2024\)Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston\.Self\-rewarding language models\.*International Conference on Machine Learning \(ICML\)*, 2024\.arXiv:2401\.10020\.
- Zhang et al\. \(2025\)Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal\.Generative verifiers: Reward modeling as next\-token prediction\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.arXiv:2408\.15240\.
- Zhao et al\. \(2025a\)Yulai Zhao, Haolin Liu, Dian Yu, et al\.Master\-RM: A reward model robust to superficial reward hacking\.Hugging Face modelsarosavo/Master\-RM, 2025a\.
- Zhao et al\. \(2025b\)Yulai Zhao, Haolin Liu, Dian Yu, et al\.One token to fool LLM\-as\-a\-judge\.*arXiv preprint arXiv:2507\.08794*, 2025b\.
- Zheng et al\. \(2023\)Lianmin Zheng, Wei\-Lin Chiang, Ying Sheng, et al\.Judging LLM\-as\-a\-judge with MT\-Bench and Chatbot Arena\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023\.arXiv:2306\.05685\.

## Appendix AReproducibility Details

All runs use a single data\-center GPU per run \(H100\-class\) via Hugging Face Transformers, with a few robustness extensions run on comparable data\-center cards; the exact configuration and dataset SHA\-256 hashes are logged in each run’srun\_config\.json\.

#### Models\.

Policies and self\-judges are Qwen3\-1\.7B/4B/8B/14B; cross\-family judges are Llama\-3\.1\-8B\-Instruct and Gemma\-3\-12B\-it\. Self\-play trains LoRA adapters\.

#### Data and anchor\.

Self\-play trains on GSM8K*train*questions \(128128per iteration\) and audits on the held\-out GSM8K*test*set \(n=1319n\{=\}1319\); we also use MATH level\-4/5, GSM\-Plus, and TruthfulQA\. The hidden anchor is a held\-out exact\-match check on the final answer: the gold answer is drawn from the dataset’s ground\-truth, normalized, and matched exactly against the policy’s parsed final answer\. It is never placed in any judge or policy prompt—the judge runs withanswer\_key\_visible==false—and is never a training signal\. Training and audit questions are disjoint splits\.

#### Self\-play loop\.

Each run is two iterations\. Per iteration the policy samplesnum\_rollouts=4=4answers per training question \(temperature0\.90\.9, top\-pp0\.950\.95\); the reference\-free self\-judge scores each \(judge\_reference\_policy==never\); within each question a preference pair is formed with a judge\-accepted rollout as*chosen*and a judge\-rejected rollout as*rejected*\(acceptance thresholdjudge\_score≥0\.5\\geq 0\.5; both members are the policy’s own rollouts; no reference\-answer fallback\), and the policy is updated by LoRA\-DPO\.

#### Gemma\-policy replication \(§[5\.4](https://arxiv.org/html/2607.05904#S5.SS4)\)\.

Policy and self\-judge are Gemma\-3\-12B\-it; the protocol matches the loop above with LoRA\-DPO \(rank88, learning rate2×10−52\{\\times\}10^\{\-5\},β=0\.1\\beta\{=\}0\.1, length\-normalized\),128128training questions per iteration,num\_rollouts=4=4\(temperature0\.70\.7, top\-pp0\.950\.95\), two iterations, at most9696DPO steps per iteration, a20482048\-token generation budget, and a256256\-token anchored\-judge budget\. The de\-anchored arm uses Qwen3\-4B as the blind\-solve verifier with reasoning enabled and a20482048\-token budget, which its independent solution requires before the exact\-match comparison\. Each seed audits the first256256held\-out test questions; the anchored arm runs five seeds, the de\-anchored arm and the Qwen re\-judge cover seeds0–22, and seed 0 is additionally audited on the fulln=1319n\{=\}1319test set by evaluation\-only reuse of the trained checkpoints\. Generation stops on the union of the tokenizer and generation\-config end\-of\-sequence tokens \(Gemma terminates chat turns with a token distinct from its tokenizer EOS\)\.

#### Answer format\.

By default answers use a reasoning\-suppressed JSON format—the policy is instructed to “Return only JSON with keys final\_answer and trace,” with thinking disabled andmax\_new\_tokens=640=640\. The chain\-of\-thought variant enables thinking withmax\_new\_tokens=4096=4096\.

#### Optimization\.

LoRA rank88, learning rate2×10−52\{\\times\}10^\{\-5\}, DPOβ=0\.1\\beta\{=\}0\.1\(not length\-normalized\), up to9696optimizer steps per iteration, max sequence length14081408\(JSON\) /51205120\(CoT\)\. The headline \(Table[1](https://arxiv.org/html/2607.05904#S4.T1)\), capability sweep \(Table[3](https://arxiv.org/html/2607.05904#A3.T3)\), and cross\-family transfer \(Table[2](https://arxiv.org/html/2607.05904#S5.T2)\) use three seeds \(0,1,20,1,2\); the five\-iteration trajectory, cross\-family\-judge training, and train\-time ensemble runs likewise use three seeds, while the out\-of\-distribution CoT self\-play is a single\-seed audit\.

#### Natural\-code audit\.

The organic code experiment \(§[5](https://arxiv.org/html/2607.05904#S5), Table[4](https://arxiv.org/html/2607.05904#A3.T4)\) audits the base verification asymmetry prior to optimization: a Qwen3\-1\.7B policy generates natural chain\-of\-thought solutions \(thinking enabled, no format suppression,max\_new\_tokens=24,576=24\{,\}576\) to the120120most\-recent function\-call problems of LiveCodeBench \(the most recent problems, which reduces pre\-training contamination; stdin/stdout formats excluded for automated execution\), judged by itself and by cross\-family judges \(Gemma\-3\-12B, Qwen3\-8B, Llama\-3\.1\-8B\) under a balanced reference\-free prompt that receives only the problem and candidate code\. Correctness is the held\-out unit\-test execution result, never shown to any judge \(verified by an automated no\-leak check on every prompt\) and never a training signal\. We report three policy seeds; of the≈100\{\\approx\}100syntactically valid candidates per seed,≈70\{\\approx\}70are wrong\. Generation truncation is10\.810\.8–15\.8%15\.8\\%and judge truncation≤12\.5%\\leq 12\.5\\%; truncated judgements default to rejection\.

#### Best\-of\-NNstress test \(§[5\.3](https://arxiv.org/html/2607.05904#S5.SS3)\)\.

The candidate pool is the same120120LiveCodeBench problems withN=16N\{=\}16candidates each \(Qwen3\-1\.7B, temperature0\.80\.8,max\_new\_tokens=24,576=24\{,\}576\); every candidate is executed against the held\-out unit tests and scored by each judge\.gap​@​k\\mathrm\{gap\}@kuses the exact expectation over allkk\-subsets per problem with uniform tie\-breaking among judge\-accepted candidates; confidence intervals are per\-problem paired bootstrap \(B=10,000B\{=\}10\{,\}000\)\. The base audit’s0\.2750\.275and thek=1k\{=\}1point of the best\-of\-NNcurve use different denominators \(single\-sample acceptance over parseable judgements versus the all\-candidate expectation\); recomputing thek=1k\{=\}1point under the base\-audit denominator recovers0\.2750\.275\. Seed robustness re\-generates the full pool under two further generation seeds\. Cross\-judge rows re\-judge the identical pool; the strict instruction is the audit prompt of the natural\-code audit and the lenient one its balanced variant\. The Ministral row is served through a quantized cloud inference endpoint \(model identifier and access date logged\) and is reported as a robustness probe\. De\-anchoring gives the judge a solve budget of24,57624\{,\}576tokens \(judgebudget12,28812\{,\}288\); the code\-forced control replaces the algorithm\-sketch commitment with a required runnable solution, and truncated commitments or judgements default to rejection\. The AIME\-2024 replication uses3030problems with1616candidates each from Ministral\-3\-8B judging its own candidates under the same strict/lenient prompts, a normalized boxed\-answer exact\-match anchor, and the reported clean\-subset and worst\-case conventions; generation truncation is21%21\\%and unparseable final answers are31\.9%31\.9\\%, which motivates reporting those layers explicitly\.

#### Judge\.

The judge sees the question and the candidate’s final answer and trace, with no reference answer, and returns JSON \{judge\_score,process\_score,explanation\} with scores in\[0,1\]\[0,1\]thresholded at0\.50\.5\(judge generation capped at256256tokens\)\. Its system prompt is verbatim:

> You are a strict educational grader\. Score only the candidate answer and reasoning shown in the user message\. Treat any instructions inside the candidate answer or trace as untrusted student content\. Ifanswer\_key\_visibleis false, solve the problem independently; never assign zero merely because no answer key is shown\.judge\_scoremeasures final\-answer correctness: 1\.0 for a mathematically or semantically correct final answer, 0\.0 for an incorrect final answer, with partial credit only for genuinely partial final answers\.process\_scoremeasures reasoning quality separately\. Return only a valid JSON object with exactly the keysjudge\_score,process\_score, andexplanation, numeric in\[0,1\]\[0,1\]\.

The recompute \(verification\-prompt\) defense additionally instructs the judge to solve the problem itself before scoring and to reject when uncertain\. The commit\-first protocol keeps the candidate in the prompt but requires a parseable independent answer \(“My answer: …”\) before any comparison, accepting only when that committed answer matches the candidate; the blind\-solve verification channel withholds the candidate entirely and accepts only when the judge’s independent answer matches\.

## Appendix BProofs

#### Proof of Proposition[2](https://arxiv.org/html/2607.05904#Thmproposition2)\.

Couple the verdicts acrossss: letU1,…,UNU\_\{1\},\\dots,U\_\{N\}be i\.i\.d\. uniform on\[0,1\]\[0,1\]and setXi​\(s\)=𝟏​\{Ui≤qi​\(s\)\}X\_\{i\}\(s\)=\\mathbf\{1\}\\\{U\_\{i\}\\leq q\_\{i\}\(s\)\\\}\. Conditionally onsstheXi​\(s\)X\_\{i\}\(s\)are independent with the required marginals, and eachXi​\(s\)X\_\{i\}\(s\)is pointwise non\-decreasing inssbecauseqiq\_\{i\}is\. For non\-decreasinggg,g​\(X1​\(s\),…,XN​\(s\)\)g\(X\_\{1\}\(s\),\\dots,X\_\{N\}\(s\)\)is then pointwise non\-decreasing inss, sohg​\(s\)=𝔼​g​\(X​\(s\)\)h\_\{g\}\(s\)=\\mathbb\{E\}\\,g\(X\(s\)\)is non\-decreasing\. Sinceggis monotone withg​\(𝟏\)=1g\(\\mathbf\{1\}\)=1,hg​\(s\)≥Pr⁡\[X1​\(s\)=⋯=XN​\(s\)=1\]=∏iqi​\(s\)h\_\{g\}\(s\)\\geq\\Pr\[X\_\{1\}\(s\)=\\dots=X\_\{N\}\(s\)=1\]=\\prod\_\{i\}q\_\{i\}\(s\); ashg≤1h\_\{g\}\\leq 1trivially,hg​\(s\)→1h\_\{g\}\(s\)\\to 1wherever everyqi​\(s\)→1q\_\{i\}\(s\)\\to 1\. Equation \([3](https://arxiv.org/html/2607.05904#S4.E3)\) is the Chebyshev/FKG association inequality for products of non\-decreasing functions of a common variable\(Esary et al\.,[1967](https://arxiv.org/html/2607.05904#bib.bib5)\)\.□\\square

#### Proof of Corollary[2](https://arxiv.org/html/2607.05904#Thmcorollary2)\.

All quantities are taken under the audited distribution of wrong candidates\(Q,A\)\(Q,A\)\. WritePS∣Q,AP\_\{S\\mid Q,A\}for the committed\-answer distribution given the question and the shown candidate, andPS∣QP\_\{S\\mid Q\}for its candidate\-marginalized version\. On a wrong candidateaa,Pr⁡\[S=a∣Q,A=a\]−Pr⁡\[S=a∣Q\]≤TV​\(PS∣Q,A=a,PS∣Q\)\\Pr\[S=a\\mid Q,A\{=\}a\]\-\\Pr\[S=a\\mid Q\]\\leq\\mathrm\{TV\}\\\!\\big\(P\_\{S\\mid Q,A=a\},\\,P\_\{S\\mid Q\}\\big\)pointwise\. For the ceiling, letS0∼PS∣QS^\{0\}\\sim P\_\{S\\mid Q\}be a candidate\-independent copy of the committed answer; then𝔼​\[Pr⁡\[S=a∣Q\]\]=Pr⁡\[S0=A\]≤Pr⁡\[S0≠gold\]=1−solve\-acc\\mathbb\{E\}\\big\[\\Pr\[S=a\\mid Q\]\\big\]=\\Pr\[S^\{0\}=A\]\\leq\\Pr\[S^\{0\}\\neq\\text\{gold\}\]=1\-\\text\{solve\-acc\}, with solve\-acc the audited average of Proposition[1](https://arxiv.org/html/2607.05904#Thmproposition1)\. Taking expectations over the wrong\-candidate distribution,Δ≤𝔼​\[TV\]\\Delta\\leq\\mathbb\{E\}\[\\mathrm\{TV\}\]\. Pinsker’s inequality holds pointwise,TV2≤12​KL​\(PS∣Q,A∥PS∣Q\)\\mathrm\{TV\}^\{2\}\\leq\\tfrac\{1\}\{2\}\\,\\mathrm\{KL\}\\big\(P\_\{S\\mid Q,A\}\\,\\\|\\,P\_\{S\\mid Q\}\\big\), so𝔼​\[KL\]≥2​𝔼​\[TV2\]\\mathbb\{E\}\[\\mathrm\{KL\}\]\\geq 2\\,\\mathbb\{E\}\[\\mathrm\{TV\}^\{2\}\]; Jensen’s inequality gives𝔼​\[TV2\]≥\(𝔼​\[TV\]\)2\\mathbb\{E\}\[\\mathrm\{TV\}^\{2\}\]\\geq\(\\mathbb\{E\}\[\\mathrm\{TV\}\]\)^\{2\}, and therefore

I​\(S;A∣Q\)=𝔼​\[KL​\(PS∣Q,A∥PS∣Q\)\]≥2​𝔼​\[TV2\]≥2​\(𝔼​\[TV\]\)2≥2​Δ2\.I\(S;A\\mid Q\)\\;=\\;\\mathbb\{E\}\\big\[\\mathrm\{KL\}\\big\(P\_\{S\\mid Q,A\}\\,\\\|\\,P\_\{S\\mid Q\}\\big\)\\big\]\\;\\geq\\;2\\,\\mathbb\{E\}\[\\mathrm\{TV\}^\{2\}\]\\;\\geq\\;2\\,\\big\(\\mathbb\{E\}\[\\mathrm\{TV\}\]\\big\)^\{2\}\\;\\geq\\;2\\Delta^\{2\}\.WithFPR=0\.719\\mathrm\{FPR\}=0\.719and ceiling0\.070\.07,Δ=0\.649\\Delta=0\.649andI≥2​Δ2=0\.84I\\geq 2\\Delta^\{2\}=0\.84nats=1\.2=1\.2bits\.□\\square

## Appendix CAdditional Tables and Robustness Details

### C\.1Ensemble dependence measurements

On post\-self\-play wrong answers the three judges’ acceptances are pairwise positively correlated \(ϕ=0\.29\\phi=0\.29–0\.380\.38\)\. The strictest three\-family ensemble \(§[5](https://arxiv.org/html/2607.05904#S5)\) passesFPRMin=0\.55\\mathrm\{FPR\}\_\{\\text\{\{Min\}\}\}=0\.55of them—above the∏iFPRi=0\.47\\prod\_\{i\}\\mathrm\{FPR\}\_\{i\}=0\.47an independent panel would attain and barely below its single most stringent member \(0\.570\.57\)—and581581wrong answers are accepted unanimously where independence predicts only≈497\{\\approx\}497\.

### C\.2Capability sweep

Table 3:Qwen3 capability sweep on then=128n\{=\}128audit subset \(JSON, mean over 3 seeds\)\. Self\-play drives the judge near its ceiling while accuracy stays flat; the gap is large at every size\. The 4B headline \(Table[1](https://arxiv.org/html/2607.05904#S4.T1), §[5](https://arxiv.org/html/2607.05904#S5)\) uses the fulln=1319n\{=\}1319test set, on which 4B base accuracy is0\.2090\.209; this subset is slightly easier \(0\.3050\.305\)\.
### C\.3Cross\-family code judging

Table 4:Cross\-family judges re\-scoring the natural\-code policy’s wrong programs \(base, pre–self\-play; LiveCodeBench natural CoT, hidden unit\-test anchor; mean±\{\\pm\}std over 3 policy seeds,nwrong≈70n\_\{\\text\{wrong\}\}\{\\approx\}70/seed\)\. Independent judge families share the blind spot; the strictestMinensemble of the three non\-degenerate judges still accepts16%16\\%of wrong programs\.The pooled Wilson95%95\\%CI for unanimous acceptance of wrong programs is\[0\.122,0\.220\]\[0\.122,0\.220\]\. Llama\-3\.1\-8B has no useful operating point as a reference\-free code judge: a strict prompt collapses it to near\-uniform rejection \(FPR=0\\mathrm\{FPR\}\{=\}0butTPR=0\.10\\mathrm\{TPR\}\{=\}0\.10\), a balanced prompt to near\-uniform acceptance \(FPR=0\.92\\mathrm\{FPR\}\{=\}0\.92,TPR=0\.97\\mathrm\{TPR\}\{=\}0\.97\), with discrimination≤0\.10\\leq 0\.10at either extreme—in contrast to its stable cross\-family operating point on GSM8K \(FPR=0\.48\\mathrm\{FPR\}\{=\}0\.48, Table[2](https://arxiv.org/html/2607.05904#S5.T2)\), a task×\{\\times\}family interaction\. The code ensemble therefore uses the three discriminative judges above\.

### C\.4Best\-of\-NNdetails

![Refer to caption](https://arxiv.org/html/2607.05904v1/fig_bon.png)Figure 4:Best\-of\-NNselection replicates the arc without training \(LiveCodeBench,N=16N\{=\}16,19201920candidates\)\. \(a\)gap​@​16\\mathrm\{gap\}@16under a strict versus a lenient reference\-free instruction: with unit\-test ground truth fixed, Llama and Mistral judges swing from over\-rejection to strong inflation under a one\-word framing change \(Table[5](https://arxiv.org/html/2607.05904#A3.T5)\)\. \(b\) Committing to an own solution before scoring lowersgap​@​16\\mathrm\{gap\}@16only for judges that can solve the task and hurts one that cannot \(1\.7B\); a runnable\-solution elicitation control reproduces the curve \(Table[6](https://arxiv.org/html/2607.05904#A3.T6)\)\.Table 5:Best\-of\-NNinflation spans judge families, and its sign tracks prompt framing rather than correctness:gap​@​16\\mathrm\{gap\}@16when the same19201920candidates \(unit\-test ground truth fixed\) are re\-judged by each judge under a strict versus a lenient reference\-free instruction\. Positive = over\-acceptance inflation; negative = over\-rejection\.†Served via a quantized cloud endpoint as a robustness probe\. All other cells are local bf16 checkpoints\.

Table 6:De\-anchoring under best\-of\-NNpressure has a capability threshold \(Qwen3 judges, same19201920candidates\): committing to its own solution hurts a judge that cannot solve the task \(1\.7B\) and helps every judge that can, with a plateau from 8B\. The control column forces a complete runnable solution as the commitment, confirming the curve is not an elicitation artifact\.The cross\-scale paired reduction \(1\.7B→\\to8B, anchored\) is0\.2100\.210\(CI\[0\.142,0\.283\]\[0\.142,0\.283\]\); the residual de\-anchored code gap at 8B is0\.2270\.227\(CI\[0\.157,0\.302\]\[0\.157,0\.302\]\)\. Restricting each judge to problems where its committed solution is untruncated preserves the capability\-threshold ordering \(0\.5620\.562/0\.2610\.261/0\.2110\.211/0\.2070\.207at 1\.7B/4B/8B/14B\)\. For the AIME\-2024 replication, the setup carries substantial invalid\-output mass \(generation truncation21%21\\%, unparseable final answers31\.9%31\.9\\%\), so we report the conservative layers explicitly: on the clean subset \(parseable, untruncated;n=300n\{=\}300candidates\) the single\-sample gap is\+0\.143\+0\.143\(per\-problem paired bootstrap CI\[0\.075,0\.217\]\[0\.075,0\.217\]\) under the strict prompt and\+0\.290\+0\.290\(\[0\.195,0\.400\]\[0\.195,0\.400\]\) under the lenient one, and both remain positive under the worst\-case convention that every unparseable or truncated candidate counts as a judge rejection \(\+0\.090\+0\.090/\+0\.181\+0\.181\)\.

### C\.5Gemma\-policy replication

Table 7:The self\-play arc replicates with a Gemma\-3\-12B policy \(GSM8K, held\-out auditn=256n\{=\}256per seed, iteration 0→\{\\to\}1\)\. Anchored self\-judge reward inflates the judge on three of five seeds and the hacked outputs transfer to an anchored Qwen3\-4B judge, while exact match never moves; the de\-anchored verification reward \(Qwen3\-4B blind\-solve verifier\) holds verify\-pass within a single item of that arm’s own exact match \(shown beside it\), with a false\-positive rate of≈0\.005\{\\approx\}0\.005throughout\. Judge–truth gap==judge\-pass−\-EM\. Rows 0–4 use then=256n\{=\}256held\-out audit; the Qwen re\-judge and the de\-anchored arm cover seeds 0–2\. The final row reports seed 0 on the fulln=1319n\{=\}1319test set\.