Binarization Flattens the Score Space

# Binarization Flattens the Score Space
Source: [https://arxiv.org/html/2609.35797](https://arxiv.org/html/2609.35797)
Jacob Cole††thanks:Work conducted independently of the author’s research at UC Berkeley\.

###### Abstract

Large language model \(LLM\) judges are often used as rewards to train policies on objectives that deterministic verifiers cannot capture\. However, these rewards are often collapsed to pass/fail \(\{0, 1\}\), which reports the verdict but not how well a response met each criterion\. We model each pass/fail verdict as a score on an unreported scale, compared with one cutoff\. A*stretch*of that scale moves every score proportionally toward or away from the cutoff, but never across it, so no verdict changes\. A policy is therefore free to apply any stretch without changing anything the panel reports\. Under a joint\-Gaussian model, a third grade adds a second threshold and removes this affine stretch ambiguity\. On MATH and SciBench outputs from one seven\-criterion judge, all 14 constructed criterionwise stretches were invisible after binarization but visible with three grades\. Atn=1,024n=1\{,\}024, a test given both population laws had at least96\.5%96\.5\\%power at a1\.5×1\.5\\timesstress\. Retaining grades closes one blind spot created by binarization, but verdicts alone remain insufficient as some changes are still indistinguishable from genuine improvement\. These include arbitrary within\-grade changes and fixed\-covariance, loading\-aligned mean shifts—the signature of a sycophancy\-shaped lift the panel reads as competence\. The shared\-factor reference approximation fit MATH and SciBench but not HealthBench, delineating its empirical scope\. We recommend keeping at least three grades \(for example, asking the judge whether each criterion is fully, partially, or not met and rewarding \{0, 0\.5, 1\}\), and externally validating gains along the remaining direction, which no finer scale removes\.

## 1Introduction

LLM\-based evaluators are increasingly used as post\-training rewards because they can represent objectives and user preferences that are difficult to express with programmatic verifiers\. When policies optimize against these rewards during post\-training, judge errors can propagate into model weights and downstream behavior\. When the judge’s output channel supplies the reward, optimization can select for policy changes along directions the channel cannot distinguish\. Therefore, validity is not only a property of the judge, but also of its deployment system\. Binary rubric rewards are already used in judge\-guided RL\. For example, Compute as Teacher scores each criterion yes/no and rewards the fraction satisfied\[[Jayalath et al\., 2025](https://arxiv.org/html/2609.35797#bib.bib3)\]\. Prior work shows that these rewards can rise while intended behavior stagnates or degrades, including in controlled demonstrations of reward hacking\[[Amodei et al\., 2016](https://arxiv.org/html/2609.35797#bib.bib9);[Skalse et al\., 2022](https://arxiv.org/html/2609.35797#bib.bib10);[Gao et al\., 2023](https://arxiv.org/html/2609.35797#bib.bib11);[Karwowski et al\., 2024](https://arxiv.org/html/2609.35797#bib.bib12);[Wang et al\., 2026](https://arxiv.org/html/2609.35797#bib.bib15);[Mahmoud et al\., 2026](https://arxiv.org/html/2609.35797#bib.bib16)\]\. Related work studies rubric reward design and mitigation\[[Gunjal et al\., 2026](https://arxiv.org/html/2609.35797#bib.bib2);[Yang et al\., 2026](https://arxiv.org/html/2609.35797#bib.bib5)\], practical judge failures such as position bias\[[Zheng et al\., 2023](https://arxiv.org/html/2609.35797#bib.bib1);[Shi et al\., 2025](https://arxiv.org/html/2609.35797#bib.bib14)\], and correlated errors in heterogeneous panels\[[Verga et al\., 2024](https://arxiv.org/html/2609.35797#bib.bib4);[Kohli, 2026](https://arxiv.org/html/2609.35797#bib.bib22)\]\. We frame this as a construct\-validity question:*what changes can the evaluator’s output channel identify at all, and which remain indistinguishable from genuine improvement?*

We study the strongest verdict\-only evidence: the*joint verdict distribution*over all criterion grades, including margins and cross\-criterion agreement\. Policies with the same joint verdict distribution are indistinguishable without held\-out gold or other external evidence\.

![Refer to caption](https://arxiv.org/html/2609.35797v1/figure1_grading_aha.png)Figure 1:Why a third grade helps\.\(a\)The space of binary verdict evidence, seen face\-on\. A benign run and a hacked run \(one that only rescales scores around the pass cutoff\) trace one path: every verdict agrees at every step\.\(b\)The same runs against one statistic the panel never reports,rr\. The benign run stays atr=0r=0; the hacked runs ride above and below, one squeezing scores toward the cutoff and one stretching them away, at identical verdicts throughout\. Any statistic that moves with the rescaling would do, and the middle grade is one of them\.\(c\)With three grades the two runs differ, but merging 0 and 0\.5 into a single fail recovers identical binary evidence\. Under Gaussian score laws, holding both thresholds fixed forces the rescaling to be the identity; within\-grade changes can still hide\.Figure[1](https://arxiv.org/html/2609.35797#S1.F1)illustrates why retaining a middle grade matters\. Each criterion is scored as 0, 0\.5, or 1; binarization treats both 0 and 0\.5 as failure and only 1 as passing\. Rescaling scores around the pass cutpoint \(a*stretch*\), or moving an output from00to0\.50\.5, can therefore change graded scores without changing any binary verdict, even jointly across criteria\. Retaining0\.50\.5exposes this motion, and under Gaussian score laws two fixed thresholds eliminate the stretch \(Theorem[1](https://arxiv.org/html/2609.35797#Thmtheorem1)\); three grades is a floor, but not a target\. The loading\-aligned mean shift is an equality of the score laws themselves, so it stays invisible at every scoring resolution, up to and including a continuous score\. Within\-grade changes persist as long as any quantization does\. Neither is removed by a finer scale, so ruling them out requires evidence from outside the verdict channel\.

#### Contributions\.

- \(1\)We identify latent score changes that leave the full joint binary verdict distribution unchanged\.
- \(2\)We prove a minimal Gaussian\-model fix: retain one middle grade and its second threshold\.
- \(3\)We verify the fix with MATH and SciBench ablations and characterize residual changes requiring external validation\. All proofs appear in Appendix[A](https://arxiv.org/html/2609.35797#A1)\.

## 2Observation model and identification question

We call each\(judge,criterion\)\(\\text\{judge\},\\text\{criterion\}\)output channel a*panel member*\. The theory permits members from one or many judges, while the experiments instantiate multiple criteria from one fixed judge for consistency\. For outputiiand memberjj, let the latent judge score be

Si​j=λj​θi\+εi​j,θi=m\+c​Zi,S\_\{ij\}=\\lambda\_\{j\}\\theta\_\{i\}\+\\varepsilon\_\{ij\},\\qquad\\theta\_\{i\}=m\+\\sqrt\{c\}\\,Z\_\{i\},whereλj\\lambda\_\{j\}is the member loading andZi,εi​1,…,εi​JZ\_\{i\},\\varepsilon\_\{i1\},\\ldots,\\varepsilon\_\{iJ\}are independent centered Gaussians with variances1,ψ1,…,ψJ1,\\psi\_\{1\},\\ldots,\\psi\_\{J\}\. HenceΣ⁡\(c\)=diag⁡\(ψ\)\+c​λ​λ⊤\\Sigma\(c\)=\\operatorname\{diag\}\(\\psi\)\+c\\lambda\\lambda^\{\\top\}\. Atc=1c=1, calibration setsψj=1−λj2\\psi\_\{j\}=1\-\\lambda\_\{j\}^\{2\}, standardizing scores only there\. A binary member returnsYi​j=𝟏\{Si​j\>τj\}Y\_\{ij\}=\\mathbf\{1\}\\\{S\_\{ij\}\>\\tau\_\{j\}\\\}\. A graded member instead fixes−∞=τj,0<⋯<τj,Kj=\+∞\-\\infty=\\tau\_\{j,0\}<\\cdots<\\tau\_\{j,K\_\{j\}\}=\+\\inftyand reports the interval containingSi​jS\_\{ij\}\. These are measurement assumptions, not claims that the shared factor is gold quality or the optimum\. Section[5](https://arxiv.org/html/2609.35797#S5)also applies the threshold\-preserving transformation directly to realized scores\.

We model ordinary \(non\-hacked\) improvement as changes in the shared factor’s meanmmand variancecc\. WritingΘref=\{\(m,c\):m∈ℝ,c\>0\}\\Theta\_\{\\mathrm\{ref\}\}=\\\{\(m,c\):m\\in\\mathbb\{R\},c\>0\\\}, we call the corresponding two\-parameter surface of joint verdict distributions the*reference orbit*\. This is deliberately narrow: genuine gains may be criterion\-specific and fall outside it\. Reference\-arm gains on MATH and SciBench are better explained by this family than by no motion, although the fit is approximate and fails on HealthBench\.

###### Definition 1\(Reference compatibility\)\.

A policy change is*detectable from joint verdicts*when its joint verdict distribution does not equal any distribution in the reference family\. Under the joint\-Gaussian model, all cumulative margins and latent\-score dependence determine the joint distribution\. Outside the joint\-Gaussian model, differences in those summaries remain sufficient evidence of departure, but matching them need not establish equality of the joint distributions\. Any change in a grade margin or pairwise grade table is therefore detectable from verdicts\.

The policy may move latent judge scores but cannot change the declared panel parameters, cutpoints, or residual process\. This fixed\-instrument assumption is what makes the blind set auditable before deployment\.

## 3Main result: what grading fixes

### 3\.1One threshold leaves the score scale unidentified

###### Observation 1\(Threshold\-preserving transformations are invisible\)\.

Let each criterion assign grades by comparing a latent score with fixed thresholds\. Any strictly increasing coordinatewise transformation that keeps those thresholds fixed leaves every grade \(and thus the joint verdict distribution\) unchanged for every sampled output under a shared coupling\. This means binary grading pins the transformation at one point while three grades pin it at two\. This family is sufficient, not exhaustive; invisible within\-cell rearrangements need not be monotone\.

This observation is distribution\-free\. Under Gaussian pre\- and post\-change score laws, equivalence is affine, so two fixed thresholds force the identity \(Theorem[1](https://arxiv.org/html/2609.35797#Thmtheorem1)\); the one\-factor model is not required\.

###### Theorem 1\(Gaussian equivalence of joint verdict distributions\)\.

LetS∼NJ​\(μ,Σ\)S\\sim N\_\{J\}\(\\mu,\\Sigma\)andS†∼NJ​\(μ†,Σ†\)S^\{\\dagger\}\\sim N\_\{J\}\(\\mu^\{\\dagger\},\\Sigma^\{\\dagger\}\)have positive definite covariances, the same fixed finite cutpoints, and fixed category labels\.\(i\)If every member is binary with threshold vectorτ\\tau, the joint verdict distributions agree if and only if there exists a positive diagonalDDwith

μ†−τ=D⁡\(μ−τ\),Σ†=D​Σ​D\.\\mu^\{\\dagger\}\-\\tau=D\(\\mu\-\\tau\),\\qquad\\Sigma^\{\\dagger\}=D\\Sigma D\.Moreover,DDis unique\.

\(ii\)With graded members, the same relations hold about each member’s first finite cutpoint, but necessarilyDj​j=1D\_\{jj\}=1wheneverKj≥3K\_\{j\}\\geq 3\. Conversely, these restricted relations imply equality of the joint verdict distributions\.\([Proof: App\.A\.2\.](https://arxiv.org/html/2609.35797#A1.SS2)\)

#### What this means\.

Each binary coordinate permits a positive affine stretch around its pass threshold, a classical ordinal\-probit indeterminacy\[[Muthén, 1984](https://arxiv.org/html/2609.35797#bib.bib6);[Olsson, 1979](https://arxiv.org/html/2609.35797#bib.bib7);[Chib and Greenberg, 1998](https://arxiv.org/html/2609.35797#bib.bib8)\]; when verdicts supply the optimization reward, it becomes an exactly characterizable blind set\. Even the full joint verdict distribution cannot reveal it\. A second fixed threshold forces the stretch to one\. The one\-factor model enters only when we characterize which reference\-like changes remain invisible\.

Design result: preserve grades rather than binarizing them\.Under joint\-Gaussian scores, a binary criterion adds information and one affine stretch ambiguity; a second finite threshold removes it\. Thus adding binary criteria buys information and blindness together, and identifiability is bounded by the coarsest member\. Without Gaussianity, extra grades constrain more points but cannot reveal arbitrary within\-grade changes\.

### 3\.2The blind direction that remains

###### Corollary 1\(Maximality of loading\-aligned mean attacks\)\.

Fix\(m,c\)∈Θref\(m,c\)\\in\\Theta\_\{\\mathrm\{ref\}\}and consider a deterministic mean shiftSatt∼NJ​\(m​λ\+a,Σ⁡\(c\)\)S^\{\\mathrm\{att\}\}\\sim N\_\{J\}\(m\\lambda\+a,\\Sigma\(c\)\)\. If some pairj≠kj\\neq khasλj​λk≠0\\lambda\_\{j\}\\lambda\_\{k\}\\neq 0andψj\+ψk\>0\\psi\_\{j\}\+\\psi\_\{k\}\>0, the attacked joint binary or fixed\-cutpoint ordinal verdict distribution equals a reference distribution if and only ifa=δ​λa=\\delta\\lambdaand\(m′,c′\)=\(m\+δ,c\)\(m^\{\\prime\},c^\{\\prime\}\)=\(m\+\\delta,c\)for someδ∈ℝ\\delta\\in\\mathbb\{R\}\.\([Proof: App\.A\.3\.](https://arxiv.org/html/2609.35797#A1.SS3)\)

#### What this means\.

After the affine stretch ambiguity is removed, a fixed\-covariance mean attack is invisible exactly when it moves every criterion in proportion to that criterion’s sensitivity to the shared factor\. To the panel, that direction looks exactly like “the model got better\.” It has the same signature as a sycophancy\-shaped stylistic lift that reads as competence across every criterion\. Grading therefore changes the blind set from many criterion\-specific distortions to one interpretable reference\-like direction in this attack class\.

### 3\.3What grading cannot fix

The remaining limitation does not rely on Gaussianity or on the number of grades\.

###### Proposition 1\(Pathwise imitation of reference motion\)\.

Fix any jointly distributed latent array\{\(Zi,εi​1,…,εi​J\)\}i∈ℐ\\\{\(Z\_\{i\},\\varepsilon\_\{i1\},\\ldots,\\varepsilon\_\{iJ\}\)\\\}\_\{i\\in\\mathcal\{I\}\}\. No Gaussianity, independence, or identical sampling is required\. For any target\(m′,c′\)\(m^\{\\prime\},c^\{\\prime\}\), the intervention

Δi​j=λj​\[\(m′−m\)\+\(c′−c\)​Zi\]\\Delta\_\{ij\}=\\lambda\_\{j\}\\\!\\left\[\(m^\{\\prime\}\-m\)\+\(\\sqrt\{c^\{\\prime\}\}\-\\sqrt\{c\}\)Z\_\{i\}\\right\]satisfiesSi​j​\(m,c\)\+Δi​j=Si​j​\(m′,c′\)S\_\{ij\}\(m,c\)\+\\Delta\_\{ij\}=S\_\{ij\}\(m^\{\\prime\},c^\{\\prime\}\)pathwise\. Under any fixed memberwise quantizers, every fixed\-horizon, sequential, or verdict\-adaptive rule therefore receives the same evidence from the intervention and the matched reference process\. Adaptive prompt selection additionally requires the identity conditional on every selectable prompt and history\.\([Proof: App\.A\.4\.](https://arxiv.org/html/2609.35797#A1.SS4)\)

#### What this means\.

This identity marks the irreducible boundary after the avoidable binary stretch ambiguity is removed\. If external evidence labels the matched intervention as reward hacking, a verdict\-only level\-α\\alphatest has power at mostα\\alphafor every sample size\. Whenc′=cc^\{\\prime\}=c, the shift is simply\(m′−m\)​λ\(m^\{\\prime\}\-m\)\\lambda\. Changing scale additionally requires the realized shared factor or an exact proxy\. This is an existence result; whether ordinary training finds such a policy is open\.

## 4Off the blind set: three detection regimes

Figure[2](https://arxiv.org/html/2609.35797#S4.F2)c–d shows the two nonzero regimes: changing margins or cross\-criterion dependence; the third is zero\-power motion within the reference family\.

Figure 2:Taxonomy of verdict\-only evidence\.\(a\)Reference motion improves verdicts and gold\.\(b\)A blind change preserves the joint verdict law while gold worsens\.\(c\)Green and red grids show reference and changed pairwise agreement at fixed criterion pass totals\.\(d\)Margin changes are detectable departures\.###### Theorem 2\(Local information geometry\)\.

Minimum KL separation from the reference family is quadratic for first\-order motion away from it, quartic for first\-order tangent motion with nonzero normal relative acceleration, and zero for paths inside the family\. Under the standard i\.i\.d\. heuristicn​DKL=O⁡\(1\)nD\_\{\\mathrm\{KL\}\}=O\(1\), these correspond, respectively, ton−1/2n^\{\-1/2\}\-scale detection,n−1/4n^\{\-1/4\}\-scale detection, and zero power\.\([Full statement and proof: App\.A\.5\.](https://arxiv.org/html/2609.35797#A1.SS5)\)

We test only the first and zero\-power regimes\.

## 5Evidence: the fix, its scope, and its boundary

We trainQwen/Qwen3\-4B\-Instruct\-2507with GRPO\[[Shao et al\., 2024](https://arxiv.org/html/2609.35797#bib.bib19)\]on MATH\-500\[[Hendrycks et al\., 2021](https://arxiv.org/html/2609.35797#bib.bib17);[Lightman et al\., 2023](https://arxiv.org/html/2609.35797#bib.bib18)\], SciBench\[[Wang et al\., 2024](https://arxiv.org/html/2609.35797#bib.bib20)\], and HealthBench\[[Arora et al\., 2025](https://arxiv.org/html/2609.35797#bib.bib21)\], scored by a fixedQwen/Qwen3\.5\-27Bjudge on seven criteria \(four for HealthBench\)\. Gold is withheld from training and panel analysis\. Appendix[B](https://arxiv.org/html/2609.35797#A2)reports the protocol; Appendix[C](https://arxiv.org/html/2609.35797#A3)gives training and reproducibility details\.

#### Model\-implied stretch stress\.

We freeze the seven\-criterion geometry\. Rescaling one fitted criterion at a time about its binary cutpoint is a theorem sanity check: binary complete\-law total variation \(TV\) is zero for all1414criterion–domain pairs at each stress level\. With three grades and a1\.5×1\.5\\timesstress, TV is0\.02340\.0234–0\.11460\.1146on MATH and0\.02480\.0248–0\.11170\.1117on SciBench\. Atn=1,024n=1\{,\}024, estimated power for a level\-0\.050\.05likelihood\-ratio test given both population laws was at least96\.5%96\.5\\%on MATH and97\.7%97\.7\\%on SciBench across all 14 experiments \(4,000 repetitions\)\. These fitted\-law stresses are summarized in Figure[3](https://arxiv.org/html/2609.35797#S5.F3)\.

#### Realized\-score stress\.

Applying the same threshold\-centered transformation directly to realized scores leaves every binary verdict vector unchanged pathwise, while empirical three\-grade TV is0\.0700\.070–0\.2760\.276across transforms, so the effect is not merely a fitted\-law artifact\.

![Refer to caption](https://arxiv.org/html/2609.35797v1/figure_grading_resolution_review.png)Figure 3:Grading resolution removes the fitted binary stretch ambiguity\.Each point rescales one criterion\. Binary complete\-law TV is zero throughout; a third grade reveals every direction\.
#### The residual blind direction is realizable\.

On frozen MATH geometry, a constructed genuine\-gain law raises the shared factor while an exploit law lowers it and applies the matched loading\-aligned translation\. Mapping the shared factor through the study’s gold\-accuracy link predicts that the two constructed policy laws differ by 18\.25 percentage points in implied gold accuracy, yet their panel rewards match and their complete2,1872\{,\}187\-cell joint grade tables are numerically identical \(population TV1\.6×10−161\.6\\times 10^\{\-16\}\)\. In a separate construction perturbed away from that exact equality, TV rises to0\.03280\.0328and estimated known\-law oracle power is0\.93870\.9387atn=1,024n=1\{,\}024\. Grading removes the affine stretch ambiguity, but not this reference\-like construction\.

#### The orbit is an approximation\.

On post\-calibration reference windows, orbit prediction improves margins and dependence over a no\-motion baseline on MATH and SciBench, but residuals remain above sampling floors \(Table[1](https://arxiv.org/html/2609.35797#S5.T1)\)\. HealthBench does not beat the no\-motion null, perhaps because heterogeneous clinical tasks induce criterion\-specific rather than shared\-factor motion; we therefore do not interpret its orbit coordinates\.

Table 1:Reference\-arm orbit stress test\.Raw orbit/no\-motion errors and sampling floors \(matched\-sample noise under a held\-fixed fitted law\) on post\-calibration windows\. Scale gap is\|c^marg−c^dep\|\|\\hat\{c\}\_\{\\mathrm\{marg\}\}\-\\hat\{c\}\_\{\\mathrm\{dep\}\}\|; threshold entries report max raw/residual drift and RMS floor; explained fractions aggregate over thresholds\. HealthBench beats the no\-motion null on neither margins nor dependence; we read this as the scope condition failing and do not interpret its orbit coordinates\.
#### Cutpoint stability is monitored\.

Fixed cutpoints are required forD=ID=I\. Residual drift exceeds its sampling floor on MATH \(0\.2360\.236vs\.0\.0290\.029\) and SciBench \(0\.2030\.203vs\.0\.0370\.037\); monitor drift and middle\-grade occupancy and recalibrate as needed\.

#### Departure was reachable; natural exact blindness was not\.

Ten signed presentation\-over\-soundness trajectories depart, but four candidate\-pool searches find no natural exact blind attack\. The blind set is constructible but was not observed in ordinary training\.

We claim\.\(1\) Binary panels have a coordinatewise affine stretch ambiguity under the Gaussian model; fixed ternary grading removes it\. \(2\) The effect appears for every criterion in two frozen real geometries\. \(3\) The remaining loading\-aligned family is exact in the declared attack class\. \(4\) Off\-orbit information scales with Fisher\-normal displacement\.We do not claim\.That grading detects all reward hacking; a validated detector; or that orbit membership implies benignity\. Stresses are constructed; the judge and residual correlation are unvalidated; replay is same\-run and inexact; and HealthBench fails the reference\-motion check\.

## 6What a practitioner should do

#### Preserve resolution\.

When grades are available, do not collapse them to pass/fail\. Under the joint\-Gaussian model, three fixed grades and two fixed thresholds are the fewest that suffice; finer scales, up to a continuous score, are no worse but remove no further stretch and add cutpoints to hold fixed \(App\.[B](https://arxiv.org/html/2609.35797#A2)\)\. This excludes binary verifiers, and no resolution resolves within\-grade or loading\-aligned shifts\.

#### Monitor reduced verdict statistics\.

Under the Gaussian model, grade margins and pairwise tables identify\(m,c\)\(m,c\)without the sparse3J3^\{J\}\-cell joint grade table \(2,1872\{,\}187cells here\)\. Freeze the instrument, fit by pairwise composite likelihood, and bootstrap the residual to flag departures\.

#### Pair verdicts with external evidence\.

On\-orbit silence is uninformative, and off\-orbit movement may be genuine improvement\. Treat departure as a trigger for held\-out gold, an independent judge, or response inspection and not as a diagnosis\.

## References

- Amodeiet al\.\(2016\)D\. Amodei, C\. Olah, J\. Steinhardt, P\. Christiano, J\. Schulman, and D\. ManéConcrete problems in AI safety\.Note:arXiv:1606\.06565Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Aroraet al\.\(2025\)R\. K\. Arora, J\. Wei, R\. S\. Hicks, P\. Bowman, J\. Quiñonero\-Candela, F\. Tsimpourlas, M\. Sharman, M\. Shah, A\. Vallone, A\. Beutel, J\. Heidecke, and K\. SinghalHealthBench: evaluating large language models towards improved human health\.Note:arXiv:2505\.08775Cited by:[§5](https://arxiv.org/html/2609.35797#S5.p1.1)\.
- Chib and Greenberg \(1998\)S\. Chib and E\. GreenbergAnalysis of multivariate probit models\.Biometrika85\(2\),pp\. 347–361\.External Links:[Document](https://dx.doi.org/10.1093/biomet/85.2.347)Cited by:[§3\.1](https://arxiv.org/html/2609.35797#S3.SS1.SSS0.Px1.p1.1)\.
- Gaoet al\.\(2023\)L\. Gao, J\. Schulman, and J\. HiltonScaling laws for reward model overoptimization\.InInternational Conference on Machine Learning,Note:arXiv:2210\.10760Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Gunjalet al\.\(2026\)A\. Gunjal, A\. Wang, E\. Lau, V\. Nath, Y\. He, B\. Liu, and S\. HendryxRubrics as rewards: reinforcement learning beyond verifiable domains\.InInternational Conference on Learning Representations,Note:arXiv:2507\.17746Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Hendryckset al\.\(2021\)D\. Hendrycks, C\. Burns, S\. Kadavath, A\. Arora, S\. Basart, E\. Tang, D\. Song, and J\. SteinhardtMeasuring mathematical problem solving with the MATH dataset\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Note:arXiv:2103\.03874Cited by:[§5](https://arxiv.org/html/2609.35797#S5.p1.1)\.
- Jayalathet al\.\(2025\)D\. Jayalath, S\. Goel, T\. Foster, P\. Jain, S\. Gururangan, C\. Zhang, A\. Goyal, and A\. ScheltenCompute as teacher: turning inference compute into reference\-free supervision\.InNeurIPS 2025 Workshop: Efficient Reasoning,Note:arXiv:2509\.14234Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Karwowskiet al\.\(2024\)J\. Karwowski, O\. Hayman, X\. Bai, K\. Kiendlhofer, C\. Griffin, and J\. SkalseGoodhart’s law in reinforcement learning\.InInternational Conference on Learning Representations,Note:arXiv:2310\.09144Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Kohli \(2026\)G\. KohliNine judges, two effective votes: correlated errors undermine LLM evaluation panels\.Note:arXiv:2605\.29800Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.Note:arXiv:2305\.20050Cited by:[§5](https://arxiv.org/html/2609.35797#S5.p1.1)\.
- Mahmoudet al\.\(2026\)A\. Mahmoud, M\. Rezaei, Z\. Wang, A\. Gunjal, B\. Liu, and Y\. HeReward hacking in rubric\-based reinforcement learning\.Note:arXiv:2605\.12474Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Muthén \(1984\)B\. MuthénA general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators\.Psychometrika49\(1\),pp\. 115–132\.External Links:[Document](https://dx.doi.org/10.1007/BF02294210)Cited by:[§3\.1](https://arxiv.org/html/2609.35797#S3.SS1.SSS0.Px1.p1.1)\.
- Olsson \(1979\)U\. OlssonMaximum likelihood estimation of the polychoric correlation coefficient\.Psychometrika44\(4\),pp\. 443–460\.External Links:[Document](https://dx.doi.org/10.1007/BF02296207)Cited by:[§3\.1](https://arxiv.org/html/2609.35797#S3.SS1.SSS0.Px1.p1.1)\.
- Plackett \(1954\)R\. L\. PlackettA reduction formula for normal multivariate integrals\.Biometrika41\(3/4\),pp\. 351–360\.External Links:[Document](https://dx.doi.org/10.1093/biomet/41.3-4.351),[Link](https://academic.oup.com/biomet/article-abstract/41/3-4/351/230622)Cited by:[§A\.1](https://arxiv.org/html/2609.35797#A1.SS1.p3.1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.Note:arXiv:2402\.03300Cited by:[Appendix B](https://arxiv.org/html/2609.35797#A2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.35797#S5.p1.1)\.
- Shiet al\.\(2025\)L\. Shi, C\. Ma, W\. Liang, X\. Diao, W\. Ma, and S\. VosoughiJudging the judges: a systematic study of position bias in LLM\-as\-a\-judge\.InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics,pp\. 292–314\.Note:arXiv:2406\.07791Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Skalseet al\.\(2022\)J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. KruegerDefining and characterizing reward gaming\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 9460–9471\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/3d719fee332caa23d5038b8a90e81796-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Vergaet al\.\(2024\)P\. Verga, S\. Hofstätter, S\. Althammer, Y\. Su, A\. Piktus, A\. Arkhangorodsky, M\. Xu, N\. White, and P\. LewisReplacing judges with juries: evaluating LLM generations with a panel of diverse models\.Note:arXiv:2404\.18796Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Wanget al\.\(2024\)X\. Wang, Z\. Hu, P\. Lu, Y\. Zhu, J\. Zhang, S\. Subramaniam, A\. R\. Loomba, S\. Zhang, Y\. Sun, and W\. WangSciBench: evaluating college\-level scientific problem\-solving abilities of large language models\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 50622–50649\.External Links:[Link](https://proceedings.mlr.press/v235/wang24z.html)Cited by:[§5](https://arxiv.org/html/2609.35797#S5.p1.1)\.
- Wanget al\.\(2026\)X\. Wang, Z\. Hao, S\. Hou, H\. Peng, J\. Li, and X\. WangReproducing, analyzing, and detecting reward hacking in rubric\-based reinforcement learning\.Note:arXiv:2606\.04923Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Yanget al\.\(2026\)M\. Yang, X\. Guo, U\. Tyagi, M\. Zhang, R\. Dumitru, S\. Hou, Y\. He, D\. Y\. Zhang, and Y\. LiuRubric dropout: a simple way to mitigate reward hacking in rubric\-as\-reward rl\.Note:arXiv:2608\.11669Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.Note:arXiv:2306\.05685Cited by:[§1](https://arxiv.org/html/2609.35797#S1.p1.1)\.

## Appendix AProofs

This appendix proves every formal result stated in the main text\. We first record the one property of the reference family used in two of the proofs\.

### A\.1Reference\-orbit identification

WriteΨ=diag⁡\(ψ\)\\Psi=\\operatorname\{diag\}\(\\psi\),Σ⁡\(c\)=Ψ\+c​λ​λ⊤\\Sigma\(c\)=\\Psi\+c\\lambda\\lambda^\{\\top\}, and

sj​\(c\)2=Var⁡\(Sj\)=ψj\+c​λj2\.s\_\{j\}\(c\)^\{2\}=\\operatorname\{Var\}\(S\_\{j\}\)=\\psi\_\{j\}\+c\\lambda\_\{j\}^\{2\}\.For a finite cutpointτj​ℓ\\tau\_\{j\\ell\}, letpj​ℓ=Pr⁡\(Sj\>τj​ℓ\)p\_\{j\\ell\}=\\Pr\(S\_\{j\}\>\\tau\_\{j\\ell\}\), and letRj​k​\(c\)=Corr⁡\(Sj,Sk\)R\_\{jk\}\(c\)=\\operatorname\{Corr\}\(S\_\{j\},S\_\{k\}\)\.

###### Lemma 1\(Identification of the reference orbit\)\.

Suppose some pairj≠kj\\neq ksatisfiesλj​λk≠0\\lambda\_\{j\}\\lambda\_\{k\}\\neq 0andψj\+ψk\>0\\psi\_\{j\}\+\\psi\_\{k\}\>0\. Then the joint Gaussian verdict distribution identifies\(m,c\)\(m,c\)\. The map\(m,c\)↦qm,c\(m,c\)\\mapsto q\_\{m,c\}of reference parameters to verdict\-cell probabilities is smooth, one\-to\-one, and has rank two at every interior law\.

###### Proof\.

For the informative pair,

Rj​k​\(c\)=c​λj​λksj​\(c\)​sk​\(c\)\.R\_\{jk\}\(c\)=\\frac\{c\\lambda\_\{j\}\\lambda\_\{k\}\}\{s\_\{j\}\(c\)s\_\{k\}\(c\)\}\.The factor multiplyingλj​λk\\lambda\_\{j\}\\lambda\_\{k\}is one\-to\-one incc, since

dd​c​log⁡csj​\(c\)​sk​\(c\)=12​c​\{ψjsj​\(c\)2\+ψksk​\(c\)2\}\>0\.\\frac\{d\}\{dc\}\\log\\frac\{c\}\{s\_\{j\}\(c\)s\_\{k\}\(c\)\}=\\frac\{1\}\{2c\}\\left\\\{\\frac\{\\psi\_\{j\}\}\{s\_\{j\}\(c\)^\{2\}\}\+\\frac\{\\psi\_\{k\}\}\{s\_\{k\}\(c\)^\{2\}\}\\right\\\}\>0\.ThusRj​k​\(c\)R\_\{jk\}\(c\)identifiescc\. At any finite cutpoint,

pj​ℓ=Φ⁡\(m​λj−τj​ℓsj​\(c\)\),m=sj​\(c\)​Φ−1​\(pj​ℓ\)\+τj​ℓλj,p\_\{j\\ell\}=\\Phi\\\!\\left\(\\frac\{m\\lambda\_\{j\}\-\\tau\_\{j\\ell\}\}\{s\_\{j\}\(c\)\}\\right\),\\qquad m=\\frac\{s\_\{j\}\(c\)\\Phi^\{\-1\}\(p\_\{j\\ell\}\)\+\\tau\_\{j\\ell\}\}\{\\lambda\_\{j\}\},so a nonzero loading then identifiesmm\.

It remains only to connect these quantities to the joint verdict distribution\. Its univariate margins give everypj​ℓp\_\{j\\ell\}\. For the informative pair, a bivariate cumulative cell probability has the formΦ2​\(a,b,Rj​k\)\\Phi\_\{2\}\(a,b;R\_\{jk\}\), with finitea,ba,balready known from the margins\. Plackett’s identity\[[Plackett, 1954](https://arxiv.org/html/2609.35797#bib.bib13)\]gives∂Φ2​\(a,b,ρ\)/∂ρ=ϕ2​\(a,b,ρ\)\>0\\partial\\Phi\_\{2\}\(a,b;\\rho\)/\\partial\\rho=\\phi\_\{2\}\(a,b;\\rho\)\>0, so that cell probability identifiesRj​kR\_\{jk\}\. This proves injectivity\. Smoothness is immediate\. Thecc\-derivative has a nonzero correlation component, whereas themm\-derivative has zero correlation component and a nonzero margin component, so the two derivatives are linearly independent\. ∎

### A\.2Gaussian equivalence of joint verdict distributions

###### Proof of Theorem[1](https://arxiv.org/html/2609.35797#Thmtheorem1)\.

We prove the mixed binary–graded statement; the all\-binary case is its special case\. Write

σj2=Σj​j,\(σj†\)2=Σj​j†,\\sigma\_\{j\}^\{2\}=\\Sigma\_\{jj\},\\qquad\(\\sigma\_\{j\}^\{\\dagger\}\)^\{2\}=\\Sigma^\{\\dagger\}\_\{jj\},and letR,R†R,R^\{\\dagger\}be the corresponding correlation matrices\. Equality of the marginal verdict distributions implies, at every finite cutpoint,

τj​ℓ−μjσj=τj​ℓ−μj†σj†\.\\frac\{\\tau\_\{j\\ell\}\-\\mu\_\{j\}\}\{\\sigma\_\{j\}\}=\\frac\{\\tau\_\{j\\ell\}\-\\mu\_\{j\}^\{\\dagger\}\}\{\\sigma\_\{j\}^\{\\dagger\}\}\.Indeed, both sides are obtained by applyingΦ−1\\Phi^\{\-1\}to the same cumulative category probability\.

Forj≠kj\\neq k, equality of a bivariate cumulative verdict probability at one finite cutpoint from each member gives

Φ2​\(aj​ℓ,ak​r,Rj​k\)=Φ2​\(aj​ℓ,ak​r,Rj​k†\),\\Phi\_\{2\}\(a\_\{j\\ell\},a\_\{kr\};R\_\{jk\}\)=\\Phi\_\{2\}\(a\_\{j\\ell\},a\_\{kr\};R\_\{jk\}^\{\\dagger\}\),where the standardized cutpoints agree by \(A\.1\)\. Positive definiteness puts both correlations in\(−1,1\)\(\-1,1\), and Plackett’s identity makes the bivariate normal cdf strictly increasing in its correlation\. HenceR†=RR^\{\\dagger\}=R\.

Setdj=σj†/σj\>0d\_\{j\}=\\sigma\_\{j\}^\{\\dagger\}/\\sigma\_\{j\}\>0\. If memberjjis binary, \(A\.1\) at its only finite thresholdτ¯j\\bar\{\\tau\}\_\{j\}yields

μj†−τ¯j=dj​\(μj−τ¯j\)\.\\mu\_\{j\}^\{\\dagger\}\-\\bar\{\\tau\}\_\{j\}=d\_\{j\}\(\\mu\_\{j\}\-\\bar\{\\tau\}\_\{j\}\)\.IfKj≥3K\_\{j\}\\geq 3, subtracting \(A\.1\) at two distinct finite cutpoints givesσj†=σj\\sigma\_\{j\}^\{\\dagger\}=\\sigma\_\{j\}, sodj=1d\_\{j\}=1, and then \(A\.1\) givesμj†=μj\\mu\_\{j\}^\{\\dagger\}=\\mu\_\{j\}\. Finally,

Σj​k†=σj†​σk†​Rj​k†=dj​dk​Σj​k\.\\Sigma\_\{jk\}^\{\\dagger\}=\\sigma\_\{j\}^\{\\dagger\}\\sigma\_\{k\}^\{\\dagger\}R\_\{jk\}^\{\\dagger\}=d\_\{j\}d\_\{k\}\\Sigma\_\{jk\}\.Thus the stated relations hold with the unique matrixD=diag⁡\(d1,…,dJ\)D=\\operatorname\{diag\}\(d\_\{1\},\\ldots,d\_\{J\}\)\.

Conversely, suppose those relations hold, choosingτ¯j\\bar\{\\tau\}\_\{j\}as the sole threshold for a binary member and the first finite cutpoint for a graded one\. The Gaussian vector

S~=τ¯\+D⁡\(S−τ¯\)\\widetilde\{S\}=\\bar\{\\tau\}\+D\(S\-\\bar\{\\tau\}\)has meanμ†\\mu^\{\\dagger\}and covarianceΣ†\\Sigma^\{\\dagger\}, hence the law ofS†S^\{\\dagger\}\. Positive scaling about a binary threshold preserves its label\. Every graded coordinate hasdj=1d\_\{j\}=1and is unchanged\. Therefore the entire labeled verdict vector is preserved pathwise, which proves equality of the joint verdict distributions\. ∎

### A\.3The remaining loading\-aligned family

###### Proof of Corollary[1](https://arxiv.org/html/2609.35797#Thmcorollary1)\.

Suppose the attacked joint verdict distribution equals a reference distribution at\(m′,c′\)\(m^\{\\prime\},c^\{\\prime\}\)\. For the pair in the corollary, the determinant of its covariance block is

c​λj2​ψk\+c​λk2​ψj\+ψj​ψk\>0,c\\lambda\_\{j\}^\{2\}\\psi\_\{k\}\+c\\lambda\_\{k\}^\{2\}\\psi\_\{j\}\+\\psi\_\{j\}\\psi\_\{k\}\>0,so the pair is nonsingular\. Equality of its bivariate verdict distribution identifies its latent correlation by the same Plackett argument as above\. The one\-to\-one correlation formula in Lemma[1](https://arxiv.org/html/2609.35797#Thmlemma1)then forcesc′=cc^\{\\prime\}=c\.

The attacked and reference marginal variances are consequently equal for every member\. Equality of a cumulative probability at any common finite cutpoint identifies the corresponding marginal means, giving

m​λ\+a=m′​λ\.m\\lambda\+a=m^\{\\prime\}\\lambda\.Thereforea=\(m′−m\)​λa=\(m^\{\\prime\}\-m\)\\lambda\. Conversely, ifa=δ​λa=\\delta\\lambda, then the attacked latent law is exactly the reference law at\(m\+δ,c\)\(m\+\\delta,c\)\. This proves both necessity and sufficiency\. ∎

### A\.4Pathwise imitation

###### Proof of Proposition[1](https://arxiv.org/html/2609.35797#Thmproposition1)\.

Direct substitution gives, simultaneously for everyi,ji,j,

Si​j​\(m,c\)\+Δi​j\\displaystyle S\_\{ij\}\(m,c\)\+\\Delta\_\{ij\}=λj​\(m\+c​Zi\)\+εi​j\+λj​\{\(m′−m\)\+\(c′−c\)​Zi\}\\displaystyle=\\lambda\_\{j\}\(m\+\\sqrt\{c\}\\,Z\_\{i\}\)\+\\varepsilon\_\{ij\}\+\\lambda\_\{j\}\\\{\(m^\{\\prime\}\-m\)\+\(\\sqrt\{c^\{\\prime\}\}\-\\sqrt\{c\}\)Z\_\{i\}\\\}=λj​\(m′\+c′​Zi\)\+εi​j=Si​j​\(m′,c′\)\.\\displaystyle=\\lambda\_\{j\}\(m^\{\\prime\}\+\\sqrt\{c^\{\\prime\}\}Z\_\{i\}\)\+\\varepsilon\_\{ij\}=S\_\{ij\}\(m^\{\\prime\},c^\{\\prime\}\)\.Fixed quantizers therefore produce identical verdict arrays under this coupling\. Any fixed\-horizon or stopping\-time rule is a measurable function of that array and, if randomized, an auxiliary random seed that can be shared under the coupling\. Its output is consequently identical pathwise\. The same induction covers adaptive prompt selection when the identity holds conditionally for every prompt and history the rule can select\. ∎

### A\.5Local information geometry

#### Full statement\.

Letℳ=\{qη:η∈Θref\}\\mathcal\{M\}=\\\{q\_\{\\eta\}:\\eta\\in\\Theta\_\{\\mathrm\{ref\}\}\\\}be the reference cell probabilities and letΠ𝒯\\Pi\_\{\\mathcal\{T\}\}project onto the reference family’s tangent space in Fisher geometry\. For a smooth joint\-verdict\-distribution pathptp\_\{t\}through an interior reference distribution, writeu=p˙0u=\\dot\{p\}\_\{0\}andu⟂=\(I−Π𝒯\)​uu\_\{\\perp\}=\(I\-\\Pi\_\{\\mathcal\{T\}\}\)u\. Then

infηDKL\(pt∥qη\)=t22∥u⟂∥p02\+o\(t2\)\.\\inf\_\{\\eta\}D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|q\_\{\\eta\}\)=\\tfrac\{t^\{2\}\}\{2\}\\\|u\_\{\\perp\}\\\|\_\{p\_\{0\}\}^\{2\}\+o\(t^\{2\}\)\.Ifu⟂=0u\_\{\\perp\}=0, the leading term ist48​‖κ‖p02\+o⁡\(t4\)\\tfrac\{t^\{4\}\}\{8\}\\\|\\kappa\\\|\_\{p\_\{0\}\}^\{2\}\+o\(t^\{4\}\), whereκ\\kappais the normal relative acceleration after subtracting the reference family’s curvature\.

Let𝒴\\mathcal\{Y\}be the finite verdict alphabet andp0=qη0p\_\{0\}=q\_\{\\eta\_\{0\}\}an interior reference distribution\. On the zero\-sum tangent space, use the Fisher inner product

⟨v,w⟩p0=∑y∈𝒴v⁡\(y\)​w​\(y\)p0​\(y\)\.\\langle v,w\\rangle\_\{p\_\{0\}\}=\\sum\_\{y\\in\\mathcal\{Y\}\}\\frac\{v\(y\)w\(y\)\}\{p\_\{0\}\(y\)\}\.WriteQ=D​qη0Q=Dq\_\{\\eta\_\{0\}\}for the Jacobian of the reference map andB=D2​qη0B=D^\{2\}q\_\{\\eta\_\{0\}\}for its second derivative\. Then𝒯=range⁡\(Q\)\\mathcal\{T\}=\\operatorname\{range\}\(Q\)\. Ifu∈𝒯u\\in\\mathcal\{T\}, the rank\-two property gives a uniqueaawithu=Q​au=Qa; withv=p¨0v=\\ddot\{p\}\_\{0\}, the normal relative acceleration in Theorem[2](https://arxiv.org/html/2609.35797#Thmtheorem2)is

κ=\(I−Π𝒯\)​\{v−B⁡\[a,a\]\}\.\\kappa=\(I\-\\Pi\_\{\\mathcal\{T\}\}\)\\\{v\-B\[a,a\]\\\}\.
###### Proof of Theorem[2](https://arxiv.org/html/2609.35797#Thmtheorem2)\.

For positive lawsp,q→p0p,q\\to p\_\{0\}on a finite alphabet, Taylor expansion gives

DKL\(p∥q\)=12∥p−q∥p02\+o\(∥p−q∥p02\)\.D\_\{\\mathrm\{KL\}\}\(p\\\|q\)=\\frac\{1\}\{2\}\\lVert p\-q\\rVert\_\{p\_\{0\}\}^\{2\}\+o\(\\lVert p\-q\\rVert\_\{p\_\{0\}\}^\{2\}\)\.The linear term vanishes because both laws sum to one\.

First consider the local infimum\. For a reference curveη⁡\(t\)=η0\+t​b\+o⁡\(t\)\\eta\(t\)=\\eta\_\{0\}\+tb\+o\(t\),

pt−qη⁡\(t\)=t⁡\(u−Q​b\)\+o⁡\(t\)\.p\_\{t\}\-q\_\{\\eta\(t\)\}=t\(u\-Qb\)\+o\(t\)\.Minimizing the leading Fisher norm overbbprojectsuuonto𝒯\\mathcal\{T\}\. Equation \(A\.4\) therefore gives

infη​near​η0DKL\(pt∥qη\)=t22∥u⟂∥p02\+o\(t2\)\.\\inf\_\{\\eta\\ \\mathrm\{near\}\\ \\eta\_\{0\}\}D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|q\_\{\\eta\}\)=\\frac\{t^\{2\}\}\{2\}\\lVert u\_\{\\perp\}\\rVert\_\{p\_\{0\}\}^\{2\}\+o\(t^\{2\}\)\.Formally,QQhas full column rank by Lemma[1](https://arxiv.org/html/2609.35797#Thmlemma1), so the Hessian ofη↦DKL\(p0∥qη\)\\eta\\mapsto D\_\{\\mathrm\{KL\}\}\(p\_\{0\}\\\|q\_\{\\eta\}\)is the positive\-definite matrixQ⊤​diag⁡\(p0−1\)​QQ^\{\\top\}\\operatorname\{diag\}\(p\_\{0\}^\{\-1\}\)Q\. The implicit\-function theorem thus supplies the unique smooth local minimizer used in \(A\.5\)\.

Now supposeu=Q​au=Qa\. A second\-order reference curve has the formη⁡\(t\)=η0\+t​a\+t2​b/2\+o⁡\(t2\)\\eta\(t\)=\\eta\_\{0\}\+ta\+t^\{2\}b/2\+o\(t^\{2\}\), and hence

pt−qη⁡\(t\)=t22​\{v−B⁡\[a,a\]−Q​b\}\+o⁡\(t2\)\.p\_\{t\}\-q\_\{\\eta\(t\)\}=\\frac\{t^\{2\}\}\{2\}\\\{v\-B\[a,a\]\-Qb\\\}\+o\(t^\{2\}\)\.Minimizing overbbremoves the tangent component and leavesκ\\kappafrom \(A\.3\)\. Applying \(A\.4\) again yields

infη​near​η0DKL\(pt∥qη\)=t48∥κ∥p02\+o\(t4\)\.\\inf\_\{\\eta\\ \\mathrm\{near\}\\ \\eta\_\{0\}\}D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|q\_\{\\eta\}\)=\\frac\{t^\{4\}\}\{8\}\\lVert\\kappa\\rVert\_\{p\_\{0\}\}^\{2\}\+o\(t^\{4\}\)\.
It remains to show that the infima in \(A\.5\)–\(A\.6\) are global\. Fix a neighborhoodUUofη0\\eta\_\{0\}\. If a sequenceηn=\(mn,cn\)∉U\\eta\_\{n\}=\(m\_\{n\},c\_\{n\}\)\\notin UsatisfiedTV⁡\(qηn,p0\)→0\\operatorname\{TV\}\(q\_\{\\eta\_\{n\}\},p\_\{0\}\)\\to 0, all of its univariate and bivariate verdict margins would converge to those ofp0p\_\{0\}\. At the informative pair, continuous inversion of the bivariate Gaussian cdf would giveRj​k​\(cn\)→Rj​k​\(c0\)R\_\{jk\}\(c\_\{n\}\)\\to R\_\{jk\}\(c\_\{0\}\)\. The strict monotonicity in Lemma[1](https://arxiv.org/html/2609.35797#Thmlemma1)impliescn→c0c\_\{n\}\\to c\_\{0\}, and any finite cutpoint with nonzero loading then gives

mn=sj​\(cn\)​Φ−1​\(pj​ℓ,n\)\+τj​ℓλj⟶m0,m\_\{n\}=\\frac\{s\_\{j\}\(c\_\{n\}\)\\Phi^\{\-1\}\(p\_\{j\\ell,n\}\)\+\\tau\_\{j\\ell\}\}\{\\lambda\_\{j\}\}\\longrightarrow m\_\{0\},contradictingηn∉U\\eta\_\{n\}\\notin U\. HenceδU:=infη∉UTV⁡\(p0,qη\)\>0\\delta\_\{U\}:=\\inf\_\{\\eta\\notin U\}\\operatorname\{TV\}\(p\_\{0\},q\_\{\\eta\}\)\>0\. For small enoughtt, the triangle inequality and Pinsker’s inequality give

infη∉UDKL\(pt∥qη\)≥δU2/2,\\inf\_\{\\eta\\notin U\}D\_\{\\mathrm\{KL\}\}\(p\_\{t\}\\\|q\_\{\\eta\}\)\\geq\\delta\_\{U\}^\{2\}/2,whereas the local comparison tends to zero\. Thus no parameter outsideUUcan attain or approach the global infimum, so the global and local infima agree\. This completes the proof\. ∎

## Appendix BExperimental protocol and evidence scope

#### Policies, tasks, and judge\.

Qwen/Qwen3\-4B\-Instruct\-2507policies are trained with GRPO\[[Shao et al\., 2024](https://arxiv.org/html/2609.35797#bib.bib19)\]on MATH\-500, SciBench, and HealthBench\. One fixedQwen/Qwen3\.5\-27Bjudge scores each response on seven criteria for MATH and SciBench and four for HealthBench\. MATH and SciBench use the grading and binarization rule reported in Section[1](https://arxiv.org/html/2609.35797#S1)\. Task\-specific external gold is withheld from both training and verdict\-only analysis\.

#### Frozen measurement geometry\.

For the grading\-resolution ablation, thresholds, loadings, and residual variances are estimated from8,1928\{,\}192early rollouts per domain and then held fixed\. Finite thresholds are recovered from cumulative graded margins by probit inversion; pairwise latent correlations are obtained from the polychoric Gaussian model; and the one\-factor geometry is fit to the resulting latent correlation structure\. Middle\-grade occupancy pools all seven criteria:9,526/57,344=16\.6%9\{,\}526/57\{,\}344=16\.6\\%on MATH and9,721/57,344=17\.0%9\{,\}721/57\{,\}344=17\.0\\%on SciBench; each criterion contributes8,1928\{,\}192ratings\. Post\-calibration replay fits the earliest at most eight steps of each run, excludes calibration\-overlapping evaluation windows, and then freezes the geometry\. The replay covers five reference\-designated and five stress\-test runs in each of MATH and SciBench, plus five reference\-designated and three adversarial stress runs in HealthBench\. Reference designation is a protocol label and does not by itself establish semantic benignity\.

#### Why three grades, and why not more\.

Three is the fewest that works, and under the joint\-Gaussian model it is also enough\. In the proof of Theorem[1](https://arxiv.org/html/2609.35797#Thmtheorem1), two distinct finite cutpoints forceσj†=σj\\sigma\_\{j\}^\{\\dagger\}=\\sigma\_\{j\}, hencedj=1d\_\{j\}=1andμj†=μj\\mu\_\{j\}^\{\\dagger\}=\\mu\_\{j\}: the entire marginal score law of that member is pinned onceKj≥3K\_\{j\}\\geq 3, so a fourth or fifth grade removes no further stretch\. Identification is memberwise, so the operative rule is that no member may be binary, not that every member carry exactly three grades; a single binary member reintroduces one stretch on its own coordinate\. A fully continuous score would not reach what remains\. Both residual results are equalities of the score law itself rather than quantization artifacts: the loading\-aligned attack of Corollary[1](https://arxiv.org/html/2609.35797#Thmcorollary1)satisfiesNJ​\(m​λ\+δ​λ,Σ⁡\(c\)\)=NJ​\(\(m\+δ\)​λ,Σ⁡\(c\)\)N\_\{J\}\(m\\lambda\+\\delta\\lambda,\\Sigma\(c\)\)=N\_\{J\}\(\(m\+\\delta\)\\lambda,\\Sigma\(c\)\), which is exactly the reference law at\(m\+δ,c\)\(m\+\\delta,c\), and the intervention of Proposition[1](https://arxiv.org/html/2609.35797#Thmproposition1)matches the reference process pathwise\. Neither is visible to any function of the scores, at any resolution\. Finer scales also carry costs our own measurements show are not hypothetical:D=ID=Irequires every cutpoint to stay fixed, and residual cutpoint drift already exceeds its sampling floor \(0\.2360\.236vs\.0\.0290\.029on MATH,0\.2030\.203vs\.0\.0370\.037on SciBench\); identifying a cutpoint requires mass on both of its sides, and the middle grade already falls to7\.0%7\.0\\%on the thinnest criterion; and the joint grade table grows asKJK^\{J\}, from2,1872\{,\}187cells atK=3K=3andJ=7J=7to78,12578\{,\}125atK=5K=5, which is why Section[6](https://arxiv.org/html/2609.35797#S6)recommends margins and pairwise tables instead\. We did not test judges that emit continuous scores, so this comparison is a statement about the model and not an empirical one\.

#### Distances, sampling floor, and oracle power\.

For categorical distributionsppandqq, total variation is

TV⁡\(p,q\)=12​∑y\|p⁡\(y\)−q⁡\(y\)\|\.\\operatorname\{TV\}\(p,q\)=\\tfrac\{1\}\{2\}\\sum\_\{y\}\|p\(y\)\-q\(y\)\|\.Known\-law power uses the likelihood\-ratio test between two fixed population distributions at level0\.050\.05, estimated from 4,000 Monte Carlo repetitions withn=1,024n=1\{,\}024samples per repetition\. It is an upper benchmark because an operational monitor must estimate its reference law\. Sampling floors are generated from matched\-size parametric samples of a held\-fixed fitted law, without rerunning the full fit; they are descriptive finite\-sample noise levels, not confidence bounds\.

#### Constructed and observed searches\.

The model\-implied stress rescales one fitted criterion at a time about its binary threshold, leaving all other coordinates fixed\. The direct realized\-score stress applies the corresponding transformation to observed scores\. The residual\-direction construction compares a shared\-factor gain with a matched loading\-aligned translation and then perturbs transversely away from that exact equality\. Candidate\-pool searches use four\-to\-eight\-step windows \(approximately4,0964\{,\}096–8,1928\{,\}192rollouts\)\. The reweighting harm gate requires at least a five\-point gold\-accuracy loss, and the component tolerance bounds every transverse moment residual by0\.0250\.025\. For the weak\-verifier screen, breadth requires high\-verifier incorrect selections on at least 10 prompts and in all five policy seeds; validity requires at least 99% parse\-valid and at most 1% unscorable responses; and truncation permits at most 1% hard\-truncated responses\. Gold\-blind reweighting gains3\.073\.07percentage points of signed reward and loses4\.404\.40points of gold accuracy, missing both gates\. Presentation\-only selection raises both presentation \(6\.226\.22points\) and gold accuracy \(5\.665\.66points\)\. The matched\-control construction stops before reading gold because it cannot optimize the required control under the frozen geometry\. Weak\-verifier best\-of\-NNgains0\.15560\.1556proxy score, but its accuracy is0\.520\.52points above matched random \(95% seed\-first, prompt\-second percentile\-bootstrap CI\[−3\.13,4\.69\]\[\-3\.13,4\.69\]\) and its breadth, validity, and truncation gates fail\. These pool\-specific failures do not establish that trained policies can never enter the blind set\.

#### Additional operational disclosures\.

Output resolution\.Three fixed grades; two cutpoints held fixed; no pass/fail collapse\.Oracle benchmark\.Atn=1,024n=1\{,\}024, three grades reveal all 14 criterion–domain pairs at the1\.5×1\.5\\timesstress with known\-law power at least96\.5%96\.5\\%; this is not operational power\.External trigger\.For a residual\-direction gain or failed fit, require held\-out gold, an independent judge, or response inspection\.

## Appendix CReproducibility details for the confirmatory runs

#### Scope, pairing, and intervention\.

This section documents the MATH and SciBench confirmatory trajectories used in the replay analyses\. Each domain crosses five registered seeds\{1,2,3,4,5\}\\\{1,2,3,4,5\\\}with benign and hacked arms, for ten runs per domain and twenty runs total\. Within a seed, the two arms use the samedata\.seedand shuffled data order\. Each run starts afresh fromQwen/Qwen3\-4B\-Instruct\-2507; checkpoint resumption is disabled\. The only arm\-level intervention is the criterion weighting in the training reward\. For MATH, the hacked weights arewpresentation=1\.0w\_\{\\rm presentation\}=1\.0,wthoroughness=0\.0w\_\{\\rm thoroughness\}=0\.0, andwsoundness=−0\.6w\_\{\\rm soundness\}=\-0\.6\. For SciBench, they arewpresentation=1\.0w\_\{\\rm presentation\}=1\.0andwsoundness=−0\.6w\_\{\\rm soundness\}=\-0\.6\. In both benign arms the weight variable is unset, which invokes the uniform mean of all seven criteria\. All other data, optimization, sampling, and serving settings are shared across arms\.

Table[C](https://arxiv.org/html/2609.35797#A3.SS0.SSS0.Px1)reports untruncated MATH windows \(436 steps; 135 remain after the coherence\-collapse gate\) and SciBench windows ending near collapse \(180 steps; 149 remain\)\.

Table 2:Confirmatory run matrix\.“Post\-reference” excludes the fixed reference epoch at steps 0–8\. Configured horizons and retained artifact lengths are separated because two MATH benign runs retain legacy 150\-step trajectories\.
#### GRPO and optimizer\.

We use GRPO \(algorithm\.adv\_estimator=grpo\)\. Each optimizer step samples 64 prompts with 16 policy rollouts per prompt, yielding 1,024 responses per step\. The PPO mini\-batch size is 32 and the per\-GPU micro\-batch size is 4\. The full\-parameter actor uses AdamW with learning rate3×10−63\\times 10^\{\-6\}, betas\(0\.9,0\.999\)\(0\.9,0\.999\), weight decay 0\.01, a constant schedule, 15 warmup steps, and gradient\-norm clipping at 1\.0\. Gradient checkpointing is enabled\. FSDP parameter and optimizer offload are enabled for the actor, and parameter offload is enabled for the reference policy\. The actor KL loss is enabled with coefficient10−310^\{\-3\}and the low\-variance KL form; KL is not added to the reward and the entropy coefficient is zero\.

#### Policy and judge sampling\.

Training and validation policy sampling both use temperature 0\.7,top\_p=0\.8\\texttt\{top\\\_p\}=0\.8,top\_k=20\\texttt\{top\\\_k\}=20, and sampling enabled\. The maximum response length is 2,048 tokens; maximum prompt length is 1,024 tokens for MATH and 1,536 for SciBench\. The policy rollout engine uses a 4,096\-token model context and at most 16,384 batched tokens\. MATH applies the same training\-only response\-length penalty in both arms: it begins at length 5,500, is hard at 8,000, and has coefficient 1\.0\. SciBench has no length penalty\.

The fixedQwen/Qwen3\.5\-27Bjudge is served separately with vLLM on two GPUs using tensor parallelism 2, bfloat16, and a 32,768\-token server context\. Judge calls use temperature 1\.0, at most 7,700 generated tokens, and thinking disabled\. An incomplete seven\-criterion parse is retried up to three times\. The task\-specific gold signal is computed independently and is excluded from both the reward and verdict\-only monitoring\.

#### Checkpoint and analysis\-window policy\.

There is deliberately no best\-checkpoint selection\. Checkpoints are written every 25 optimizer steps, validation runs every 20 steps, andresume\_mode=disable\. A monitor alarm never halts a hacked run; the configured horizon is the planned stopping rule, and any incidental early termination is retained and reported\. The analysis selects a trajectory window rather than a checkpoint\. Its fixed reference epoch is steps 0–8, and the clean window ends immediately before coherence collapse, defined for this analysis as the first post\-reference step whose mean log\-probability per token is below the run’s reference mean by 0\.5\. Executed horizons are determined by the frozen launcher configurations and recorded rollout artifacts\.

#### Compute and runtime accounting\.

One run uses one node with six H100 GPUs: four for policy training and two for the judge server\. The job timeout is 86,400 seconds and at most three runs are concurrent, for a peak allocation of 18 H100s\. The repository artifacts do not record per\-run start and end timestamps, so we do not report measured H100\-hours\. For scale, one 24\-hour attempt for each of the 20 cells would be a configured ceiling of20×6×24=2,88020\\times 6\\times 24=2\{,\}880H100\-hours, excluding pilots and retries; actual consumption is lower for trajectories that end before the timeout\.