Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning

# Abstention as an Action Can Kill Both the Reward Gradient and the KL Anchor: Collapse Law and Repair for Error-Penalized Reinforcement Learning
Source: [https://arxiv.org/html/2608.00301](https://arxiv.org/html/2608.00301)
###### Abstract

Error\-penalized scoring rules \(\+1\+1for a correct answer,−λ\-\\lambdafor a wrong one,0for abstaining\) are increasingly prescribed against hallucination: a rational agent facing such a rule answers exactly when its correctness probability exceeds Chow’s thresholdt∗=λ/\(1\+λ\)t^\{\\ast\}=\\lambda/\(1\+\\lambda\)\. We prove that a KL\-anchored gradient learner can do the opposite\. When abstention is a discrete action, the reward gradient and the anchor’s restoring force are throttled by the same gate\-saturation factor and die together: under explicit conditions \(among them, blanket answering loses score in expectation and prompts share a bounded readout\) the model drifts toward refusing everything, its mean training reward rising to zero like1/t1/tin training timett, so the curve reads as improvement while coverage collapses\. The advantage estimator compounds the failure: in its sparse\-answer regime, group normalization silently replaces every designed penalty with an effective penalty of one, moving the learned threshold fromt∗t^\{\\ast\}to1/21/2\. The repair is structural: train a mandatory confidence report with a strictly proper score plus a correctness reward, and abstain only at deployment by thresholding the report\. The always\-emitted report has no gate to saturate, so no shared factor can kill its reward gradient and its anchor together, and its calibrated optimum is attracting\. Simulations confirm every prediction, and experiments on language models at two scales confirm the mechanism live: the rule silences questions the models demonstrably still solve within ten optimizer steps, an ablation isolates the cause, and report\-level training raises coverage, accuracy, and calibration together\.

††footnotetext:These authors contributed equally\.## 1Introduction

Language models hallucinate in part because we pay them to\.Kalaiet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib1)\)argue that binary\-graded training and evaluation reward confident guessing over honest abstention, and prescribe the classical remedy: penalize errors, give partial or zero credit to abstention, and state the stakes\. The prescription is spreading through evaluation practice, and a growing line of work adopts it directly as a reinforcement\-learning objective, rewarding answers\+1/−λ/0\+1/\{\-\\lambda\}/0\(Weiet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib4); Songet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib11); Wanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib9)\)\.

For a*rational*agent the prescription is provably correct: facing the rule\(\+1,−λ,0\)\(\+1,\-\\lambda,0\), the optimal policy answers exactly when its correctness probability exceeds Chow’s thresholdt∗=λ/\(1\+λ\)t^\{\\ast\}=\\lambda/\(1\+\\lambda\)\(Chow[1970](https://arxiv.org/html/2608.00301#bib.bib31)\)\. This paper concerns what the same rule does to a*gradient learner*: a policy dragged toward the payoff table by policy\-gradient fine\-tuning with a KL anchor to its base model, the standard RLHF configuration\(Christianoet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib28); Stiennonet al\.[2020](https://arxiv.org/html/2608.00301#bib.bib29); Ouyanget al\.[2022](https://arxiv.org/html/2608.00301#bib.bib30)\)\. We show the two can produce opposite outcomes for policies whose prompts share a bounded readout, and that the difference has a clean mechanical cause\.

#### The mechanism in one identity\.

Under a penalty rule, abstention is a discrete*action*: the model answers with probabilityσ​\(v​\(x\)\)\\sigma\(v\(x\)\), wherevvis a learned gate logit andσ\\sigmathe logistic function\. Every gradient of the expected score with respect to the gate parameters carries the saturation factorσ′​\(v\)\\sigma^\{\\prime\}\(v\)\. So far this is ordinary softmax\-policy\-gradient saturation\(Meiet al\.[2020](https://arxiv.org/html/2608.00301#bib.bib16); Razinet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib15)\)\. The new observation concerns the anchor\. For Bernoulli gates,

dd​v​KL​\(Bern​\(σ​\(v\)\)∥Bern​\(σ​\(v0\)\)\)=σ′​\(v\)​\(v−v0\),\\frac\{d\}\{dv\}\\,\\mathrm\{KL\}\\big\(\\mathrm\{Bern\}\(\\sigma\(v\)\)\\,\\big\\\|\\,\\mathrm\{Bern\}\(\\sigma\(v\_\{0\}\)\)\\big\)\\;=\\;\\sigma^\{\\prime\}\(v\)\\,\(v\-v\_\{0\}\),\(1\)so the KL anchor’s restoring force carries the*same*factor\. If the base model answers nearly everything and blanket answering loses score in expectation \(precisely the situation the penalty rule is designed to punish\), thenvvfalls,σ′​\(v\)\\sigma^\{\\prime\}\(v\)dies, and the anchor’s pull back toward the base dies with it\. The model does not converge to calibrated threshold abstention; it drifts toward answering nothing, ever more slowly, along aβ\\beta\-independent power law, onto a plateau that is metastable rather than absorbing: escape exists, but arrives exponentially late in the gain it must accumulate\. Practitioners’ intuition that “the KL term keeps the policy near its base” fails structurally: KL regularization, ordinarily a stabilizer\(Geistet al\.[2019](https://arxiv.org/html/2608.00301#bib.bib19); Vieillardet al\.[2020](https://arxiv.org/html/2608.00301#bib.bib20)\), provides no restoring force on exactly the coordinate that is collapsing \(the reverse KL would, but it is not what RLHF uses\)\.

#### Contributions\.

1. 1\.Mechanism\.A single factor, the gate’s saturationσ′​\(v\)\\sigma^\{\\prime\}\(v\), throttles the reward gradient and the KL anchor’s restoring force alike: the two die together at the all\-abstain vertex\. Collapse needs three ingredients \(blanket answering loses score, prompts share a bounded readout, abstention is action\-level\); removing any one eliminates the collapse in the analyzed class\. The mechanism lifts from the two\-token gate to sequence policies with branch\-separable content\.
2. 2\.Law\.Under these conditions the collapse follows a finite\-time law: the mean training reward rises to zero like1/t1/tup to log factors, at a time stable in the anchor strength\.
3. 3\.Estimator\.The advantage estimator selects the law: group normalization steepens the decay to1/t21/t^\{2\}above a knee at answer rate≈1/G\{\\approx\}1/G\(GGthe group size\) and, below it, silently replaces every designed penaltyλ\\lambdabyλeff=1\\lambda\_\{\\mathrm\{eff\}\}=1; dynamic resampling conserves the collapse per rollout, yielding an estimator\-indexed menu of falsifiable predictions\.
4. 4\.Repair\.Move abstention from the action space to a mandatory confidence report, trained with a strictly proper score plus a correctness reward and thresholded only at deployment: the report channel has no shared vanishing factor, and its calibrated optimum is interior and attracting\.
5. 5\.Evidence\.Enumeration and simulation confirm every prediction; live tests on a 1\.5B model reproduce the collapse on questions the model provably still solves, isolate its cause by ablation, and measure both gradient norms at zero there, while a single\-seed 7B run adds the predicted late escape\.

## 2Setting

Promptsxxcarry a correctness probabilityq​\(x\)∈\[0,1\]q\(x\)\\in\[0,1\]for the frozen base model’s best candidate answer\. A decision layer observes an internal signalsswith marginal densityp​\(s\)p\(s\)and posteriorq¯​\(s\)=𝔼​\[q∣s\]\\bar\{q\}\(s\)=\\mathbb\{E\}\[q\\mid s\]\. Under the penalty rule\(\+1,−λ,0\)\(\+1,\-\\lambda,0\)the expected score of answering atssism​\(q¯​\(s\)\)m\(\\bar\{q\}\(s\)\), wherem​\(u\):=\(1\+λ\)​u−λm\(u\):=\(1\+\\lambda\)u\-\\lambda, andg​\(s\):=p​\(s\)​m​\(q¯​\(s\)\)g\(s\):=p\(s\)\\,m\(\\bar\{q\}\(s\)\)is the*answering\-gain density*\. The optimal deterministic policy answers iffg​\(s\)\>0g\(s\)\>0, i\.e\. iffq¯​\(s\)\>t∗=λ/\(1\+λ\)\\bar\{q\}\(s\)\>t^\{\\ast\}=\\lambda/\(1\+\\lambda\)\(Chow’s rule\), attainingU∗=∫g\+U^\{\\ast\}=\\int g\_\{\+\}with rational coveragecov∗=∫g\>0p\\mathrm\{cov\}^\{\\ast\}=\\int\_\{g\>0\}p\.

#### Two mechanisms, matched heads\.

Both mechanisms read the same bounded featureϕ​\(s\)=σ​\(w​s\+a\)∈\(0,1\)\\phi\(s\)=\\sigma\(ws\+a\)\\in\(0,1\)through a four\-parameter head; boundedness reflects how common confidence readouts \(sigmoid or softmax heads, verbalized scales\) are built\.

*Action\-level \(penalty rule\)\.*Gate logitvθ​\(s\)=c4​ϕ​\(s\)\+c0v\_\{\\theta\}\(s\)=c\_\{4\}\\phi\(s\)\+c\_\{0\}withθ=\(w,a,c4,c0\)\\theta=\(w,a,c\_\{4\},c\_\{0\}\); during training the model*samples*answer/abstain with probabilityσ​\(vθ​\(s\)\)\\sigma\(v\_\{\\theta\}\(s\)\), and the objective is

Jβ​\(θ\)=\\displaystyle J\_\{\\beta\}\(\\theta\)\\;=\\;∫σ​\(vθ\)​g​𝑑s\\displaystyle\\int\\sigma\(v\_\{\\theta\}\)\\,g\\,ds−β​∫p​KL​\(Bern​\(σ​\(vθ\)\)∥Bern​\(σ​\(v0\)\)\)​𝑑s,\\displaystyle\-\\beta\\int p\\;\\mathrm\{KL\}\\big\(\\mathrm\{Bern\}\(\\sigma\(v\_\{\\theta\}\)\)\\,\\\|\\,\\mathrm\{Bern\}\(\\sigma\(v\_\{0\}\)\)\\big\)\\,ds,\(2\)withv0:=vθ0v\_\{0\}:=v\_\{\\theta\_\{0\}\}the base gate andβ\>0\\beta\>0the anchor strength\. The KL in \([2](https://arxiv.org/html/2608.00301#S2.E2)\) is the gate term of the sequence\-level KL \(the chain rule adds branch\-conditional terms that vanish when branch content matches the base\); Theorem[2](https://arxiv.org/html/2608.00301#Thmtheorem2)treats the full sequence KL with trainable content\.

*Report\-level \(proper rule\)\.*Confidencecθ′​\(s\)=κ0\+\(κ1−κ0\)​ϕ​\(s\)c\_\{\\theta^\{\\prime\}\}\(s\)=\\kappa\_\{0\}\+\(\\kappa\_\{1\}\-\\kappa\_\{0\}\)\\phi\(s\)withθ′=\(w,a,κ0,κ1\)\\theta^\{\\prime\}=\(w,a,\\kappa\_\{0\},\\kappa\_\{1\}\)andκ0,κ1∈\[0,1\]\\kappa\_\{0\},\\kappa\_\{1\}\\in\[0,1\], soc∈\[0,1\]c\\in\[0,1\]\(interior optima in our calibrations; a projected flow preserves the box at the same rates toward the constrained optimum\); the model answers*every*question, reportscc, and is scored by the Brier rule1−\(Y−c\)21\-\(Y\-c\)^\{2\},Y∼Bern​\(q¯\)Y\\sim\\mathrm\{Bern\}\(\\bar\{q\}\)\. Its per\-question output is deterministic givenss, so no action\-policy KL exists; we anchor it with the proximal termβ2​‖θ′−θ0′‖2\\tfrac\{\\beta\}\{2\}\\\|\\theta^\{\\prime\}\-\\theta^\{\\prime\}\_\{0\}\\\|^\{2\}\(function\-space anchors on the report leave every conclusion unchanged; Proposition[8](https://arxiv.org/html/2608.00301#Thmproposition8)\)\.

*Deployment\.*Both greedy: the penalty policy answers iffvθ​\(s\)\>0v\_\{\\theta\}\(s\)\>0and the report policy iffcθ′​\(s\)\>t∗c\_\{\\theta^\{\\prime\}\}\(s\)\>t^\{\\ast\}; both target the same decision region\.

*Initialization\.*Matched, overconfident, weakly discriminative: the base answers≈88%\{\\approx\}88\\%of questions while the true posterior spans\[0\.2,0\.9\]\[0\.2,0\.9\]\(Guoet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib40); Tianet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib38)\)\.

#### Standing assumption and scope\.

\(B1\):D0:=−∫g​𝑑s\>0D\_\{0\}:=\-\\int g\\,ds\>0; equivalently𝔼​\[q\]<t∗\\mathbb\{E\}\[q\]<t^\{\\ast\}: blanket answering loses score in expectation\. It is necessary for collapse\. Two further conditions are necessary, and we state them as sharply as the first\.

###### Proposition 1\(Tabular policies do not collapse\)\.

Under vanilla policy gradient with one free logit per prompt, the flowv˙​\(s\)=σ′​\(v​\(s\)\)​\[m​\(q¯​\(s\)\)−β​\(v​\(s\)−v0​\(s\)\)\]\\dot\{v\}\(s\)=\\sigma^\{\\prime\}\(v\(s\)\)\[\\,m\(\\bar\{q\}\(s\)\)\-\\beta\(v\(s\)\-v\_\{0\}\(s\)\)\\,\]has stationary pointv∗​\(s\)=v0​\(s\)\+m​\(q¯​\(s\)\)/βv^\{\\ast\}\(s\)=v\_\{0\}\(s\)\+m\(\\bar\{q\}\(s\)\)/\\beta\. On every prompt withm​\(q¯​\(s\)\)≠0m\(\\bar\{q\}\(s\)\)\\neq 0the equilibrium decision converges to Chow’s rule asβ↓0\\beta\\downarrow 0; at finiteβ\\betait is correct outside a band of widthO​\(β\)O\(\\beta\)inmmaround the threshold \(uniformly, whenever\|m\|≥γ\>β​‖v0‖∞\|m\|\\geq\\gamma\>\\beta\\\|v\_\{0\}\\\|\_\{\\infty\}\)\.

The population condition \(B1\) can drag profitable prompts down only because the biasc0c\_\{0\}is common to all of them\. The paper’s thesis is the conjunction:*a discrete abstain action supplies a vertex that attracts on theq¯<t∗\\bar\{q\}<t^\{\\ast\}prompts; a shared bounded readout transmits the attraction to theq¯\>t∗\\bar\{q\}\>t^\{\\ast\}prompts; and the KL anchor, which is supposed to arrest exactly such drifts, is throttled by the same factor that drives them\.*

## 3Anatomy of the Collapse

We take the collapse apart in three passes: the static mechanism at the abstain vertex, the finite\-time law it forces under gradient flow, and the estimator dependence that sets the law’s exponent and its effective penalty\.

### One Factor Throttles Both Forces

###### Lemma 1\(Shared throttling factor\)\.

For the objective \([2](https://arxiv.org/html/2608.00301#S2.E2)\),

∇θJβ=∫σ′​\(vθ\)​\[g−β​p​\(vθ−v0\)\]​∇θvθ​d​s\.\\nabla\_\{\\theta\}J\_\{\\beta\}=\\\!\\int\\\!\\sigma^\{\\prime\}\(v\_\{\\theta\}\)\\Big\[g\-\\beta p\\,\(v\_\{\\theta\}\-v\_\{0\}\)\\Big\]\\nabla\_\{\\theta\}v\_\{\\theta\}\\,ds\.Reward and anchor share the factorσ′​\(vθ\)\\sigma^\{\\prime\}\(v\_\{\\theta\}\): where the gate saturates, both vanish \(differentiate \([2](https://arxiv.org/html/2608.00301#S2.E2)\) under the integral; the KL term is \([1](https://arxiv.org/html/2608.00301#S1.E1)\) pointwise\)\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x1.png)Figure 1:The geometry of the failure\. \(a\) A hard threshold needs diverging logit norm; the loss lobe ofggoutweighs its gain lobe \(B1\)\. \(b\) Sharpness is free under the true KL, not under a proximal anchor\. \(c\) The report level needs anO​\(1\)O\(1\)\-norm fit, its sharpness supplied byt∗t^\{\\ast\}\.###### Proposition 2\(The anchor cannot exclude the vertex\)\.

Bernoulli KL to a fixed base is bounded, and at the all\-abstain vertex \(σ​\(v\)≡0\\sigma\(v\)\\equiv 0\) it costs exactlyKL¯⟂:=𝔼p​\[log⁡11−σ​\(v0\)\]<∞\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}:=\\mathbb\{E\}\_\{p\}\[\\log\\frac\{1\}\{1\-\\sigma\(v\_\{0\}\)\}\]<\\infty\. Hence, under\(B1\),Jβ​\(all\-abstain\)−Jβ​\(base\)=L​\(0\)−β​KL¯⟂\>0J\_\{\\beta\}\(\\text\{all\-abstain\}\)\-J\_\{\\beta\}\(\\text\{base\}\)=L\(0\)\-\\beta\\,\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}\>0wheneverβ<βcrit:=L​\(0\)/KL¯⟂\\beta<\\beta\_\{\\mathrm\{crit\}\}:=L\(0\)/\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}, whereL​\(0\)=∫σ​\(v0\)​\(−g\)L\(0\)=\\int\\sigma\(v\_\{0\}\)\(\-g\)is the base policy’s expected loss\. Total abstention strictly beats the base for every anchor strength below an explicit threshold, with no dynamics required\. BecauseKL¯⟂\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}grows only logarithmically in the base answer rate,βcrit\\beta\_\{\\mathrm\{crit\}\}sits above typical RLHF coefficients \(β∈\[10−3,5×10−2\]\\beta\\in\[10^\{\-3\},5\\times 10^\{\-2\}\], with reward in the rule’s own units\) across realistic calibrations\.

###### Proposition 3\(The dichotomy at the vertex is local\)\.

Fix a prompt with posteriorq¯\\bar\{q\}; evaluate the reward gradient at the degenerate vertex of the output’s range\.\(a\)*Action level:*∂v𝔼​\[r\]=σ′​\(v\)​m​\(q¯\)→0\\partial\_\{v\}\\mathbb\{E\}\[r\]=\\sigma^\{\\prime\}\(v\)\\,m\(\\bar\{q\}\)\\to 0asv→−∞v\\to\-\\infty, with the sign ofmm\. The abstain vertex is*attracting on exactly the prompts withq¯<t∗\\bar\{q\}<t^\{\\ast\}*and repelling on the rest\.\(b\)*Report level:*for any strictly proper score with Savage representation𝔼​\[S​\(c\)\]=Γ​\(c\)\+Γ′​\(c\)​\(q¯−c\)\\mathbb\{E\}\[S\(c\)\]=\\Gamma\(c\)\+\\Gamma^\{\\prime\}\(c\)\(\\bar\{q\}\-c\),∂c𝔼​\[S\]=Γ′′​\(c\)​\(q¯−c\)\\partial\_\{c\}\\mathbb\{E\}\[S\]=\\Gamma^\{\\prime\\prime\}\(c\)\(\\bar\{q\}\-c\)\. The expected score strictly increases from the vertex towardq¯\\bar\{q\}, so the confidence vertex is*never*attracting, and it is strictly repelling with gradient≥ζ​q¯\\geq\\zeta\\bar\{q\}wheneverΓ′′≥ζ\>0\\Gamma^\{\\prime\\prime\}\\geq\\zeta\>0near the boundary \(Brier:Γ′′≡2\\Gamma^\{\\prime\\prime\}\\equiv 2\)\.

###### Corollary 1\(The regularizer class that dies at the boundary\)\.

LetR​\(πθ\)R\(\\pi\_\{\\theta\}\)be any functional of the policy withπ​∂πR→0\\pi\\,\\partial\_\{\\pi\}R\\to 0asπ→0\\pi\\to 0\. Then∂vR=π​\(1−π\)​∂πR→0\\partial\_\{v\}R=\\pi\(1\-\\pi\)\\,\\partial\_\{\\pi\}R\\to 0: its gradient vanishes on the absorbing boundary with the reward’s\. The class contains the entropy bonus and RLHF’s anchorKL​\(πθ∥π0\)\\mathrm\{KL\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{0\}\)\(both logarithmic\), but*not*the reverse KLKL​\(π0∥πθ\)\\mathrm\{KL\}\(\\pi\_\{0\}\\\|\\pi\_\{\\theta\}\), whose vertex cost diverges and which floors the answer rate atΘ​\(β\)\\Theta\(\\beta\): the direction of the KL is load\-bearing, and RLHF’s points the collapsing way\. Balancing throttled drift against throttled restoring force leaves only an exponentially small floor,Pfloor≍e−Deff/βP\_\{\\mathrm\{floor\}\}\\asymp e^\{\-D\_\{\\mathrm\{eff\}\}/\\beta\}\(e\.g\.≈10−13\{\\approx\}10^\{\-13\}atβ=0\.01\\beta=0\.01\), reached only after timeeΘ​\(1/β\)e^\{\\Theta\(1/\\beta\)\}\.

Together the statics make the vertex cheap, locally attracting, and unrestorable \(Figure[1](https://arxiv.org/html/2608.00301#S3.F1)draws the geometry\); what they do not give is the*rate*: what an experimenter would see, and on what clock\.

### The Collapse Law under Gradient Flow

WriteJ​\(θ\):=∫σ​\(vθ\)​g​𝑑sJ\(\\theta\):=\\int\\sigma\(v\_\{\\theta\}\)\\,g\\,dsfor the reward term of \([2](https://arxiv.org/html/2608.00301#S2.E2)\), andL​\(t\):=−J​\(θt\)L\(t\):=\-J\(\\theta\_\{t\}\)\. Since an answered prompt earnsm​\(q¯\)m\(\\bar\{q\}\)and an abstained one earns0,LLis*minus the mean training reward*, the curve every RL run already logs\. Define the observable*anchor share*ρ​\(t\):=β​∇J⋅∇𝒦/‖∇J‖2\\rho\(t\):=\\beta\\,\\nabla J\\\!\\cdot\\\!\\nabla\\mathcal\{K\}/\\\|\\nabla J\\\|^\{2\}, with𝒦\\mathcal\{K\}the population KL term of \([2](https://arxiv.org/html/2608.00301#S2.E2)\), and the weighted gate meanσ¯​\(t\):=∫σ2​\(v\)​\(−g\)/∫σ​\(v\)​\(−g\)\\bar\{\\sigma\}\(t\):=\\int\\sigma^\{2\}\(v\)\(\-g\)\\big/\\\!\\int\\sigma\(v\)\(\-g\)\. Letr​\(s\):=c4​ϕ​\(s\)r\(s\):=c\_\{4\}\\phi\(s\),Λ​\(t\):=𝔼p​\[er\]\\Lambda\(t\):=\\mathbb\{E\}\_\{p\}\[e^\{r\}\]\(the*tilt*\),Deff​\(t\):=∫er​\(−g\)D\_\{\\mathrm\{eff\}\}\(t\):=\\int e^\{r\}\(\-g\), and letttiltt\_\{\\mathrm\{tilt\}\}be the zero of the tilted marginm¯tilt:=𝔼tilt​\[m​\(q¯\)\]\\bar\{m\}\_\{\\mathrm\{tilt\}\}:=\\mathbb\{E\}\_\{\\mathrm\{tilt\}\}\[m\(\\bar\{q\}\)\]under the measure∝er​p\\propto e^\{r\}p\.

###### Theorem 1\(Collapse law on the invariant\)\.

Consider gradient flowθ˙=∇Jβ\\dot\{\\theta\}=\\nabla J\_\{\\beta\}on \([2](https://arxiv.org/html/2608.00301#S2.E2)\) under\(B1\), from an initialization withL​\(0\)\>0L\(0\)\>0\.\(i\)*\(Envelope; one observable hypothesis\.\)*Ifρ​\(τ\)≤12\\rho\(\\tau\)\\leq\\tfrac\{1\}\{2\}on\[0,t\]\[0,t\], then

1L​\(t\)≥1L​\(0\)\+12​∫0t\(1−σ¯​\(τ\)\)2​𝑑τ\.\\frac\{1\}\{L\(t\)\}\\;\\geq\\;\\frac\{1\}\{L\(0\)\}\+\\frac\{1\}\{2\}\\int\_\{0\}^\{t\}\\big\(1\-\\bar\{\\sigma\}\(\\tau\)\\big\)^\{2\}d\\tau\.\(ii\)*\(Two\-sided finite\-time law\.\)*Assume additionally, on a window\[T0,c⋆​ttilt\]\[T\_\{0\},\\,c\_\{\\star\}t\_\{\\mathrm\{tilt\}\}\]withc⋆<1c\_\{\\star\}<1: the parameter\-space certificateβ​‖θτ−θ0‖∞≤12​Deff​\(τ\)\\beta\\\|\\theta\_\{\\tau\}\-\\theta\_\{0\}\\\|\_\{\\infty\}\\leq\\tfrac\{1\}\{2\}D\_\{\\mathrm\{eff\}\}\(\\tau\); the measured signc4​\(τ\)≤0c\_\{4\}\(\\tau\)\\leq 0; a tilt boundΛ​\(τ\)≤Λ¯<1\\Lambda\(\\tau\)\\leq\\bar\{\\Lambda\}<1; and the explicit small\-β\\betaconditionβ​\|c4​\(0\)\|​Λ¯≤14​\(1−Λ¯\)​D¯​\(c⋆\)\\beta\\,\|c\_\{4\}\(0\)\|\\,\\bar\{\\Lambda\}\\leq\\tfrac\{1\}\{4\}\(1\-\\bar\{\\Lambda\}\)\\,\\underline\{D\}\(c\_\{\\star\}\), whereD¯​\(c⋆\):=min\[0,c⋆​ttilt\]⁡Deff\\underline\{D\}\(c\_\{\\star\}\):=\\min\_\{\[0,c\_\{\\star\}t\_\{\\mathrm\{tilt\}\}\]\}D\_\{\\mathrm\{eff\}\}\. Thenσ¯→0\\bar\{\\sigma\}\\to 0in Cesàro mean, the readout gain grows as\|c4\|=O​\(log⁡t\)\|c\_\{4\}\|=O\(\\log t\), and

c1t​log2⁡t≤L​\(t\)≤c2t,T0≤t≤c⋆​ttilt,\\frac\{c\_\{1\}\}\{t\\log^\{2\}t\}\\;\\leq\\;L\(t\)\\;\\leq\\;\\frac\{c\_\{2\}\}\{t\},\\qquad T\_\{0\}\\leq t\\leq c\_\{\\star\}t\_\{\\mathrm\{tilt\}\},withc1,c2c\_\{1\},c\_\{2\}depending only on tracked quantities of the run\.\(iii\)*\(Endpoint\.\)*LLcrosses zero at a timeZLZ\_\{L\}withinO​\(1\)O\(1\)absolute time ofttiltt\_\{\\mathrm\{tilt\}\}, provided an explicit error\-dominance condition holds at the crossing \(it does on our runs\);ZLZ\_\{L\}stabilizes asβ↓0\\beta\\downarrow 0\(measured:3\.2%3\.2\\%over two decades\)\.

###### Proof idea\.

The exact softmax identity gives∂c0J=−\(1−σ¯\)​L\\partial\_\{c\_\{0\}\}J=\-\(1\-\\bar\{\\sigma\}\)L; underρ≤12\\rho\\leq\\tfrac\{1\}\{2\},L˙≤−12​‖∇J‖2≤−12​\(1−σ¯\)2​L2\\dot\{L\}\\leq\-\\tfrac\{1\}\{2\}\\\|\\nabla J\\\|^\{2\}\\leq\-\\tfrac\{1\}\{2\}\(1\-\\bar\{\\sigma\}\)^\{2\}L^\{2\}, which integrates to \(i\)\. For \(ii\), the plateau approximationσ′​\(v\)≍ev\\sigma^\{\\prime\}\(v\)\\asymp e^\{v\}turns the bias flow intodd​t​e−c0≥Deff/2\\frac\{d\}\{dt\}e^\{\-c\_\{0\}\}\\geq D\_\{\\mathrm\{eff\}\}/2, and a gain\-growth lemma derives\|c4\|=O​\(log⁡t\)\|c\_\{4\}\|=O\(\\log t\)from the sign, tilt, and small\-β\\betainputs rather than assuming it\. Full proofs of all results are in the appendix\. ∎

Two readings matter\.*First*, the hypotheses are measurements, not conveniences:ρ\\rhois computable from logged gradients \(on our runs it peaks at0\.0080\.008\), and the sign, tilt, and small\-β\\betainputs are checkable per run; every hypothesis held on every run reported in this paper, simulated and live\. Read this way, part \(i\) is an a priori envelope from a single observable, and part \(ii\) is a certified finite\-time law for any run whose logs pass its checks\.*Second*, the law is onLL; the answer ratePPobeysP=L/\|m¯tilt\|⋅\(1\+o​\(1\)\)P=L/\|\\bar\{m\}\_\{\\mathrm\{tilt\}\}\|\\cdot\(1\+o\(1\)\)away from the endpoint, and its exponent is*not*constant, so predictions should be stated onLL\.

###### Proposition 4\(The proximal surrogate understates the pathology\)\.

Replace the KL in \([2](https://arxiv.org/html/2608.00301#S2.E2)\) byβ2​‖θ−θ0‖2\\tfrac\{\\beta\}\{2\}\\\|\\theta\-\\theta\_\{0\}\\\|^\{2\}\. The restoring force no longer carriesσ′​\(v\)\\sigma^\{\\prime\}\(v\); the flow converges exponentially to a stable equilibrium with answer\-rate floorP∞=Θ​\(β​log⁡\(1/β\)\)P\_\{\\infty\}=\\Theta\(\\beta\\log\(1/\\beta\)\)\. Analyses that model RLHF’s KL by a proximal term are therefore*not conservative*here: true KL is strictly more permissive of collapse than its weight\-space stand\-in\.

###### Proposition 5\(Bounded readouts delay greedy recovery\)\.

Withv​\(s\)=c4​ϕ​\(s\)\+c0v\(s\)=c\_\{4\}\\phi\(s\)\+c\_\{0\},ϕ∈\[0,1\]\\phi\\in\[0,1\], greedy decoding answers a positive\-measure set iff the gauge\-invariant scalarsupsv​\(s\)=max⁡\(c4,0\)\+c0\\sup\_\{s\}v\(s\)=\\max\(c\_\{4\},0\)\+c\_\{0\}is positive\. The tilt available to the flow isΛ​\(t\)≤e\|c4\|\\Lambda\(t\)\\leq e^\{\|c\_\{4\}\|\}; under the gain growth of Theorem[1](https://arxiv.org/html/2608.00301#Thmtheorem1)\(ii\), reaching gainMMtakes timeeΩ​\(M\)e^\{\\Omega\(M\)\}\. A bounded readout thus delays escape exponentially in the gain the escape must accumulate, without creating an absorbing state\. Gates driven by unbounded hidden\-state projections sit outside the class and escape\.

#### Lifting beyond the frozen gate\.

The lifting theorem removes both restrictions at once: full sequence KL and trainable content, for mixture\-structured policies\.

###### Theorem 2\(Gate\-to\-sequence lifting; abridged, with full statement and proof in the appendix\)\.

Letπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)be a sequence policy with refusal eventAA, answer and refusal probabilitiesπans=1−πθ​\(A∣x\)\\pi\_\{\\mathrm\{ans\}\}=1\-\\pi\_\{\\theta\}\(A\\mid x\)andπabs=πθ​\(A∣x\)\\pi\_\{\\mathrm\{abs\}\}=\\pi\_\{\\theta\}\(A\\mid x\), gate logitvv, answer\-branch correctnessqθq\_\{\\theta\}, and answer\-/refusal\-conditional KLsKans,KabsK\_\{\\mathrm\{ans\}\},K\_\{\\mathrm\{abs\}\}to base\.\(i\)Exactly,KLseq=KLgate\+πans​Kans\+πabs​Kabs\\mathrm\{KL\}\_\{\\mathrm\{seq\}\}=\\mathrm\{KL\}\_\{\\mathrm\{gate\}\}\+\\pi\_\{\\mathrm\{ans\}\}K\_\{\\mathrm\{ans\}\}\+\\pi\_\{\\mathrm\{abs\}\}K\_\{\\mathrm\{abs\}\}, and for any mixture parameterization

∂vJβ=σ′​\(v\)​\[m​\(qθ\)−β​\(v−v0\)−β​\(Kans−Kabs\)\]:\\partial\_\{v\}J\_\{\\beta\}=\\sigma^\{\\prime\}\(v\)\\big\[m\(q\_\{\\theta\}\)\-\\beta\(v\-v\_\{0\}\)\-\\beta\(K\_\{\\mathrm\{ans\}\}\-K\_\{\\mathrm\{abs\}\}\)\\big\]:*every*gate\-channel term is throttled byσ′​\(v\)\\sigma^\{\\prime\}\(v\), and the anchor taxes only the branch taken: abstaining*hides*content drift from the KL\.\(ii\)With branch\-separable content \(answer and refusal content on separate parameters, refusals initialized at base, and an explicit KL\-gradient regularity boundCK<∞C\_\{K\}<\\infty\), the refusal branch is invariant and the collapsed manifold is stationary in the limitπans→0\\pi\_\{\\mathrm\{ans\}\}\\to 0; the surviving restoring force pins*refusals*to base, never answering\.\(iii\)Under a capability ceiling preserving\(B1\), or under an explicit race inequality between the collapse clock and content drift, checkable on logged runs and stated in the appendix, the finite\-time law of Theorem[1](https://arxiv.org/html/2608.00301#Thmtheorem1)survives with lower bound weakened by one logarithm,Ω​\(1/\(t​log3⁡t\)\)\\Omega\(1/\(t\\log^\{3\}t\)\)\. The correctness channel is throttled byπans\\pi\_\{\\mathrm\{ans\}\}, so the plateau delivers onlyO​\(log⁡ttilt\)O\(\\log t\_\{\\mathrm\{tilt\}\}\)of total content signal:*the plateau starves the very channel that could avert it*\. Content shared across branches, and gate–content parameterizations with no bias coordinate, remain open; a generic single\-decoder Transformer sits outside on both counts\.

### The Advantage Estimator Rewrites the Rule

Theorem[1](https://arxiv.org/html/2608.00301#Thmtheorem1)is a statement about vanilla policy gradient \(PG\)\. Practice uses group\-based estimators: sampleGGcompletions per prompt, form advantagesA^i=\(ri−μ^r\)/σ^r\\hat\{A\}\_\{i\}=\(r\_\{i\}\-\\hat\{\\mu\}\_\{r\}\)/\\hat\{\\sigma\}\_\{r\}\(GRPO;Shaoet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib21)\) or mean\-baseline advantages \(RLOO, Dr\. GRPO;Ahmadianet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib23); Liuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib22)\)\. Becauseσ^r\\hat\{\\sigma\}\_\{r\}itself depends on the answer probabilityp=σ​\(v\)p=\\sigma\(v\), normalization changes the*exponent*and, in the sparse regime, the*rule*\. WriteM2:=q¯\+\(1−q¯\)​λ2M\_\{2\}:=\\bar\{q\}\+\(1\-\\bar\{q\}\)\\lambda^\{2\}\.

###### Proposition 6\(Drift menu\)\.

The per\-prompt expected drift of the gate coordinate satisfies:\(a\)vanilla PG and mean\-baseline estimators:Θ​\(p​m\)\\Theta\(p\\,m\), hence localLL\-exponent−1\-1;\(b\)group\-std normalization,1/G≪p≪11/G\\ll p\\ll 1:≈m​p/M2=Θ​\(p\)\\approx m\\sqrt\{p/M\_\{2\}\}=\\Theta\(\\sqrt\{p\}\), hence exponent−2\-2;\(c\)group\-std normalization,p≪1/Gp\\ll 1/G: a group contains at most one answer, whose advantage issign⁡\(r\)​G−1\\operatorname\{sign\}\(r\)\\sqrt\{G\-1\}*independently of\|r\|\|r\|*, so the drift isp​\(2​q¯−1\)​G−1p\\,\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}: the designed penalty is erased,λeff=1\\lambda\_\{\\mathrm\{eff\}\}=1for every nominalλ\\lambda, and the exponent returns to−1\-1\. The knee between\(b\)and\(c\)sits atp≈1/Gp\\approx 1/G\.

Table 1:The estimator determines the decay law and the penalty actually optimized \(tt: optimizer steps;NN: rollouts;k:=\|2​q¯−1\|​G−1k:=\|2\\bar\{q\}\-1\|\\sqrt\{G\-1\},ω≍k/G\\omega\\asymp k/G\)\. DAPO’s per\-step exponential and its rollout dilution cancel inL​\(N\)L\(N\)\(Proposition[7](https://arxiv.org/html/2608.00301#Thmproposition7)\); the bottom three rows optimizeλeff=1\\lambda\_\{\\mathrm\{eff\}\}=1\.![Refer to caption](https://arxiv.org/html/2608.00301v1/x2.png)Figure 2:The menu by exact enumeration \(q¯=0\.3\\bar\{q\}=0\.3,λ=4\.5\\lambda=4\.5\)\. \(a\) GRPO drift∝p\\propto\\sqrt\{p\}above the kneep=1/Gp=1/G\(dotted\),∝p\\propto pbelow; vanilla PG∝p\\propto pthroughout\. \(b\) Sparse regime: drift/p/pcollapses onto\(2​q¯−1\)​G−1\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}for every nominalλ\\lambda\. \(c\) Exponents converge to0\.50\.5and1\.01\.0\.*The threshold moves\.*In regime \(c\) the drift changes sign atq¯=1/2\\bar\{q\}=1/2, not att∗t^\{\\ast\}: the optimizer trains the model to answer the entire bandq¯∈\(1/2,t∗\)\\bar\{q\}\\in\(1/2,t^\{\\ast\}\)on which the rule assigns negative gain; atλ=4\.5\\lambda=4\.5this is the band\(0\.50,0\.82\)\(0\.50,0\.82\)\. Group normalization thus silently swaps in a rule whose Chow threshold is1/21/2, manufacturing the hallucinations the rule was introduced to prevent\.*The abstention rate is set byGG, notλ\\lambda\.*On that band the two regimes push in opposite directions, creating a stable fixed point atp∗=Θ​\(1/G\)p^\{\\ast\}=\\Theta\(1/G\): the abstention rate on the contested band is set by the group size\.

###### Proposition 7\(Sampling filters conserve the collapse\)\.

LetDDbe the event that a group’s reward vector is constant \(e\.g\. all\-abstain\), on whichA^i≡0\\hat\{A\}\_\{i\}\\equiv 0\. Then𝔼​\[g^\]=Pr⁡\[Dc\]​𝔼​\[g^∣Dc\]\\mathbb\{E\}\[\\hat\{g\}\]=\\Pr\[D^\{c\}\]\\,\\mathbb\{E\}\[\\hat\{g\}\\mid D^\{c\}\]identically, so a filter that discards degenerate groups and resamples \(DAPO;Yuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib24)\) conserves the expected*reward*drift per rollout exactly, for everyp,G,q¯,λp,G,\\bar\{q\},\\lambdaand every prompt distribution, while taking1/Pr⁡\[Dc\]≈1/\(G​p\)1/\\Pr\[D^\{c\}\]\\approx 1/\(Gp\)times more rollouts per step\. The anchor, applied once per optimizer step, is thereby*diluted*byPr⁡\[Dc\]\\Pr\[D^\{c\}\]on the rollout axis\. Per step the law becomes exponential,L=Θ​\(e−ω​t\)L=\\Theta\(e^\{\-\\omega t\}\)withω≍\|2​q¯−1\|/G\\omega\\asymp\|2\\bar\{q\}\-1\|/\\sqrt\{G\}; per rollout nothing changes\. Consequently no resampling scheme that discards zero\-gradient groups can repair the collapse: only changing the score or the advantage estimator can\.

#### Falsifiable predictions\.

The section compresses to five estimator\-indexed predictions: \(P1\) vanilla\-PG and mean\-baseline runs show localLL\-exponent−1\-1; \(P2\) GRPO shows−2\-2above the knee; \(P3\) the knee sits at per\-prompt answer rate≈1/G\{\\approx\}1/Gand moves withGG; \(P4\) below the knee the drift is exactlyλ\\lambda\-invariant, with amplitude\(2​q¯−1\)​G−1\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}; \(P5\) GRPO and GRPO\+DAPO trajectories coincide on the rollout axis atβ=0\\beta=0; the experiments take them to a live model \(Table[1](https://arxiv.org/html/2608.00301#S3.T1)and Figure[2](https://arxiv.org/html/2608.00301#S3.F2)display the enumerated menu\)\.

## 4The Report\-Level Repair

The repair removes abstention from the action space: answer every question, train a mandatory confidence report, threshold only at deployment\.

###### Proposition 8\(No shared vanishing factor; anchor\-robust\)\.

The population Brier objective satisfies∂κiB=−2​𝔼p​\[\(c−q¯\)​∂κic\]\\partial\_\{\\kappa\_\{i\}\}B=\-2\\mathbb\{E\}\_\{p\}\[\(c\-\\bar\{q\}\)\\,\\partial\_\{\\kappa\_\{i\}\}c\]with no action\-saturation factor\. At the calibrated optimumc≡q¯c\\equiv\\bar\{q\}the Hessian in the range parameters is−2​𝔼​\[∇c​∇c⊤\]\-2\\mathbb\{E\}\[\\nabla c\\,\\nabla c^\{\\top\}\], negative definite wheneverϕ\\phiis nonconstant on the support ofpp: an*interior attracting*equilibrium \(q¯\\bar\{q\}bounded away from0and11\), withinO​\(β\)O\(\\beta\)of calibration under the proximal anchor, losingO​\(β\)O\(\\beta\)deployment utility in general andO​\(β2\)O\(\\beta^\{2\}\)under the margin conditionPrp⁡\(\|q¯−t∗\|≤ϵ\)=O​\(ϵ\)\\Pr\_\{p\}\(\|\\bar\{q\}\-t^\{\\ast\}\|\\leq\\epsilon\)=O\(\\epsilon\), which a single transversal crossing under a bounded signal density implies\. The conclusion is anchor\-robust: under a quadratic function\-space anchor the optimum iscβ=q¯\+β2\+β​\(c0−q¯\)c\_\{\\beta\}=\\bar\{q\}\+\\tfrac\{\\beta\}\{2\+\\beta\}\(c\_\{0\}\-\\bar\{q\}\); under a Bernoulli\-KL anchor on the*report*, the anchor’s derivative diverges at the boundary and actively repels degenerate reports\. At the action level, reward and anchor share the vanishing factorσ′​\(v\)\\sigma^\{\\prime\}\(v\); at the report level they share none: the Brier gradient isΘ​\(1\)\\Theta\(1\)away from calibration\. With the feature frozen, the anchored objective in the range parameters is strongly concave: convergence is global and exponential, where the same reduction leaves the action\-level rule collapsing\.

The anchor asymmetry is structural, not elective: a deterministic report has no action distribution, hence no policy KL to anchor, and the proposition shows its conclusion is invariant to the anchor geometry that replaces it\. On the action side the anchor is part of the finding\. Under the true policy KL that RLHF uses, the action arm deploys nothing at any signal quality \(Table[2](https://arxiv.org/html/2608.00301#S5.T2)\); replacing that KL by a weight\-space proximal term, which production stacks do not use, rescues medium and strong signal in the model \(coverage0\.2340\.234and0\.4680\.468against rational0\.2590\.259and0\.4690\.469\) but still forfeits all of weak signal \(0\.0000\.000against0\.0350\.035\) and buys its sharpness with unbounded parameter norm \(Figure[1](https://arxiv.org/html/2608.00301#S3.F1)b\)\. The report level needs no anchor surgery and is near\-rational at all three signal qualities\. One scope note on “no vanishing factor”: withc=κ0\+\(κ1−κ0\)​ϕc=\\kappa\_\{0\}\+\(\\kappa\_\{1\}\-\\kappa\_\{0\}\)\\phi, gradients in the feature parameters\(w,a\)\(w,a\)still carryϕ′\\phi^\{\\prime\}and can saturate\. What the report removes is a shared factor multiplying*every*coordinate\. For any fixed feature the range parameters alone drive the deployed report to calibration at rateΘ​\(1\)\\Theta\(1\), so the output has no reachable region of zero gradient\. Whenq¯\\bar\{q\}is not representable by the head, the same gradient drivesccto its constrainedL2​\(p\)L^\{2\}\(p\)projection, still a unique attractor, interior whenever no box constraint is active, and deployment pays at most\(1\+λ\)​𝔼p​\|c−q¯\|\(1\+\\lambda\)\\,\\mathbb\{E\}\_\{p\}\|c\-\\bar\{q\}\|in utility for the approximation error\. The experiments meet this as the binding readout constraint \(in the live runs the full model trains under the composite score, the report read out by a trained linear head on the prompt state\)\.

###### Proposition 9\(The composite objective; pure properness is not enough\)\.

Score an answer \(correct with probabilityqq\) and reportccbyrα=α​1​\{correct\}\+1−\(Y−c\)2r\_\{\\alpha\}=\\alpha\\,\\mathbf\{1\}\\\{\\mathrm\{correct\}\\\}\+1\-\(Y\-c\)^\{2\}\. Then∂c𝔼​\[rα\]=2​\(q−c\)\\partial\_\{c\}\\,\\mathbb\{E\}\[r\_\{\\alpha\}\]=2\(q\-c\)and∂q𝔼​\[rα\]=α−1\+2​c\\partial\_\{q\}\\,\\mathbb\{E\}\[r\_\{\\alpha\}\]=\\alpha\-1\+2c\.\(a\)The report is driven to calibration for everyα\\alpha\.\(b\)Pure Brier \(α=0\\alpha=0\) makes the calibrated accuracy gradient2​q−1<02q\-1<0for everyq<1/2q<1/2: the model gains score by*degrading*accuracy on hard prompts and reporting the degradation honestly; these are the same prompts condition\(B1\)selects\.\(c\)Anyα\>1\\alpha\>1makes the accuracy gradient strictly positive everywhere;α=1\\alpha=1leaves a degenerate stationary point atq=c=0q=c=0\. The composite contains no discrete abstain action, every episode returns aqq\-correlated score, and the vertex analysis of Proposition[3](https://arxiv.org/html/2608.00301#Thmproposition3)\(b\)is unchanged: no collapse is introduced\.

Each term has one job \(the correctness term drives accuracy; the proper term drives the report to calibration, which the deployment threshold reads\)\. Under a correctness\-only score the report never moves and deployment has nothing to threshold\.

The design lesson, in one sentence:*train confidence continuously, with a proper score plus a correctness reward at weightα\>1\\alpha\>1, and threshold only at deployment\.*

The lesson names a family: any design that keeps answering mandatory in training and learns confidence outside the action channel, including post\-hoc calibration\(Guoet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib40)\)and conformal deployment\-time abstention\(Abbasi Yadkoriet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib42)\), inherits the immunity; the composite is the member whose calibration and accuracy directions the propositions above certify jointly\.

## 5Experiments

### Exact and Simulated Confirmation

*Mean\-field enumeration\.*All entries of Table[1](https://arxiv.org/html/2608.00301#S3.T1)are verified by exact enumeration over group outcomes: the sparse\-regime drift/p/pmatches\(2​q¯−1\)​G−1\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}to three decimals for everyλ∈\{1,2,4\.5,9\}\\lambda\\in\\\{1,2,4\.5,9\\\}, the two exponents converge to0\.50\.5and1\.01\.0asGGgrows, and the rollout\-axis conservation of Proposition[7](https://arxiv.org/html/2608.00301#Thmproposition7)holds to six significant figures\.

*Simulations\.*In a two\-type model calibrated favorably to the penalty rule \(qH,qL=0\.9,0\.2q\_\{H\},q\_\{L\}=0\.9,0\.2; true gate KL for the action arm, greedy deployment, horizonT=104T=10^\{4\},β=10−3\\beta=10^\{\-3\}\): the collapse law holds \(Figure[3](https://arxiv.org/html/2608.00301#S5.F3):LLfalls onto one curve across two decades ofβ\\beta, below the1/t1/tenvelope past the transient; the measured GRPO\-minus\-vanilla slope difference is−1\.013\-1\.013against a predicted−1\-1\); and the deployment outcome is one\-sided \(Table[2](https://arxiv.org/html/2608.00301#S5.T2)\): at strong signal the penalty rule forfeits*all*of a46\.9%46\.9\\%\-coverage optimum while the report\-level mechanism captures97%97\\%of its utility\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x3.png)Figure 3:\(a\) The invariantLLfor three anchor strengths; the upturn isttiltt\_\{\\mathrm\{tilt\}\}\. \(b\) The proximal surrogate floors the reward; true KL does not\. \(c\) Greedy coverage atλ=4\.5\\lambda=4\.5: report level \(solid\) near\-rational; action level \(dashed\) deploys nothing\.Table 2:Deployment utility and coverage atλ=4\.5\\lambda=4\.5\(β=10−3\\beta=10^\{\-3\},T=104T=10^\{4\}\)\. Atλ=1\\lambda=1this calibration has𝔼​\[q\]=0\.55\>t∗\\mathbb\{E\}\[q\]=0\.55\>t^\{\\ast\}, \(B1\) fails, and neither mechanism collapses\.*An empirical anchor\.*For base policies with a sharp monotone confidence threshold, a*negative*error\-penalized score implies initial drift toward collapse: at least3333of more than3636public frontier models satisfy this at the deployedλ=1\\lambda=1\(Jacksonet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib46)\), and no singleλ\\lambdais training\-safe across models \(derivation and scope in the appendix\)\.

### Language\-Model Experiments

We test the theory on Qwen2\.5\-1\.5B \(three seeds per run\) and, at scale, Qwen2\.5\-7B \(single seeds\)\(Qwen Team[2024](https://arxiv.org/html/2608.00301#bib.bib43)\), on short\-form QA from TriviaQA and PopQA\(Joshiet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib44); Mallenet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib45)\), in two tiers\.*Tier 1 \(theorem\-grade, head\-only\):*the paper’s four\-parameter gate head is trained on a frozen feature of the base model \(the answer/abstain logit difference\), with rewards from pre\-graded answer banks, so thatLL,σ¯\\bar\{\\sigma\}, andρ\\rhoare computed*exactly*per step; this tier tests the envelope of Theorem[1](https://arxiv.org/html/2608.00301#Thmtheorem1)\(i\), and sweeps the estimator menu \(P1–P5\) acrossG∈\{64,256\}G\\in\\\{64,256\\\}andλ∈\{1,4,9\}\\lambda\\in\\\{1,4,9\\\}with tolerances stated in the appendix\.*Tier 2 \(full\-parameter\):*fine\-tuning of the full model with the unbiased RLOO estimator\. The abstain action is a binary first\-token gate in the model’s own vocabulary, with no additional head: the first generated position is restricted to a designated answer/refusal pair \(refusal immediately ends the sequence\), so the pair is the entire action space there, and the per\-prompt answer probability is read exactly from it in one forward pass\. The division of labor is deliberate: Tier 1 tests the estimator menu against exact enumerated references, and Tier 2 tests the mechanism with an estimator that cannot manufacture a collapse\. Tier 2 compares action\-level abstention training against report\-level composite training \(α=2\\alpha=2\) on a two\-tier prompt mixture satisfying \(B1\), with two controls: per\-prompt capability certified at the capability checkpoints by forced\-answer rollouts \(Clopper–Pearson lower bound abovet∗t^\{\\ast\}\), so gate collapse is not confounded with capability loss; and an attribution control that zeroes the low\-accuracy tier’s reward\-channel advantages, keeping its KL channel and paired randomness\. A per\-checkpoint gate\-versus\-content KL decomposition monitors the branch\-length alternative\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x4.png)Figure 4:Live\-model dynamics on the capability\-certified cohort\. \(a\) The penalty rule collapses the median answer probability within ten steps; the attribution control never moves\. The 7B run collapses, plateaus, then escapes late, its paired control rising to full answering\. \(b\) Under the composite objective, coverage rises jointly with accuracy and calibration\.Tier\-1 outcomes\.On the real pool every prediction lands within tolerance: the drift\-menu local slopes are0\.5070\.507\(group\-normalized; predicted12\\tfrac\{1\}\{2\}\) and0\.9510\.951\(mean\-baseline; predicted11\); the knee scales by3\.793\.79forG=64→256G=64\{\\to\}256\(predicted≈4\{\\approx\}4\), landing within1\.1%1\.1\\%of the enumerated reference; below the knee the drift isλ\\lambda\-invariant at every grid point; the first\-window mean reward rises in every head\-only configuration \(p≤7×10−4p\\leq 7\{\\times\}10^\{\-4\}\); and the envelope of Theorem[1](https://arxiv.org/html/2608.00301#Thmtheorem1)\(i\) is falsified in no seed\. Two comparisons are structurally unavailable on this pool \(the sparse\-regime collapse leaves no common eligible window for the trajectory\-slope contrast; the bimodal base gate exhausts the dynamic\-sampling budget from the first step\)\. The full table is in the appendix\.

Tier\-2 outcomes\.On a held\-out500500\-prompt high\-accuracy cohort, the median answer probability falls from1\.0001\.000to≤0\.008\{\\leq\}0\.008within ten optimizer steps and to≤2×10−5\{\\leq\}2\{\\times\}10^\{\-5\}by step80008000\(exact readouts, not sampling estimates\), in three of three seeds, while forced\-answer correctness on those prompts holds at0\.940\.94–0\.970\.97throughout \(256256rollouts per prompt; one\-sided95%95\\%Clopper–Pearson lower bounds abovet∗t^\{\\ast\}at every capability checkpoint for461/433/432461/433/432of the500500\)\. The attribution control isolates the cause: zeroing only the low\-accuracy tier’s reward\-channel advantages, keeping the KL channel and the paired randomness identical, holds the cohort median at1\.0001\.000at every probe point, in all three paired seeds\. At the collapsed terminal state, on a fixed diagnostic batch, the task\-gradient norm is0and the gate\-KL\-anchor gradient norm is below2×10−102\{\\times\}10^\{\-10\}, against a task\-gradient norm of108108in the paired control at the same step: the reward gradient and the restoring force die together, and only where the collapse happened \(Figure[4](https://arxiv.org/html/2608.00301#S5.F4)a\)\.

The report level, and two boundaries measured at scale\.Trained on the same mixture with the same estimator \(three seeds\), the report channel moves opposite to the gate: coverage0\.47→0\.780\.47\\to 0\.78–0\.810\.81, accuracy0\.43→0\.690\.43\\to 0\.69, Brier0\.27→0\.220\.27\\to 0\.22, jointly \(Figure[4](https://arxiv.org/html/2608.00301#S5.F4)b\), the direction the interior attracting optimum of Proposition[9](https://arxiv.org/html/2608.00301#Thmproposition9)predicts\. One caveat scopes this: at this scale the binding constraint is the readout, not the mechanism \(a linear prompt\-state probe opens at coverage0\.470\.47on a tier whose true correctness exceeds0\.90\.9\), so the 1\.5B runs certify the direction of the report channel rather than near\-rational deployed utility; the 7B run closes most of that gap\. Single\-seed runs at 7B sharpen two boundaries\. First, the overconfident\-base premise is a property of scale and template rather than of the mechanism: offered the abstain option, the 7B base answers only0\.510\.51of its answerable questions \(against0\.910\.91without it\), already partway toward Chow’s rule\. \(B1\) still holds on the training mixture and the saturation throttle is fully open at a half\-answering start, so collapse remains the prediction, and it arrives: the 7B gate collapses by step1010and sits near10−410^\{\-4\}for roughly60006000steps, silencing even a base that already abstains on half of what it knows, while the paired control, started from the same gate, climbs to0\.930\.93within the same ten steps and to full answering just after\. The failed premise removes only the motivating overconfident scenario, not the collapse\. Second, the escape is real, and late: the run leaves the plateau between steps6\.36\.3k and7\.97\.9k, recovering to median1\.01\.0with the task gradient revived \(131131, against0at the 1\.5B terminal\)\. The recovery is margin\-selective \(terminal median1\.01\.0, mean0\.650\.65\), and the 1\.5B horizon shows no escape: both regimes of the escape\-time’s exponential sensitivity \(Proposition[5](https://arxiv.org/html/2608.00301#Thmproposition5)\) are observed\. The 7B report channel ends at coverage0\.980\.98, accuracy0\.940\.94, Brier0\.050\.05\.

## 6Related Work

Beyond the ternary\-reward line already cited, RL\-trained abstention includes advantage reweighting\(Panet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib5)\), knowledge\-boundary, uncertainty\-aware, and clarification rewards\(Gaoet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib6); Zenget al\.[2026](https://arxiv.org/html/2608.00301#bib.bib7); Zhaiet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib8); Xuet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib10)\), and confidence\-shaped rewards\(Zhanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib12); Liet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib13); Yanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib14)\)\. Several report, as engineering obstacles, phenomena our theory derives: an “abstention trap,” advantage bias, vanishing gradients, refusal erosion\(Songet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib11)\)\. Reported ternary\-reward successes\(Weiet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib4); Panet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib5)\)are reconciled with the theory in the limitations below\.Kalai and Vempala \([2024](https://arxiv.org/html/2608.00301#bib.bib2)\)prove calibrated models must hallucinate andKalaiet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib1)\)prescribe error\-penalized scoring; our results are the dynamical caveat to that program\. Closest to our repair,Damaniet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib3)\)train correctness plus a Brier score on reported confidence\. Remark[1](https://arxiv.org/html/2608.00301#Thmremark1)and Proposition[9](https://arxiv.org/html/2608.00301#Thmproposition9)supply why the action\-level alternative fails and why the correctness weight must exceed the properness weight\. GRPO, its critiques, and its filters\(Shaoet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib21); Liuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib22); Ahmadianet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib23); Yuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib24); Huet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib25)\)supply the estimator menu\. Reward overoptimization\(Gaoet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib26); Skalseet al\.[2022](https://arxiv.org/html/2608.00301#bib.bib27)\)is orthogonal, since our score*is*the deployment utility\. Chow’s rule and selective prediction\(Chow[1970](https://arxiv.org/html/2608.00301#bib.bib31); El\-Yaniv and Wiener[2010](https://arxiv.org/html/2608.00301#bib.bib32); Geifman and El\-Yaniv[2017](https://arxiv.org/html/2608.00301#bib.bib33); Kamathet al\.[2020](https://arxiv.org/html/2608.00301#bib.bib34); Renet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib35)\), confidence elicitation and calibration\(Kadavathet al\.[2022](https://arxiv.org/html/2608.00301#bib.bib36); Linet al\.[2022](https://arxiv.org/html/2608.00301#bib.bib37); Tianet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib38); Kumaran and others[2026](https://arxiv.org/html/2608.00301#bib.bib39); Guoet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib40); Kuhnet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib41)\), and conformal deployment\-time abstention\(Abbasi Yadkoriet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib42)\)support the deployment half of our design lesson\. An extended discussion is in the appendix\.

## 7Limitations and Conclusion

Reconciling with reported ternary\-reward successes\.TruthRL\(Weiet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib4)\)and TIAR\(Panet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib5)\)reduce hallucination without runaway refusal, and there is no contradiction\. Both deployλ=1\\lambda=1, where the sparse\-regime rewritingλ→λeff=1\\lambda\\to\\lambda\_\{\\mathrm\{eff\}\}=1is the identity and the over\-answering band is empty\. Collapse requires \(B1\) on the*training*mix, which curated or retrieval\-augmented data plausibly violate\. And where \(B1\) holds, a rising mean reward at typical budgets*is*the plateau’s signature, not evidence against it\.Scope of the lifting\.Theorem[2](https://arxiv.org/html/2608.00301#Thmtheorem2)’s open cases remain open as theory, but not as observation: the Tier\-2 runs sit in exactly this class, a fully shared decoder with a binary first\-token gate in its own vocabulary, and exhibit the collapse and the joint gradient death\. Free\-form refusals, with no designated decision position, remain untested\. Production estimator features \(sampled\-token variance, ratio clipping, theϵ\\epsilonfloor in the normalizer, length normalization\) remain outside the analysis\. Both live backbones are one model family \(Qwen2\.5\) on short\-form QA, with training mixtures constructed to satisfy \(B1\)\.The report channel must be trainable toward correctness\.Verbalized confidence empirically tracks answer*commitment*more than correctness\(Kumaran and others[2026](https://arxiv.org/html/2608.00301#bib.bib39)\)\. The composite supervises the report against realized correctness, but our live runs read the report out through a linear probe on the prompt state: the verbalized channel remains untested\. With an LLM judge crediting hedged text at rateν\\nu, properness degrades byO​\(ν\)O\(\\nu\): our claims are scoped to programmatic grading\.

Conclusion\.Error\-penalized abstention is the right prescription for a rational agent and, we prove, a self\-defeating one for a KL\-anchored gradient learner whose prompts share a bounded readout, with the advantage estimator, not the designed rule, setting the collapse law’s exponent, knee, and effective penalty\. The remedy is not to stop penalizing errors but to move the penalty to the deployment rule\. Scoring rules are designed for rational reporters; the geometry they induce on gradient learners is a separate design surface\.

## References

- Y\. Abbasi Yadkori, I\. Kuzborskij, D\. Stutz,et al\.\(2024\)Mitigating LLM hallucinations via conformal abstention\.External Links:2405\.01563Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§4](https://arxiv.org/html/2608.00301#S4.p5.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee,et al\.\(2024\)Back to basics: revisiting REINFORCE style optimization for learning from human feedback in LLMs\.External Links:2402\.14740Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§L\.2](https://arxiv.org/html/2608.00301#A12.SS2.p1.5),[§3](https://arxiv.org/html/2608.00301#S3.SSx3.p1.5),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- C\. K\. Chow \(1970\)On optimum recognition error and reject tradeoff\.IEEE Transactions on Information Theory16\(1\),pp\. 41–46\.Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.p2.2),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- P\. Christiano, J\. Leike, T\. B\. Brown, M\. Martic,et al\.\(2017\)Deep reinforcement learning from human preferences\.External Links:1706\.03741Cited by:[§1](https://arxiv.org/html/2608.00301#S1.p2.2)\.
- G\. Cui, Y\. Zhang, J\. Chen,et al\.\(2025\)The entropy mechanism of reinforcement learning for reasoning language models\.External Links:2505\.22617Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px2.p1.1)\.
- M\. Damani, I\. Puri, S\. Slocum,et al\.\(2025\)Beyond binary rewards: training LMs to reason about their uncertainty\.External Links:2507\.16806Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1),[Remark 1](https://arxiv.org/html/2608.00301#Thmremark1.p1.1)\.
- R\. El\-Yaniv and Y\. Wiener \(2010\)On the foundations of noise\-free selective classification\.Journal of Machine Learning Research11,pp\. 1605–1641\.Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- C\. Gao, C\. Huang, K\. Luo, Z\. Qiao,et al\.\(2026\)KARL: mitigating hallucinations in LLMs via knowledge\-boundary\-aware reinforcement learning\.External Links:2604\.22779Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- L\. Gao, J\. Schulman, and J\. Hilton \(2023\)Scaling laws for reward model overoptimization\.External Links:2210\.10760Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- Y\. Geifman and R\. El\-Yaniv \(2017\)Selective classification for deep neural networks\.External Links:1705\.08500Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- M\. Geist, B\. Scherrer, and O\. Pietquin \(2019\)A theory of regularized Markov decision processes\.External Links:1901\.11275Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.SS0.SSS0.Px1.p1.7)\.
- T\. Gneiting and A\. E\. Raftery \(2007\)Strictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\.Cited by:[Appendix G](https://arxiv.org/html/2608.00301#A7.SS0.SSS0.Px8.p1.6)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.External Links:1706\.04599Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[Appendix C](https://arxiv.org/html/2608.00301#A3.SS0.SSS0.Px4.p1.14),[§2](https://arxiv.org/html/2608.00301#S2.SS0.SSS0.Px1.p5.2),[§4](https://arxiv.org/html/2608.00301#S4.p5.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- J\. Hu, J\. K\. Liu, H\. Xu, and W\. Shen \(2025\)REINFORCE\+\+: stabilizing critic\-free policy optimization with global advantage normalization\.External Links:2501\.03262Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- D\. Jackson, W\. Keating, G\. Cameron, and M\. Hill\-Smith \(2025\)AA\-Omniscience: evaluating cross\-domain knowledge reliability in large language models\.External Links:2511\.13029Cited by:[Appendix K](https://arxiv.org/html/2608.00301#A11.p1.29),[§5](https://arxiv.org/html/2608.00301#S5.SSx1.p3.4)\.
- M\. Joshi, E\. Choi, D\. S\. Weld, and L\. Zettlemoyer \(2017\)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension\.External Links:1705\.03551Cited by:[Appendix L](https://arxiv.org/html/2608.00301#A12.p1.2),[§5](https://arxiv.org/html/2608.00301#S5.SSx2.p1.7)\.
- S\. Kadavath, T\. Conerly, A\. Askell, T\. Henighan,et al\.\(2022\)Language models \(mostly\) know what they know\.External Links:2207\.05221Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- A\. T\. Kalai, O\. Nachum, S\. S\. Vempala, and E\. Zhang \(2025\)Why language models hallucinate\.External Links:2509\.04664Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- A\. T\. Kalai and S\. S\. Vempala \(2024\)Calibrated language models must hallucinate\.External Links:2311\.14648Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- A\. Kamath, R\. Jia, and P\. Liang \(2020\)Selective question answering under domain shift\.External Links:2006\.09462Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- R\. Kirk, I\. Mediratta, C\. Nalmpantis, J\. Luketina,et al\.\(2024\)Understanding the effects of RLHF on LLM generalisation and diversity\.External Links:2310\.06452Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px2.p1.1)\.
- L\. Kuhn, Y\. Gal, and S\. Farquhar \(2023\)Semantic uncertainty: linguistic invariances for uncertainty estimation in natural language generation\.External Links:2302\.09664Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- D\. Kumaranet al\.\(2026\)Reported confidence in LLMs tracks commitment more than correctness\.External Links:2606\.29490Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1),[§7](https://arxiv.org/html/2608.00301#S7.p1.5)\.
- J\. Li, Y\. Wang, Y\. Chen,et al\.\(2025\)Knowledge\-level consistency reinforcement learning: dual\-fact alignment for long\-form factuality\.External Links:2509\.23765Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- S\. Lin, J\. Hilton, and O\. Evans \(2022\)Teaching models to express their uncertainty in words\.External Links:2205\.14334Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- Z\. Liu, C\. Chen, W\. Li, P\. Qi,et al\.\(2025\)Understanding R1\-zero\-like training: a critical perspective\.External Links:2503\.20783Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[Appendix G](https://arxiv.org/html/2608.00301#A7.SS0.SSS0.Px8.p1.6),[§3](https://arxiv.org/html/2608.00301#S3.SSx3.p1.5),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- A\. Mallen, A\. Asai, V\. Zhong, R\. Das,et al\.\(2023\)When not to trust language models: investigating effectiveness of parametric and non\-parametric memories\.External Links:2212\.10511Cited by:[Appendix L](https://arxiv.org/html/2608.00301#A12.p1.2),[§5](https://arxiv.org/html/2608.00301#S5.SSx2.p1.7)\.
- J\. Mei, C\. Xiao, C\. Szepesvári, and D\. Schuurmans \(2020\)On the global convergence rates of softmax policy gradient methods\.External Links:2005\.06392Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.SS0.SSS0.Px1.p1.4)\.
- L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida,et al\.\(2022\)Training language models to follow instructions with human feedback\.External Links:2203\.02155Cited by:[§1](https://arxiv.org/html/2608.00301#S1.p2.2)\.
- M\. Pan, S\. Zhao, N\. Zhang, P\. Shin,et al\.\(2026\)TIAR: trajectory\-informed advantage reweighting for LLM abstention learning\.External Links:2605\.25850Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1),[§7](https://arxiv.org/html/2608.00301#S7.p1.5)\.
- Qwen Team \(2024\)Qwen2\.5 technical report\.External Links:2412\.15115Cited by:[Appendix L](https://arxiv.org/html/2608.00301#A12.p1.2),[§5](https://arxiv.org/html/2608.00301#S5.SSx2.p1.7)\.
- N\. Razin, H\. Zhou, O\. Saremi, V\. Thilak,et al\.\(2024\)Vanishing gradients in reinforcement finetuning of language models\.External Links:2310\.20703Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.SS0.SSS0.Px1.p1.4)\.
- J\. Ren, J\. Luo, Y\. Zhao, K\. Krishna,et al\.\(2023\)Out\-of\-distribution detection and selective generation for conditional language models\.External Links:2209\.15558Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- J\. Schulman \(2020\)Approximating KL divergence\.Note:http://joschu\.net/blog/kl\-approx\.htmlAccessed: 2026\-07\-11Cited by:[§L\.2](https://arxiv.org/html/2608.00301#A12.SS2.p1.5)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu,et al\.\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2608.00301#S3.SSx3.p1.5),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- J\. Skalse, N\. H\. R\. Howe, D\. Krasheninnikov, and D\. Krueger \(2022\)Defining and characterizing reward hacking\.External Links:2209\.13085Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- L\. Song, T\. Shi, and J\. Zhao \(2025\)The hallucination tax of reinforcement finetuning\.External Links:2505\.13988Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- N\. Stiennon, L\. Ouyang, J\. Wu, D\. M\. Ziegler,et al\.\(2020\)Learning to summarize from human feedback\.External Links:2009\.01325Cited by:[§1](https://arxiv.org/html/2608.00301#S1.p2.2)\.
- K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma,et al\.\(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.External Links:2305\.14975Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px5.p1.1),[Appendix C](https://arxiv.org/html/2608.00301#A3.SS0.SSS0.Px4.p1.14),[§2](https://arxiv.org/html/2608.00301#S2.SS0.SSS0.Px1.p5.2),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- N\. Vieillard, T\. Kozuno, B\. Scherrer, O\. Pietquin,et al\.\(2020\)Leverage the average: an analysis of KL regularization in RL\.External Links:2003\.14089Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.SS0.SSS0.Px1.p1.7)\.
- Y\. Wang, Z\. Yang, W\. Ma, Z\. Sui,et al\.\(2025\)Enhancing reliability across short and long\-form QA via reinforcement learning\.External Links:2512\.08944Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.p1.1)\.
- Z\. Wei, X\. Yang, K\. Sun, J\. Wang,et al\.\(2025\)TruthRL: incentivizing truthful LLMs via reinforcement learning\.External Links:2509\.25760Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2608.00301#S1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1),[§7](https://arxiv.org/html/2608.00301#S7.p1.5)\.
- R\. J\. Williams \(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8,pp\. 229–256\.Cited by:[§L\.2](https://arxiv.org/html/2608.00301#A12.SS2.p1.5)\.
- H\. Xu, Z\. Zhu, S\. Zhang, D\. Ma,et al\.\(2024\)Rejection improves reliability: training LLMs to refuse unknown questions using RL from knowledge feedback\.External Links:2403\.18349Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- J\. Yang, J\. Tu, H\. Liu,et al\.\(2025\)BARREL: boundary\-aware reasoning for factual and reliable LRMs\.External Links:2505\.13529Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan,et al\.\(2025\)DAPO: an open\-source LLM reinforcement learning system at scale\.External Links:2503\.14476Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px4.p1.1),[Appendix G](https://arxiv.org/html/2608.00301#A7.SS0.SSS0.Px7.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1),[Proposition 7](https://arxiv.org/html/2608.00301#Thmproposition7.p1.8.8)\.
- X\. Zeng, J\. Huang, C\. Xie, G\. Nan,et al\.\(2026\)UCPO: uncertainty\-aware policy optimization\.External Links:2601\.22648Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- S\. Zhai, J\. Liang, and D\. Kang \(2026\)Abstain\-R1: calibrated abstention and post\-refusal clarification via verifiable RL\.External Links:2604\.17073Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.
- B\. Zhang, Z\. Chen, B\. Song,et al\.\(2025\)ConfClip: confidence\-weighted and clipped reward for reinforcement learning in LLMs\.External Links:2509\.17730Cited by:[Appendix A](https://arxiv.org/html/2608.00301#A1.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.00301#S6.p1.1)\.

*This appendix contains the extended related\-work discussion, the complete proofs, the full lifting theorem, the complete estimator and report\-level analyses, negative results, simulation details, the leaderboard derivation, and the language\-model experiment design\. Results restated from the main text are cited by their main\-text numbers \(“Theorem 1 of the main text”\); results stated only in this appendix are numbered within their sections \(e\.g\., Theorem F\.1\)\.*

## Appendix AExtended Related Work

The main text’s related\-work section compresses the following discussion\.

#### RL for abstention and hallucination\.

Recent work trains abstention with RL: ternary\-reward GRPO\(Weiet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib4)\), advantage reweighting for abstention\(Panet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib5)\), knowledge\-boundary\-aware rewards\(Gaoet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib6)\), uncertainty\-decoupled advantages\(Zenget al\.[2026](https://arxiv.org/html/2608.00301#bib.bib7)\), joint abstention and clarification rewards\(Zhaiet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib8)\), reliability rewards for unanswerable questions\(Wanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib9)\), RL from knowledge feedback\(Xuet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib10)\), and confidence\-shaped rewards\(Zhanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib12); Liet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib13); Yanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib14)\)\. Several of these report, as engineering obstacles, phenomena our theory derives: an “abstention trap”\(Gaoet al\.[2026](https://arxiv.org/html/2608.00301#bib.bib6)\), advantage bias under uncertainty rewards\(Zenget al\.[2026](https://arxiv.org/html/2608.00301#bib.bib7)\), vanishing gradients from coarse rewards\(Zhanget al\.[2025](https://arxiv.org/html/2608.00301#bib.bib12)\), and refusal erosion under standard RL fine\-tuning\(Songet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib11)\)\. We offer a common mechanism, rates, and conditions for when such training fails, together with a repair derived from the mechanism rather than patched onto it\.Weiet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib4)\)andPanet al\.\([2026](https://arxiv.org/html/2608.00301#bib.bib5)\)report ternary\-reward successes on real models; the main text’s limitations reconcile this with the theory\.

#### Vanishing gradients and RL fine\-tuning dynamics\.

Meiet al\.\([2020](https://arxiv.org/html/2608.00301#bib.bib16)\)give softmax policy\-gradient rates governed by the optimal action’s probability;Razinet al\.\([2024](https://arxiv.org/html/2608.00301#bib.bib15)\)show reward fine\-tuning stalls on inputs where the reward’s standard deviation under the policy is small\. Our contribution is not thatσ′\\sigma^\{\\prime\}can vanish, but the co\-death identity \(equation \(1\) of the main text\): the*regularizer*believed to hold the policy near its base carries the same vanishing factor, an interaction absent from both analyses and specific to action\-level abstention\. Entropy collapse in RL for reasoning\(Cuiet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib17)\)and RLHF’s diversity loss\(Kirket al\.[2024](https://arxiv.org/html/2608.00301#bib.bib18)\)are adjacent self\-throttling phenomena on a different quantity \(policy entropy\); our collapse can occur with entropy intact\.

#### Scoring rules and the statistical account of hallucination\.

Kalai and Vempala \([2024](https://arxiv.org/html/2608.00301#bib.bib2)\)prove calibrated models must hallucinate;Kalaiet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib1)\)locate persistence in binary\-graded incentives and prescribe error\-penalized scoring\. Our results are the dynamical caveat to that program: statically optimal scoring rules can be dynamically unreachable for a KL\-anchored gradient learner\. Closest to our repair,Damaniet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib3)\)train with correctness plus a Brier score on reported confidence and prove the objective’s optimum is accurate and calibrated; our analysis supplies the missing half:*why*the action\-level alternative fails, why properness must act on a continuous report channel rather than a gated action \(clipping the training signal belowt∗t^\{\\ast\}reinstates the failure; Remark 1 of the main text\), and why the correctness weight must strictly exceed the properness weight \(Proposition 9 of the main text\)\.

#### KL\-regularized RL, estimators, and reward hacking\.

KL regularization is ordinarily a stabilizer with provable averaging benefits\(Geistet al\.[2019](https://arxiv.org/html/2608.00301#bib.bib19); Vieillardet al\.[2020](https://arxiv.org/html/2608.00301#bib.bib20)\); we exhibit a structural exception\. GRPO\(Shaoet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib21)\), its critiques\(Liuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib22); Ahmadianet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib23)\), DAPO\(Yuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib24)\), and global normalization\(Huet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib25)\)supply the estimator menu; reward overoptimization\(Gaoet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib26); Skalseet al\.[2022](https://arxiv.org/html/2608.00301#bib.bib27)\)is an orthogonal failure channel \(our score*is*the deployment utility, so there is no proxy gap\)\.

#### Selective prediction and calibration\.

The thresholdt∗t^\{\\ast\}is Chow’s rule\(Chow[1970](https://arxiv.org/html/2608.00301#bib.bib31)\); risk–coverage foundations are due toEl\-Yaniv and Wiener \([2010](https://arxiv.org/html/2608.00301#bib.bib32)\); Geifman and El\-Yaniv \([2017](https://arxiv.org/html/2608.00301#bib.bib33)\), with NLP instantiations in selective QA\(Kamathet al\.[2020](https://arxiv.org/html/2608.00301#bib.bib34); Renet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib35)\)\. Confidence elicitation\(Kadavathet al\.[2022](https://arxiv.org/html/2608.00301#bib.bib36); Linet al\.[2022](https://arxiv.org/html/2608.00301#bib.bib37); Tianet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib38)\), its commitment bias\(Kumaran and others[2026](https://arxiv.org/html/2608.00301#bib.bib39)\), calibration\(Guoet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib40); Kuhnet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib41)\), and conformal deployment\-time abstention\(Abbasi Yadkoriet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib42)\)together support the deployment half of our design lesson: the report channel is trainable, and thresholding belongs at inference\.

## Appendix BNotation, Conventions, and Observables

Throughout,σ​\(z\)=1/\(1\+e−z\)\\sigma\(z\)=1/\(1\+e^\{\-z\}\),σ′=σ​\(1−σ\)\\sigma^\{\\prime\}=\\sigma\(1\-\\sigma\)\. Prompts carry correctnessq​\(x\)q\(x\); the decision layer sees a signals∼p​\(s\)s\\sim p\(s\)with posteriorq¯​\(s\)=𝔼​\[q∣s\]\\bar\{q\}\(s\)=\\mathbb\{E\}\[q\\mid s\]; the penalty rule is\(\+1,−λ,0\)\(\+1,\-\\lambda,0\)with marginm​\(u\)=\(1\+λ\)​u−λm\(u\)=\(1\+\\lambda\)u\-\\lambda, thresholdt∗=λ/\(1\+λ\)t^\{\\ast\}=\\lambda/\(1\+\\lambda\), and gain densityg​\(s\)=p​\(s\)​m​\(q¯​\(s\)\)g\(s\)=p\(s\)\\,m\(\\bar\{q\}\(s\)\)\. The bounded feature isϕ​\(s\)=σ​\(w​s\+a\)∈\(0,1\)\\phi\(s\)=\\sigma\(ws\+a\)\\in\(0,1\); the action\-level gate isvθ​\(s\)=c4​ϕ​\(s\)\+c0v\_\{\\theta\}\(s\)=c\_\{4\}\\phi\(s\)\+c\_\{0\},θ=\(w,a,c4,c0\)\\theta=\(w,a,c\_\{4\},c\_\{0\}\); the report head iscθ′​\(s\)=κ0\+\(κ1−κ0\)​ϕ​\(s\)c\_\{\\theta^\{\\prime\}\}\(s\)=\\kappa\_\{0\}\+\(\\kappa\_\{1\}\-\\kappa\_\{0\}\)\\phi\(s\),θ′=\(w,a,κ0,κ1\)\\theta^\{\\prime\}=\(w,a,\\kappa\_\{0\},\\kappa\_\{1\}\)\. The anchored objective is

Jβ​\(θ\)\\displaystyle J\_\{\\beta\}\(\\theta\)=∫σ​\(vθ\)​g−β​𝒦​\(θ\),\\displaystyle=\\int\\sigma\(v\_\{\\theta\}\)\\,g\\;\-\\;\\beta\\,\\mathcal\{K\}\(\\theta\),𝒦​\(θ\)\\displaystyle\\mathcal\{K\}\(\\theta\):=∫p​KL​\(Bern​\(σ​\(vθ\)\)∥Bern​\(σ​\(v0\)\)\)\.\\displaystyle:=\\int p\\,\\mathrm\{KL\}\\big\(\\mathrm\{Bern\}\(\\sigma\(v\_\{\\theta\}\)\)\\,\\\|\\,\\mathrm\{Bern\}\(\\sigma\(v\_\{0\}\)\)\\big\)\.\(3\)Standing condition\(B1\):D0=−∫g\>0D\_\{0\}=\-\\int g\>0, equivalently𝔼​\[q\]<t∗\\mathbb\{E\}\[q\]<t^\{\\ast\}\.

#### Observables\.

L​\(t\):=−J​\(θt\)=∫σ​\(v\)​\(−g\)L\(t\):=\-J\(\\theta\_\{t\}\)=\\int\\sigma\(v\)\(\-g\)\(minus the mean training reward\); answer rateP​\(t\)=∫σ​\(v\)​pP\(t\)=\\int\\sigma\(v\)\\,p; anchor shareρ​\(t\):=β​∇J⋅∇𝒦/‖∇J‖2\\rho\(t\):=\\beta\\,\\nabla J\\cdot\\nabla\\mathcal\{K\}/\\\|\\nabla J\\\|^\{2\}; weighted gate meanσ¯​\(t\):=∫σ2​\(−g\)/∫σ​\(−g\)\\bar\{\\sigma\}\(t\):=\\int\\sigma^\{2\}\(\-g\)\\big/\\\!\\int\\sigma\(\-g\)\(the weightσ​\(−g\)\\sigma\(\-g\)is signed; no sign ofσ¯\\bar\{\\sigma\}is ever assumed\)\. Withr​\(s\):=c4​ϕ​\(s\)r\(s\):=c\_\{4\}\\phi\(s\): tiltΛ​\(t\)=𝔼p​\[er\]\\Lambda\(t\)=\\mathbb\{E\}\_\{p\}\[e^\{r\}\]; effective driftDeff​\(t\)=∫er​\(−g\)D\_\{\\mathrm\{eff\}\}\(t\)=\\int e^\{r\}\(\-g\); tilted marginm¯tilt=𝔼tilt​\[m​\(q¯\)\]\\bar\{m\}\_\{\\mathrm\{tilt\}\}=\\mathbb\{E\}\_\{\\mathrm\{tilt\}\}\[m\(\\bar\{q\}\)\]under the measure∝er​p\\propto e^\{r\}p, so thatDeff=−Λ​m¯tiltD\_\{\\mathrm\{eff\}\}=\-\\Lambda\\,\\bar\{m\}\_\{\\mathrm\{tilt\}\}exactly\.ZLZ\_\{L\}denotes the zero ofLL;Zm=ttiltZ\_\{m\}=t\_\{\\mathrm\{tilt\}\}the zero ofm¯tilt\\bar\{m\}\_\{\\mathrm\{tilt\}\};G±:=∫g±G\_\{\\pm\}:=\\int g\_\{\\pm\};D¯​\(c⋆\):=min\[0,c⋆​ttilt\]⁡Deff\\underline\{D\}\(c\_\{\\star\}\):=\\min\_\{\[0,c\_\{\\star\}t\_\{\\mathrm\{tilt\}\}\]\}D\_\{\\mathrm\{eff\}\}; first momentM1:=∫\|s\|​\|g\|​𝑑s<∞M\_\{1\}:=\\int\|s\|\|g\|\\,ds<\\infty\. All measured values quoted below are taken at the working calibration \(Section[J](https://arxiv.org/html/2608.00301#A10)\); “the run” means\(μ,λ,β\)=\(1,4\.5,10−3\)\(\\mu,\\lambda,\\beta\)=\(1,4\.5,10^\{\-3\}\)\.

#### The parameter\-space certificate\.

Several derivations use the a\-priori condition

β​‖θt−θ0‖∞≤12​Deff​\(t\),t≤c⋆​ttilt,c⋆<1,\\beta\\,\\\|\\theta\_\{t\}\-\\theta\_\{0\}\\\|\_\{\\infty\}\\ \\leq\\ \\tfrac\{1\}\{2\}\\,D\_\{\\mathrm\{eff\}\}\(t\),\\qquad t\\leq c\_\{\\star\}t\_\{\\mathrm\{tilt\}\},\\ c\_\{\\star\}<1,\(4\)which certifies the observable hypothesisρ≤12\\rho\\leq\\tfrac\{1\}\{2\}of Theorem 1 up to theΘ\\Thetaconstant of∇𝒦=Θ​\(ec0​‖θ−θ0‖∞\)\\nabla\\mathcal\{K\}=\\Theta\(e^\{c\_\{0\}\}\\\|\\theta\-\\theta\_\{0\}\\\|\_\{\\infty\}\)\. The certificate is conservative by about two orders of magnitude: atβ=10−3\\beta=10^\{\-3\}it lapses att=7\.8×103t=7\.8\\times 10^\{3\}\(againstttilt=1\.15×104t\_\{\\mathrm\{tilt\}\}=1\.15\\times 10^\{4\}\), while the measured shareρ\\rhopeaks at0\.00800\.0080att≈2×103t\\approx 2\\times 10^\{3\}, crosses zero neart≈7×103t\\approx 7\\times 10^\{3\}as the reward and anchor gradients de\-align, and reaches−0\.011\-0\.011just beforettiltt\_\{\\mathrm\{tilt\}\}: the anchor never spends more than2%2\\%of its allowed budget\.

## Appendix CProofs of the Static Results

#### Threshold identity\.

g​\(s\)=p​\(s\)​\[\(1\+λ\)​q¯​\(s\)−λ\]g\(s\)=p\(s\)\[\(1\+\\lambda\)\\bar\{q\}\(s\)\-\\lambda\]andp\>0p\>0, sog​\(s\)\>0⇔q¯​\(s\)\>λ/\(1\+λ\)g\(s\)\>0\\iff\\bar\{q\}\(s\)\>\\lambda/\(1\+\\lambda\)\. Hence the penalty rule’s optimal decision region and the calibrated\-confidence threshold region coincide, and the two mechanisms are compared on the same target\. ∎

#### Proof of Lemma 1 \(shared throttling factor\)\.

ForP=Bern​\(π\)P=\\mathrm\{Bern\}\(\\pi\),Q=Bern​\(π0\)Q=\\mathrm\{Bern\}\(\\pi\_\{0\}\),∂πKL=logit​π−logit​π0\\partial\_\{\\pi\}\\mathrm\{KL\}=\\mathrm\{logit\}\\,\\pi\-\\mathrm\{logit\}\\,\\pi\_\{0\}\. Withπ=σ​\(v\)\\pi=\\sigma\(v\),logit​π=v\\mathrm\{logit\}\\,\\pi=vandd​π/d​v=σ′​\(v\)d\\pi/dv=\\sigma^\{\\prime\}\(v\), sodd​v​KL=σ′​\(v\)​\(v−v0\)\\tfrac\{d\}\{dv\}\\mathrm\{KL\}=\\sigma^\{\\prime\}\(v\)\(v\-v\_\{0\}\); the gradient formula follows by differentiating \([3](https://arxiv.org/html/2608.00301#A2.E3)\) under the integral\. ∎

#### Proof of Proposition 2 \(the anchor cannot exclude the vertex\)\.

Bernoulli KL to a fixed base is bounded:KL​\(Bern​\(σ​\(v\)\)∥Bern​\(σ​\(v0\)\)\)≤max⁡\{log⁡1σ​\(v0\),log⁡11−σ​\(v0\)\}\\mathrm\{KL\}\(\\mathrm\{Bern\}\(\\sigma\(v\)\)\\\|\\mathrm\{Bern\}\(\\sigma\(v\_\{0\}\)\)\)\\leq\\max\\\{\\log\\frac\{1\}\{\\sigma\(v\_\{0\}\)\},\\log\\frac\{1\}\{1\-\\sigma\(v\_\{0\}\)\}\\\}for everyvv\. At the all\-abstain vertex the cost is exactlyKL¯⟂=𝔼p​\[log⁡11−σ​\(v0\)\]\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}=\\mathbb\{E\}\_\{p\}\[\\log\\frac\{1\}\{1\-\\sigma\(v\_\{0\}\)\}\], soJβ​\(all\-abstain\)−Jβ​\(base\)=L​\(0\)−β​KL¯⟂J\_\{\\beta\}\(\\text\{all\-abstain\}\)\-J\_\{\\beta\}\(\\text\{base\}\)=L\(0\)\-\\beta\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}, positive iffβ<βcrit=L​\(0\)/KL¯⟂\\beta<\\beta\_\{\\mathrm\{crit\}\}=L\(0\)/\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}\. SinceKL¯⟂≈log⁡11−pans\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}\\approx\\log\\frac\{1\}\{1\-p\_\{\\mathrm\{ans\}\}\}grows only logarithmically in the base answer ratepansp\_\{\\mathrm\{ans\}\}, the threshold decays slowly: on the leaderboard\-calibrated model of Section[K](https://arxiv.org/html/2608.00301#A11),βcrit=0\.139,0\.116,0\.089,0\.060,0\.041\\beta\_\{\\mathrm\{crit\}\}=0\.139,\\,0\.116,\\,0\.089,\\,0\.060,\\,0\.041atpans=80%,88%,95%,99%,99\.9%p\_\{\\mathrm\{ans\}\}=80\\%,88\\%,95\\%,99\\%,99\.9\\%, above typical RLHF coefficients throughout \(atpans=99\.9%p\_\{\\mathrm\{ans\}\}=99\.9\\%,β=0\.05\\beta=0\.05the static argument alone no longer suffices and one must appeal to the dynamics\)\. Two glosses\. \(i\) The statement is that the anchor cannot*exclude*the vertex from the reachable set; that the vertex is the*destination*is the dynamical claim of Theorem 1\. \(ii\) The comparison is not a dichotomy at the global level: under \(B1\) the abstain vertex \(reward0\) beats the base \(−D0\-D\_\{0\}\), but the*confidence*vertexc≡0c\\equiv 0also beats an overconfident base under the Brier score\. Only the local gradient at the vertex separates the mechanisms \(Proposition 3\)\. ∎

#### Proof of Proposition 3 \(local dichotomy\), with the tokenization caveat\.

\(a\)∂v𝔼​\[r\]=∂v\[σ​\(v\)​m​\(q¯\)\]=σ′​\(v\)​m​\(q¯\)→0\\partial\_\{v\}\\mathbb\{E\}\[r\]=\\partial\_\{v\}\[\\sigma\(v\)m\(\\bar\{q\}\)\]=\\sigma^\{\\prime\}\(v\)m\(\\bar\{q\}\)\\to 0asv→−∞v\\to\-\\infty, withsign=sign⁡m\\operatorname\{sign\}=\\operatorname\{sign\}m\. \(b\) By the Savage representation of a strictly proper score,𝔼​\[S​\(c\)\]=Γ​\(c\)\+Γ′​\(c\)​\(q¯−c\)\\mathbb\{E\}\[S\(c\)\]=\\Gamma\(c\)\+\\Gamma^\{\\prime\}\(c\)\(\\bar\{q\}\-c\)withΓ\\Gammastrictly convex, so∂c𝔼​\[S\]=Γ′′​\(c\)​\(q¯−c\)\\partial\_\{c\}\\mathbb\{E\}\[S\]=\\Gamma^\{\\prime\\prime\}\(c\)\(\\bar\{q\}\-c\); integrating \(Γ′\\Gamma^\{\\prime\}strictly increasing\) the expected score strictly increases fromc=0c=0towardq¯\\bar\{q\}, so the vertex is never optimal and has no basin; if additionallyΓ′′≥ζ\>0\\Gamma^\{\\prime\\prime\}\\geq\\zeta\>0near the boundary, the vertex gradient is≥ζ​q¯\>0\\geq\\zeta\\bar\{q\}\>0\. The lower bound is not automatic for every proper score:Γ​\(c\)=c4\\Gamma\(c\)=c^\{4\}is strictly proper yet has∂c𝔼​\[S\]=12​c2​\(q¯−c\)=0\\partial\_\{c\}\\mathbb\{E\}\[S\]=12c^\{2\}\(\\bar\{q\}\-c\)=0atc=0c=0, a first\-order\-degenerate \(but still non\-attracting\) vertex\.*Caveat\.*When the confidence is a sampled token, the vertex gradient in*logit*space carries a softmax factor and vanishes degenerately; this is harmless in practice only because language models initialize overconfident, not underconfident\(Guoet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib40); Tianet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib38)\), an empirical fact rather than a theorem\. ∎

#### Proof of Corollary 1 \(the regularizer class\), with the floor\.

∂vR=\(∂π/∂v\)​∂πR=π​\(1−π\)​∂πR→0\\partial\_\{v\}R=\(\\partial\\pi/\\partial v\)\\,\\partial\_\{\\pi\}R=\\pi\(1\-\\pi\)\\partial\_\{\\pi\}R\\to 0wheneverπ​∂πR→0\\pi\\,\\partial\_\{\\pi\}R\\to 0\. Membership: entropy has∂πH=log⁡1−ππ\\partial\_\{\\pi\}H=\\log\\frac\{1\-\\pi\}\{\\pi\}and the forward anchor∂πKL=logit​π−logit​π0\\partial\_\{\\pi\}\\mathrm\{KL\}=\\mathrm\{logit\}\\,\\pi\-\\mathrm\{logit\}\\,\\pi\_\{0\}, both logarithmic, hence in the class; the reverse anchorKL​\(π0∥πθ\)\\mathrm\{KL\}\(\\pi\_\{0\}\\\|\\pi\_\{\\theta\}\)has∂vR=π−π0→−π0≠0\\partial\_\{v\}R=\\pi\-\\pi\_\{0\}\\to\-\\pi\_\{0\}\\neq 0and diverging vertex cost, hence outside; so isR=−log⁡πR=\-\\log\\pi\(∂vR→−1\\partial\_\{v\}R\\to\-1\)\. The class fixes no rate:∂πR=1/\(π​log⁡\(1/π\)\)\\partial\_\{\\pi\}R=1/\(\\pi\\log\(1/\\pi\)\)qualifies yet gives∂vR≍1/log⁡\(1/π\)\\partial\_\{v\}R\\asymp 1/\\log\(1/\\pi\); entropy and forward KL themselves vanish likeΘ​\(π​log⁡\(1/π\)\)\\Theta\(\\pi\\log\(1/\\pi\)\), one logarithm slower than the reward’sΘ​\(π\)\\Theta\(\\pi\), and that logarithm is the\|v−v0\|\|v\-v\_\{0\}\|in the floor computation\. \(i\)*Entropy bonuses do not help*:d​H/d​v=−v​σ′​\(v\)dH/dv=\-v\\,\\sigma^\{\\prime\}\(v\)carries the same factor as the KL gradient and is of the same order on the plateau\. \(ii\)*The floor is exponentially small*: balancingσ′​\(v\)​Deff\\sigma^\{\\prime\}\(v\)D\_\{\\mathrm\{eff\}\}againstβ​σ′​\(v\)​\|v−v0\|\\beta\\sigma^\{\\prime\}\(v\)\|v\-v\_\{0\}\|gives\|c0\|≍Deff/β\|c\_\{0\}\|\\asymp D\_\{\\mathrm\{eff\}\}/\\betaandPfloor≍e−Deff/βP\_\{\\mathrm\{floor\}\}\\asymp e^\{\-D\_\{\\mathrm\{eff\}\}/\\beta\}\(identicallye−Deff/τe^\{\-D\_\{\\mathrm\{eff\}\}/\\tau\}for an entropy bonus of weightτ\\tau\), reached only after timeeΘ​\(1/β\)e^\{\\Theta\(1/\\beta\)\}in the frozen\-readout bias dynamics; numerically, freezing the readout gain, the predicted stationary pointsc0=v0−D0/βc\_\{0\}=v\_\{0\}\-D\_\{0\}/\\beta\(KL\) andc0=−D0/τc\_\{0\}=\-D\_\{0\}/\\tau\(entropy\) are reproduced to three decimals\. The reverse KL floors atP=Θ​\(β\)P=\\Theta\(\\beta\)\(balanceev​\|m\|=β​π0e^\{v\}\|m\|=\\beta\\pi\_\{0\}\), one logarithm below the proximal floor of Proposition 4\. ∎

#### Proof of Proposition 1 \(tabular policies do not collapse\)\.

With one logit per prompt the flow isv˙​\(s\)=σ′​\(v​\(s\)\)​\[m​\(q¯​\(s\)\)−β​\(v​\(s\)−v0​\(s\)\)\]\\dot\{v\}\(s\)=\\sigma^\{\\prime\}\(v\(s\)\)\[m\(\\bar\{q\}\(s\)\)\-\\beta\(v\(s\)\-v\_\{0\}\(s\)\)\]; sinceσ′\>0\\sigma^\{\\prime\}\>0, stationary points solvem=β​\(v−v0\)m=\\beta\(v\-v\_\{0\}\), i\.e\.v∗​\(s\)=v0​\(s\)\+m​\(q¯​\(s\)\)/βv^\{\\ast\}\(s\)=v\_\{0\}\(s\)\+m\(\\bar\{q\}\(s\)\)/\\beta, and the bracket is decreasing invv, sov∗v^\{\\ast\}is attracting\.sign⁡v∗=sign⁡m\\operatorname\{sign\}v^\{\\ast\}=\\operatorname\{sign\}miffm​\(m\+β​v0\)\>0m\(m\+\\beta v\_\{0\}\)\>0, which fails only on a band of widthO​\(β\)O\(\\beta\)inmmaround zero; a uniform margin\|m\|≥γ\>β​‖v0‖∞\|m\|\\geq\\gamma\>\\beta\\\|v\_\{0\}\\\|\_\{\\infty\}removes it, and for every prompt withm≠0m\\neq 0the decision converges to Chow’s rule asβ↓0\\beta\\downarrow 0\. Numerically, atβ=10−3\\beta=10^\{\-3\}the tabular terminal policy agrees with Chow’s rule on100%100\\%of the evaluation grid \(the misclassified band, of width≈2×10−3\\approx 2\\times 10^\{\-3\}inmm, falls between grid points\) and attains coverage0\.2590\.259against a rational0\.2590\.259\. ∎

## Appendix DProof of Theorem 1 \(Collapse Law\)

#### \(The identity\.\)

For a softmax policy and reward linear in the policy,∂J/∂zj=πj​\(rj−J\)\\partial J/\\partial z\_\{j\}=\\pi\_\{j\}\(r\_\{j\}\-J\)exactly\. On a prompt withm<0m<0, the abstain action is optimal \(rabs=0=J∗r\_\{\\mathrm\{abs\}\}=0=J^\{\\ast\}\), so with gap𝒢=σ​\(v\)​\|m\|\\mathcal\{G\}=\\sigma\(v\)\|m\|,\|∂vJ\|=\(1−σ​\(v\)\)​𝒢\|\\partial\_\{v\}J\|=\(1\-\\sigma\(v\)\)\\,\\mathcal\{G\}\. In the population,∂c0J=∫σ′​\(v\)​g=∫σ​\(1−σ\)​g\\partial\_\{c\_\{0\}\}J=\\int\\sigma^\{\\prime\}\(v\)g=\\int\\sigma\(1\-\\sigma\)g, whileL=∫σ​\(−g\)L=\\int\\sigma\(\-g\); withσ¯\\bar\{\\sigma\}as in Section[B](https://arxiv.org/html/2608.00301#A2),∂c0J=−\(1−σ¯\)​L\\partial\_\{c\_\{0\}\}J=\-\(1\-\\bar\{\\sigma\}\)L*exactly*\. Measured\|∂c0J\|/L=0\.971,0\.995,0\.9987,0\.9997\|\\partial\_\{c\_\{0\}\}J\|/L=0\.971,\\,0\.995,\\,0\.9987,\\,0\.9997att=30,102,3×102,103t=30,10^\{2\},3\\times 10^\{2\},10^\{3\}\. The exponent below is a property of the*parameterization class*: under gradient flow in the logit \(or any fixed linear reparameterization\) it is−1\-1; a nonlinear reparameterization can change it \(flowingθ\>0\\theta\>0withv=−log⁡θv=\-\\log\\thetagives𝒢≍t−1/3\\mathcal\{G\}\\asymp t^\{\-1/3\}\)\. Everything below is at the standard logit parameterization\.

#### \(σ¯\\bar\{\\sigma\}control\.\)

The weightσ​\(−g\)\\sigma\(\-g\)is signed, soσ¯∈\(0,1\)\\bar\{\\sigma\}\\in\(0,1\)is not automatic \(measured,σ¯\\bar\{\\sigma\}touches−0\.0008\-0\.0008att=6×103t=6\\times 10^\{3\}; nearZLZ\_\{L\}it can diverge, whicht≤c⋆​ttiltt\\leq c\_\{\\star\}t\_\{\\mathrm\{tilt\}\}excludes\)\. Since\|g\|=−g\+2​g\+\|g\|=\-g\+2g\_\{\+\},

\|σ¯\|≤supsσ​\(v​\(s\)\)⋅\(1\+2​∫σ​g\+L\)\.\|\\bar\{\\sigma\}\|\\ \\leq\\ \\sup\_\{s\}\\sigma\(v\(s\)\)\\cdot\\Big\(1\+\\frac\{2\\int\\sigma g\_\{\+\}\}\{L\}\\Big\)\.Work in the run’s gaugec4≤0c\_\{4\}\\leq 0, soer≤1e^\{r\}\\leq 1andsupsσ​\(v\)≤ec0\\sup\_\{s\}\\sigma\(v\)\\leq e^\{c\_\{0\}\}\. Fromσ​\(v\)≥ev​\(1−esupsv\)\\sigma\(v\)\\geq e^\{v\}\(1\-e^\{\\sup\_\{s\}v\}\),L≥ec0​\(Deff−ec0​∫er​g−\)≥12​ec0​DeffL\\geq e^\{c\_\{0\}\}\(D\_\{\\mathrm\{eff\}\}\-e^\{c\_\{0\}\}\\\!\\int e^\{r\}g\_\{\-\}\)\\geq\\tfrac\{1\}\{2\}e^\{c\_\{0\}\}D\_\{\\mathrm\{eff\}\}onceec0≤D¯​\(c⋆\)/\(2​G−\)e^\{c\_\{0\}\}\\leq\\underline\{D\}\(c\_\{\\star\}\)/\(2G\_\{\-\}\), which the bias envelope below grants fort≥T0​\(c⋆\):=2​Cb​G−/D¯​\(c⋆\)2t\\geq T\_\{0\}\(c\_\{\\star\}\):=2C\_\{b\}G\_\{\-\}/\\underline\{D\}\(c\_\{\\star\}\)^\{2\}\(Cb:=2/\(1−Λ¯\)C\_\{b\}:=2/\(1\-\\bar\{\\Lambda\}\)\)\. Then∫σ​g\+/L≤2​G\+/D¯​\(c⋆\)\\int\\sigma g\_\{\+\}/L\\leq 2G\_\{\+\}/\\underline\{D\}\(c\_\{\\star\}\), so onT0≤t≤c⋆​ttiltT\_\{0\}\\leq t\\leq c\_\{\\star\}t\_\{\\mathrm\{tilt\}\},

\|σ¯​\(t\)\|≤\(1\+4​G\+D¯​\(c⋆\)\)​ec0​\(t\)=Oc⋆​\(1/t\)\.\|\\bar\{\\sigma\}\(t\)\|\\ \\leq\\ \\Big\(1\+\\frac\{4G\_\{\+\}\}\{\\underline\{D\}\(c\_\{\\star\}\)\}\\Big\)e^\{c\_\{0\}\(t\)\}\\ =\\ O\_\{c\_\{\\star\}\}\(1/t\)\.No sign ofσ¯\\bar\{\\sigma\}is needed; a negative excursion only speeds the certified decay\.

#### \(Gain growth:\|c4\|=Oc⋆​\(log⁡t\)\|c\_\{4\}\|=O\_\{c\_\{\\star\}\}\(\\log t\), derived\.\)

Under \([4](https://arxiv.org/html/2608.00301#A2.E4)\) and the sign inputc4≤0c\_\{4\}\\leq 0on the plateau \(only the sign is used\),c˙4=∫σ′​\(v\)​ϕ​\[g−β​p​\(v−v0\)\]\\dot\{c\}\_\{4\}=\\int\\sigma^\{\\prime\}\(v\)\\phi\[g\-\\beta p\(v\-v\_\{0\}\)\]\. Pointwise\|v−v0\|≤2​‖θ−θ0‖∞\+\|c4​\(0\)\|\|v\-v\_\{0\}\|\\leq 2\\\|\\theta\-\\theta\_\{0\}\\\|\_\{\\infty\}\+\|c\_\{4\}\(0\)\|, so \([4](https://arxiv.org/html/2608.00301#A2.E4)\) givesβ​‖v−v0‖∞≤Deff\+β​\|c4​\(0\)\|\\beta\\\|v\-v\_\{0\}\\\|\_\{\\infty\}\\leq D\_\{\\mathrm\{eff\}\}\+\\beta\|c\_\{4\}\(0\)\|; withϕ≤1\\phi\\leq 1,σ′≤ev\\sigma^\{\\prime\}\\leq e^\{v\},er≤1e^\{r\}\\leq 1:

\|c˙4\|≤\(G\+\+2​G−\+β​\|c4​\(0\)\|\)​ec0\.\|\\dot\{c\}\_\{4\}\|\\ \\leq\\ \\big\(G\_\{\+\}\+2G\_\{\-\}\+\\beta\|c\_\{4\}\(0\)\|\\big\)\\,e^\{c\_\{0\}\}\.For the bias envelope,c˙0≤ec0​\[−Deff\+ec0​G−\+\(Deff\+β​\|c4​\(0\)\|\)​Λ\]\\dot\{c\}\_\{0\}\\leq e^\{c\_\{0\}\}\[\-D\_\{\\mathrm\{eff\}\}\+e^\{c\_\{0\}\}G\_\{\-\}\+\(D\_\{\\mathrm\{eff\}\}\+\\beta\|c\_\{4\}\(0\)\|\)\\Lambda\]; once the tilt inputΛ​\(τ\)≤Λ¯<1\\Lambda\(\\tau\)\\leq\\bar\{\\Lambda\}<1holds past the transient \(measuredΛ≤0\.3\\Lambda\\leq 0\.3fromt≥102t\\geq 10^\{2\};Λ<1\\Lambda<1is automatic forc4<0c\_\{4\}<0, a quantitative bound is not\) and the small\-β\\betacondition

β​\|c4​\(0\)\|​Λ¯≤14​\(1−Λ¯\)​D¯​\(c⋆\)\\beta\\,\|c\_\{4\}\(0\)\|\\,\\bar\{\\Lambda\}\\ \\leq\\ \\tfrac\{1\}\{4\}\\,\(1\-\\bar\{\\Lambda\}\)\\,\\underline\{D\}\(c\_\{\\star\}\)\(5\)holds \(a trajectory\-level hypothesis*not*implied by \([4](https://arxiv.org/html/2608.00301#A2.E4)\); on the run it holds with factor1313–8484to spare\),e−c0e^\{\-c\_\{0\}\}grows at rate≳\(1−Λ¯\)​D¯​\(c⋆\)/2\\gtrsim\(1\-\\bar\{\\Lambda\}\)\\underline\{D\}\(c\_\{\\star\}\)/2past a transient, whenceec0​\(τ\)≤min⁡\{ec0​\(0\),Cb/\(τ​D¯​\(c⋆\)\)\}e^\{c\_\{0\}\(\\tau\)\}\\leq\\min\\\{e^\{c\_\{0\}\(0\)\},C\_\{b\}/\(\\tau\\underline\{D\}\(c\_\{\\star\}\)\)\\\}\. Integrating\|c˙4\|\|\\dot\{c\}\_\{4\}\|against this envelope gives\|c4​\(t\)\|≤\|c4​\(0\)\|\+Oc⋆​\(log⁡t\)\|c\_\{4\}\(t\)\|\\leq\|c\_\{4\}\(0\)\|\+O\_\{c\_\{\\star\}\}\(\\log t\)\. The same bounds control the anchor gradient:β​\|∂c0𝒦\|,β​\|∂c4𝒦\|≤\(Deff\+β​\|c4​\(0\)\|\)​ec0=Oc⋆​\(L\)\\beta\|\\partial\_\{c\_\{0\}\}\\mathcal\{K\}\|,\\beta\|\\partial\_\{c\_\{4\}\}\\mathcal\{K\}\|\\leq\(D\_\{\\mathrm\{eff\}\}\+\\beta\|c\_\{4\}\(0\)\|\)e^\{c\_\{0\}\}=O\_\{c\_\{\\star\}\}\(L\), whileβ​\|∂w𝒦\|,β​\|∂a𝒦\|\\beta\|\\partial\_\{w\}\\mathcal\{K\}\|,\\beta\|\\partial\_\{a\}\\mathcal\{K\}\|carry an extra\|c4\|\|c\_\{4\}\|and anss\-factor bounded in first\-moment form \(ϕ′≤14\\phi^\{\\prime\}\\leq\\tfrac\{1\}\{4\};sups\|ϕ′​\(w​s\+a\)​s\|\\sup\_\{s\}\|\\phi^\{\\prime\}\(ws\+a\)s\|can blow up asw→0w\\to 0,𝔼p​\|s\|\\mathbb\{E\}\_\{p\}\|s\|cannot\), soβ​‖∇𝒦‖=Oc⋆​\(\(1\+\|c4\|\)​L\)\\beta\\\|\\nabla\\mathcal\{K\}\\\|=O\_\{c\_\{\\star\}\}\(\(1\+\|c\_\{4\}\|\)L\)\. The certified constants are far from sharp; the measured fit is\|c4\|=0\.469​log⁡t−0\.390\|c\_\{4\}\|=0\.469\\log t\-0\.390withR2=0\.9998R^\{2\}=0\.9998\(againstR2=0\.9739R^\{2\}=0\.9739for the best power law\)\.

#### \(Upper bound and envelope\.\)

L˙=−∇J⋅∇Jβ=−‖∇J‖2\+β​∇J⋅∇𝒦\\dot\{L\}=\-\\nabla J\\cdot\\nabla J\_\{\\beta\}=\-\\\|\\nabla J\\\|^\{2\}\+\\beta\\nabla J\\cdot\\nabla\\mathcal\{K\}\. Under the anchor\-share hypothesisρ≤12\\rho\\leq\\tfrac\{1\}\{2\}\(for which \([4](https://arxiv.org/html/2608.00301#A2.E4)\) is the up\-to\-Θ\\Thetacertificate\),

L˙≤−12​‖∇J‖2≤−12​\(∂c0J\)2=−12​\(1−σ¯\)2​L2,\\dot\{L\}\\ \\leq\\ \-\\tfrac\{1\}\{2\}\\\|\\nabla J\\\|^\{2\}\\ \\leq\\ \-\\tfrac\{1\}\{2\}\(\\partial\_\{c\_\{0\}\}J\)^\{2\}\\ =\\ \-\\tfrac\{1\}\{2\}\(1\-\\bar\{\\sigma\}\)^\{2\}L^\{2\},so1/L​\(t\)≥1/L​\(0\)\+12​∫0t\(1−σ¯\)2​𝑑τ1/L\(t\)\\geq 1/L\(0\)\+\\tfrac\{1\}\{2\}\\int\_\{0\}^\{t\}\(1\-\\bar\{\\sigma\}\)^\{2\}d\\tau: the envelope, needing nothing beyondρ≤12\\rho\\leq\\tfrac\{1\}\{2\}\(atβ=0\\beta=0the factor12\\tfrac\{1\}\{2\}disappears\)\. The upgrade toL=O​\(1/t\)L=O\(1/t\)needslim inft1t​∫0t\(1−σ¯\)2\>0\\liminf\_\{t\}\\tfrac\{1\}\{t\}\\int\_\{0\}^\{t\}\(1\-\\bar\{\\sigma\}\)^\{2\}\>0, supplied byσ¯\\bar\{\\sigma\}control from \([4](https://arxiv.org/html/2608.00301#A2.E4)\) with the sign, tilt, and small\-β\\betainputs\. No pointwiseL≤\(\(1−σ¯\)2​t\)−1L\\leq\(\(1\-\\bar\{\\sigma\}\)^\{2\}t\)^\{\-1\}follows \(that would hold the time\-varyingσ¯\\bar\{\\sigma\}at its smallest value; measuredL​t=1\.26\>1Lt=1\.26\>1att=30t=30is the counterexample\); the finite\-ttbound keeps the intercept1/L​\(0\)1/L\(0\)\.

#### \(Lower bound\.\)

∂c0J,∂c4J=O​\(L\)\\partial\_\{c\_\{0\}\}J,\\partial\_\{c\_\{4\}\}J=O\(L\)\(ϕ∈\[0,1\]\\phi\\in\[0,1\]\), but∂wJ=∫σ′​\(v\)​c4​ϕ′​\(w​s\+a\)​s​g\\partial\_\{w\}J=\\int\\sigma^\{\\prime\}\(v\)c\_\{4\}\\phi^\{\\prime\}\(ws\+a\)s\\,gis bounded withϕ′≤14\\phi^\{\\prime\}\\leq\\tfrac\{1\}\{4\}andM1<∞M\_\{1\}<\\infty:\|∂wJ\|≤14​\|c4\|​ec0​M1=Oc⋆​\(\|c4\|​L\)\|\\partial\_\{w\}J\|\\leq\\tfrac\{1\}\{4\}\|c\_\{4\}\|e^\{c\_\{0\}\}M\_\{1\}=O\_\{c\_\{\\star\}\}\(\|c\_\{4\}\|L\), likewise∂aJ\\partial\_\{a\}J\. With\|c4\|=O​\(log⁡t\)\|c\_\{4\}\|=O\(\\log t\)\(gain growth\) one gets‖∇J‖2=O​\(L2​log2⁡t\)\\\|\\nabla J\\\|^\{2\}=O\(L^\{2\}\\log^\{2\}t\), and withβ​‖∇𝒦‖=Oc⋆​\(\(1\+\|c4\|\)​L\)\\beta\\\|\\nabla\\mathcal\{K\}\\\|=O\_\{c\_\{\\star\}\}\(\(1\+\|c\_\{4\}\|\)L\)the cross term is controlled in absolute value*whatever its sign*:\|L˙\|≤‖∇J‖2\+β​‖∇J‖​‖∇𝒦‖=O​\(L2​log2⁡t\)\|\\dot\{L\}\|\\leq\\\|\\nabla J\\\|^\{2\}\+\\beta\\\|\\nabla J\\\|\\\|\\nabla\\mathcal\{K\}\\\|=O\(L^\{2\}\\log^\{2\}t\), soL=Ω​\(1/\(t​log2⁡t\)\)L=\\Omega\(1/\(t\\log^\{2\}t\)\)\. Both bounds requiret≤c⋆​ttiltt\\leq c\_\{\\star\}t\_\{\\mathrm\{tilt\}\}, and every constant degrades asc⋆→1c\_\{\\star\}\\to 1\(the two lobes ofggcancel inLLatttiltt\_\{\\mathrm\{tilt\}\}but not in∫σ′​\|g\|\\int\\sigma^\{\\prime\}\|g\|\)\.

#### \(Observables, and the endpoint\.\)

Deff=−Λ​m¯tiltD\_\{\\mathrm\{eff\}\}=\-\\Lambda\\bar\{m\}\_\{\\mathrm\{tilt\}\}by definition;L=ec0​Deff​\(1\+εL\)L=e^\{c\_\{0\}\}D\_\{\\mathrm\{eff\}\}\(1\+\\varepsilon\_\{L\}\)andP=Λ​ec0​\(1\+εP\)P=\\Lambda e^\{c\_\{0\}\}\(1\+\\varepsilon\_\{P\}\)with the*exact*error functions

εL=−∫er​σ​\(v\)​\(−g\)Deff,εP=−∫er​σ​\(v\)​pΛ,\\varepsilon\_\{L\}=\-\\frac\{\\int e^\{r\}\\sigma\(v\)\(\-g\)\}\{D\_\{\\mathrm\{eff\}\}\},\\qquad\\varepsilon\_\{P\}=\-\\frac\{\\int e^\{r\}\\sigma\(v\)\\,p\}\{\\Lambda\},satisfying\|εP\|≤ev¯\|\\varepsilon\_\{P\}\|\\leq e^\{\\bar\{v\}\}and\|εL\|≤ev¯​\(1\+2​G\+/D¯​\(c⋆\)\)\|\\varepsilon\_\{L\}\|\\leq e^\{\\bar\{v\}\}\(1\+2G\_\{\+\}/\\underline\{D\}\(c\_\{\\star\}\)\)on the window \(v¯:=supsv=c0\\bar\{v\}:=\\sup\_\{s\}v=c\_\{0\}in the gaugec4≤0c\_\{4\}\\leq 0\)\. Hence

P=L\|m¯tilt\|⋅1\+εP1\+εLP=\\frac\{L\}\{\|\\bar\{m\}\_\{\\mathrm\{tilt\}\}\|\}\\cdot\\frac\{1\+\\varepsilon\_\{P\}\}\{1\+\\varepsilon\_\{L\}\}exactly away fromZL∪ZmZ\_\{L\}\\cup Z\_\{m\}, and differentiating the defining integrals term by term givest​\(\|ε˙L\|\+\|ε˙P\|\)=O​\(ev¯​log⁡t\)=o​\(1\)t\(\|\\dot\{\\varepsilon\}\_\{L\}\|\+\|\\dot\{\\varepsilon\}\_\{P\}\|\)=O\(e^\{\\bar\{v\}\}\\log t\)=o\(1\): the slope identityd​log⁡P/d​log⁡t=d​log⁡L/d​log⁡t−d​log⁡\|m¯tilt\|/d​log⁡t\+o​\(1\)\\mathrm\{d\}\\log P/\\mathrm\{d\}\\log t=\\mathrm\{d\}\\log L/\\mathrm\{d\}\\log t\-\\mathrm\{d\}\\log\|\\bar\{m\}\_\{\\mathrm\{tilt\}\}\|/\\mathrm\{d\}\\log t\+o\(1\), which the run reproduces to three decimals in every instrumented window\. BecauseΛ∈\[t−Θ​\(1\),tΘ​\(1\)\]\\Lambda\\in\[t^\{\-\\Theta\(1\)\},t^\{\\Theta\(1\)\}\]under\|c4\|=Θ​\(log⁡t\)\|c\_\{4\}\|=\\Theta\(\\log t\), the aggregate exponent ofPPis*not*determined by the theorem while that ofLLis\.

*Transversality, reduced to a checkable condition\.*DeffD\_\{\\mathrm\{eff\}\}does not depend onc0c\_\{0\}; fori∈\{w,a,c4\}i\\in\\\{w,a,c\_\{4\}\\\},∂iDeff=−∫\(∂iv\)erg=:−Ai\\partial\_\{i\}D\_\{\\mathrm\{eff\}\}=\-\\int\(\\partial\_\{i\}v\)e^\{r\}g=:\-A\_\{i\}, while the flow movesθ˙i=ec0​\(Ai\+ei\)\\dot\{\\theta\}\_\{i\}=e^\{c\_\{0\}\}\(A\_\{i\}\+e\_\{i\}\)with explicit plateau\-approximation errorseie\_\{i\}\. The chain rule gives

D˙eff=−ec0​\(Aw2\+Aa2\+Ac42\)−ec0​∑iAi​ei:\\dot\{D\}\_\{\\mathrm\{eff\}\}\\;=\\;\-\\,e^\{c\_\{0\}\}\\big\(A\_\{w\}^\{2\}\+A\_\{a\}^\{2\}\+A\_\{c\_\{4\}\}^\{2\}\\big\)\\;\-\\;e^\{c\_\{0\}\}\\sum\_\{i\}A\_\{i\}e\_\{i\}:minus a sum of squares\. Consequently \(i\)DeffD\_\{\\mathrm\{eff\}\}is monotone decreasing on the plateau up to the stated errors \(verified: exactD˙eff<0\\dot\{D\}\_\{\\mathrm\{eff\}\}<0on all of\[30,1\.2×104\]\[30,1\.2\\times 10^\{4\}\], matching−ec0​∑Ai2\-e^\{c\_\{0\}\}\\sum A\_\{i\}^\{2\}to0\.990\.99–1\.011\.01\); \(ii\) at a zeroZZofDeffD\_\{\\mathrm\{eff\}\}, single\-crossing ofgginϕ\\phi\(automatic for strictly monotoneq¯\\bar\{q\}andw≠0w\\neq 0\) gives\|Ac4\(Z\)\|=∫er\|ϕ−ϕ∗\|\|g\|=:δ\(Z\)\>0\|A\_\{c\_\{4\}\}\(Z\)\|=\\int e^\{r\}\|\\phi\-\\phi^\{\\ast\}\|\|g\|=:\\delta\(Z\)\>0\(measuredδ​\(Zm\)=0\.0299\\delta\(Z\_\{m\}\)=0\.0299to five decimals\)\. Provided the*error\-dominance condition*∑i\|Ai​ei\|≤12​∑iAi2\\sum\_\{i\}\|A\_\{i\}e\_\{i\}\|\\leq\\tfrac\{1\}\{2\}\\sum\_\{i\}A\_\{i\}^\{2\}holds nearZZ, a smallness condition that must be checked separately \(it is*not*implied by \([4](https://arxiv.org/html/2608.00301#A2.E4)\), whose right side vanishes atZmZ\_\{m\}; measured ratio1\.011\.01\), the zero is transversal with rate carrying its ownec0e^\{c\_\{0\}\}, and combining with the exact identity\|Deff​\(ZL\)\|≤ec0​∫e2​r​\|g\|\|D\_\{\\mathrm\{eff\}\}\(Z\_\{L\}\)\|\\leq e^\{c\_\{0\}\}\\int e^\{2r\}\|g\|atZLZ\_\{L\}, the twoec0e^\{c\_\{0\}\}factors cancel:

\|ZL−Zm\|≤2​∫e2​r​\|g\|δ​\(Zm\)2​\(1\+o​\(1\)\)=O​\(1\)\|Z\_\{L\}\-Z\_\{m\}\|\\ \\leq\\ \\frac\{2\\int e^\{2r\}\|g\|\}\{\\delta\(Z\_\{m\}\)^\{2\}\}\\,\\big\(1\+o\(1\)\\big\)\\;=\\;O\(1\)in*absolute*time units\. Measured: first\-order prediction\|Deff​\(ZL\)\|/\|D˙eff​\(Zm\)\|=11\.5\|D\_\{\\mathrm\{eff\}\}\(Z\_\{L\}\)\|/\|\\dot\{D\}\_\{\\mathrm\{eff\}\}\(Z\_\{m\}\)\|=11\.5time units against a measured gapZL−Zm=11\.5Z\_\{L\}\-Z\_\{m\}=11\.5\(Zm=11452\.3Z\_\{m\}=11452\.3,ZL=11463\.8Z\_\{L\}=11463\.8:0\.10%0\.10\\%\)\. ∎

#### The four zeros, and which are observable\.

The bias obeysc˙0=ec0​Λ​m¯tilt\+β​ec0​Θ​\(\|c0\|\)\\dot\{c\}\_\{0\}=e^\{c\_\{0\}\}\\Lambda\\bar\{m\}\_\{\\mathrm\{tilt\}\}\+\\beta e^\{c\_\{0\}\}\\Theta\(\|c\_\{0\}\|\), the anchor correction positive\. Distinguish:ZLZ\_\{L\}\(zero ofLL; directly observable, no gauge\);ZmZ\_\{m\}\(zero ofm¯tilt\\bar\{m\}\_\{\\mathrm\{tilt\}\}; observable by stratifying on confidence;0\.10%0\.10\\%fromZLZ\_\{L\}on the run\);ZAZ\_\{A\}\(zero ofc˙0\\dot\{c\}\_\{0\};*not*observable, sincec0c\_\{0\}is a coordinate of our parameterization\);ZCZ\_\{C\}\(zero ofd​log⁡P/d​log⁡t\\mathrm\{d\}\\log P/\\mathrm\{d\}\\log t\)\. The anchor makesZA<ZmZ\_\{A\}<Z\_\{m\}, with measured gapsZm−ZA=2285,981,355,96,17Z\_\{m\}\-Z\_\{A\}=2285,\\,981,\\,355,\\,96,\\,17atβ=10−2,3×10−3,10−3,3×10−4,10−4\\beta=10^\{\-2\},3\\times 10^\{\-3\},10^\{\-3\},3\\times 10^\{\-4\},10^\{\-4\}; the gaps are*not*proportional toβ\\beta\(ratios vary by a factor of two\), so no proportionality is claimed\.ZLZ\_\{L\}itself moves withβ\\beta:8745,10709,11464,11756,118428745,\\,10709,\\,11464,\\,11756,\\,11842over the same grid: a26%26\\%shift atβ=10−2\\beta=10^\{\-2\}but3\.2%3\.2\\%between10−310^\{\-3\}and10−410^\{\-4\}\. The defensible statement is:*the mean reward crosses zero at a time that stabilizes asβ↓0\\beta\\downarrow 0, to within5%5\\%forβ≤10−3\\beta\\leq 10^\{\-3\}\.*

#### Gauge\-dependence, and the coincidence of zeros\.

The decomposition of thePP\-slope into a bias term and a tilt term is gauge\-dependent \(the reparameterization of Section[E](https://arxiv.org/html/2608.00301#A5)moves mass betweenc0c\_\{0\}andc4c\_\{4\}while fixingv​\(⋅\)v\(\\cdot\)pointwise\); the observablesLL,PP,m¯tilt\\bar\{m\}\_\{\\mathrm\{tilt\}\}are not, so the observed “compensation” between the two terms needs no proof\. The falsifiable endpoint prediction is a*coincidence of zeros*, not the value of any exponent: over successive plateau windows the aggregate sloped​log⁡P/d​log⁡t\\mathrm\{d\}\\log P/\\mathrm\{d\}\\log truns−1\.016,−0\.838,−0\.618,−0\.270,\+0\.068\-1\.016,\\,\-0\.838,\\,\-0\.618,\\,\-0\.270,\\,\+0\.068whilem¯tilt\\bar\{m\}\_\{\\mathrm\{tilt\}\}runs−1\.318→\+0\.013\-1\.318\\to\+0\.013\. The slope flattens and flips exactly asm¯tilt\\bar\{m\}\_\{\\mathrm\{tilt\}\}does, and it is the coincidence of the two zero crossings that is the prediction\.

#### Where the plateau ends, mechanically\.

The tiltec4​ϕe^\{c\_\{4\}\\phi\}up\-weights the region ofϕ\\phiselected by the sign ofc4c\_\{4\}, and the trained head ordersϕ\\phiso that this is the*profitable*region \(on the run,w<0w<0andc4<0c\_\{4\}<0: the mean ofϕ\\phiis0\.2820\.282ong\+g\_\{\+\}against0\.9150\.915ong−g\_\{\-\}, ande−\|c4\|​ϕe^\{\-\|c\_\{4\}\|\\phi\}up\-weights exactly the former\)\. Hencem¯tilt\\bar\{m\}\_\{\\mathrm\{tilt\}\}increases with\|c4\|\|c\_\{4\}\|and crosses zero at a value that is a constant of the problem; because\|c4\|\|c\_\{4\}\|grows likelog⁡t\\log t,ttiltt\_\{\\mathrm\{tilt\}\}is finite and, in the limitβ↓0\\beta\\downarrow 0,β\\beta\-independent\. The model escapes by*re\-weighting*its answering toward the profitable prompts, which is also why the escape is invisible to greedy deployment until much later \(Section[E](https://arxiv.org/html/2608.00301#A5)\)\.

#### A certified slope band with no unknown constants\.

PutR​\(t\):=‖∇J‖2/\(∂c0J\)2≥1R\(t\):=\\\|\\nabla J\\\|^\{2\}/\(\\partial\_\{c\_\{0\}\}J\)^\{2\}\\geq 1\. Exactly,

−R​\(1−σ¯\)2​t​L≤d​log⁡Ld​log⁡t≤−12​\(1−σ¯\)2​t​L,\-R\\,\(1\-\\bar\{\\sigma\}\)^\{2\}\\,tL\\ \\leq\\ \\frac\{\\mathrm\{d\}\\log L\}\{\\mathrm\{d\}\\log t\}\\ \\leq\\ \-\\tfrac\{1\}\{2\}\(1\-\\bar\{\\sigma\}\)^\{2\}\\,tL,the left edge an edge only whereρ≥0\\rho\\geq 0\. All three factors are measurable; att=102,3×102,103,3×103t=10^\{2\},3\\times 10^\{2\},10^\{3\},3\\times 10^\{3\}the lower edges are−1\.315,−1\.132,−1\.095,−1\.239\-1\.315,\-1\.132,\-1\.095,\-1\.239against exact slopes−1\.310,−1\.125,−1\.086,−1\.229\-1\.310,\-1\.125,\-1\.086,\-1\.229\. The gap between slope and edge, in units of the edge,*is*the anchor shareρ​\(t\)\\rho\(t\)\(identically\), measured at0\.40\.4–0\.8%0\.8\\%across the windows and matching the independently computedρ\\rhoto five decimals: this is the one place the anchor’s effect is measured rather than bounded\. The band*widens*likelog2⁡t\\log^\{2\}t:R−1=Θ​\(c42\)R\-1=\\Theta\(c\_\{4\}^\{2\}\), which is the lower bound’s polylog in observable form, an independent and fit\-free prediction\.

## Appendix EBounded Readouts and the Two Escapes

#### Proof of Proposition 5\.

supsv​\(s\)=max⁡\(c4,0\)\+c0\\sup\_\{s\}v\(s\)=\\max\(c\_\{4\},0\)\+c\_\{0\}sinceϕ\\phiranges over\(0,1\)\(0,1\); greedy decoding answers a positive\-measure set iff this exceeds0\. The scalar is gauge\-invariant: the reparameterization\(w,a,c4,c0\)↦\(−w,−a,−c4,c0\+c4\)\(w,a,c\_\{4\},c\_\{0\}\)\\mapsto\(\-w,\-a,\-c\_\{4\},c\_\{0\}\+c\_\{4\}\)leavesv​\(⋅\)v\(\\cdot\)pointwise unchanged while flipping the sign ofc4c\_\{4\}\. For an unbounded featurev=u​s\+bv=us\+bwithu\>0u\>0and full\-supportss, greedy answers the tail\{s\>−b/u\}\\\{s\>\-b/u\\\}for every finitebb\. For the tilt: escape requires∫er​g\+\\int e^\{r\}g\_\{\+\}to outgrow∫er​g−\\int e^\{r\}g\_\{\-\}; withr=c4​ϕr=c\_\{4\}\\phi,ϕ∈\[0,1\]\\phi\\in\[0,1\],er≤e\|c4\|e^\{r\}\\leq e^\{\|c\_\{4\}\|\}uniformly, so under gain growth \(\|c4\|=Oc⋆​\(log⁡t\)\|c\_\{4\}\|=O\_\{c\_\{\\star\}\}\(\\log t\)\) the available tilt is polynomial inttand reaching gainMMtakes timeeΩ​\(M\)e^\{\\Omega\(M\)\}; for a Gaussian feature the tilteu2/2e^\{u^\{2\}/2\}is super\-polynomial at the same gain growth\. ∎

#### Two escapes\.

The*sampling*policy escapes atttiltt\_\{\\mathrm\{tilt\}\}, whereDeffD\_\{\\mathrm\{eff\}\}crosses zero;*greedy*coverage recovers only whensupsv​\(s\)\\sup\_\{s\}v\(s\)crosses zero, strictly later \(Figure[5](https://arxiv.org/html/2608.00301#A5.F5)\)\. On the runc4<0c\_\{4\}<0throughout \(−1\.78\-1\.78att=102t=10^\{2\},−19\.5\-19\.5att=105t=10^\{5\}\), so greedy recovery waits forc0c\_\{0\}itself to turn positive, betweent=3×104t=3\\times 10^\{4\}and5×1045\\times 10^\{4\}at the working point,*after*the sampling escape atttilt≈1\.15×104t\_\{\\mathrm\{tilt\}\}\\approx 1\.15\\times 10^\{4\}\. Exit times are erratic in the signal quality because they depend on how far pastttiltt\_\{\\mathrm\{tilt\}\}the recovery ofc0c\_\{0\}must run: atβ=10−3\\beta=10^\{\-3\}the first time greedy coverage is positive is≤10\\leq 10forλ≤3\.5\\lambda\\leq 3\.5at everyμ\\mu; atλ=4\\lambda=4it is2\.1×1042\.1\\times 10^\{4\}\(μ=1\\mu\{=\}1\),7\.4×1037\.4\\times 10^\{3\}\(μ=2\\mu\{=\}2\),\>107\{\>\}10^\{7\}\(μ=0\.5\\mu\{=\}0\.5\); atλ=4\.5\\lambda=4\.5,3\.3×1043\.3\\times 10^\{4\}\(μ=1\\mu\{=\}1\) and\>107\{\>\}10^\{7\}forμ∈\{0\.5,2\}\\mu\\in\\\{0\.5,2\\\}\. Exit times vary by less than a factor22acrossβ∈\[10−4,10−2\]\\beta\\in\[10^\{\-4\},10^\{\-2\}\], as the asymptoticβ\\beta\-independence ofttiltt\_\{\\mathrm\{tilt\}\}predicts; the non\-monotonicity inμ\\muis real \(a ridge in the exit\-time landscape, Figure[6](https://arxiv.org/html/2608.00301#A5.F6)\), spanning more than five orders of magnitude across a band ofλ\\lambdaonly two units wide\.*The plateau is a slow manifold, not a trap; its length relative to any realistic training budget, not absorption, is the claim\.*

![Refer to caption](https://arxiv.org/html/2608.00301v1/x5.png)Figure 5:\(a\) Terminal policy logits atT=104T=10^\{4\},μ=1\\mu=1,λ=4\.5\\lambda=4\.5: with a bounded readoutv​\(s\)<0v\(s\)<0for everyss\(greedy abstains on everything\); with an unbounded feature the logit crosses zero and greedy answers a tail\. \(b\) The tilt factor required for escape diverges for unbounded features and is capped for bounded ones\.![Refer to caption](https://arxiv.org/html/2608.00301v1/x6.png)Figure 6:A horizon\-free view:tentert\_\{\\mathrm\{enter\}\}is the first time greedy coverage falls below2%2\\%of rational,texitt\_\{\\mathrm\{exit\}\}the first time thereafter it exceeds20%20\\%\. \(a\)log10⁡texit\\log\_\{10\}t\_\{\\mathrm\{exit\}\}under true KL with a bounded head atβ=10−3\\beta=10^\{\-3\}; green cells never collapse\. \(b\) Plateau lengthtexit/tentert\_\{\\mathrm\{exit\}\}/t\_\{\\mathrm\{enter\}\}explodes past a criticalλ\\lambdathat increases with signal qualityμ\\mu, non\-monotonically; the ridge is displayed rather than smoothed\.

## Appendix FThe Lifting Theorem in Full

###### Theorem F\.1\(Gate\-to\-sequence lifting; full statement\)\.

Letπθ​\(y∣x\)\\pi\_\{\\theta\}\(y\\mid x\)be a sequence policy,AAthe refusal event,πans​\(x\)=1−πθ​\(A∣x\)\\pi\_\{\\mathrm\{ans\}\}\(x\)=1\-\\pi\_\{\\theta\}\(A\\mid x\), and anchor the objective with the full sequence KL\. WriteKans\(x\)=KL\(πθ\(⋅∣x,Ac\)∥π0\(⋅∣x,Ac\)\)K\_\{\\mathrm\{ans\}\}\(x\)=\\mathrm\{KL\}\(\\pi\_\{\\theta\}\(\\cdot\\mid x,A^\{c\}\)\\\|\\pi\_\{0\}\(\\cdot\\mid x,A^\{c\}\)\),Kabs​\(x\)K\_\{\\mathrm\{abs\}\}\(x\)likewise for the refusal branch, andqθ​\(x\)q\_\{\\theta\}\(x\)for the answer\-conditional correctness\.

Part I \(exact identities; any mixture parameterization\)\.For every policy,

KLseq​\(x\)=KLgate​\(x\)\+πans​Kans\+πabs​Kabs,\\mathrm\{KL\}\_\{\\mathrm\{seq\}\}\(x\)=\\mathrm\{KL\}\_\{\\mathrm\{gate\}\}\(x\)\+\\pi\_\{\\mathrm\{ans\}\}K\_\{\\mathrm\{ans\}\}\+\\pi\_\{\\mathrm\{abs\}\}K\_\{\\mathrm\{abs\}\},whereKLgate=KL​\(Bern​\(πans\)∥Bern​\(πans0\)\)\\mathrm\{KL\}\_\{\\mathrm\{gate\}\}=\\mathrm\{KL\}\(\\mathrm\{Bern\}\(\\pi\_\{\\mathrm\{ans\}\}\)\\\|\\mathrm\{Bern\}\(\\pi^\{0\}\_\{\\mathrm\{ans\}\}\)\)\. If the gate has a logitvvon which the branch conditionals do not depend, then

∂vJβ=σ′​\(v\)​\[m​\(qθ\)−β​\(v−v0\)−β​\(Kans−Kabs\)\]:\\partial\_\{v\}J\_\{\\beta\}=\\sigma^\{\\prime\}\(v\)\\big\[m\(q\_\{\\theta\}\)\-\\beta\(v\-v\_\{0\}\)\-\\beta\(K\_\{\\mathrm\{ans\}\}\-K\_\{\\mathrm\{abs\}\}\)\\big\]:every term carriesσ′​\(v\)\\sigma^\{\\prime\}\(v\); Lemma 1, Proposition 2 \(the all\-abstain state with refusals at base costs exactlyKL¯⟂\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}under the sequence KL, soβcrit\\beta\_\{\\mathrm\{crit\}\}is unchanged\) and Corollary 1 lift verbatim withmeff=m​\(qθ\)−β​\(Kans−Kabs\)m\_\{\\mathrm\{eff\}\}=m\(q\_\{\\theta\}\)\-\\beta\(K\_\{\\mathrm\{ans\}\}\-K\_\{\\mathrm\{abs\}\}\)\. The anchor taxes only the branch taken: once answer content has drifted \(Kans\>KabsK\_\{\\mathrm\{ans\}\}\>K\_\{\\mathrm\{abs\}\}\), its gate component pushes*toward*abstention: abstaining hides content drift from the KL\. \(In the closed model this shift is second\-order:ZLZ\_\{L\}moves by<0\.1<0\.1time units when theKansK\_\{\\mathrm\{ans\}\}term is switched off; it becomes first\-order when content drifts exogenously\.\)

Part II \(branch\-separable content\)\.Assumeψ=\(ψans,ψabs\)\\psi=\(\\psi\_\{\\mathrm\{ans\}\},\\psi\_\{\\mathrm\{abs\}\}\)withqθ,Kansq\_\{\\theta\},K\_\{\\mathrm\{ans\}\}depending only onψans\\psi\_\{\\mathrm\{ans\}\}andKabsK\_\{\\mathrm\{abs\}\}only onψabs\\psi\_\{\\mathrm\{abs\}\}; refusals initialized at base; the answer branch within bounded KL of base,K¯0:=supxKans​\(x,0\)<∞\\overline\{K\}\_\{0\}:=\\sup\_\{x\}K\_\{\\mathrm\{ans\}\}\(x,0\)<\\infty; and the regularity boundCK:=supx,t\(\|∂ψansqθ\|\+‖∇ψansKans‖\)<∞C\_\{K\}:=\\sup\_\{x,t\}\(\|\\partial\_\{\\psi\_\{\\mathrm\{ans\}\}\}q\_\{\\theta\}\|\+\\\|\\nabla\_\{\\psi\_\{\\mathrm\{ans\}\}\}K\_\{\\mathrm\{ans\}\}\\\|\)<\\infty, an explicit assumption \(it holds in the Bernoulli content model, whose correctness probabilities stay in a compact subinterval of\(0,1\)\(0,1\); it does*not*follow from a bound on the scalar correctness\)\. Then the refusal branch is invariant \(Kabs≡0K\_\{\\mathrm\{abs\}\}\\equiv 0along the flow\), the collapsed manifold\{πans=0,ψabs=ψabs,0\}\\\{\\pi\_\{\\mathrm\{ans\}\}=0,\\ \\psi\_\{\\mathrm\{abs\}\}=\\psi\_\{\\mathrm\{abs\},0\}\\\}is stationary in the limit, and the value term is controlled along the flow by the path bound

supxKans​\(x,t\)≤K¯0\+\(1\+λ\+β\)​CK2​∫0tP​\(τ\)​𝑑τ\.\\sup\_\{x\}K\_\{\\mathrm\{ans\}\}\(x,t\)\\ \\leq\\ \\overline\{K\}\_\{0\}\+\(1\+\\lambda\+\\beta\)\\,C\_\{K\}^\{2\}\\\!\\int\_\{0\}^\{t\}P\(\\tau\)\\,d\\tau\.The only force surviving off the manifold restores*refusals*to base, never answering\. With content*shared*across branches this fails:−β​πabs​∇ψKabs\-\\beta\\pi\_\{\\mathrm\{abs\}\}\\nabla\_\{\\psi\}K\_\{\\mathrm\{abs\}\}isO​\(β\)O\(\\beta\)and unthrottled \(πabs→1\\pi\_\{\\mathrm\{abs\}\}\\to 1\); that case is open\.

Part III \(finite\-time sandwich, lifted\)\.Assume additionallyρfull:=β​∇J⋅∇KLseq/‖∇J‖2≤12\\rho\_\{\\mathrm\{full\}\}:=\\beta\\,\\nabla J\\cdot\\nabla\\mathrm\{KL\}\_\{\\mathrm\{seq\}\}/\\\|\\nabla J\\\|^\{2\}\\leq\\tfrac\{1\}\{2\}on the full gradient; a capability ceilingqθ≤qmaxq\_\{\\theta\}\\leq q\_\{\\max\}with ceiling\-\(B1\)\(D¯max​\(c⋆\):=minwindow​∫er​\(−gmax\)\>0\\underline\{D\}^\{\\max\}\(c\_\{\\star\}\):=\\min\_\{\\mathrm\{window\}\}\\int e^\{r\}\(\-g\_\{\\max\}\)\>0forgmax:=p​\[\(1\+λ\)​qmax−λ\]g\_\{\\max\}:=p\[\(1\+\\lambda\)q\_\{\\max\}\-\\lambda\]\); and the sign, tilt, and small\-β\\betainputs of Theorem 1\. Then the envelope and theO​\(1/t\)O\(1/t\)upper bound survive verbatim, and the certified lower bound weakens by one logarithm,L=Ω​\(1/\(t​log3⁡t\)\)L=\\Omega\(1/\(t\\log^\{3\}t\)\), with constants carryingCKC\_\{K\}andK¯0\\overline\{K\}\_\{0\}; the window hypothesissupx,τ≤c⋆​ttiltKans≤K¯<∞\\sup\_\{x,\\tau\\leq c\_\{\\star\}t\_\{\\mathrm\{tilt\}\}\}K\_\{\\mathrm\{ans\}\}\\leq\\overline\{K\}<\\infty\(automatic in the Bernoulli content model\) restoreslog2⁡t\\log^\{2\}t\. Without a ceiling, collapse persists whenever the race inequality

ΔB1\>\[\(1\+λ\)​Cq​\(T\)\+β​C∇K​\(T\)\]​∫0TP​\(τ\)​𝑑τ\\Delta\_\{\\mathrm\{B1\}\}\\;\>\\;\\big\[\(1\+\\lambda\)\\,C\_\{q\}\(T\)\+\\beta\\,C\_\{\\nabla K\}\(T\)\\big\]\\int\_\{0\}^\{T\}\\\!P\(\\tau\)\\,d\\tauholds atT=ZLT=Z\_\{L\}, whereΔB1\\Delta\_\{\\mathrm\{B1\}\}is the\(B1\)\-breaking parameter distance andCq,C∇KC\_\{q\},C\_\{\\nabla K\}splitCKC\_\{K\}by force\. The criterion is a*logged\-data certificate*, stated in the run’s own parameterization and norm \(the constants are not reparameterization\-invariant\): it certifies a given run, not a function\-space property\.

###### Proof\.

*\(Part I\)*The identity is the chain rule for KL over the partition\{A,Ac\}\\\{A,A^\{c\}\\\}, plus∂v\[σ​\(v\)​Kans\+\(1−σ​\(v\)\)​Kabs\]=σ′​\(v\)​\(Kans−Kabs\)\\partial\_\{v\}\[\\sigma\(v\)K\_\{\\mathrm\{ans\}\}\+\(1\-\\sigma\(v\)\)K\_\{\\mathrm\{abs\}\}\]=\\sigma^\{\\prime\}\(v\)\(K\_\{\\mathrm\{ans\}\}\-K\_\{\\mathrm\{abs\}\}\), the conditionals beingvv\-independent in a mixture parameterization\. At the all\-abstain state with refusals at base,πans=0\\pi\_\{\\mathrm\{ans\}\}=0andKabs=0K\_\{\\mathrm\{abs\}\}=0, soKLseq​\(x\)=log⁡11−σ​\(v0​\(x\)\)\\mathrm\{KL\}\_\{\\mathrm\{seq\}\}\(x\)=\\log\\frac\{1\}\{1\-\\sigma\(v\_\{0\}\(x\)\)\}pointwise\.

*\(Part II\)*Under separability,∇ψabs\\nabla\_\{\\psi\_\{\\mathrm\{abs\}\}\}of the reward and ofKansK\_\{\\mathrm\{ans\}\}vanish identically, soψ˙abs=−β​𝔼x​\[πabs​∇ψabsKabs\]\\dot\{\\psi\}\_\{\\mathrm\{abs\}\}=\-\\beta\\mathbb\{E\}\_\{x\}\[\\pi\_\{\\mathrm\{abs\}\}\\nabla\_\{\\psi\_\{\\mathrm\{abs\}\}\}K\_\{\\mathrm\{abs\}\}\], zero at base where everyKabs​\(x\)K\_\{\\mathrm\{abs\}\}\(x\)is minimized:ψabs\\psi\_\{\\mathrm\{abs\}\}is invariant\. On the content channels theπans→0\\pi\_\{\\mathrm\{ans\}\}\\to 0limit follows by dominated convergence \(integrands bounded byCKC\_\{K\},sup\|m′\|\\sup\|m^\{\\prime\}\|\)\. The gate channel carries the*value*term−β​σ′​\(v\)​Kans\-\\beta\\sigma^\{\\prime\}\(v\)K\_\{\\mathrm\{ans\}\}, whichCKC\_\{K\}does not control along an unbounded trajectory; the flow structure does:ψ˙ans=𝔼x​\[πans​\(\(1\+λ\)​∂ψq−β​∇ψKans\)\]\\dot\{\\psi\}\_\{\\mathrm\{ans\}\}=\\mathbb\{E\}\_\{x\}\[\\pi\_\{\\mathrm\{ans\}\}\(\(1\+\\lambda\)\\partial\_\{\\psi\}q\-\\beta\\nabla\_\{\\psi\}K\_\{\\mathrm\{ans\}\}\)\]gives‖ψ˙ans‖≤\(1\+λ\+β\)​CK​P​\(t\)\\\|\\dot\{\\psi\}\_\{\\mathrm\{ans\}\}\\\|\\leq\(1\+\\lambda\+\\beta\)C\_\{K\}P\(t\), whence the displayed path bound, and𝔼x​\[σ′​\(v\)​Kans\]≤P​\(t\)​supxKans​\(x,t\)\\mathbb\{E\}\_\{x\}\[\\sigma^\{\\prime\}\(v\)K\_\{\\mathrm\{ans\}\}\]\\leq P\(t\)\\sup\_\{x\}K\_\{\\mathrm\{ans\}\}\(x,t\), which on any window withP=Oc⋆​\(1/t\)P=O\_\{c\_\{\\star\}\}\(1/t\)isO​\(log⁡t/t\)→0O\(\\log t/t\)\\to 0;𝔼x​\[πans​Kans\]\\mathbb\{E\}\_\{x\}\[\\pi\_\{\\mathrm\{ans\}\}K\_\{\\mathrm\{ans\}\}\]obeys the same bound, so the collapse path’s sequence\-anchor cost remains the gate’sKL¯⟂\\overline\{\\mathrm\{KL\}\}\_\{\\perp\}alone\. \(Measured, on the trainable\-content run:K¯0=5\.6×10−4\\overline\{K\}\_\{0\}=5\.6\\times 10^\{\-4\};supxKans​\(x,ZL\)=1\.6×10−3\\sup\_\{x\}K\_\{\\mathrm\{ans\}\}\(x,Z\_\{L\}\)=1\.6\\times 10^\{\-3\}against2\.3×10−32\.3\\times 10^\{\-3\}from the measured displacement and5\.4×10−35\.4\\times 10^\{\-3\}fully certified; the gate value term peaks at7\.5×10−57\.5\\times 10^\{\-5\}and is3\.3×10−83\.3\\times 10^\{\-8\}atZLZ\_\{L\}\.\) With sharedψ\\psithe first step fails:∇ψKabs≠0\\nabla\_\{\\psi\}K\_\{\\mathrm\{abs\}\}\\neq 0once answer\-side updates have movedψ\\psi\.

*\(Part III\)*Pointwiseqθ≤qmaxq\_\{\\theta\}\\leq q\_\{\\max\}withmmincreasing givesDeff​\(qθ\)≥D¯max​\(c⋆\)D\_\{\\mathrm\{eff\}\}\(q\_\{\\theta\}\)\\geq\\underline\{D\}^\{\\max\}\(c\_\{\\star\}\)on the window, andmeff≤m​\(qmax\)m\_\{\\mathrm\{eff\}\}\\leq m\(q\_\{\\max\}\)\(−β​Kans≤0\-\\beta K\_\{\\mathrm\{ans\}\}\\leq 0dropped,Kabs≡0K\_\{\\mathrm\{abs\}\}\\equiv 0\), so the bias envelope,σ¯\\bar\{\\sigma\}control, and gain growth run as in Section[D](https://arxiv.org/html/2608.00301#A4)withD¯max\\underline\{D\}^\{\\max\}\-type constants\.*Envelope:*underρfull≤12\\rho\_\{\\mathrm\{full\}\}\\leq\\tfrac\{1\}\{2\}\(the hypothesis must sit on the full gradient because the content channels contribute their own cross termβ​∂ψJ⋅∂ψKLseq\\beta\\,\\partial\_\{\\psi\}J\\cdot\\partial\_\{\\psi\}\\mathrm\{KL\}\_\{\\mathrm\{seq\}\}, not sign\-definite\),L˙≤−12​\(1−σ¯\)2​L2\\dot\{L\}\\leq\-\\tfrac\{1\}\{2\}\(1\-\\bar\{\\sigma\}\)^\{2\}L^\{2\}, the identity∂c0J=−\(1−σ¯\)​L\\partial\_\{c\_\{0\}\}J=\-\(1\-\\bar\{\\sigma\}\)Lholding at each instant for the currentgg\.*Lower bound:*pastT0T\_\{0\},P=L\|m¯tilt\|⋅1\+εP1\+εL≤2​LD¯max​\(c⋆\)P=\\frac\{L\}\{\|\\bar\{m\}\_\{\\mathrm\{tilt\}\}\|\}\\cdot\\frac\{1\+\\varepsilon\_\{P\}\}\{1\+\\varepsilon\_\{L\}\}\\leq\\frac\{2L\}\{\\underline\{D\}^\{\\max\}\(c\_\{\\star\}\)\}, so\|∂ψJ\|≤\(1\+λ\)​CK​P=Oc⋆​\(L\)\|\\partial\_\{\\psi\}J\|\\leq\(1\+\\lambda\)C\_\{K\}P=O\_\{c\_\{\\star\}\}\(L\)and the anchor’s content component isOc⋆​\(β​L\)O\_\{c\_\{\\star\}\}\(\\beta L\)\. The anchor’s gate coordinates carry, beyondσ′​\(v\)​\(v−v0\)\\sigma^\{\\prime\}\(v\)\(v\-v\_\{0\}\), the value termσ′​\(v\)​\(Kans−Kabs\)\\sigma^\{\\prime\}\(v\)\(K\_\{\\mathrm\{ans\}\}\-K\_\{\\mathrm\{abs\}\}\): onc0,c4c\_\{0\},c\_\{4\}\(\|∂v\|≤1\|\\partial v\|\\leq 1\) it isOc⋆​\(β​L​log⁡t\)O\_\{c\_\{\\star\}\}\(\\beta L\\log t\)by the path bound; onw,aw,a,∂wv=c4​ϕ′​\(w​s\+a\)​s\\partial\_\{w\}v=c\_\{4\}\\phi^\{\\prime\}\(ws\+a\)scarries the extra\|c4\|\|c\_\{4\}\|, givingOc⋆​\(β​L​log2⁡t\)O\_\{c\_\{\\star\}\}\(\\beta L\\log^\{2\}t\); these coordinates arec4​ϕ′c\_\{4\}\\phi^\{\\prime\}\-weighted, notϕ\\phi\-weighted, so no a fortiori argument applies\. Hence‖∇J‖=Oc⋆​\(L​log⁡t\)\\\|\\nabla J\\\|=O\_\{c\_\{\\star\}\}\(L\\log t\)butβ​‖∇KLseq‖=Oc⋆​\(L​log2⁡t\)\\beta\\\|\\nabla\\mathrm\{KL\}\_\{\\mathrm\{seq\}\}\\\|=O\_\{c\_\{\\star\}\}\(L\\log^\{2\}t\), so\|L˙\|=Oc⋆​\(L2​log3⁡t\)\|\\dot\{L\}\|=O\_\{c\_\{\\star\}\}\(L^\{2\}\\log^\{3\}t\)andL=Ω​\(1/\(t​log3⁡t\)\)L=\\Omega\(1/\(t\\log^\{3\}t\)\); boundedKansK\_\{\\mathrm\{ans\}\}on the window makes every value termO​\(\|c4\|​L\)O\(\|c\_\{4\}\|L\)and restoreslog2⁡t\\log^\{2\}t\. The total content signal is∫\|∂ψJ\|=Oc⋆​\(log⁡ttilt\)\\int\|\\partial\_\{\\psi\}J\|=O\_\{c\_\{\\star\}\}\(\\log t\_\{\\mathrm\{tilt\}\}\), which yields the race criterion:‖ψ˙ans‖≤\[\(1\+λ\)​Cq\+β​C∇K\]​P​\(t\)\\\|\\dot\{\\psi\}\_\{\\mathrm\{ans\}\}\\\|\\leq\[\(1\+\\lambda\)C\_\{q\}\+\\beta C\_\{\\nabla K\}\]P\(t\)pointwise, so if the displayed inequality holds atT=ZLT=Z\_\{L\}, \(B1\) holds on\[0,ZL\]\[0,Z\_\{L\}\]and the collapse completes first\. ∎

#### Numerical instantiation\.

Simulated with a trainable correctness shiftqu=q0\+\(qmax−q0\)​σ​\(u\)q\_\{u\}=q\_\{0\}\+\(q\_\{\\max\}\-q\_\{0\}\)\\sigma\(u\)and minimal Bernoulli content KL: under the ceiling \(𝔼​\[qmax\]=0\.65<t∗\\mathbb\{E\}\[q\_\{\\max\}\]=0\.65<t^\{\\ast\}\) the collapse law is unchanged, withL​t=0\.71,0\.51Lt=0\.71,\\,0\.51att=102,103t=10^\{2\},10^\{3\}against the frozen\-content0\.71,0\.510\.71,\\,0\.51, and content moves𝔼​\[q\]\\mathbb\{E\}\[q\]by10−410^\{\-4\}during the plateau; without any ceiling \(qmax=0\.95q\_\{\\max\}=0\.95, \(B1\) breakable\) collapse still wins: \(B1\) first fails att=2\.9×104t=2\.9\\times 10^\{4\}, a factor2\.82\.8*after*ZL=1\.0×104Z\_\{L\}=1\.0\\times 10^\{4\}, the model having moved𝔼​\[q\]\\mathbb\{E\}\[q\]by0\.0050\.005of the needed0\.270\.27\. Race certificate on that run:∫0ZLP=12\.4\\int\_\{0\}^\{Z\_\{L\}\}P=12\.4; window constantsCq=0\.023C\_\{q\}=0\.023,C∇K=0\.0031C\_\{\\nabla K\}=0\.0031give right side1\.541\.54againstΔB1=4\.71\\Delta\_\{\\mathrm\{B1\}\}=4\.71\(actual movement0\.560\.56\), so the run is certified; the cruder global constants give12\.812\.8and fail to certify despite the conclusion holding: the certificate’s sharpness depends on the Lipschitz constants supplied\.*The plateau starves the very channel that could avert it\.*

## Appendix GEstimator Analysis in Full

#### Derivation of the drift menu \(Proposition 6\)\.

GRPO formsA^i=\(ri−μ^r\)/σ^r\\hat\{A\}\_\{i\}=\(r\_\{i\}\-\\hat\{\\mu\}\_\{r\}\)/\\hat\{\\sigma\}\_\{r\}within groups ofGG; since∑iA^i=0\\sum\_\{i\}\\hat\{A\}\_\{i\}=0, the per\-prompt expected gate drift is𝔼​\[S\]/G\\mathbb\{E\}\[S\]/GwithS=∑answeringA^iS=\\sum\_\{\\mathrm\{answering\}\}\\hat\{A\}\_\{i\}, which we evaluate by exact enumeration over group outcomes\(k,j\)\(k,j\)\(kkanswers,jjcorrect among them, probabilities\(Gk\)​pk​\(1−p\)G−k​\(kj\)​q¯j​\(1−q¯\)k−j\\binom\{G\}\{k\}p^\{k\}\(1\-p\)^\{G\-k\}\\binom\{k\}\{j\}\\bar\{q\}^\{j\}\(1\-\\bar\{q\}\)^\{k\-j\}, population\-variance normalizer with denominatorGGandϵ=0\\epsilon=0; degenerate groupsA^≡0\\hat\{A\}\\equiv 0\)\. \(a\) For vanilla PG and mean\-baseline estimators the drift isΘ​\(p​m\)\\Theta\(p\\,m\)at everypp\(mean\-baseline: exactlyp​m​\(1−p\)​\(G−1\)/Gp\\,m\\,\(1\-p\)\(G\-1\)/G, which the enumeration reproduces to10−1510^\{\-15\}\)\. \(b\) For1/G≪p≪11/G\\ll p\\ll 1, the sample statistics concentrate:μ^r≈p​m\\hat\{\\mu\}\_\{r\}\\approx pm,σ^r≈p​M2\\hat\{\\sigma\}\_\{r\}\\approx\\sqrt\{pM\_\{2\}\}withM2=q¯\+\(1−q¯\)​λ2M\_\{2\}=\\bar\{q\}\+\(1\-\\bar\{q\}\)\\lambda^\{2\}, so drift≈m​p/M2\\approx m\\sqrt\{p/M\_\{2\}\}; the enumerated log\-log slopes approach0\.50\.5from below asGGgrows \(0\.451/0\.468/0\.4790\.451/0\.468/0\.479atG=64/256/1024G=64/256/1024\), the deficit being theO​\(1/G\)O\(1/G\)statistic\-correlation bias\. \(c\) Forp≪1/Gp\\ll 1/G, a group has at most one answer with probability≈G​p\\approx Gp; for the single\-answer groupr=\(r,0,…,0\)r=\(r,0,\\dots,0\):μ^=r/G\\hat\{\\mu\}=r/G,σ^2=r2​\(G−1\)/G2\\hat\{\\sigma\}^\{2\}=r^\{2\}\(G\-1\)/G^\{2\}, soA^1=sign⁡\(r\)​G−1\\hat\{A\}\_\{1\}=\\operatorname\{sign\}\(r\)\\sqrt\{G\-1\}*independently of\|r\|\|r\|*\. Hence drift→p​\(2​q¯−1\)​G−1\\to p\\,\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}: the penalty magnitude is erased,λeff=1\\lambda\_\{\\mathrm\{eff\}\}=1\. Enumerated drift/p/pmatches\(2​q¯−1\)​G−1\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}to three decimals for everyλ∈\{1,2,4\.5,9\}\\lambda\\in\\\{1,2,4\.5,9\\\}\.

#### Rate saturation inλ\\lambda\(above the knee\)\.

m/M2→−1−q¯m/\\sqrt\{M\_\{2\}\}\\to\-\\sqrt\{1\-\\bar\{q\}\}asλ→∞\\lambda\\to\\infty: enumeration atq¯=0\.3\\bar\{q\}=0\.3,G=256G=256,p=0\.05p=0\.05gives drift/p=−0\.378,−0\.582,−0\.705,−0\.751,−0\.788,−0\.793/\\sqrt\{p\}=\-0\.378,\-0\.582,\-0\.705,\-0\.751,\-0\.788,\-0\.793forλ=1,2,4\.5,9,50,200\\lambda=1,2,4\.5,9,50,200\(limit−0\.837\-0\.837; Figure[7](https://arxiv.org/html/2608.00301#A7.F7)\), while vanilla drift/p=m/p=mgrows linearly from−0\.40\-0\.40to−34\.7\-34\.7\. The sign is preserved \(sign⁡\(m/M2\)=sign⁡m\\operatorname\{sign\}\(m/\\sqrt\{M\_\{2\}\}\)=\\operatorname\{sign\}m\): regime \(b\) distorts the rate, not the threshold\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x7.png)Figure 7:Rate saturation above the knee: the population\-normalized amplitude drift/p/\\sqrt\{p\}\(enumerated atp=0\.05p=0\.05,G=256G=256\) saturates at−1−q¯\-\\sqrt\{1\-\\bar\{q\}\}\(dotted\) as the nominalλ\\lambdagrows\. Past the knee, making the penalty harsher stops making the collapse faster\.
#### Threshold shift and the1/G1/Gfixed point \(below the knee\)\.

The regime\-\(c\) drift∝\(2​q¯−1\)\\propto\(2\\bar\{q\}\-1\)changes sign atq¯=1/2\\bar\{q\}=1/2, not att∗t^\{\\ast\}\(Figure[8](https://arxiv.org/html/2608.00301#A7.F8)\): the optimizer answers the entire bandq¯∈\(1/2,t∗\)\\bar\{q\}\\in\(1/2,t^\{\\ast\}\)on which the rule assigns negative gain \(atλ=4\.5\\lambda=4\.5:\(0\.5,0\.818\)\(0\.5,0\.818\)\)\. On that band regimes \(b\) and \(c\) push in opposite directions, so the drift crosses zero from above at a stablep∗=Θ​\(1/G\)p^\{\\ast\}=\\Theta\(1/G\)\(constant increasing inq¯\\bar\{q\}\); a tabular policy under GRPO therefore lands on a three\-tier terminal profile \(0forq¯<1/2\\bar\{q\}<1/2;Θ​\(1/G\)\\Theta\(1/G\)on the band;11abovet∗t^\{\\ast\}\):*the abstention rate on the contested band is set by the group size, not by the penalty\.*

#### Proof of Proposition 7 \(conservation\), and its scope\.

LetDDbe the degenerate event \(constant group reward\)\. Tautologically𝔼​\[g^\]=𝔼​\[g^​𝟏​\{Dc\}\]=Pr⁡\[Dc\]​𝔼​\[g^∣Dc\]\\mathbb\{E\}\[\\hat\{g\}\]=\\mathbb\{E\}\[\\hat\{g\}\\mathbf\{1\}\\\{D^\{c\}\\\}\]=\\Pr\[D^\{c\}\]\\,\\mathbb\{E\}\[\\hat\{g\}\\mid D^\{c\}\]\. GRPO spendsGGrollouts per step and moves by𝔼​\[g^\]\\mathbb\{E\}\[\\hat\{g\}\]; the resampling filter spendsG/Pr⁡\[Dc\]G/\\Pr\[D^\{c\}\]rollouts per step and moves by𝔼​\[g^∣Dc\]\\mathbb\{E\}\[\\hat\{g\}\\mid D^\{c\}\]: the expected*reward*drift per rollout is𝔼​\[g^\]/G\\mathbb\{E\}\[\\hat\{g\}\]/Gfor both, for everyp,G,q¯,λp,G,\\bar\{q\},\\lambdaand every prompt distribution, with the acceptance weight cancelling the per\-group rollout cost prompt by prompt\. Hence the reward\-only \(β=0\\beta=0\) mean\-field expected trajectories coincide pointwise in rollout countNN\(confirmed to six significant figures:P=2\.007716×10−3P=2\.007716\\times 10^\{\-3\}atN=104N=10^\{4\}under both\)\. The anchor does*not*cancel: it is applied once per optimizer step, and the filter takesPr⁡\[Dc\]≈G​p\\Pr\[D^\{c\}\]\\approx Gpas many steps per rollout, diluting the anchor’s restoring force by that factor \(atG=8G=8,p=10−3p=10^\{\-3\}:125×125\\times\)\. Scope: the statement covers filters that discard*zero\-gradient*groups; filters that discard groups with nonzero gradient \(variance thresholds, curricula\) change the drift and are outside it, as are clip\-higher and token\-level losses \(objective modifications\)\. Per optimizer step the sparse\-regime law becomes exponential,c˙0→\(2​q¯−1\)​G−1/G\\dot\{c\}\_\{0\}\\to\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}/G, i\.e\.L=Θ​\(e−ω​t\)L=\\Theta\(e^\{\-\\omega t\}\),ω≍\|2​q¯−1\|/G\\omega\\asymp\|2\\bar\{q\}\-1\|/\\sqrt\{G\}; per rollout, nothing changes \(Figure[9](https://arxiv.org/html/2608.00301#A7.F9)\)\. ∎

###### Proposition G\.1\(The anchor under a sampling filter: a barrier, not a floor\)\.

In the homogeneous scalar model \(one effective prompt, logitc0c\_\{0\},p=σ​\(c0\)≈ec0p=\\sigma\(c\_\{0\}\)\\approx e^\{c\_\{0\}\}in the sparse regime, knee atc0knee=−log⁡Gc\_\{0\}^\{\\mathrm\{knee\}\}=\-\\log G\), the filtered flow is

c˙0=−kG\+β​ec0​\(v0−c0\),k=\|2​q¯−1\|​G−1\.\\dot\{c\}\_\{0\}=\-\\frac\{k\}\{G\}\+\\beta\\,e^\{c\_\{0\}\}\(v\_\{0\}\-c\_\{0\}\),\\qquad k=\|2\\bar\{q\}\-1\|\\sqrt\{G\-1\}\.Assumev0\+log⁡G\>1v\_\{0\}\+\\log G\>1\(equivalentlyc0knee<v0−1c\_\{0\}^\{\\mathrm\{knee\}\}<v\_\{0\}\-1\), so the anchor term is strictly increasing on the sparse regime and largest at the knee\. Comparing anchor and drive there gives

βcritfilter=kv0\+log⁡G,\\beta^\{\\mathrm\{filter\}\}\_\{\\mathrm\{crit\}\}=\\frac\{k\}\{\\,v\_\{0\}\+\\log G\\,\},with an all\-or\-nothing structure \(every sparse\-regime root is repelling sinced​c˙0/d​c0\>0\\mathrm\{d\}\\dot\{c\}\_\{0\}/\\mathrm\{d\}c\_\{0\}\>0there\): forβ\>βcritfilter\\beta\>\\beta^\{\\mathrm\{filter\}\}\_\{\\mathrm\{crit\}\}a trajectory descending from above meetsc˙0\>0\\dot\{c\}\_\{0\}\>0at the knee and, provided the above\-knee field also points down at the knee \(a two\-sided sign condition we check by enumeration rather than prove\), stalls atp≈1/Gp\\approx 1/G, an*observable*answer rate; forβ<βcritfilter\\beta<\\beta^\{\\mathrm\{filter\}\}\_\{\\mathrm\{crit\}\}it passes the knee and no stationary point exists thereafter\. Numericallyβcritfilter=0\.259\\beta^\{\\mathrm\{filter\}\}\_\{\\mathrm\{crit\}\}=0\.259atG=8G=8and0\.5160\.516atG=64G=64\(q¯=0\.3\\bar\{q\}=0\.3,v0=2v\_\{0\}=2; the ratio1\.991\.99matches theG/log⁡G\\sqrt\{G\}/\\log Gscaling\); realisticβ≤10−2\\beta\\leq 10^\{\-2\}sit1\.41\.4–1\.71\.7orders below, so collapse proceeds\. The anchor is not weakened into a lower floor; it is converted into a switch\.

#### Zero gradients are exact, not small\.

When every sample in a group abstains, allGGrewards equal0,σ^r=0\\hat\{\\sigma\}\_\{r\}=0, andA^i≡0\\hat\{A\}\_\{i\}\\equiv 0: the prompt’s contribution is identically zero\. The fraction of prompts in this state is\(1−p\)G\(1\-p\)^\{G\}: atp=10−3p=10^\{\-3\},99\.2%99\.2\\%forG=8G=8and77\.4%77\.4\\%forG=256G=256\.

#### From drift to decay law\.

WithL≍ec0​DeffL\\asymp e^\{c\_\{0\}\}D\_\{\\mathrm\{eff\}\}andDeffD\_\{\\mathrm\{eff\}\}slowly varying,L˙/L=c˙0​\(1\+o​\(1\)\)\\dot\{L\}/L=\\dot\{c\}\_\{0\}\(1\+o\(1\)\), so the menu’s drift scalings convert to laws in one line each:

c˙0∝−ec0\\displaystyle\\dot\{c\}\_\{0\}\\propto\-e^\{c\_\{0\}\}⇒L∼t−1;c˙0∝−ec0/2⇒L∼t−2;\\displaystyle\\Rightarrow L\\sim t^\{\-1\};\\qquad\\dot\{c\}\_\{0\}\\propto\-e^\{c\_\{0\}/2\}\\Rightarrow L\\sim t^\{\-2\};c˙0→−c\\displaystyle\\dot\{c\}\_\{0\}\\to\-c\\⇒L∼e−c​t\.\\displaystyle\\Rightarrow L\\sim e^\{\-ct\}\.\(GRPO is not a gradient flow onJJ, so Theorem 1’s envelope does not apply to it; the transfer above is what does\.\)

#### Two practical notes on dynamic sampling\.

\(i\) The published method also removes the KL term entirely\(Yuet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib24)\), so the anchor\-dilution channel is moot in practice; the conservation of the reward drift is the operative content\. \(ii\) The resampling budget is capped in practice: onceG​pGpfalls below the cap the batch cannot be filled and training stalls, so the real curve tracks the unfiltered estimator on the rollout axis and then halts\.

#### Group normalization and the report channel\.

The report\-level mechanism survives group\-based estimation for a reason worth separating from determinism: within a prompt allGGsamples answer, so normalization rescales a strictly proper score by a prompt\-level statistic\. For*fixed*scale this preserves properness\(Gneiting and Raftery[2007](https://arxiv.org/html/2608.00301#bib.bib49)\); butμ^r,σ^r\\hat\{\\mu\}\_\{r\},\\hat\{\\sigma\}\_\{r\}are sample statistics correlated withrir\_\{i\}\(exactly the bias identified byLiuet al\.\([2025](https://arxiv.org/html/2608.00301#bib.bib22)\)\), so under group\-standard\-deviation normalization properness is preserved only up to anO​\(1/G\)O\(1/G\)bias, and*exactly*under mean\-only baselines \(RLOO, Dr\. GRPO\)\. This gives the design lesson a second leg:*move abstention to the report level, and drop the standard\-deviation normalization\.*Action\-level rules enjoy no such invariance at all, because the abstain baseline0and the penalty−λ\-\\lambdaare coupled through the action distribution, which is the content of this section\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x8.png)Figure 8:The fixed point that group normalization creates\. \(a\) Forq¯\\bar\{q\}inside the band\(1/2,t∗\)\(1/2,t^\{\\ast\}\)the GRPO drift is negative abovep≈1/Gp\\approx 1/Gand positive below it: a stable interior fixed point \(circles\); forq¯\>t∗\\bar\{q\}\>t^\{\\ast\}the drift is positive everywhere\. \(b\) Terminal profile of a tabular policy under GRPO: three tiers, with the answering threshold atq¯=1/2\\bar\{q\}=1/2rather than the rule’st∗=0\.818t^\{\\ast\}=0\.818; the shaded band is where the rule assigns negative gain and the optimizer answers anyway\. \(c\) The fixed point scales asΘ​\(1/G\)\\Theta\(1/G\)\.![Refer to caption](https://arxiv.org/html/2608.00301v1/x9.png)Figure 9:Conservation of collapse per rollout\. \(a\) Against optimizer steps the two estimators obey different laws: unfiltered on log–log, filtered on semi\-log \(inset\)\. \(b\) Against rollouts consumed, the two curves coincide with each other and withG/\(k​N\)G/\(kN\)to six significant figures \(reward\-only,β=0\\beta=0; with the anchor on, the fields differ by thePr⁡\[Dc\]\\Pr\[D^\{c\}\]\-diluted restoring term\)\. The paired test \(separation in steps, coincidence in samples\) requires no curve fitting, and no plausible confound produces both\.

## Appendix HThe Report\-Level Mechanism in Full

#### Boundary versus interior\.

Under action\-level abstention two information channels die together at the vertex: the action channel’s per\-episode Fisher information about the gate logit isIv=𝔼p​\[σ′​\(v\)\]=Θ​\(P​\(answer\)\)→0I\_\{v\}=\\mathbb\{E\}\_\{p\}\[\\sigma^\{\\prime\}\(v\)\]=\\Theta\(P\(\\mathrm\{answer\}\)\)\\to 0, and the correctness channel is throttled byP​\(answer\)P\(\\mathrm\{answer\}\)because the abstain branch returns a constant \(for the feature parameters\(w,a\)\(w,a\)the Fisher matrix carries additional Jacobian factors, so the honest statement isI​\(w,a\)=O​\(P​\(answer\)\)I\(w,a\)=O\(P\(\\mathrm\{answer\}\)\)absent a nondegeneracy floor\)\. Under report\-level scoring every episode returns aqq\-correlated score, and the driving term∂wB=−2​𝔼p​\[\(c−q¯\)​\(κ1−κ0\)​σ′​\(w​s\+a\)​s\]\\partial\_\{w\}B=\-2\\mathbb\{E\}\_\{p\}\[\(c\-\\bar\{q\}\)\(\\kappa\_\{1\}\-\\kappa\_\{0\}\)\\sigma^\{\\prime\}\(ws\+a\)s\]loses its discriminative factor only if the reported range collapses \(κ1→κ0\\kappa\_\{1\}\\to\\kappa\_\{0\}\), an*interior*point of parameter space where the negative\-definite Hessian below supplies restoring force, versus the*boundary*pointP→0P\\to 0where, by Corollary 1, every policy\-space restoring force in its class vanishes\. Boundary versus interior is the exact content of “kills\.” \(The two sides are analogous, not identical, objects: a per\-episode Fisher information about the gate logit on one side, the curvature of the population objective in the report’s range parameters on the other; a unified comparison would fix one observation likelihood for both mechanisms, which we have not done\. The invariant content is the boundary\-versus\-interior distinction; Figure[10](https://arxiv.org/html/2608.00301#A8.F10)a shows the two quantities falling as one curve\.\)

![Refer to caption](https://arxiv.org/html/2608.00301v1/x10.png)Figure 10:\(a\) Boundary versus interior: the action\-level rule’s per\-episode information𝔼p​\[σ′​\(v\)\]\\mathbb\{E\}\_\{p\}\[\\sigma^\{\\prime\}\(v\)\]tracksP​\(answer\)P\(\\mathrm\{answer\}\)to zero, while the report\-level objective’s curvature𝔼p​\[ϕ2\+\(1−ϕ\)2\]\\mathbb\{E\}\_\{p\}\[\\phi^\{2\}\+\(1\-\\phi\)^\{2\}\]is bounded below; its degenerate direction is interior, where restoring force is strictly positive\. \(b\) Two escapes: the sampled rate turns up atttiltt\_\{\\mathrm\{tilt\}\}, whereDeffD\_\{\\mathrm\{eff\}\}\(gray, right axis\) crosses zero; greedy coverage recovers strictly later; the report\-level mechanism never leaves near\-rational coverage\.
#### Proof of Proposition 8 \(report equilibrium; anchors\)\.

𝔼​\[\(Y−c\)2∣s\]=\(c−q¯\)2\+q¯​\(1−q¯\)\\mathbb\{E\}\[\(Y\-c\)^\{2\}\\mid s\]=\(c\-\\bar\{q\}\)^\{2\}\+\\bar\{q\}\(1\-\\bar\{q\}\), soB​\(θ′\)=const−𝔼p​\[\(cθ′−q¯\)2\]B\(\\theta^\{\\prime\}\)=\\mathrm\{const\}\-\\mathbb\{E\}\_\{p\}\[\(c\_\{\\theta^\{\\prime\}\}\-\\bar\{q\}\)^\{2\}\], with∂κ0B=−2​𝔼p​\[\(c−q¯\)​\(1−ϕ\)\]\\partial\_\{\\kappa\_\{0\}\}B=\-2\\mathbb\{E\}\_\{p\}\[\(c\-\\bar\{q\}\)\(1\-\\phi\)\],∂κ1B=−2​𝔼p​\[\(c−q¯\)​ϕ\]\\partial\_\{\\kappa\_\{1\}\}B=\-2\\mathbb\{E\}\_\{p\}\[\(c\-\\bar\{q\}\)\\phi\]: no action\-saturation factor\. At the realizing parameter \(which exists in the working model:q¯​\(s\)=qL\+\(qH−qL\)​σ​\(2​μ​s\)\\bar\{q\}\(s\)=q\_\{L\}\+\(q\_\{H\}\-q\_\{L\}\)\\sigma\(2\\mu s\), realized by\(w,a,κ0,κ1\)=\(2​μ,0,qL,qH\)\(w,a,\\kappa\_\{0\},\\kappa\_\{1\}\)=\(2\\mu,0,q\_\{L\},q\_\{H\}\)\) the Hessian is−2​𝔼​\[∇c​∇c⊤\]≺0\-2\\mathbb\{E\}\[\\nabla c\\,\\nabla c^\{\\top\}\]\\prec 0\(components of∇c\\nabla clinearly independent inL2​\(p\)L^\{2\}\(p\)\), and the implicit function theorem applied to∇B​\(θ′\)−β​\(θ′−θ0′\)=0\\nabla B\(\\theta^\{\\prime\}\)\-\\beta\(\\theta^\{\\prime\}\-\\theta^\{\\prime\}\_\{0\}\)=0yields an attracting anchored equilibrium withinO​\(β\)O\(\\beta\); sinceq¯\\bar\{q\}is strictly increasing withq¯′​\(s∗\)\>0\\bar\{q\}^\{\\prime\}\(s^\{\\ast\}\)\>0andg​\(s∗\)=0g\(s^\{\\ast\}\)=0, anO​\(β\)O\(\\beta\)perturbation ofccmoves the deployment threshold byO​\(β\)O\(\\beta\)and costsO​\(β2\)O\(\\beta^\{2\}\)utility \(the operative condition is the margin boundPrp⁡\(\|q¯−t∗\|≤ϵ\)=O​\(ϵ\)\\Pr\_\{p\}\(\|\\bar\{q\}\-t^\{\\ast\}\|\\leq\\epsilon\)=O\(\\epsilon\), implied here by the single transversal crossing,q¯′​\(s∗\)\>0\\bar\{q\}^\{\\prime\}\(s^\{\\ast\}\)\>0, withppbounded near it: the anchored solutions above satisfy‖c−q¯‖∞=O​\(β\)\\\|c\-\\bar\{q\}\\\|\_\{\\infty\}=O\(\\beta\)uniformly, so disagreement requires\|q¯−t∗\|=O​\(β\)\|\\bar\{q\}\-t^\{\\ast\}\|=O\(\\beta\), a set of massO​\(β\)O\(\\beta\)on which the per\-point gap\(1\+λ\)​\|q¯−t∗\|\(1\+\\lambda\)\|\\bar\{q\}\-t^\{\\ast\}\|is itselfO​\(β\)O\(\\beta\); without the margin bound, the generic\(1\+λ\)​𝔼p​\|c−q¯\|=O​\(β\)\(1\+\\lambda\)\\mathbb\{E\}\_\{p\}\|c\-\\bar\{q\}\|=O\(\\beta\)stands\)\.*Function\-space anchors\.*\(a\) Quadraticβ2​∫p​\(c−c0\)2\\tfrac\{\\beta\}\{2\}\\\!\\int p\(c\-c\_\{0\}\)^\{2\}: pointwise maximization givescβ=q¯\+β2\+β​\(c0−q¯\)c\_\{\\beta\}=\\bar\{q\}\+\\tfrac\{\\beta\}\{2\+\\beta\}\(c\_\{0\}\-\\bar\{q\}\)\. \(b\) Bernoulli KL between reported confidences:h​\(c\)=2​\(q¯−c\)−β​\(logit​c−logit​c0\)h\(c\)=2\(\\bar\{q\}\-c\)\-\\beta\(\\mathrm\{logit\}\\,c\-\\mathrm\{logit\}\\,c\_\{0\}\)is strictly decreasing withh→±∞h\\to\\pm\\inftyat the endpoints, so a unique rootcβc\_\{\\beta\}strictly betweenq¯\\bar\{q\}andc0c\_\{0\}, and\|cβ−q¯\|≤β2​Λ0\|c\_\{\\beta\}\-\\bar\{q\}\|\\leq\\tfrac\{\\beta\}\{2\}\\Lambda\_\{0\}withΛ0=sups\|logit​q¯−logit​c0\|<∞\\Lambda\_\{0\}=\\sup\_\{s\}\|\\mathrm\{logit\}\\,\\bar\{q\}\-\\mathrm\{logit\}\\,c\_\{0\}\|<\\inftyforq¯\\bar\{q\}bounded away from\{0,1\}\\\{0,1\\\}\(numerically‖cβ−q¯‖∞=1\.08​β\\\|c\_\{\\beta\}\-\\bar\{q\}\\\|\_\{\\infty\}=1\.08\\betaatβ=10−2\\beta=10^\{\-2\}, inside the bound2\.2​β2\.2\\beta\)\. In both cases the sameO​\(β2\)O\(\\beta^\{2\}\)deployment loss under the same margin condition\.*Misspecification\.*Ifq¯\\bar\{q\}is not representable by the head, the population objective is, in the range parameters, a concave quadratic; its maximizer over the box\[0,1\]2\[0,1\]^\{2\}is the constrainedL2​\(p\)L^\{2\}\(p\)projection ofq¯\\bar\{q\}, unique and attracting under the projected flow wheneverϕ\\phiis nonconstant on the support ofpp\(then𝔼​\[∇c​∇c⊤\]≻0\\mathbb\{E\}\[\\nabla c\\,\\nabla c^\{\\top\}\]\\succ 0\), and interior exactly when no box constraint is active; on the deployment disagreement set\{sign⁡\(c−t∗\)≠sign⁡\(q¯−t∗\)\}\\\{\\operatorname\{sign\}\(c\-t^\{\\ast\}\)\\neq\\operatorname\{sign\}\(\\bar\{q\}\-t^\{\\ast\}\)\\\}the threshold lies betweenccandq¯\\bar\{q\}, so\|m​\(q¯\)\|=\(1\+λ\)​\|q¯−t∗\|≤\(1\+λ\)​\|q¯−c\|\|m\(\\bar\{q\}\)\|=\(1\+\\lambda\)\|\\bar\{q\}\-t^\{\\ast\}\|\\leq\(1\+\\lambda\)\|\\bar\{q\}\-c\|pointwise, and the deployed\-utility gap to Chow’s rule is at most\(1\+λ\)​𝔼p​\|c−q¯\|\(1\+\\lambda\)\\,\\mathbb\{E\}\_\{p\}\|c\-\\bar\{q\}\|\. The structural point: at the action level, reward and anchor share the vanishing factorσ′​\(v\)\\sigma^\{\\prime\}\(v\); at the report level they share none: the Brier gradient2​\(q¯−c\)2\(\\bar\{q\}\-c\)isΘ​\(1\)\\Theta\(1\)away from calibration, and the report\-KL derivative*diverges*at the boundary, repelling degenerate reports; the direction of the KL, load\-bearing at the action level, is moot here\. Scope: these are statements about the report*value*; for a sampled confidence token the logit\-space caveat of Section[C](https://arxiv.org/html/2608.00301#A3)applies, and overconfident initialization is the condition that discharges it\. ∎

#### Frozen\-feature training\-properness, with an explicit clock\.

Freeze\(w,a\)\(w,a\)at any values with non\-constantϕ\\phiand flowx=\(κ0,κ1\)x=\(\\kappa\_\{0\},\\kappa\_\{1\}\)underBβ​\(x\)=B​\(x\)−β2​‖x−x0‖2B\_\{\\beta\}\(x\)=B\(x\)\-\\tfrac\{\\beta\}\{2\}\\\|x\-x\_\{0\}\\\|^\{2\}\.BBis a concave quadratic with Hessian−2​M\-2M,M=𝔼p​\[u​u⊤\]≻0M=\\mathbb\{E\}\_\{p\}\[uu^\{\\top\}\]\\succ 0foru=\(1−ϕ,ϕ\)⊤u=\(1\-\\phi,\\phi\)^\{\\top\}, soBβB\_\{\\beta\}is\(2​λmin​\(M\)\+β\)\(2\\lambda\_\{\\min\}\(M\)\+\\beta\)\-strongly concave and the flow converges to its unique maximizer from every initialization inℝ2\\mathbb\{R\}^\{2\}at ratee−\(2​λmin​\(M\)\+β\)​te^\{\-\(2\\lambda\_\{\\min\}\(M\)\+\\beta\)t\}, with‖xβ∗−x∗‖≤β​‖x∗−x0‖/\(2​λmin​\(M\)\)\\\|x^\{\\ast\}\_\{\\beta\}\-x^\{\\ast\}\\\|\\leq\\beta\\\|x^\{\\ast\}\-x\_\{0\}\\\|/\(2\\lambda\_\{\\min\}\(M\)\)\(a projected variant on\[0,1\]2\[0,1\]^\{2\}keepsc∈\[0,1\]c\\in\[0,1\]throughout at the same rate: projection onto a convex set is nonexpansive, so the strong\-concavity contraction is preserved\)\. On the working model2​λmin​\(M\)=0\.552\\lambda\_\{\\min\}\(M\)=0\.55, so the matched overconfident initialization reaches10−310^\{\-3\}parameter error byT≈11T\\approx 11, against the action\-level rule, which the*same*frozen\-feature reduction leaves collapsing with recovery only ateΘ​\(1/β\)e^\{\\Theta\(1/\\beta\)\}\. The mechanism separation is thus*proved*, not simulated, in the frozen\-feature subfamily; the joint four\-parameter basin is nonconvex and remains an empirical statement \(Section[J](https://arxiv.org/html/2608.00301#A10)\)\. ∎

#### Proof of Proposition 9 \(composite objective\)\.

𝔼​\[\(Y−c\)2\]=q−2​q​c\+c2\\mathbb\{E\}\[\(Y\-c\)^\{2\}\]=q\-2qc\+c^\{2\}gives𝔼​\[rα\]=α​q\+1−q\+2​q​c−c2\\mathbb\{E\}\[r\_\{\\alpha\}\]=\\alpha q\+1\-q\+2qc\-c^\{2\}, whence the displayed partials\. \(a\)∂c=2​\(q−c\)\\partial\_\{c\}=2\(q\-c\)drives the report to calibration for everyα\\alpha\. \(b\) Atα=0\\alpha=0the calibrated score is1−q\+q21\-q\+q^\{2\}, U\-shaped with minimum atq=12q=\\tfrac\{1\}\{2\}: the accuracy gradient2​q−12q\-1is negative for everyq<12q<\\tfrac\{1\}\{2\}; andq<12q<\\tfrac\{1\}\{2\}is, up to the value oft∗t^\{\\ast\}, the region \(B1\) selects: the defect of a bare proper score lands on the same questions as the penalty rule’s pathology\. \(c\) On the calibration manifold the accuracy gradient isα−1\+2​q\\alpha\-1\+2q; at the accuracy vertexq=c=0q=c=0it equalsα−1\\alpha\-1: attracting iffα<1\\alpha<1, degenerate atα=1\\alpha=1, repelling iffα\>1\\alpha\>1\. This is the same local vertex test as Proposition 3, applied in theqqcoordinate to our own proposal, which is why we requireα\>1\\alpha\>1strictly\. The composite has no discrete abstain action, every episode returns aqq\-correlated score, and the addedα​q\\alpha qterm is independent ofcc, leaving the report\-vertex analysis unchanged\. ∎

#### The clipping ablation, quantified\.

Clipping the Brier training signal belowt∗t^\{\\ast\}\(constant score, zero gradient there\) reinstates the dead zone: simulation exhibits a sliding\-mode equilibrium on the threshold \(learned confidence terminating at0\.74990\.7499againstt∗=0\.750t^\{\\ast\}=0\.750at moderateβ\\beta, and above it at largeβ\\beta, where the model wrongly answers\), and deployment utility pointwise weakly worse than unclipped at everyβ\\betatested, though the rule remains proper throughout, if no longer strictly: below the clip all reports tie, so truth\-telling is still a maximizer, just not the unique one\. The operative variable is the reachable region of identically zero training gradient, not properness\.

## Appendix INegative Results and Degenerate Cases

#### Proof of Proposition 4 \(proximal floor\)\.

With the proximal anchor,c˙0=ec0​\(−Deff\)​\(1\+o​\(1\)\)\+β​\(c0​\(0\)−c0\)\\dot\{c\}\_\{0\}=e^\{c\_\{0\}\}\(\-D\_\{\\mathrm\{eff\}\}\)\(1\+o\(1\)\)\+\\beta\(c\_\{0\}\(0\)\-c\_\{0\}\); the restoring term does not vanish asc0→−∞c\_\{0\}\\to\-\\infty, and settingc˙0=0\\dot\{c\}\_\{0\}=0givesec0=β​\(c0​\(0\)−c0\)/Deffe^\{c\_\{0\}\}=\\beta\(c\_\{0\}\(0\)\-c\_\{0\}\)/D\_\{\\mathrm\{eff\}\}, a stable root atc0=log⁡β\+log⁡log⁡\(1/β\)\+O​\(1\)c\_\{0\}=\\log\\beta\+\\log\\log\(1/\\beta\)\+O\(1\), approached exponentially fast; thenP∞=Θ​\(β​log⁡\(1/β\)\)P\_\{\\infty\}=\\Theta\(\\beta\\log\(1/\\beta\)\)with bounded tilt\.*The surrogate is not conservative: true KL is more permissive of collapse than the weight\-space stand\-in\.*∎

#### A norm–sharpness impossibility statement, and why it is demoted\.

Under the proximal surrogate, stationarity bounds‖θe−θ0‖≤G1/β\\\|\\theta\_\{e\}\-\\theta\_\{0\}\\\|\\leq G\_\{1\}/\\beta, and anε\\varepsilon\-optimal stochastic policy needs logit normΘ​\(log⁡\(1/ε\)\)\\Theta\(\\log\(1/\\varepsilon\)\), so no equilibrium beatsΔ​\(β\)=γ​δ028​e−δ0​G1/β\\Delta\(\\beta\)=\\frac\{\\gamma\\delta\_\{0\}^\{2\}\}\{8\}e^\{\-\\delta\_\{0\}G\_\{1\}/\\beta\}\-suboptimality\. Both halves fail as a statement about RLHF: \(i\) under*true*KL the norm budget does not exist \(Bernoulli KL is bounded uniformly in‖θ‖\\\|\\theta\\\|, so hard\-threshold policies are legitimate limiting stationary points\); \(ii\) even under the surrogate the bound is numerically vacuous at realistic anchoring \(Δ=2\.3×10−3\\Delta=2\.3\\times 10^\{\-3\}against an actual loss0\.5270\.527atβ=0\.3\\beta=0\.3;Δ≈10−22\\Delta\\approx 10^\{\-22\}atβ=10−2\\beta=10^\{\-2\}\)\. All of the interesting failure is dynamical\. We retain the statement because the way it fails under true KL is what pointed us to Lemma 1\.

#### Parameterization\-dependence of the exponent\.

The1/t1/tlaw is a property of the logit parameterization class: any fixed linear or redundant softmax\-logit reparameterization changes only constants \(flowing both logits rather than their difference halves the envelope constants\), while the nonlinearv=−log⁡θv=\-\\log\\theta,θ\>0\\theta\>0, givesθ˙∝\(1\+θ\)−2\\dot\{\\theta\}\\propto\(1\+\\theta\)^\{\-2\}and𝒢≍t−1/3\\mathcal\{G\}\\asymp t^\{\-1/3\}\. All exponent claims are therefore stated at the standard parameterization\.

## Appendix JSimulation Details

#### Calibration\.

Two latent types\(qH,qL\)=\(0\.9,0\.2\)\(q\_\{H\},q\_\{L\}\)=\(0\.9,0\.2\)\(𝔼​\[q\]=0\.55\\mathbb\{E\}\[q\]=0\.55\), signals∣H∼N​\(μ,1\)s\\mid H\\sim N\(\\mu,1\),s∣L∼N​\(−μ,1\)s\\mid L\\sim N\(\-\\mu,1\);μ\\muis the base model’s self\-knowledge quality \(the main text’s weak, medium, and strong signal levels areμ=0\.5,1,2\\mu=0\.5,1,2\)\. The calibration is deliberately favorable to the penalty rule and held fixed; the collapse threshold is crossed by varyingλ\\lambda, and \(B1\)⇔𝔼​\[q\]<t∗\\iff\\mathbb\{E\}\[q\]<t^\{\\ast\}fails atλ=1\\lambda=1for this calibration \(so theλ=1\\lambda=1rows of the deployment table show the no\-collapse regime, a property of the chosen accuracy rather than ofλ=1\\lambda=1; the empirical anchoring across deployed models is Section[K](https://arxiv.org/html/2608.00301#A11)\)\.

#### Matched comparison\.

Every panel uses: true gate\-level policy KL for the penalty arm; the proximal anchor the report arm structurally admits \(it has no action distribution, hence no policy KL; the asymmetry is structural, not a choice, and function\-space anchors leave the conclusions unchanged, Section[H](https://arxiv.org/html/2608.00301#A8)\); identical four\-parameter bounded heads; matched overconfident, weakly discriminative initialization \(base answers≈88%\\approx 88\\%; reported confidences in\[0\.70,0\.95\]\[0\.70,0\.95\]against true posteriors in\[0\.2,0\.9\]\[0\.2,0\.9\]\); greedy deployment for both; horizonT=104T=10^\{4\};β=10−3\\beta=10^\{\-3\}unless stated\. Reported quantities are absolute utility and coverage, never the ratioU/U∗U/U^\{\\ast\}\(meaningless whereU<0U<0orU∗≈0U^\{\\ast\}\\approx 0\)\. The horizon\-free phase diagram over\(μ,λ,β\)\(\\mu,\\lambda,\\beta\)is Figure[11](https://arxiv.org/html/2608.00301#A10.F11)\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x11.png)Figure 11:Horizon\-free phase diagram over\(μ,λ\)\(\\mu,\\lambda\)\. Colour islog10\\log\_\{10\}of the time to competence, the first time the greedily deployed policy attains half ofU∗U^\{\\ast\}; black cells never become competent within10610^\{6\}\. Top: action\-level abstention under true policy KL\. Bottom: report\-level abstention\. Both greedy, identical bounded heads; the two rows’β\\betamultiply different objects \(policy KL above, weight\-space proximal term below\) because a deterministic report policy has no action distribution; the asymmetry is structural, not a choice\.
#### Headline outcomes\.

\(i\) The invariantLLforβ∈\{10−4,10−3,10−2\}\\beta\\in\\\{10^\{\-4\},10^\{\-3\},10^\{\-2\}\\\}falls onto one curve, below theβ=0\\beta=0envelope past the transient \(L​t=1\.26Lt=1\.26att=30t=30against the intercept\-carrying bound1\.531\.53;1\.14,1\.05,1\.01,1\.0021\.14,1\.05,1\.01,1\.002bounds against measured0\.71,0\.57,0\.51,0\.320\.71,0\.57,0\.51,0\.32att=102,3×102,103,6×103t=10^\{2\},3\\times 10^\{2\},10^\{3\},6\\times 10^\{3\}; the anchored guarantee carries the12\\tfrac\{1\}\{2\}and tends to22, the measured anchor share never exceeding0\.0080\.008\)\. \(ii\) Estimator contrast atG=256G=256on a homogeneous calibration where all estimators collapse:d​log⁡L/d​log⁡t=−1\.490\\mathrm\{d\}\\log L/\\mathrm\{d\}\\log t=\-1\.490\(vanilla\) and−2\.503\-2\.503\(GRPO\); both absolute values carry the same finite\-time depression, and their*difference*is−1\.013\-1\.013against a predicted−1\-1; the difference, not the absolute exponent, is the stable statistic\. \(iii\) Deployment atλ=4\.5\\lambda=4\.5,T=104T=10^\{4\}: the action\-level arm deploys utility0\.0000\.000and coverage0\.0000\.000at every signal qualityμ∈\{0\.5,1,2\}\\mu\\in\\\{0\.5,1,2\\\}, forfeiting up toU∗=0\.199U^\{\\ast\}=0\.199and46\.9%46\.9\\%coverage; the report\-level arm attains0\.0045/0\.074/0\.1940\.0045/0\.074/0\.194against optima0\.0045/0\.0738/0\.19940\.0045/0\.0738/0\.1994\(9494–100%100\\%\), with coverage within22points of rational\. \(iv\) Atλ=1\\lambda=1\(\(B1\) fails\) both arms match the optimum: collapse is a property of the \(B1\) regime, not of training per se\. \(v\) The unbounded\-head control escapes byt≈104t\\approx 10^\{4\}and recoversU∗U^\{\\ast\}exactly, while the bounded head shows zero greedy coverage out to10710^\{7\}atμ=0\.5\\mu=0\.5: the bounded readout is what makes the plateau consequential\. \(vi\) The anchor is load\-bearing on the action side: swapping the true KL for the weight\-space proximal term rescues deployment atμ∈\{1,2\}\\mu\\in\\\{1,2\\\}\(coverage0\.234/0\.4680\.234/0\.468against rational0\.259/0\.4690\.259/0\.469,U=0\.073/0\.199U=0\.073/0\.199\) but not atμ=0\.5\\mu=0\.5\(coverage0\.0000\.000against rational0\.0350\.035\), measured atT=104T=10^\{4\},β=10−3\\beta=10^\{\-3\}\.

## Appendix KAn Empirical Sufficient Condition on Public Leaderboards

The blanket\-answering condition \(B1\) is𝔼​\[q\]<t∗\\mathbb\{E\}\[q\]<t^\{\\ast\}, which public leaderboards do not identify: they report accuracy*conditional*on the model’s own abstentions, bounding𝔼​\[q\]\\mathbb\{E\}\[q\]only within\[acc,acc\+abst\]\[\\mathrm\{acc\},\\mathrm\{acc\}\+\\mathrm\{abst\}\]\. What is identified is a sufficient condition for the*initial drift*\. The drift at initialization isc˙0​\(0\)=∫σ′​\(v0​\(s\)\)​g​\(s\)​𝑑s\\dot\{c\}\_\{0\}\(0\)=\\int\\sigma^\{\\prime\}\(v\_\{0\}\(s\)\)g\(s\)\\,ds, andσ′\\sigma^\{\\prime\}concentrates on the base policy’s*marginal*questions, not its confident ones\. Suppose the base policy is describable by a sharp monotone confidence threshold,v0​\(s\)=k​\(q¯​\(s\)−θ0\)v\_\{0\}\(s\)=k\(\\bar\{q\}\(s\)\-\\theta\_\{0\}\)withkklarge enough thatσ′​\(v0\)\\sigma^\{\\prime\}\(v\_\{0\}\)localizes on\{q¯≈θ0\}\\\{\\bar\{q\}\\approx\\theta\_\{0\}\\\}at a scale on whichggvaries little \(a Laplace\-type concentration condition; with monotoneq¯\\bar\{q\}the sign conclusion holds in thek→∞k\\to\\inftylimit, and we*assume*it at finitekk; this is the least rigorous step in this section\)\. Thensign⁡c˙0​\(0\)=sign⁡\(θ0−t∗\)\\operatorname\{sign\}\\dot\{c\}\_\{0\}\(0\)=\\operatorname\{sign\}\(\\theta\_\{0\}\-t^\{\\ast\}\), and sinceq¯\>θ0\\bar\{q\}\>\\theta\_\{0\}on the attempted set,

θ0≤𝔼​\[q¯∣attempted\]=accacc\+err\.\\theta\_\{0\}\\ \\leq\\ \\mathbb\{E\}\[\\bar\{q\}\\mid\\text\{attempted\}\]\\ =\\ \\frac\{\\mathrm\{acc\}\}\{\\mathrm\{acc\}\+\\mathrm\{err\}\}\.Combining withacc−λ​err\>0⇔accacc\+err\>t∗\\mathrm\{acc\}\-\\lambda\\,\\mathrm\{err\}\>0\\iff\\frac\{\\mathrm\{acc\}\}\{\\mathrm\{acc\}\+\\mathrm\{err\}\}\>t^\{\\ast\}:a negative error\-penalized score \(conditional accuracy below the Chow threshold\) implies, within the threshold class, that the initial drift of penalty\-rule training points toward collapse\.\(Figure[12](https://arxiv.org/html/2608.00301#A11.F12)\.\) The implication is one\-directional, and the converse genuinely fails \(a base withθ0=0\.30\\theta\_\{0\}=0\.30,k=30k=30has conditional accuracy0\.532\>t∗0\.532\>t^\{\\ast\}and a positive score, yet negative drift\): the count*undercounts*the affected set\. On the AA\-Omniscience leaderboard\(Jacksonet al\.[2025](https://arxiv.org/html/2608.00301#bib.bib46)\), which scores\+1/−1/0\+1/\-1/0over more than3636frontier models, only three scored above zero at the launch snapshot: at least3333satisfy the observable half of the condition atλ=1\\lambda=1\. Becauseλcrit=acc/err\\lambda\_\{\\mathrm\{crit\}\}=\\mathrm\{acc\}/\\mathrm\{err\}varies across models \(e\.g\.1\.171\.17,1\.001\.00,0\.790\.79,0\.730\.73for the four models whose triples we analyzed\), no singleλ\\lambdais training\-safe for a heterogeneous population\. Scope: the behavioral half, that refusals follow a sharp monotone confidence threshold, is an assumption; models whose abstentions are driven by formatting or safety policies sit outside the certificate, and the benchmark itself is a measurement instrument whose rankings this analysis does not question\.

![Refer to caption](https://arxiv.org/html/2608.00301v1/x12.png)Figure 12:\(a\) Each model’s score lineacc−λ​err\\mathrm\{acc\}\-\\lambda\\,\\mathrm\{err\}; its zero isλcrit=acc/err\\lambda\_\{\\mathrm\{crit\}\}=\\mathrm\{acc\}/\\mathrm\{err\}, and at the deployedλ=1\\lambda=1the sign of the score is the published index\. \(b\) The same fact in the \(conditional accuracy,λ\\lambda\) plane: a negative index*implies*collapse drift; a positive one does not preclude it\.#### Re\-scoring published evaluations\.

Re\-scoring published \(accuracy, error, abstention\) triples underscore​\(λ\)=acc−λ​err\\mathrm\{score\}\(\\lambda\)=\\mathrm\{acc\}\-\\lambda\\,\\mathrm\{err\}across four public cohorts yields eight pairwise rank reversals atλ<2\.3\\lambda<2\.3; the informative ones cross atλ=0\.041\\lambda=0\.041andλ=0\.834\\lambda=0\.834\(a16%16\\%\-accuracy model overtaking a39%39\\%\-accuracy one\)\. Reversals as such are guaranteed \(score lines with different slopes must cross somewhere\); what the re\-scoring measures is*where*they fall, and both informative crossings sit below the deployedλ=1\\lambda=1: a leaderboard that fixes a singleλ\\lambdais implicitly choosing among rankings that flip within the range of stakes it already spans\.

## Appendix LLanguage\-Model Experiment Details

Base models: Qwen2\.5\-1\.5B \(all multi\-seed runs\) and Qwen2\.5\-7B \(single\-seed runs at scale, with its own calibration of the pools and heads\)\(Qwen Team[2024](https://arxiv.org/html/2608.00301#bib.bib43)\); short\-form QA pools drawn from TriviaQA and PopQA\(Joshiet al\.[2017](https://arxiv.org/html/2608.00301#bib.bib44); Mallenet al\.[2023](https://arxiv.org/html/2608.00301#bib.bib45)\), graded programmatically against reference answers, with a grading\-error audit \(manual audit of auto\-wrong and auto\-correct samples per cell; error massesδFN,δFP\\delta\_\{\\mathrm\{FN\}\},\\delta\_\{\\mathrm\{FP\}\}reported with confidence intervals, and the margin correctionmtrue=mobs\+\(1\+λ\)​\(δFN−δFP\)m\_\{\\mathrm\{true\}\}=m\_\{\\mathrm\{obs\}\}\+\(1\+\\lambda\)\(\\delta\_\{\\mathrm\{FN\}\}\-\\delta\_\{\\mathrm\{FP\}\}\)applied as a sensitivity analysis; conclusions whose sign depends on the correction are labeled grader\-sensitive\)\.

#### Infrastructure\.

All runs are single\-node, single\-GPU jobs on a Slurm\-managed cluster, each on one NVIDIA A100\-SXM4\-80GB \(primary\) or H100 80GB GPU, with9696GB host memory and88CPU cores per GPU job, Rocky Linux 8\.10, Python 3\.9\.9, PyTorch 2\.8\.0 \(CUDA 12\.8, cuDNN 9\.10\.2\), transformers 4\.57\.6, tokenizers 0\.22\.2, datasets 4\.5\.0, accelerate 1\.10\.1, numpy 2\.0\.2, scipy 1\.13\.1\. Backbones \(Qwen2\.5\-1\.5B/7B\-Instruct\) run in bfloat16 with dropout disabled and log\-probabilities computed in fp32; sampling is neutralized against the shipped generation configurations so rollouts come from the policy itself\. GPU jobs total637\.5637\.5GPU\-hours \(longest single80008000\-step run≈35\{\\approx\}35h wall\-clock\); Tier\-1 runs never call the language model and reproduce on CPU in minutes\. Checkpointing is exact\-resume, with randomness keyed per \(seed, step\)\.

### L\.1Tier 1: Head\-Only Gate Dynamics on Frozen Features

The paper’s four\-parameter gate headvθ​\(x\)=c4​σ​\(w​z​\(x\)\+a\)\+c0v\_\{\\theta\}\(x\)=c\_\{4\}\\sigma\(wz\(x\)\+a\)\+c\_\{0\}is trained on a frozen scalar featurez​\(x\)z\(x\)of the base model \(the position\-one logit difference between the two single\-token decision symbols under an answer/abstain instruction template\), with the answer/abstain decision implemented as an explicit two\-token gate: the first generated position is restricted to the two decision symbols, and the per\-prompt answer probability is read*exactly*from the masked softmax, with no sampling error and no zero\-inflation\. Content and rewards come from pre\-generated, pre\-graded answer banks, so thatL​\(t\)L\(t\),σ¯​\(t\)\\bar\{\\sigma\}\(t\),ρ​\(t\)\\rho\(t\), andDeff​\(t\)D\_\{\\mathrm\{eff\}\}\(t\)are exact pool sums at every step, and the reference distribution for the gate KL is the base head under the identical mask\. This tier tests: \(i\) the integral envelope of Theorem 1\(i\), both as a deterministic expected\-gradient flow \(numerically integrated ODE; theorem\-grade, tolerance\-only\) and under SGD sampling \(sampling\-robustness, multi\-seed\); \(ii\) the estimator menu, by sweeping the target per\-prompt answer rate across half\-decade points10−510^\{\-5\}–10−110^\{\-1\}via bias offsets, atG∈\{64,256\}G\\in\\\{64,256\\\}andλ∈\{1,4,9\}\\lambda\\in\\\{1,4,9\\\}, for vanilla policy gradient, mean\-baseline, and group\-standard\-deviation normalization, against*zero\-parameter enumerated references*computed from the pool’s own empirical correctness distribution\. The tolerances: a GRPO\-minus\-vanilla local slope difference outside\[−1\.3,−0\.7\]\[\-1\.3,\-0\.7\]above the knee falsifies the−2\-2prediction; a measured knee outside a factor22of the enumerated location, or aGG\-scaling ratio outside\[2,8\]\[2,8\]forG=64→256G=64\{\\to\}256, falsifies the knee prediction; below the knee, drift/p/poutside\[2/3,1\.5\]\[2/3,1\.5\]of\(2​q¯−1\)​G−1\(2\\bar\{q\}\-1\)\\sqrt\{G\-1\}, or aλ\\lambda\-ratio \(λ=9\\lambda=9vs44\) outside\[0\.8,1\.25\]\[0\.8,1\.25\], falsifies theλeff=1\\lambda\_\{\\mathrm\{eff\}\}=1erasure; the dynamic\-sampling comparison atβ=0\\beta=0tests rollout\-axis coincidence\. Raw\-gradient probes at frozen checkpoints \(fixed prompt subsets, fixed rollout counts\) provide the optimizer\-independent drift statistic\.

### L\.2Tier 2: Full\-Parameter Fine\-Tuning

Full\-parameter RL fine\-tuning with an*unbiased*estimator \(leave\-one\-out baselines\) on a two\-tier prompt mixture,80%80\\%low\-accuracy \(q^∈\[0\.15,0\.35\]\\hat\{q\}\\in\[0\.15,0\.35\]\) and20%20\\%high\-accuracy \(q^≥0\.90\\hat\{q\}\\geq 0\.90\), chosen so that the aggregate satisfies \(B1\) while a certified subpopulation is unambiguously worth answering\. The sequence\-level KL to the frozen reference is implemented as a reward\-side term \(the summed per\-token log\-ratio\) through its own leave\-one\-out advantage channel, i\.e\. the update direction is1G​∑i\(Air−β​AiK\)​∇log⁡πθ​\(yi∣x\)\\frac\{1\}\{G\}\\sum\_\{i\}\(A\_\{i\}^\{r\}\-\\beta A\_\{i\}^\{K\}\)\\nabla\\log\\pi\_\{\\theta\}\(y\_\{i\}\\mid x\)with separate baselines for the reward and KL channels; this is an unbiased gradient of the anchored objective\(Williams[1992](https://arxiv.org/html/2608.00301#bib.bib50); Schulman[2020](https://arxiv.org/html/2608.00301#bib.bib51); Ahmadianet al\.[2024](https://arxiv.org/html/2608.00301#bib.bib23)\); the reference is masked identically to the policy at the gate position, and the forced end\-of\-sequence token after a refusal is excluded from the KL term\.

*Action\-level arm\.*The penalty rule\(\+1,−λ,0\)\(\+1,\-\\lambda,0\)with the two\-token decision interface of Tier 1, now carried by the full model’s own next\-token distribution with no added parameters: the first generated position is restricted to the answer/refusal pair, refusal is a single discrete action \(immediate end of sequence\), and the per\-prompt answer probability is read exactly from the renormalized pair in one forward pass\. Predictions: mean training reward negative and rising \(the collapse signature\), per\-prompt answer rates on the high\-accuracy tier crushed despite intact capability\.

*Capability certification\.*The central confound, that the gate closed because the model got worse, is excluded by construction: at the crossing checkpoints and at the end of training, each high\-tier prompt receives256256forced\-answer rollouts, and the analysis conditions on the subset whose per\-prompt one\-sided95%95\\%Clopper–Pearson lower confidence bound on correctness exceedst∗t^\{\\ast\}at every capability checkpoint \(retaining a positive answering margin\)\. Drag\-down is claimed only if the median gate probability on that certified subset falls below half its initial value at consecutive checkpoints\.

*Attribution control\.*A paired run \(common random numbers\) in which the low\-accuracy tier’s*reward\-channel*advantages are zeroed while its KL channel is retained: if the high\-tier gate collapse disappears, the collapse is attributed to the shared readout transmitting the low tier’s negative drift \(the mechanism\) rather than to any high\-tier\-local force\.

*Report\-level arm\.*Same data, same estimator, no abstain action \(the first position is forced to the answer symbol\): a prompt\-level confidence head is trained with the composite objective \(α=2\\alpha=2; score channel through the group advantage on the composite reward with the report value detached, plus the pathwise Brier gradient through the head\), the sequence KL applied to answer content only; abstention is applied only at deployment by thresholding the learned confidence att∗t^\{\\ast\}\. Predictions: no decay of mean reward under either estimator; final accuracy within0\.050\.05and Brier score within0\.050\.05of their early\-training values \(translation\-invariant, component\-wise criteria\); deployment coverage and utility near the certified\-capability optimum\.

### L\.3Results

#### Head\-only gate dynamics\.

Every quantitative tolerance above is met:

Below the knee the drift isλ\\lambda\-invariant at every grid point, the amplitude ratios above the knee fall within their tolerance bands, and the first\-window mean training reward rises in every sweep configuration \(p≤7×10−4p\\leq 7\{\\times\}10^\{\-4\}\)\. The integral envelope of Theorem 1\(i\) holds along the exact expected\-gradient flow and under SGD sampling in every seed\. Two comparisons are structurally unavailable on this pool: the trajectory\-slope contrast \(the sparse\-regime collapse leaves no common fitting window\) and the dynamic\-sampling comparison \(the real base gate is bimodal, so zero\-variance groups exhaust the resampling budget from the first step\)\. No measured quantity contradicts a prediction\.

#### Full\-model collapse, per seed\.

The certified cohort’s median gate probability falls below half its initial value at consecutive checkpoints in three of three seeds, with461/433/432461/433/432of the500500prompts certified; the paired control’s median never crosses and stays above0\.80\.8of its start in three of three pairs\. Step\-1010cohort medians are0\.0059/0\.0076/0\.00370\.0059/0\.0076/0\.0037and terminal medians5×10−7/1\.6×10−5/3\.7×10−75\{\\times\}10^\{\-7\}/1\.6\{\\times\}10^\{\-5\}/3\.7\{\\times\}10^\{\-7\}, while the control’s cohort median is exactly1\.0001\.000at every probe point\. Cohort mean forced\-answer correctness at the four capability checkpoints:0\.959→0\.965→0\.967→0\.9390\.959\\to 0\.965\\to 0\.967\\to 0\.939\. Capability at the two crossing checkpoints is measured at the exact steps, from replayed checkpoints whose weights are bit\-identical to the run’s: the per\-prompt bound clearst∗t^\{\\ast\}for498/486/495498/486/495of the500500prompts at step1010and497/491/495497/491/495at step1313, with cohort mean correctness0\.950\.95–0\.970\.97there\. Terminal diagnostic\-batch gradient norms per seed: task0/0/00/0/0; gate\-KL anchor6×10−13/1\.8×10−10/2\.6×10−136\{\\times\}10^\{\-13\}/1\.8\{\\times\}10^\{\-10\}/2\.6\{\\times\}10^\{\-13\}; the control’s task\-gradient norm at the same step is108108\. The mean reward rises over the first window in two of three seeds; the third collapses before the window closes\. At 7B \(single seed\), the base answers0\.510\.51of its answerable questions when offered the abstain option \(0\.910\.91without it\); the cohort median collapses by step1010, sits near10−410^\{\-4\}for roughly60006000steps, and leaves the plateau between the63106310and79437943probe points, recovering to terminal median1\.01\.0with mean0\.650\.65and a revived task gradient of131131; the paired control rises from the same0\.5550\.555start to0\.9300\.930by step1010and1\.0001\.000thereafter, never dipping\.

#### Report\-level runs\.

At 1\.5B \(three seeds each\), a30003000\-step AdamW run \(lr10−610^\{\-6\}; an optimizer\-robustness variant of the same objective\) moves coverage0\.466→0\.598/0\.604/0\.6020\.466\\to 0\.598/0\.604/0\.602and accuracy0\.41→0\.570\.41\\to 0\.57with Brier flat near0\.270\.27, and the80008000\-step plain\-SGD run reaches coverage0\.778/0\.810/0\.7940\.778/0\.810/0\.794, accuracy0\.6850\.685, Brier0\.220\.22\. At 7B \(single seed\) coverage passes0\.80\.8by step631631and ends at0\.9760\.976, accuracy0\.9420\.942, Brier0\.0520\.052at step80008000\. The 7B confidence head fits its calibration split9×9\{\\times\}better than at 1\.5B \(MSE0\.00250\.0025vs\.0\.02200\.0220\) yet still opens at coverage0\.4220\.422: linear\-probe ranking improves with scale, absolute calibration att∗=0\.818t^\{\\ast\}=0\.818does not, so the readout remains the binding constraint at both scales\.

#### Grading\.

Answers are graded programmatically\. The manual audit re\-judged all2424cells \(penalty and attribution\-control arms×\\timesthree seeds×\\timestwo checkpoints×\\timestwo tiers\),100100auto\-wrong and5050auto\-correct answers per cell,36003600rows in total\. The error profile is one\-sided: false negatives \(aliases, prefixes, sentence\-wrapped and cross\-language answers the string matcher missed\) carry per\-cell massδFN∈\[0\.002,0\.033\]\\delta\_\{\\mathrm\{FN\}\}\\in\[0\.002,0\.033\]on the high tier and\[0\.046,0\.131\]\[0\.046,0\.131\]on the low tier, while false positives are near\-absent \(δFP≤0\.020\\delta\_\{\\mathrm\{FP\}\}\\leq 0\.020per cell, zero in1818of2424\)\. Measured correctness therefore understates true correctness throughout: the pooled margin correction\(1\+λ\)​\(δFN−δFP\)\(1\+\\lambda\)\(\\delta\_\{\\mathrm\{FN\}\}\-\\delta\_\{\\mathrm\{FP\}\}\)is\+0\.058\+0\.058to\+0\.161\+0\.161on three high\-tier strata \(−0\.057\-0\.057on the fourth\) and\+0\.376\+0\.376to\+0\.619\+0\.619on the low tier; the capability certificates are conservative \(δFP≈0\\delta\_\{\\mathrm\{FP\}\}\\approx 0leaves them no inflation channel\); and the corrected low\-tier accuracy \(≈0\.30\{\\approx\}0\.30–0\.350\.35against a measured0\.230\.23\) leaves \(B1\) a wide margin, the corrected mixture mean≈0\.46\{\\approx\}0\.46sitting far belowt∗=0\.818t^\{\\ast\}=0\.818\. With95%95\\%Clopper–Pearson intervals on the audited fractions and unsure judgments counted both ways, no margin’s sign flips within any cell’s band: no conclusion is grader\-sensitive\.