OpenJev-RLCD：一个可用的 RLCD 实现

# OpenJev-RLCD: A Working RLCD Implementation
Source: [https://arxiv.org/html/2609.38850](https://arxiv.org/html/2609.38850)
###### Abstract

Decision models such as Jev answer questions about an agent’s state with probabilities, and those probabilities are only useful if they are calibrated: a system that automates the decisions it is confident about must be right when it says it is\. The open\-source reproductions of this idea are trained by supervised fine\-tuning and repaired with temperature scaling, while standard reinforcement learning from verifiable rewards makes reasoning models sharply overconfident\. We describe a working implementation of*reinforcement learning for calibrated decisions*\(RLCD\) for reasoning models and the analysis that led to it\. The model samples a rationale and we*read*the answer distribution it commits to afterwards; every rationale is scored by a strictly proper scoring rule of that distribution\. A one\-line variance identity explains why the popular alternative—scoring the mixture of several samples—rewards rationales for disagreeing and degenerates the reasoning, and shows that RLVR is exactly the mixture objective without its diversity term\. The per\-rationale objective has a calibrated optimum, but optimized naively it either switches reasoning off or is drowned out by high\-variance policy gradients\. The resulting recipe is*calibrate, then reinforce*: first train the post\-rationale decision with a proper score on the model’s own rationales, then optimize the reasoning with the same proper score as reward\. With Qwen3\-1\.7B on two real reasoning tasks \(3 seeds, paired tests\), RLCD matches or beats SFT, RFT/STaR and GRPO—each followed by temperature scaling—in accuracy and beats all of them in selective prediction \(area under the risk–coverage curve\); on GSM8K answer verification a single RLCD query can decide 81% of the items at≤\\leq5% error, versus 19% for GRPO\. On a task whose uncertainty is annotator disagreement, the same objective provably cannot beat cross\-entropy and, left alone, discovers that it should stop reasoning\. We release code, all run logs and an API\-compatible server\. Codes are available at: https://github\.com/ZimmyGao/openjev\-rlcd\.

## 1Introduction

A calibrated decision model returns, for a question about a state \(“will this plan succeed?”, “is this answer correct?”, “which option is right?”\), a probability distribution over answers whose confidence matches its accuracy\. The value of such a model lies less in its accuracy than in what calibration enables: a downstream agent can automate the decisions the model is sure about and escalate the rest, and it can do so from the model’s own confidence, without labels\. TypeSafe AI’s Jev\[[17](https://arxiv.org/html/2609.38850#bib.bib1)\]markets exactly this capability under the name*reinforcement learning for calibrated decisions*\(RLCD\), without disclosing a training algorithm\. The open\-source ecosystem that followed \(several hundred repositories within weeks\) trains scorers by supervised fine\-tuning \(SFT\) and fixes their confidence post hoc with temperature scaling \(TS\)\[[8](https://arxiv.org/html/2609.38850#bib.bib4)\]\.

Reinforcement learning is the natural tool for decisions whose outcomes are only observed after the fact, and it is how reasoning models are trained today\[[15](https://arxiv.org/html/2609.38850#bib.bib9)\]\. But the reward used for reasoning—1 if the answer is correct—is not a proper scoring rule of anything the model reports, and RL with it produces models that are confidently wrong\. A number of recent works therefore reward*sets*of samples \(duplicate penalties, collision terms, group calibration rewards\)\. This paper asks what a working RLCD for reasoning models is, and answers it with an analysis, a recipe and a controlled empirical study\.

#### Contributions\.

- •A readout formulation and a variance identity \(§[3](https://arxiv.org/html/2609.38850#S3)\)\.We score the answer distributionu⁡\(r\)u\(r\)that a reasoning model commits to after its own rationalerr\. For the Brier score,J⁡\(𝔼r​u\)=𝔼r​J​\(u\)\+Varr​\(u\)J\(\\mathbb\{E\}\_\{r\}u\)=\\mathbb\{E\}\_\{r\}J\(u\)\+\\mathrm\{Var\}\_\{r\}\(u\): scoring the mixture of samples pays a bonus for rationales that disagree, whether or not the disagreement is grounded\. With one\-hot answers the same identity writes the calibrated group reward as RLVR plus a Gini\-diversity bonus whose coefficient must be exactly one\. Scoring each rationale’s own distribution removes the bonus and has the calibrated optimumu⁡\(r\)=q⁡\(x\)u\(r\)=q\(x\)for every rationale, so a single query is calibrated\.
- •Two failure modes and the recipe that avoids them \(§[3\.2](https://arxiv.org/html/2609.38850#S3.SS2)\)\.Optimized end to end, the per\-rationale objective either abandons reasoning \(the “System\-One” collapse; prevented by a KL anchor\) or is drowned by the policy\-gradient term, whose gradient is 7–12×\\timeslarger than the readout’s and dominates the optimizer\.*Calibrate, then reinforce*: train the readout first, then reinforce the reasoning with the proper score as reward, anchored at the calibrated model\.
- •A controlled comparison on real tasks \(§[5](https://arxiv.org/html/2609.38850#S5)\)\.On MMLU\-Pro and on GSM8K answer verification, with three seeds and paired tests on every test item, RLCD matches or beats SFT\+TS, RFT/STaR\+TS \(with and without KL\) and GRPO\+TS in accuracy and Brier score, and beats all of them in area under the risk–coverage curve\. Forked\-checkpoint controls show when reinforcing the reasoning helps \(when reasoning is computation\) and why its reward must be proper \(a correctness reward undoes the calibrated readout\)\.
- •Where RL cannot help \(§[5\.3](https://arxiv.org/html/2609.38850#S5.SS3)\)\.When the uncertainty is annotator disagreement \(ChaosNLI\), reasoning cannot reduce it; the per\-rationale objective discovers this and turns reasoning off, and no RL variant beats cross\-entropy on the same labels\.

## 2Setting

A questionxxhasKKanswer options and an outcomeY∼q\(⋅∣x\)Y\\sim q\(\\cdot\\mid x\);qqis one\-hot when there is a right answer \(epistemic uncertainty\) and spread when humans disagree \(aleatoric uncertainty\)\. A reasoning model with parametersθ\\thetasamples a rationaler∼πθ\(⋅∣x\)r\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\), which ends in the markerAnswer:; we then*read*its next\-token distribution over theKKoption tokens,

uθ​\(x,r\)=softmax⁡\(logitsθ​\(x,r\)​\[option tokens\]\)∈ΔK,u\_\{\\theta\}\(x,r\)=\\mathrm\{softmax\}\\big\(\\text\{logits\}\_\{\\theta\}\(x,r\)\[\\text\{option tokens\}\]\\big\)\\in\\Delta\_\{K\},instead of sampling an answer from it\. The model’s decision distribution is the mixturepθ​\(x\)=𝔼r​uθ​\(x,r\)p\_\{\\theta\}\(x\)=\\mathbb\{E\}\_\{r\}u\_\{\\theta\}\(x,r\); a single query returnsuθ​\(x,r\)u\_\{\\theta\}\(x,r\)for one sampled rationale\. We use the \(shifted, negated\) Brier scoreJ⁡\(u,y\)=2​uy−‖u‖2=1−‖u−ey‖2J\(u,y\)=2u\_\{y\}\-\\\|u\\\|^\{2\}=1\-\\\|u\-e\_\{y\}\\\|^\{2\}, whose expectationJ⁡\(u,q\)=𝔼Y∼q​J​\(u,Y\)=2​u⊤​q−‖u‖2J\(u,q\)=\\mathbb\{E\}\_\{Y\\sim q\}J\(u,Y\)=2u^\{\\top\}q\-\\\|u\\\|^\{2\}is uniquely maximized atu=qu=q\[[2](https://arxiv.org/html/2609.38850#bib.bib2),[7](https://arxiv.org/html/2609.38850#bib.bib3)\]; the log score behaves the same way in our experiments\.

## 3Method

### 3\.1What to score: the mixture or each rationale

###### Proposition 1\(Variance decomposition\)\.

For any distribution over rationales and anyqq, withp=𝔼r​u​\(r\)p=\\mathbb\{E\}\_\{r\}u\(r\),

J⁡\(p,q\)=𝔼r​J​\(u⁡\(r\),q\)\+𝔼r​‖u⁡\(r\)−p‖2\.J\(p,q\)\\;=\\;\\mathbb\{E\}\_\{r\}\\,J\\big\(u\(r\),q\\big\)\\;\+\\;\\mathbb\{E\}\_\{r\}\\big\\\|u\(r\)\-p\\big\\\|^\{2\}\.

###### Proof\.

J⁡\(p,q\)−𝔼r​J​\(u⁡\(r\),q\)=𝔼r​‖u⁡\(r\)‖2−‖p‖2=𝔼r​‖u⁡\(r\)−p‖2J\(p,q\)\-\\mathbb\{E\}\_\{r\}J\(u\(r\),q\)=\\mathbb\{E\}\_\{r\}\\\|u\(r\)\\\|^\{2\}\-\\\|p\\\|^\{2\}=\\mathbb\{E\}\_\{r\}\\\|u\(r\)\-p\\\|^\{2\}, sinceJJis affine inuuexcept for−‖u‖2\-\\\|u\\\|^\{2\}\. ∎

The mixture objectiveJ⁡\(p,q\)J\(p,q\)—the objective behind scoring groups of samples—therefore equals the average per\-rationale score plus a bonus for rationales that disagree\. Nothing in the bonus asks the disagreement to be grounded: a policy can collect it by randomizing its conclusions\. Two consequences follow\.

###### Corollary 1\(RLVR is the mixture objective without its diversity term\)\.

If the answer is sampled,u⁡\(r\)=eAu\(r\)=e\_\{A\}withA∼pA\\sim p, then𝔼A​J​\(eA,q\)=2​p⊤​q−1=2​Pr⁡\(A=Y\)−1\\mathbb\{E\}\_\{A\}J\(e\_\{A\},q\)=2p^\{\\top\}q\-1=2\\Pr\(A=Y\)\-1is the RLVR objective andVar=1−‖p‖2\\mathrm\{Var\}=1\-\\\|p\\\|^\{2\}is the Gini impurity ofpp: the calibrated group reward is “RLVR \+ Gini bonus”\. With any coefficientλ<1\\lambda<1on the bonus the optimum is no longerp=qp=q; withλ=0\\lambda=0\(RLVR\) it is the mode\.

###### Proposition 2\(Per\-rationale objective\)\.

LetJλ​\(θ\)=𝔼x​𝔼r​J​\(uθ​\(x,r\),q⁡\(x\)\)\+λ​𝔼x​Varr​\(uθ\)J\_\{\\lambda\}\(\\theta\)=\\mathbb\{E\}\_\{x\}\\mathbb\{E\}\_\{r\}J\(u\_\{\\theta\}\(x,r\),q\(x\)\)\+\\lambda\\,\\mathbb\{E\}\_\{x\}\\mathrm\{Var\}\_\{r\}\(u\_\{\\theta\}\)\. If the outcome is independent of the model’s own rationale given the question \(Y⟂r\|xY\\perp r\\mid x, true wheneverYYis produced by the world\), then for everyλ<1\\lambda<1the unique maximizer over readouts isu⁡\(x,r\)=q⁡\(x\)u\(x,r\)=q\(x\)for every rationalerr; forλ=1\\lambda=1every readout with𝔼r​u​\(x,r\)=q⁡\(x\)\\mathbb\{E\}\_\{r\}u\(x,r\)=q\(x\)is optimal\.

###### Proof\.

Jλ=\(1−λ\)​𝔼r​J​\(u⁡\(r\),q\)\+λ​J​\(p,q\)J\_\{\\lambda\}=\(1\-\\lambda\)\\mathbb\{E\}\_\{r\}J\(u\(r\),q\)\+\\lambda J\(p,q\)by Proposition[1](https://arxiv.org/html/2609.38850#Thmproposition1)\. Both terms are maximized byu⁡\(r\)≡qu\(r\)\\equiv q; the first is strictly concave in eachu⁡\(r\)u\(r\), which gives uniqueness forλ<1\\lambda<1\. ∎

We therefore use the per\-rationale objective \(λ=0\\lambda=0\): a single query is calibrated at the optimum, and the objective no longer pays for disagreement\. Rationales still matter—a better rationale lets a finite\-capacity model computeq⁡\(x\)q\(x\)more accurately—but they are never rewarded for being random\.

#### Estimator\.

WithMMrationales per question and one outcomeYY, writeRi=J⁡\(uθ​\(x,ri\),Y\)R\_\{i\}=J\(u\_\{\\theta\}\(x,r\_\{i\}\),Y\)\. The gradient of𝔼r​J\\mathbb\{E\}\_\{r\}Jsplits into a pathwise part through the readout and a score\-function part for the rationales,

∇^=1M​∑i=1M∇θRi\+1M​∑i=1M\(Ri−1M−1​∑j≠iRj\)​∇θ​log⁡πθ​\(ri∣x\),\\widehat\{\\nabla\}=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\nabla\_\{\\theta\}R\_\{i\}\\;\+\\;\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\Big\(R\_\{i\}\-\\frac\{1\}\{M\-1\}\\sum\_\{j\\neq i\}R\_\{j\}\\Big\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(r\_\{i\}\\mid x\),with a leave\-one\-out baseline\[[10](https://arxiv.org/html/2609.38850#bib.bib6),[1](https://arxiv.org/html/2609.38850#bib.bib7)\]\. The estimator is unbiased; so are its variants with a KL penalty or per\-rationale costs added as shaping terms, and the Rao\-Blackwellized estimator of the mixture objective \(Appendix[A](https://arxiv.org/html/2609.38850#A1)\)\. We verified all of them against exact enumeration on tabular models \(maximum error<10−15<10^\{\-15\}\)\. Readinguuinstead of sampling the answer is a Rao\-Blackwellization of the sample\-only estimator\[[11](https://arxiv.org/html/2609.38850#bib.bib8)\]; with an empty rationale it reduces to training a direct scorer with the Brier score\.

### 3\.2Two ways the per\-rationale objective fails

#### It switches reasoning off\.

Reasoning models conclude their rationales decisively, souuafter a rationale is nearly one\-hot and the Brier score punishes every confident mistake\. Hedging after an*empty*rationale is learned faster than hedging after reasoning, and the policy gradient moves all mass to the empty rationale\. On MMLU\-Pro this happens within about 100 steps \(Figure[1](https://arxiv.org/html/2609.38850#S3.F1), red\): all four rationales become the bare marker and the model turns into a direct scorer\. A KL anchor to the reasoning model \(β=0\.04\\beta=0\.04, blue\) prevents the collapse and lets the model learn to hedge*after*reasoning\.

#### It is drowned by the policy gradient\.

Even with the anchor, adding the score\-function term to training from scratch makes every metric worse than training the readout alone, monotonically in its weight \(Table[4](https://arxiv.org/html/2609.38850#S5.T4)\)\. The gradient norm of the score\-function term is 7–12×\\timesthat of the pathwise term \(RMS over training\), so it dominates Adam’s second\-moment estimate: the useful, low\-variance readout signal is taken in steps sized by the noise\. Its direction is also unhelpful early on: while the readout is still miscalibrated, the reward mostly teaches rationales to sound certain\.

Figure 1:The System\-One collapse\(MMLU\-Pro, per\-rationale objective, trained end to end\)\. Without a KL anchor the four rationales per question become identical and empty after∼\\sim100 steps \(left: variance ofuuacross rationales\) while the readout learns to hedge \(right\)\. With a KL anchor the model keeps reasoning and learns to hedge after it\. The no\-KL run was stopped at step 150, after the collapse\.

### 3\.3The recipe: calibrate, then reinforce

Algorithm 1RLCD for a reasoning model \(one step;MMrationales for each question in the batch\)1:sample

r1,…,rM∼πθ\(⋅∣x\)r\_\{1\},\\dots,r\_\{M\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)untilAnswer:; read

ui=uθ​\(x,ri\)u\_\{i\}=u\_\{\\theta\}\(x,r\_\{i\}\); observe

YY
2:

Ri←J⁡\(ui,Y\)R\_\{i\}\\leftarrow J\(u\_\{i\},Y\)⊳\\trianglerightany strictly proper score of the read\-out distribution

3:ifstage 1 \(calibrate the decision\)then

4:

ℒ←−1M∑iRi\\mathcal\{L\}\\leftarrow\-\\frac\{1\}\{M\}\\sum\_\{i\}R\_\{i\}⊳\\trianglerightpathwise only: rationales are sampled on\-policy but not reinforced

5:elsestage 2 \(reinforce the reasoning\), starting from the stage\-1 model

θ1\\theta\_\{1\}
6:

Ai←1M​\(Ri−β​log⁡πθ​\(ri\)πθ1​\(ri\)\)−leave\-one\-out meanA\_\{i\}\\leftarrow\\frac\{1\}\{M\}\\big\(R\_\{i\}\-\\beta\\log\\frac\{\\pi\_\{\\theta\}\(r\_\{i\}\)\}\{\\pi\_\{\\theta\_\{1\}\}\(r\_\{i\}\)\}\\big\)\-\\text\{leave\-one\-out mean\}
7:

ℒ←−1M∑iRi−c∑isg\(Ai\)logπθ\(ri∣x\)\\mathcal\{L\}\\leftarrow\-\\frac\{1\}\{M\}\\sum\_\{i\}R\_\{i\}\-c\\sum\_\{i\}\\mathrm\{sg\}\(A\_\{i\}\)\\log\\pi\_\{\\theta\}\(r\_\{i\}\\mid x\)⊳\\trianglerightc=0\.3c=0\.3,β=0\.04\\beta=0\.04

8:endif

9:update

θ\\thetawith Adam on

ℒ\\mathcal\{L\}

Algorithm[1](https://arxiv.org/html/2609.38850#alg1)separates the two roles\. Stage 1 trains the decision the model reports after its own reasoning with a proper score; it is the pathwise half of the RLCD gradient, i\.e\. the deterministic policy gradient of the proper\-score reward with respect to the reported distribution, with the rationale treated as an on\-policy latent variable\. Stage 2 then reinforces the reasoning with the*same*proper score as reward, a down\-weighted score\-function term, and a KL anchor at the calibrated model, so that the reward is informative and the anchor protects what stage 1 learned\. Both stages observe exactly the outcomeYYof each question, like every baseline\.

## 4Experimental setup

#### Tasks\.

*MMLU\-Pro*\[[18](https://arxiv.org/html/2609.38850#bib.bib17)\]: 10\-option questions, a random 4000/500/1500 split of the 12k\-question benchmark \(we evaluate on the first 1000 test and 300 dev questions\); the task is knowledge\-heavy\.*GSM8K\-Verify*: a yes/no decision built from GSM8K\[[4](https://arxiv.org/html/2609.38850#bib.bib18)\]—“is this proposed final answer correct?”—where the proposal is a sampled solution of Qwen3\-0\.6B \(53% of the proposals are correct on train, 50% on test\); train/dev come from GSM8K\-train \(4000/300 used\), test = all 1260 GSM8K\-test problems with a parsable proposal\. Reasoning is essential: on a 400\-item probe the base model reaches 56% answering directly and 87% after reasoning\.*ChaosNLI*\[[12](https://arxiv.org/html/2609.38850#bib.bib19)\]: MNLI items with 100 annotator labels each \(199 dev, 400 test items\); every visit to a training item draws one fresh annotator label, and the target is the human label distribution\.

#### Model and methods\.

All methods start from Qwen3\-1\.7B\[[14](https://arxiv.org/html/2609.38850#bib.bib20)\]in non\-thinking mode, use AdamW with learning rate2⋅10−62\\cdot 10^\{\-6\}, 8 questions andM=4M=4rationales per step, and see the same sequence of questions and outcomes for a given seed\.SFT\+TS: the same model answers directly and is fine\-tuned with cross\-entropy on the label token \(400 or 800 steps; 800 steps did not improve on 400 on either task, so we report 400\)\.RFT/STaR\+TS\[[20](https://arxiv.org/html/2609.38850#bib.bib10)\]: rejection\-sampling fine\-tuning on the trajectories whose sampled answer is correct \(rationale and answer tokens\), with and without the KL anchor\.GRPO\+TS\[[15](https://arxiv.org/html/2609.38850#bib.bib9)\]: correctness reward, group\-normalized advantages, KL anchorβ=0\.04\\beta=0\.04\.RLCD: stage 1 \(400 steps, Brier or log score\), and stage 2 \(400 more steps from the stage\-1 checkpoint\)\. All baselines receive the same temperature scaling, fitted by grid search on the dev split\.

#### Metrics and statistics\.

Accuracy, Brier score, NLL and ECE of the single\-query readout after temperature scaling; the area under the risk–coverage curve \(AURC; items ranked by confidence, risk = error rate\); Cov@ϵ\\epsilon, the largest fraction of items, ranked by confidence, whose error rate is at mostϵ\\epsilon\[[6](https://arxiv.org/html/2609.38850#bib.bib16)\]; and the fitted temperatureTT\(T≈1T\\approx 1: calibrated as trained\)\. Everything is averaged over three seeds; tests are paired over test items \(per\-item values averaged over seeds\), with 95% bootstrap intervals and two\-sided sign\-flip permutationpp\-values for means\.∗\\ast:p<0\.05p<0\.05,∗⁣∗\\ast\\ast:p<0\.01p<0\.01\.

## 5Results

### 5\.1Main comparison

Table 1:Single\-query decisions after temperature scaling\(mean±\\pms\.d\. over 3 seeds\)\. Cov@ϵ\\epsilon: fraction of test items that can be decided automatically at error rate≤ϵ\\leq\\epsilon\(5% for GSM8K\-Verify, 20% for MMLU\-Pro, whose error rates are higher\)\.TT: temperature fitted on dev \(1 = calibrated as trained\)\. Best in bold\.Figure 2:Risk–coverageof single\-query decisions \(after temperature scaling, averaged over seeds\)\. Dotted: the error budget used for Cov@ϵ\\epsilonin Table[1](https://arxiv.org/html/2609.38850#S5.T1)\. Temperature scaling cannot repair the ranking of overconfident models \(GRPO, RFT\), whose error rate is flat in coverage\.Table[1](https://arxiv.org/html/2609.38850#S5.T1)and Figure[2](https://arxiv.org/html/2609.38850#S5.F2)summarize the comparison\. On GSM8K\-Verify, where reasoning is essential, SFT\+TS reaches only 70\.4% accuracy, and two\-stage RLCD improves on it by \+21\.8 points and −0\.247 Brier\. The reasoning\-based baselines are accurate but miscalibrated in ranking: after temperature scaling GRPO matches RLCD in accuracy \(91\.8% vs\. 92\.2%, n\.s\.\) and is close in Brier score \(difference −0\.015,p=0\.09p=0\.09\), yet its confidence separates right from wrong much worse \(AURC 0\.061 vs\. 0\.034, paired bootstrapp<10−3p<10^\{\-3\}\), so at a 5% error budget it can decide 19% of the items against 81% for RLCD\. RFT behaves like GRPO; with the KL anchor it reaches 91\.4% accuracy but still decides only 12% of the items at 5% error \(two\-stage RLCD vs\. RFT\+KL: Brier −0\.025, AURC −0\.041,p<0\.01p<0\.01\)\. The fitted temperatures tell the same story from the other side: GRPO and RFT needT≈7T\\approx 7, RLCDT=1\.4T=1\.4\.

On MMLU\-Pro every reasoning baseline fails to improve on SFT\+TS after temperature scaling, and without the KL anchor RFT collapses to empty rationales in all three seeds \(32\.7% accuracy, below the 38\.9% of the base model\)\. RLCD improves accuracy by \+5\.3 points over SFT\+TS and by \+5\.2 over GRPO\+TS, with Brier −0\.053 and −0\.095 \(allp<10−3p<10^\{\-3\}\)\. At a 20% error budget it can decide 28% of the items, against 16% for SFT\+TS and none for any reasoning baseline, whose confidence is uninformative after temperature scaling\. Stage 1 alone already carries most of the gain on both tasks; the choice of proper score for the readout matters little \(Brier vs\. log score\)\.

### 5\.2What reinforcing the reasoning adds

Table 2:Stage\-2 continuations forked from the same stage\-1 checkpoint\(3 seeds\), change relative to the checkpoint \(single query, after TS\)\. Anchor only: the same score\-function noise and KL anchor as RLCD\-RL but no outcome reward\.Figure 3:Stage\-2 reward matters\.Brier improvement over continuing to train the readout only, for the same 400\-step budget from the same checkpoint \(95% bootstrap intervals over test items, 3 seeds pooled\)\.To isolate stage 2, we saved the stage\-1 model for each seed and forked four 400\-step continuations from it with the same data stream \(Table[2](https://arxiv.org/html/2609.38850#S5.T2), Figure[3](https://arxiv.org/html/2609.38850#S5.F3)\)\. Three findings are consistent across seeds\.

- •Where reasoning is computation, reinforcing it helps\.On GSM8K\-Verify, continuing to train the readout alone*degrades*the model, whereas RLCD\-RL improves it; against the readout\-only continuation, RLCD\-RL gains \+3\.5 accuracy points and −0\.057 Brier \(p<10−3p<10^\{\-3\}, all three seeds individually significant\)\.
- •The outcome reward, not the anchor, does the work\.The anchor\-only control has the same score\-function noise and KL anchor but no reward; it is worse than RLCD\-RL \(GSM8K\-Verify \+4\.4 points, MMLU\-Pro Brier −0\.011, bothp<0\.01p<0\.01\) and unstable across seeds \(it collapsed in one of them\)\.
- •The reward must be proper\.Continuing with GRPO’s correctness reward is as good as RLCD\-RL on GSM8K\-Verify, but on MMLU\-Pro, where RL cannot raise accuracy much, it re\-inflates the confidence of the calibrated readout \(TTfrom 1\.4 to 3\.8\) and makes Brier score and AURC significantly worse than the checkpoint; RLCD\-RL beats it by \+1\.8 accuracy points and −0\.035 Brier \(p<10−3p<10^\{\-3\}\)\.

On MMLU\-Pro, reinforcing the reasoning with the proper score is neutral on average \(difference to the readout\-only continuation \+0\.002 Brier, n\.s\.; the sign varies by seed\): the benefit of stage 2 is specific to tasks where better reasoning changes the answer\.

### 5\.3Aleatoric uncertainty: when RL cannot help

Table 3:ChaosNLI\(400 test items\)\. The target is the distribution of 100 human labels; every method sees one fresh annotator label per visit\.Ifq⁡\(x\)q\(x\)is spread because annotators disagree, reasoning cannot reduce the uncertainty, and with an explicit readout the RLCD gradient equals the Brier gradient in expectation, so it cannot beat supervised training on the same labels\. ChaosNLI confirms both halves \(Table[3](https://arxiv.org/html/2609.38850#S5.T3)\)\. The mixture objective \(λ=1\\lambda=1\) calibrates its 16\-sample mixture \(JSD 0\.121\) by making the rationales random—they run to the length limit and a single rationale is badly calibrated—as Proposition[1](https://arxiv.org/html/2609.38850#Thmproposition1)predicts\. The per\-rationale objective instead discovers that reasoning is useless here and switches it off, reaching the same quality as cross\-entropy on direct answers \(JSD 0\.075 vs\. 0\.074\)\. We regard this as the correct behaviour of a decision objective, and as a reason to prefer a fast direct scorer when uncertainty is aleatoric\.

### 5\.4Ablations

Table 4:Weight of the score\-function term when training from scratch\(MMLU\-Pro, seed 17, per\-rationale objective with KL anchor, single query after TS\)\. More policy gradient is monotonically worse; this motivates separating the stages\.#### Score\-function weight\.

Table[4](https://arxiv.org/html/2609.38850#S5.T4): when the stages are not separated, every positive weight on the score\-function term hurts, in line with the gradient\-dilution diagnosis of §[3\.2](https://arxiv.org/html/2609.38850#S3.SS2)\.

#### Readout warm\-up alone is not enough\.

Training the readout for 200 steps and then reinforcing for 200 steps \(both from scratch\) ties the readout\-only model; the gains of Table[2](https://arxiv.org/html/2609.38850#S5.T2)require a converged readout\.

#### Who scores the rationale\.

Scoring rationales with a frozen copy of the calibrated readout instead of the policy’s own \(to prevent co\-adaptation\) was worse on GSM8K\-Verify \(seed 17: Brier\+0\.021\+0\.021, AURC\+0\.030\+0\.030\): the reasoning shortened and the policy’s own readout became overconfident\.

#### No KL anchor in stage 2\.

The reasoning degenerates after∼\\sim330 steps \(the model stops writing the marker and repeats its conclusion to the length limit\); the readout\-based reward is blind to the format of the rationale, so an anchor is necessary\.

#### Explicit\-probability RLCD\.

For a model without rationales \(a direct scorer\), RLCD with an explicit distribution has the same expected gradient as the Brier loss; on CLINC150 and ChaosNLI it never beat cross\-entropy \(CLINC150 accuracy 94\.6% vs\. 96\.3%\), and RLVR on the same models collapsed to the mode\.

## 6Related work

#### Calibration of language models\.

Post\-hoc temperature scaling\[[8](https://arxiv.org/html/2609.38850#bib.bib4)\]and verbalized or sampled confidence\[[9](https://arxiv.org/html/2609.38850#bib.bib14),[16](https://arxiv.org/html/2609.38850#bib.bib15)\]calibrate the*level*of confidence but not its ranking; selective prediction\[[6](https://arxiv.org/html/2609.38850#bib.bib16)\]evaluates exactly the ranking, which is where the methods in Table[1](https://arxiv.org/html/2609.38850#S5.T1)differ most\. Training with proper scores\[[2](https://arxiv.org/html/2609.38850#bib.bib2),[7](https://arxiv.org/html/2609.38850#bib.bib3)\]is standard for classifiers; our contribution is where to apply it in a reasoning model\.

#### RL for reasoning and calibration\.

RLVR and GRPO\[[15](https://arxiv.org/html/2609.38850#bib.bib9)\]optimize the probability of a correct sampled answer, which Corollary[1](https://arxiv.org/html/2609.38850#Thmcorollary1)identifies as the mixture objective without its diversity term; REINFORCE with leave\-one\-out baselines\[[19](https://arxiv.org/html/2609.38850#bib.bib5),[10](https://arxiv.org/html/2609.38850#bib.bib6),[1](https://arxiv.org/html/2609.38850#bib.bib7)\]is our estimator\. Concurrent work rewards verbalized confidence with the Brier score\[[5](https://arxiv.org/html/2609.38850#bib.bib13)\]; we instead score the answer distribution the model actually assigns, and analyze group rewards through Proposition[1](https://arxiv.org/html/2609.38850#Thmproposition1)\.

#### Rationales as latent variables\.

Stage 1 resembles the answer\-likelihood term of latent\-rationale training \(TRICE, LaTRO\)\[[13](https://arxiv.org/html/2609.38850#bib.bib11),[3](https://arxiv.org/html/2609.38850#bib.bib12)\]and STaR\[[20](https://arxiv.org/html/2609.38850#bib.bib10)\], which maximize the likelihood of the correct answer; the difference is the proper score of the full distribution, which is what makes the reported confidence usable\. The RFT/STaR baseline in Table[1](https://arxiv.org/html/2609.38850#S5.T1)shows that likelihood\-based cloning of correct trajectories inherits the overconfidence of RLVR\.

## 7Limitations

All experiments use a 1\.7B model, 400–800 update steps and three seeds; the effects are large relative to the seed variance, but larger models and longer training may change the balance between the two stages\. GSM8K\-Verify is a task we constructed from GSM8K with a 0\.6B proposer\. We compare with Jev only behaviorally \(the API is reproduced; the training data are not public\), and claim nothing about Jev’s internal algorithm\. Stage 2 helps on the reasoning\-essential task and is neutral on the knowledge\-heavy one; we do not yet know how to predict its benefit without running it\.

## 8Conclusion

A working RLCD for reasoning models scores what the model actually decides—the answer distribution after its own rationale—with a proper score, first to calibrate that decision and then to reinforce the reasoning that feeds it\. Scoring groups of samples instead rewards disagreement, and a correctness reward undoes calibration; both are visible in one identity\. The payoff is not a few points of accuracy but decisions that can be delegated: at the same accuracy as GRPO, a single RLCD query knows which of its answers to trust\.

## References

- \[1\]A\. Ahmadian, C\. Cremer, M\. Gallé, M\. Fadaee, J\. Kreutzer, O\. Pietquin, A\. Üstün, and S\. Hooker\(2024\)Back to basics: revisiting REINFORCE\-style optimization for learning from human feedback in LLMs\.InAnnual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[§3\.1](https://arxiv.org/html/2609.38850#S3.SS1.SSS0.Px1.p1.2),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px2.p1.1)\.
- \[2\]G\. W\. Brier\(1950\)Verification of forecasts expressed in terms of probability\.Monthly Weather Review78\(1\),pp\. 1–3\.Cited by:[§2](https://arxiv.org/html/2609.38850#S2.p1.2),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px1.p1.1)\.
- \[3\]H\. Chen, Y\. Feng, Z\. Liu, W\. Yao, A\. Prabhakar, S\. Heinecke, R\. Ho, P\. Mui, S\. Savarese, C\. Xiong, and H\. Wang\(2024\)Language models are hidden reasoners: unlocking latent reasoning capabilities via self\-rewarding\.arXiv preprint arXiv:2411\.04282\.Cited by:[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px3.p1.1)\.
- \[4\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px1.p1.1)\.
- \[5\]M\. Damani, I\. Puri, S\. Slocum, I\. Shenfeld, L\. Choshen, Y\. Kim, and J\. Andreas\(2025\)Beyond binary rewards: training LMs to reason about their uncertainty\.arXiv preprint arXiv:2507\.16806\.Cited by:[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px2.p1.1)\.
- \[6\]Y\. Geifman and R\. El\-Yaniv\(2017\)Selective classification for deep neural networks\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px3.p1.1),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px1.p1.1)\.
- \[7\]T\. Gneiting and A\. E\. Raftery\(2007\)Strictly proper scoring rules, prediction, and estimation\.Journal of the American Statistical Association102\(477\),pp\. 359–378\.Cited by:[§2](https://arxiv.org/html/2609.38850#S2.p1.2),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px1.p1.1)\.
- \[8\]C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger\(2017\)On calibration of modern neural networks\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§1](https://arxiv.org/html/2609.38850#S1.p1.1),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px1.p1.1)\.
- \[9\]S\. Kadavathet al\.\(2022\)Language models \(mostly\) know what they know\.arXiv preprint arXiv:2207\.05221\.Cited by:[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px1.p1.1)\.
- \[10\]W\. Kool, H\. van Hoof, and M\. Welling\(2019\)Buy 4 REINFORCE samples, get a baseline for free\!\.InICLR Workshop on Deep Reinforcement Learning Meets Structured Prediction,Cited by:[§3\.1](https://arxiv.org/html/2609.38850#S3.SS1.SSS0.Px1.p1.2),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px2.p1.1)\.
- \[11\]S\. Mohamed, M\. Rosca, M\. Figurnov, and A\. Mnih\(2020\)Monte Carlo gradient estimation in machine learning\.Journal of Machine Learning Research21\(132\),pp\. 1–62\.Cited by:[§3\.1](https://arxiv.org/html/2609.38850#S3.SS1.SSS0.Px1.p1.2)\.
- \[12\]Y\. Nie, X\. Zhou, and M\. Bansal\(2020\)What can we learn from collective human opinions on natural language inference data?\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px1.p1.1)\.
- \[13\]D\. Phan, M\. D\. Hoffman, D\. Dohan, S\. Douglas, T\. A\. Le, A\. Parisi, P\. Sountsov, C\. Sutton, S\. Vikram, and R\. A\. Saurous\(2023\)Training chain\-of\-thought via latent\-variable inference\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px3.p1.1)\.
- \[14\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px2.p1.1)\.
- \[15\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2609.38850#S1.p2.1),[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px2.p1.1)\.
- \[16\]K\. Tian, E\. Mitchell, A\. Zhou, A\. Sharma, R\. Rafailov, H\. Yao, C\. Finn, and C\. D\. Manning\(2023\)Just ask for calibration: strategies for eliciting calibrated confidence scores from language models fine\-tuned with human feedback\.InConference on Empirical Methods in Natural Language Processing \(EMNLP\),Cited by:[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px1.p1.1)\.
- \[17\]TypeSafe AI\(2026\)Jev: calibrated decisions for ai agents\.Note:Product documentation,[https://typesafe\.ai](https://typesafe.ai/)Cited by:[§1](https://arxiv.org/html/2609.38850#S1.p1.1)\.
- \[18\]Y\. Wang, X\. Ma, G\. Zhang, Y\. Ni, A\. Chandra, S\. Guo, W\. Ren, A\. Arulraj, X\. He, Z\. Jiang, T\. Li, M\. Ku, K\. Wang, A\. Zhuang, R\. Fan, X\. Yue, and W\. Chen\(2024\)MMLU\-Pro: a more robust and challenging multi\-task language understanding benchmark\.InAdvances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track,Cited by:[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px1.p1.1)\.
- \[19\]R\. J\. Williams\(1992\)Simple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8,pp\. 229–256\.Cited by:[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px2.p1.1)\.
- \[20\]E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman\(2022\)STaR: bootstrapping reasoning with reasoning\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§4](https://arxiv.org/html/2609.38850#S4.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2609.38850#S6.SS0.SSS0.Px3.p1.1)\.

## Appendix AEstimators and their exact checks

#### Per\-rationale objective with shaping\.

For per\-rationale termssis\_\{i\}that depend only onrir\_\{i\}\(andYY\), e\.g\.si=−βM​log⁡πθ​\(ri\)πref​\(ri\)s\_\{i\}=\-\\frac\{\\beta\}\{M\}\\log\\frac\{\\pi\_\{\\theta\}\(r\_\{i\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(r\_\{i\}\)\}or a length cost, the surrogate

ℒ=−1M∑iRi−∑isg\(RiM\+si−1M−1∑j≠i\(RjM\+sj\)\)logπθ\(ri\)\\mathcal\{L\}=\-\\frac\{1\}\{M\}\\sum\_\{i\}R\_\{i\}\-\\sum\_\{i\}\\mathrm\{sg\}\\Big\(\\tfrac\{R\_\{i\}\}\{M\}\+s\_\{i\}\-\\frac\{1\}\{M\-1\}\\sum\_\{j\\neq i\}\\big\(\\tfrac\{R\_\{j\}\}\{M\}\+s\_\{j\}\\big\)\\Big\)\\log\\pi\_\{\\theta\}\(r\_\{i\}\)has expected gradient−∇θ\(𝔼r​J​\(uθ,q\)\+𝔼r​s\)\-\\nabla\_\{\\theta\}\\big\(\\mathbb\{E\}\_\{r\}J\(u\_\{\\theta\},q\)\+\\mathbb\{E\}\_\{r\}s\\big\), because each baseline uses only the other, independent rationales\.

#### Mixture objective\.

J^=2M​∑iui​\[Y\]−1M⁡\(M−1\)​∑i≠jui⊤​uj\\widehat\{J\}=\\frac\{2\}\{M\}\\sum\_\{i\}u\_\{i\}\[Y\]\-\\frac\{1\}\{M\(M\-1\)\}\\sum\_\{i\\neq j\}u\_\{i\}^\{\\top\}u\_\{j\}is unbiased forJ⁡\(p,q\)J\(p,q\)because the rationales are independent; its gradient is the pathwise derivative ofJ^\\widehat\{J\}plus, for each rationale, a score\-function term whose reward is the part ofJ^\\widehat\{J\}that depends onrir\_\{i\}, with a baseline that does not\.

#### Checks\.

For tabular models \(R≤4R\\leq 4rationale types,K≤4K\\leq 4,M∈\{3,4\}M\\in\\\{3,4\\\}\) we enumerate allRMR^\{M\}rationale tuples and all outcomes and compare the expected surrogate gradient with the exact gradient of the target: the maximum absolute difference is below10−1510^\{\-15\}for the mixture estimator, the per\-rationale estimator, their convex combinations and the shaped versions \(test\_rlcd\_rb\.py\)\.

## Appendix BHyperparameters

## Appendix CMixture readout

Averaging the readout over four rationales improves every reasoning method; the ordering of Table[1](https://arxiv.org/html/2609.38850#S5.T1)is unchanged\.