Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data
Summary
The paper proposes an inverse reasoning LLM that generates reasoning traces conditioned on answers, enabling efficient self-distillation from answer-only data as a middle ground between SFT and RL. Using Qwen3-0.6B on Countdown and GSM8K, a single offline distillation round roughly doubles pass@1 on hard prompts and the inverse model doubles as a verifier, raising best-of-n selection accuracy from 0.753 to 0.969.
arXiv:2610.18878v1 Announce Type: new
Abstract: Post-training reasoning LLMs often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt-answer tuples and lacks the groundtruth chain-of-thought (CoT) sequence. On the other hand, reinforcement learning (with verifiable rewards, for example) can operate without CoT data but it is much more compute-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non-zero policy gradient. In this work we explore a middleground, a self-distillation technique that uses answer-only data, treating CoT as latent variables. This relies on training a partner ``inverse reasoning LLM" that generates the original's reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, with all generation happening offline and up front, making the approach particularly suited to continual post-training. Using Qwen3-0.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts and, on Countdown, outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio (with smaller gains on GSM8K); rejection sampling by construction has no signal on problems the model never solves on its own. Continued GRPO on the fresh data retains an edge at hard pass@1 but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness. When used for best-of-$n$ selection at test time, it raises selection accuracy on Countdown from 0.753 (majority voting) to 0.969.
View Cached Full Text
Cached at: 10/15/26, 06:45 AM
# Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data
Source: [https://arxiv.org/html/2610.18878](https://arxiv.org/html/2610.18878)
###### Abstract
Post\-training reasoning large language models \(LLMs\) often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt\-answer tuples and lacks the groundtruth chain\-of\-thought \(CoT\) sequence\. On the other hand, reinforcement learning \(with verifiable rewards, for example\) can operate without CoT data but it is much more compute\-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non\-zero policy gradient\. In this work we explore a middleground, a self\-distillation technique that uses answer\-only data, treating CoT as latent variables\. This relies on training a partner “inverse reasoning LLM” that generates the original’s reasoning traces conditioned on the answer\. The inverse model is trained once and reused across incoming batches of answer\-only data, with all generation happening offline and up front, making the approach particularly suited to continual post\-training\. Using Qwen3\-0\.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts \(e\.g\.0\.08→0\.190\.08\\to 0\.19on Countdown\) and, on Countdown, outperforms rejection\-sampling self\-distillation on every hard\-tail metric at every mix ratio \(with smaller gains on GSM8K\); rejection sampling by construction has no signal on problems the model never solves on its own\. Continued GRPO on the fresh data retains an edge at hard pass@1 \(0\.230\.23\) but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step\. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness\. When used for best\-of\-nnselection at test time, it raises selection accuracy on Countdown from 0\.753 \(majority voting\) to 0\.969\.
## 1Introduction
Chain\-of\-thought \(CoT\) reasoning\([Wei et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib15)\)has extended the abilities of large language models \(LLMs\) from their initial defining task of text generation to being capable of solving complex problems\. From the perspective of an LLM, there is no difference between the tokens that are part of the prompt, the CoT sequence, or the final answer\. Functionally, however, these tokens serve very different purposes: CoT tokens are largely a means to an end, providing additional computation steps that bridge the prompt and the final answer\. The asymmetry also has a counterpart at the data level, where question–answer pairs are often abundant but reasoning traces are scarce\. This has been a large factor in the focus on reinforcement learning \(RL\) techniques such as GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.18878#bib.bib5)\)over supervised finetuning \(SFT\), which would require groundtruth reasoning traces, especially on verifiable tasks\([Lambert et al\., 2024](https://arxiv.org/html/2610.18878#bib.bib4)\)\. Here, groundtruth CoT is not needed as the learning signal comes solely from whether the final answer is correct\. While these approaches have proven effective, they are typically far less compute\-efficient than supervised finetuning, requiring many rollouts per prompt before the LLM stumbles upon a correct answer that provides a learning signal\.
The starting point for our work is how we can do efficient, supervised finetuning for CoT reasoning LLMs on question\-answer pairs without reasoning data\. We do this by treating the CoT sequences as latent variables and take a variational inference approach that makes use of a learnedinverse model\. This is a separate LLM trained to generate the CoT of the original LLM given both the prompt and answer\. Using an auxiliary answer\-conditioned model has firm standing in the literature, including the earlier work of[Zelikman et al\. \(2022\)](https://arxiv.org/html/2610.18878#bib.bib14)and[Phan et al\. \(2023\)](https://arxiv.org/html/2610.18878#bib.bib13), and the very recently developed techniques of RAVR\([Lin et al\., 2026](https://arxiv.org/html/2610.18878#bib.bib12)\)and CoVRL\([Wen et al\., 2025](https://arxiv.org/html/2610.18878#bib.bib3)\), though these operate in the online RL setting rather than the SFT/offline\-RL setting we target \(see Section[3](https://arxiv.org/html/2610.18878#S3)for a detailed comparison\)\.
Our main experiments are self\-distillation applications, where the inverse model is trained from CoT on correct answers from the forward model, and then the forward model is finetuned so its CoT matches the inverse model’s\. This works with either the original data the inverse model was trained on or on new data, which can be considered an instance of offline reinforcement learning\.
As an additional use case, we find that the inverse model forms an accurate verifier of correctness, which we show can be used as a more efficient alternative to \(adaptive\) best\-of\-n sampling\([Cobbe et al\., 2021](https://arxiv.org/html/2610.18878#bib.bib9);[Aggarwal et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib7)\)\.
For validating our techniques, we run experiments using Qwen3\-0\.6B\([Yang et al\., 2025](https://arxiv.org/html/2610.18878#bib.bib10)\)with the Countdown\([Gandhi et al\., 2024](https://arxiv.org/html/2610.18878#bib.bib6)\)and GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.18878#bib.bib9)\)datasets\. In both environments we start from a model whose GRPO training has saturated and show that distillation from the inverse model roughly doubles pass@1 on*hard*prompts \(those the starting model solves at most 25% of the time\) while maintaining overall pass@8 and pass@32\. On Countdown it beats the rejection\-sampling baseline on every hard\-tail metric at every mix ratio, and on GSM8K it matches or modestly exceeds it; the hard tail is exactly the regime where rejection sampling is structurally blind since it can only train on problems the forward model already solves\. On the other hand, the inverse model can propose a correct\-leading CoT for problems the forward model never solves on its own\. Continued GRPO on fresh data remains the strongest method at hard pass@1, but distillation surpasses it at pass@8 and pass@32 without any on\-policy rollouts during training\.
## 2Method
### 2\.1Overview
SetupPrompt–answer data𝒟=\{\(P,A\)\}\\mathcal\{D\}=\\\{\(P,A\)\\\}Pretrained reasoning modelpθp\_\{\\theta\}Inverse modelSFTqφ\(C∣P,A\)q\_\{\\varphi\}\(C\\mid P,A\)on𝒟\\mathcal\{D\}to reproducepθp\_\{\\theta\}’s CoT on correct answersSelf\-distillation \(train\-time\)SFTpθp\_\{\\theta\}on triples\(P,C∼qφ\(P,A\),A\)\\big\(P,\\,C\\sim q\_\{\\varphi\}\(P,A\),\\,A\\big\)obeyingforward consistency:A∼pθ\(P,C\)A\\sim p\_\{\\theta\}\(P,C\)\. Data can come from𝒟\\mathcal\{D\}or new datasetTest\-time verifierScorepθp\_\{\\theta\}answers with the map\(P,C,A\)↦qφ\(C∣P,A\)\(P,C,A\)\\mapsto q\_\{\\varphi\}\(C\\mid P,A\),where\(C,A\)∼pθ\(P\)\(C,A\)\\sim p\_\{\\theta\}\(P\)Figure 1:Overview of our method\. Starting from answer\-only data𝒟\\mathcal\{D\}and a reasoning modelpθp\_\{\\theta\}that already solves part of it, we train an*inverse model*qφq\_\{\\varphi\}to reconstructpθp\_\{\\theta\}’s chain\-of\-thought from the answer\. The inverse model then supports two downstream uses: \(top\) self\-distillation, wherepθp\_\{\\theta\}is finetuned onqφq\_\{\\varphi\}\-generated reasoning traces that remain forward\-consistent withpθp\_\{\\theta\}, after which the finetunedpθp\_\{\\theta\}is deployed alone; and \(bottom\) test\-time verification, whereqφq\_\{\\varphi\}scores candidate answers sampled frompθp\_\{\\theta\}\.We now give an overview of our method, whose summary can be seen in Figure[1](https://arxiv.org/html/2610.18878#S2.F1)\. Consider a dataset distribution𝒟=\{\(P,A\)\}⊂𝒫×𝒜\\mathcal\{D\}=\\\{\(P,A\)\\\}\\subset\\mathcal\{P\}\\times\\mathcal\{A\}where𝒫\\mathcal\{P\}is a space of prompts and𝒜\\mathcal\{A\}is a space of answers\. Letpθp\_\{\\theta\}be an LLM with parametersθ\\theta\(we will also usepθp\_\{\\theta\}to denote the probability distribution the LLM determines on the token space\)\. We assume that the LLM is a reasoning model, i\.e\. from a promptPPan answer is arrived at by first generating a CoT sequenceCC\(commonly distinguished from the prompt and answer by being enclosed in<think\>tags\) fromPPand then generatingAAfromPPandCC\.
Our ultimate goal is to treat the CoT sequences as latent variables and findθ\\thetathat maximizes the log\-likelihood
ℓ\(θ\)=∑\(P,A\)∈𝒟logpθ\(A\|P\)\\ell\(\\theta\)=\\sum\_\{\(P,A\)\\in\\mathcal\{D\}\}\\log p\_\{\\theta\}\(A\|P\)
This is difficult because we only readily have access topθ\(C\|P\)p\_\{\\theta\}\(C\|P\)andpθ\(A\|C,P\)p\_\{\\theta\}\(A\|C,P\), and computingpθ\(A\|P\)p\_\{\\theta\}\(A\|P\)by marginalizing overCCis intractable\. However, we can take advantage of Fisher’s identity\([Cappé et al\., 2005](https://arxiv.org/html/2610.18878#bib.bib11)\)
∇θℓ\(θ0\)=∑\(P,A\)∈𝒟𝔼C∼pθ0\(⋅\|P,A\)\[∇θlogpθ\(C,A\|P\)\]\|θ=θ0\\nabla\_\{\\theta\}\\ell\(\\theta\_\{0\}\)=\\sum\_\{\(P,A\)\\in\\mathcal\{D\}\}\\mathbb\{E\}\_\{C\\sim p\_\{\\theta\_\{0\}\}\(\\cdot\|P,A\)\}\\left\.\\left\[\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(C,A\|P\)\\right\]\\right\|\_\{\\theta=\\theta\_\{0\}\}\(1\)if we can get a hold of the posteriorpθ\(C\|P,A\)p\_\{\\theta\}\(C\|P,A\)\. For this we take a variational inference approach and approximate this distribution via a separate language modelqφq\_\{\\varphi\}that generatesCCfrom\(P,A\)\(P,A\), writingqφ\(⋅∣P,A\)≈pθ\(⋅∣P,A\)q\_\{\\varphi\}\(\\cdot\\mid P,A\)\\approx p\_\{\\theta\}\(\\cdot\\mid P,A\)\. This is trained via supervised finetuning on the completions thatpθp\_\{\\theta\}got correct\.
### 2\.2Trainingqφq\_\{\\varphi\}
Naively trainingqφq\_\{\\varphi\}via gradient descent against the negative log likelihood \(NLL\) objective on correct completions,
Lnll\(P\)=−logqφ\(C\|P,A\),whereC∼pθ\(P\),A∼pθ\(P,C\),\(P,A\)∈𝒟L\_\{nll\}\(P\)=\-\\log q\_\{\\varphi\}\(C\|P,A\),~\\text\{ where \}C\\sim p\_\{\\theta\}\(P\),A\\sim p\_\{\\theta\}\(P,C\),\(P,A\)\\in\\mathcal\{D\}\(2\)turns out to not work well in practice: we found thatqφq\_\{\\varphi\}trained in this way tends to ignoreAAand will generate some reasoning traces that lead to a wrong answer\. This is likely due to the token imbalance \(of most environments, including the ones we use\) where CoT sequences are significantly longer than the answer sequences\. Therefore, to encourageqφq\_\{\\varphi\}to takeAAinto account, we supplement the NLL loss with a contrastive term\. We do this by picking a group sizeg\>1g\>1and generategg\-many completions frompθp\_\{\\theta\}\. We then filter the generated group completions to those where the group contains at least one correct and one incorrect answer\. Letℓ\+\(P\)\\ell\_\{\+\}\(P\)andℓ−\(P\)\\ell\_\{\-\}\(P\)be the average log probs \(across tokens\) thatqφq\_\{\\varphi\}assigns for all completions of a promptPPthat lead to correct and incorrect answers, respectively\. We then define the contrastive margin loss as
Lcont\(P\)=max\(m−\(ℓ\+\(P\)−ℓ−\(P\)\),0\)L\_\{cont\}\(P\)=\\max\\left\(m\-\(\\ell\_\{\+\}\(P\)\-\\ell\_\{\-\}\(P\)\),0\\right\)\(3\)wherem\>0m\>0is themarginhyperparameter\. We combine these, via a hyperparameterλ\\lambda, to form our total objective
Lq=Lnll\+λLcont,L\_\{q\}=L\_\{nll\}\+\\lambda L\_\{cont\},\(4\)which we optimize over minibatches of prompts using stochastic gradient descent\. Note that theLnllL\_\{nll\}term only takes in correct completions whileLcontL\_\{cont\}makes use of the entire group of completions\.
### 2\.3Applyingqφq\_\{\\varphi\}
Fromqφq\_\{\\varphi\}we get two applications\.
#### 2\.3\.1Distillation fromqφq\_\{\\varphi\}topθp\_\{\\theta\}
The first, which is our main focus, finetunespθp\_\{\\theta\}from the CoT traces generated byqφq\_\{\\varphi\}\. This proceeds by starting from a dataset𝒟′\\mathcal\{D\}^\{\\prime\}\(which may or may not be𝒟\\mathcal\{D\}\) of prompt\-answer pairs, and then usingqφq\_\{\\varphi\}to generate CoT traces for it\. From these, we filter to include only those that leadpθp\_\{\\theta\}to the correct answer, thus forming the dataset,
𝒟qφ,pθ′=\{\(P,C,A\)∣C∼qφ\(P,A\),A∼pθ\(P,C\),\(P,A\)∈𝒟′\}\.\\mathcal\{D\}^\{\\prime\}\_\{q\_\{\\varphi\},p\_\{\\theta\}\}=\\\{\(P,C,A\)\\mid C\\sim q\_\{\\varphi\}\(P,A\),~A\\sim p\_\{\\theta\}\(P,C\),~\(P,A\)\\in\\mathcal\{D\}^\{\\prime\}\\\}\.
which we can directly do SFT ofpθp\_\{\\theta\}on since this has complete reasoning traces with answers\. Concretely, we sample a single CoT fromqφq\_\{\\varphi\}per prompt and keep it only if it is*forward consistent*: conditioned on the prompt andqφq\_\{\\varphi\}’s CoT,pθp\_\{\\theta\}greedy\-decodes the correct answer\. We additionally dropqφq\_\{\\varphi\}\-CoTs for prompts on which everypθp\_\{\\theta\}rollout is already correct, concentratingqφq\_\{\\varphi\}’s supervision wherepθp\_\{\\theta\}needs it\.
As the optimization process updates the weights ofpθp\_\{\\theta\}, the distributional approximation accuracy ofqφq\_\{\\varphi\}may drift from the changingpθ\(⋅\|P,A\)p\_\{\\theta\}\(\\cdot\|P,A\)since𝒟qφ,pθ′\\mathcal\{D\}^\{\\prime\}\_\{q\_\{\\varphi\},p\_\{\\theta\}\}is fixed from the start \(we do not resample frompθp\_\{\\theta\}during distillation\)\. Therefore as a regularization term to keeppθp\_\{\\theta\}from drifting too much we supplement𝒟qφ,pθ′\\mathcal\{D\}^\{\\prime\}\_\{q\_\{\\varphi\},p\_\{\\theta\}\}with rejection sampling trajectories ofpθp\_\{\\theta\}alone, the set of which we will denote by𝒟RSpθ′\\mathcal\{D\}^\{\\prime\}\_\{RS\_\{p\_\{\\theta\}\}\}: we sample a group of completions frompθp\_\{\\theta\}for each prompt and keep every correct completion as its own training example\. We let the parameter0≤μ≤10\\leq\\mu\\leq 1, which we call themix ratio, denote the ratio of data in a batch that is drawn from𝒟qφ,pθ′\\mathcal\{D\}^\{\\prime\}\_\{q\_\{\\varphi\},p\_\{\\theta\}\}\(the rest coming from𝒟RSpθ′\\mathcal\{D\}^\{\\prime\}\_\{RS\_\{p\_\{\\theta\}\}\}\), with examples drawn uniformly from each pool\. We experiment and show the tradeoffs for choosing values ofμ∈\{0,0\.25,0\.5,0\.75,1\}\\mu\\in\\\{0,0\.25,0\.5,0\.75,1\\\}\.
We note importantly that in this applicationqφq\_\{\\varphi\}is used only during training: at deployment the finetunedpθp\_\{\\theta\}operates alone, leaving inference cost and interface unchanged\.
We also highlight where the generation cost is paid\. All sampling, namely the CoT traces fromqφq\_\{\\varphi\}and the rejection\-sampling trajectories frompθp\_\{\\theta\}, happens once, up front, before any weight update\. The remainder of training is plain SFT over a fixed dataset\. This is in contrast to online policy\-gradient methods such as GRPO, which must regenerate a fresh group of rollouts for every prompt at every training step because the rollouts must come from the current policy\. Our approach is thus naturally an offline one: the up\-front generations can be batched with high throughput, reused across runs \(e\.g\. across the mix\-ratio sweep in Section[4\.2](https://arxiv.org/html/2610.18878#S4.SS2)\), and the training itself never blocks on autoregressive sampling\.
#### 2\.3\.2Usingqφq\_\{\\varphi\}as a verifier ofpθp\_\{\\theta\}
The second application usesqφq\_\{\\varphi\}as a verifier ofpθp\_\{\\theta\}’s correctness\. Given a promptPP, CoTCC, and answerAAcomputed frompθp\_\{\\theta\}, we can useqφq\_\{\\varphi\}to computeqφ\(C\|P,A\)q\_\{\\varphi\}\(C\|P,A\), which is its estimate for the likelihood of the reasoning given the prompt and generated answer\. Concretely, the score we use is the length\-normalized log\-likelihood1\|C\|\+1logqφ\(C,eos∣P,A\)\\frac\{1\}\{\|C\|\+1\}\\log q\_\{\\varphi\}\(C,\\texttt\{eos\}\\mid P,A\), computed in a single forward pass; since attention among CoT tokens is causal \(Section[4](https://arxiv.org/html/2610.18878#S4)\), this is an ordinary autoregressive sequence likelihood\. Unlike the distillation application, hereqφq\_\{\\varphi\}is deployed at test time; this requires no groundtruth answer sinceqφq\_\{\\varphi\}scores the answer thatpθp\_\{\\theta\}itself generated\. Because of the contrastive term \(eq\.[3](https://arxiv.org/html/2610.18878#S2.E3)\) used in trainingqφq\_\{\\varphi\}, it is reasonable to expect that this number correlates with the correctness ofAA\. In other words, we can think of the function\(P,C,A\)↦qφ\(C\|P,A\)\(P,C,A\)\\mapsto q\_\{\\varphi\}\(C\|P,A\)as similar to the trained verifiers of[Cobbe et al\. \(2021\)](https://arxiv.org/html/2610.18878#bib.bib9)\. In Section[4\.3](https://arxiv.org/html/2610.18878#S4.SS3)we measure its performance as both a best\-of\-n and adaptive\([Aggarwal et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib7)\)sampler\.
## 3Comparison to related work
RAVR\([Lin et al\., 2026](https://arxiv.org/html/2610.18878#bib.bib12)\)and CoVRL\([Wen et al\., 2025](https://arxiv.org/html/2610.18878#bib.bib3)\), which also take a variational inference approach, are most similar to our work but their focus is on RL instead of SFT: they use the answer\-conditioned model to improve the sample efficiency of online algorithms such as GRPO, and still require continuously sampling multiple rollouts per prompt from the current policy during training\. Because of our contrastive loss \(eq\.[3](https://arxiv.org/html/2610.18878#S2.E3)\), our method also requires multiple rollouts per prompt for trainingqφq\_\{\\varphi\}, but this is incurred once up\-front and the sameqφq\_\{\\varphi\}can be used across fresh datasets as we show in Section[4\.2](https://arxiv.org/html/2610.18878#S4.SS2)\. Consistent with their findings, our results in Section[4\.1](https://arxiv.org/html/2610.18878#S4.SS1)show thatqφq\_\{\\varphi\}allows for more usable traces per generation compared to sampling frompθp\_\{\\theta\}alone\. We also highlight that both RAVR and CoVRL use the same LLM for both the forward and inverse model, using different prompt templates to switch between the two modes\. We, on the other hand, train a separate inverse model, which as we show additionally serves as an accurate verifier for the forward model \(Section[4\.3](https://arxiv.org/html/2610.18878#S4.SS3)\)\.
STaR\([Zelikman et al\., 2022](https://arxiv.org/html/2610.18878#bib.bib14)\)and TRICE\([Phan et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib13)\)are early works that used rejection sampling and answer\-conditioned reasoning\. STaR is an iterative finetuning algorithm built on rejection sampling andrationalization, which provides reasoning conditioned on the groundtruth answer, while TRICE treats CoT as a latent variable as we do\. Neither trains a separate inverse model to get an amortized posterior: STaR does in\-context proposals by injecting the answer into the prompt, and TRICE utilizes a non\-amortized Markov\-chain Monte Carlo expectation\-maximization algorithm with a memory bank\.
From a data\-efficiency perspective, we view our work as a middle ground between GRPO\-style reinforcement learning \(e\.g\. RAVR/CoVRL\) and the finetuning technique of STaR\. The former techniques are able to take advantage of rollouts that do not get the correct answer \(indeed, to get a learning signal from GRPO on the binary tasks we have been considering requires groups that have both correct and incorrect answers\), while pure SFT does not\. In our approach, these mixed rollouts are essential for the contrastive term that is needed to effectively trainqφq\_\{\\varphi\}\.
## 4Experiments and Results
We use two environments: GSM8K\([Cobbe et al\., 2021](https://arxiv.org/html/2610.18878#bib.bib9)\)and Countdown\([Gandhi et al\., 2024](https://arxiv.org/html/2610.18878#bib.bib6)\)\. We recall that the Countdown task involves finding an arithmetic expression that usesnngiven numbers and evaluates to a target number\. We programmatically generate the Countdown datasets we use, takingn=4n=4and generating 50,000 examples\. We then break this into an 80/10/10 train/val/test split\. For GSM8K we use the official train/test split, with no validation set carved out, and use the test set only for final reporting\.
For each environment, we begin with the pre\-trained Qwen3\-0\.6B model\([Yang et al\., 2025](https://arxiv.org/html/2610.18878#bib.bib10)\)and train it via GRPO on the train dataset until performance saturates, forming the forward modelpθp\_\{\\theta\}for all experiments\. Our interest is in what can be extracted*after*RL has plateaued, when further GRPO yields diminishing returns\. We then trainqφq\_\{\\varphi\}on this dataset as described in Section[2\.2](https://arxiv.org/html/2610.18878#S2.SS2)\.
While there are no real restrictions onqφq\_\{\\varphi\}architecturally, in our experiments we takeqφq\_\{\\varphi\}to be LoRA\([Hu et al\., 2022](https://arxiv.org/html/2610.18878#bib.bib8)\)\-finetuned from a copy ofpθp\_\{\\theta\}\. Separately, we drop the causality restriction on the attention layers within the prefix, which is the concatenation\[P;A\]\[P;A\]\. Explicitly, all tokens within\[P;A\]\[P;A\]attend to one another, while the CoT generation is causal, i\.e\. those tokens attend to\[P;A\]\[P;A\]and the preceding CoT tokens\. The same mask is used when trainingqφq\_\{\\varphi\}and when scoring with it as a verifier\.
Full details of our training datasets and hyperparameters can be found in Appendix[A](https://arxiv.org/html/2610.18878#A1)\.
### 4\.1FCR ofqφq\_\{\\varphi\}
The core metric we use to evaluateqφq\_\{\\varphi\}is theforward consistency rate at n \(FCR@n\\text\{FCR@\}n\)\. This is the proportion of prompts wherepθp\_\{\\theta\}generates a correct answer out ofnncompletions when conditioned on the CoT generated byqφq\_\{\\varphi\}\(the filter used when building training data, described in Section[2\.3\.1](https://arxiv.org/html/2610.18878#S2.SS3.SSS1), is the greedy variant of this check\)\. Of course, this alone is not a complete measure ofqφq\_\{\\varphi\}: ifqφq\_\{\\varphi\}just repeated the answer it is conditioned on then this makespθp\_\{\\theta\}’s job easy and FCR will be high\. However, this metric is important since when usingqφq\_\{\\varphi\}to finetunepθp\_\{\\theta\}we only use those CoT that are forward consistent\. In Table[1](https://arxiv.org/html/2610.18878#S4.T1)we report the FCR scores for the inverse models trained on each environment, at various values ofnn\. To show thatqφq\_\{\\varphi\}can extend the ability ofpθp\_\{\\theta\}, we also report FCR on the subset of prompts wherepθp\_\{\\theta\}by itself generates the correct answer at most 25% of the time, which we label asFCR\-hard\.
The results for Countdown and GSM8K are in Table[1](https://arxiv.org/html/2610.18878#S4.T1)\. We see thatqφq\_\{\\varphi\}drastically increases the set of solvable problems in Countdown, while for GSM8K the gains are concentrated on the hard subset \(and are significant, though not as extreme\)\. We believe one of the main reasons for the discrepancy is that an answer in Countdown, being a full arithmetic expression, exposes much more of the solution’s structure to reason backwards from than the single number that constitutes a GSM8K answer\. This does not makeqφq\_\{\\varphi\}vacuous on Countdown: its reasoning is trained to matchpθp\_\{\\theta\}’s and, crucially, it transfers: as we see in the following distillation experiments, finetuningpθp\_\{\\theta\}on it gives a significant improvement\. Appendix[C](https://arxiv.org/html/2610.18878#A3)shows a qualitative example of this on a prompt thatpθp\_\{\\theta\}never solves on its own:qφq\_\{\\varphi\}’s CoT is a non\-trivial backward decomposition of the answer, and after distillationpθp\_\{\\theta\}reproduces the same strategy unaided\. A second factor is that GSM8K simply has less room for improvement, with the baselinepθp\_\{\\theta\}already at 0\.94 pass@32\. For our main application of distillingqφq\_\{\\varphi\}intopθp\_\{\\theta\}, what matters most is the high FCR on hard prompts, i\.e\. extending the set of prompts from which a correct\-leading CoT can be obtained at all\. When we run SFT with mix ratioμ\>0\\mu\>0we sample CoT from bothpθp\_\{\\theta\}andqφq\_\{\\varphi\}, so prompts thatpθp\_\{\\theta\}solves but that are not forward consistent underqφq\_\{\\varphi\}are not lost\.
Table 1:Forward consistency rates ofqφq\_\{\\varphi\}compared with thepθp\_\{\\theta\}baseline \(where FCR is just the pass rate\) on all prompts and on the hard subset \(pθp\_\{\\theta\}correct at most 25% of the time\)\.
### 4\.2Distillation fromqφq\_\{\\varphi\}topθp\_\{\\theta\}
Here we do two general experiments\. In the first, to show the generalizability ofqφq\_\{\\varphi\}, we distill topθp\_\{\\theta\}using a dataset thatqφq\_\{\\varphi\}was not trained on\. This mimics the important real\-world continual learning situation where new prompt\-answer pairs arrive and we want to efficiently updatepθp\_\{\\theta\}\.
In the second experiment we distill on the same dataset thatqφq\_\{\\varphi\}\(and, via GRPO,pθp\_\{\\theta\}\) was trained on\. With the data held fixed, this experiment isolates how much additional learningqφq\_\{\\varphi\}can extract from a dataset on which GRPO performance has already saturated\.
#### 4\.2\.1Fresh data
Because of the small dataset size of GSM8K and it only having a train/test split, for this experiment we only use the Countdown dataset\. Let𝒟0\\mathcal\{D\}\_\{0\}be the dataset thatqφq\_\{\\varphi\}was trained on and𝒟1\\mathcal\{D\}\_\{1\}be a fresh dataset \(so𝒟′=𝒟1\\mathcal\{D\}^\{\\prime\}=\\mathcal\{D\}\_\{1\}in the notation of Section[2\.3\.1](https://arxiv.org/html/2610.18878#S2.SS3.SSS1)\)\.𝒟1\\mathcal\{D\}\_\{1\}consists of 40,000 Countdown problems generated with the same procedure and parameters as𝒟0\\mathcal\{D\}\_\{0\}but a different random seed, deduplicated against all splits of𝒟0\\mathcal\{D\}\_\{0\}so that no problem \(as a multiset of numbers and target\) appears in both\. This dataset is used for training only and all evaluations are on𝒟0\\mathcal\{D\}\_\{0\}’s test split\. The up\-front generation on𝒟1\\mathcal\{D\}\_\{1\}consists of 32 rollouts per prompt frompθp\_\{\\theta\}\(which supplies𝒟RSpθ′\\mathcal\{D\}^\{\\prime\}\_\{RS\_\{p\_\{\\theta\}\}\}, the all\-correct filter of Section[2\.3\.1](https://arxiv.org/html/2610.18878#S2.SS3.SSS1), and the per\-prompt difficulty estimates used in evaluation\) and one answer\-conditioned CoT per prompt fromqφq\_\{\\varphi\}\. To study the effect of the mix ratio, we run distillation for eachμ∈\{0,0\.25,0\.5,0\.75,1\}\\mu\\in\\\{0,0\.25,0\.5,0\.75,1\\\}\. Note thatμ=0\\mu=0is rejection sampling\([Yuan et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib1)\), which we use as a baseline to compare against\. For other baseline comparisons we use the performance ofpθp\_\{\\theta\}before distillation and ofpθp\_\{\\theta\}further finetuned, using GRPO, on𝒟1\\mathcal\{D\}\_\{1\}\.
We evaluate with pass@kk, the probability that at least one ofkksampled completions answers correctly, and its restriction to the hard subset defined above, hard pass@kk\. For all runs in this experiment, we choose each run’s checkpoint by its best hard pass@8 over𝒟0\\mathcal\{D\}\_\{0\}’s val split, and then report all metrics from evaluating that checkpoint once on the held\-out test split\. We repeat this for each experiment with two seeds and report the mean and standard deviation in Table[2](https://arxiv.org/html/2610.18878#S4.T2)\.
Distillation withμ≥0\.5\\mu\\geq 0\.5is strongest on every pass@8 and pass@32 metric, while the baselines retain an edge at pass@1 \(rejection sampling on all prompts, continued GRPO on the hard subset\)\. This split is consistent with what each method optimizes: training only on the model’s own reachable solutions \(rejection sampling\) or sharpening the current policy’s modes \(GRPO\) improves single\-sample precision, whereas distillingqφq\_\{\\varphi\}’s traces adds newly solvable prompts whose solutions are initially sampled with low probability, a gain that surfaces at largerkk\.
Table 2:Distillation on fresh Countdown data \(𝒟1\\mathcal\{D\}\_\{1\}\)\. Rows with mix ratioμ\>0\\mu\>0are our method;μ=0\\mu=0is the rejection\-sampling baseline, and we also compare against continued GRPO training on𝒟1\\mathcal\{D\}\_\{1\}and the starting modelpθp\_\{\\theta\}\. We report mean±\\pmstandard deviation over two seeds\.##### Efficiency
Our method is clearly more data\-efficient than traditional SFT techniques that require groundtruth CoT traces, and in this section we highlight the structural properties of it that make it more compute\-efficient than policy gradient methods \(such as GRPO\) from a continual learning perspective, where the recurring cost is ingesting each new batch of prompt\-answer data\.
*Generation cost scales with data, not with training\.*Every completion GRPO learns from must be sampled from the current policy, inside the training loop: prompts are re\-rolled\-out at every visit, so generation cost grows with training length \(and dominated it in our runs\)\. Our pipeline computes rollouts for each batch exactly once, and the artifacts are reused by every subsequent run on that batch \(e\.g\. all ten SFT runs behind Table[2](https://arxiv.org/html/2610.18878#S4.T2)used the same rollout data\)\. The inverse model itself is a one\-time cost:qφq\_\{\\varphi\}was trained on𝒟0\\mathcal\{D\}\_\{0\}and applied to𝒟1\\mathcal\{D\}\_\{1\}with no retraining\.
*Generation decouples from training\.*Because the SFT dataset is fixed by design, its rollouts never go stale: they can be produced on separate, inference\-optimized hardware concurrently with training\. Offline generation also batches continuously over the entire dataset at full GPU occupancy, whereas on\-policy generation is capped at one group per prompt and drains as short completions finish, which leads to significantly less throughput\.
*Training memory scales with one sequence, not a group\.*A GRPO update computes gradients through the log\-probabilities of every completion in a group, so group size is capped by training memory\. SFT backpropagates through a single CoT per example, however many rollouts were sampled offline\.
*Compute converts into signal on the hard tail\.*On the 17% of fresh prompts thatpθp\_\{\\theta\}never solves, every GRPO rollout group is uniformly incorrect and the gradient is identically zero however long training runs, whileqφq\_\{\\varphi\}produces forward\-consistent CoT for 30% of exactly those prompts\. Consistently, continued GRPO plateaued below theμ≥0\.5\\mu\\geq 0\.5distillation arms on hard pass@8 and pass@32 \(Table[2](https://arxiv.org/html/2610.18878#S4.T2)\) despite running for more than twice the RL that producedpθp\_\{\\theta\}\.
#### 4\.2\.2Original data
These experiments are the same as above except we distill on𝒟0\\mathcal\{D\}\_\{0\}, the datasetqφq\_\{\\varphi\}was trained on, and experiment on GSM8K as well as Countdown\. For metrics we show held\-out evaluation curves over training checkpoints \(Figure[2](https://arxiv.org/html/2610.18878#S4.F2)\) and report in Table[3](https://arxiv.org/html/2610.18878#S4.T3)the mean over the last 30% of checkpoints\. On GSM8K the hard\-tail gains over rejection sampling are smaller \(at most\+0\.03\+0\.03hard pass@32\) and several cells tie, while pass@1 falls below both rejection sampling and the starting model at high mix ratios, the same precision\-versus\-coverage tradeoff seen on fresh data\. Figure[6](https://arxiv.org/html/2610.18878#A3.F6)in Appendix[C](https://arxiv.org/html/2610.18878#A3)traces a held\-out GSM8K prompt through this experiment where the startingpθp\_\{\\theta\}fails on all 32 rollouts but the distilled model solves greedily, ruling out memorization\.
Table 3:Distillation on the original data \(𝒟0\\mathcal\{D\}\_\{0\}\) for GSM8K \(top; evaluated on the standard held\-out GSM8K test set\) and Countdown \(bottom; evaluated on the held\-out test split\)\. Values are the mean over the last 30% of training checkpoints, reported as mean±\\pmstandard deviation over two seeds\.\(a\)GSM8K, hard pass@8
\(b\)GSM8K, pass@8
\(c\)Countdown, hard pass@8
\(d\)Countdown, pass@8
Figure 2:Held\-out evaluation curves over training checkpoints on GSM8K \(top\) and Countdown \(bottom\), for each mix ratioμ\\muand the rejection\-sampling baseline \(μ=0\\mu=0\)\. Shaded bands show±1\\pm 1standard deviation across the two seeds; the dashed line is the initial modelpθp\_\{\\theta\}\. Curves for the remaining values ofkkare in Appendix[B](https://arxiv.org/html/2610.18878#A2)\.
### 4\.3qφq\_\{\\varphi\}\-verifier results
We now evaluate the second application ofqφq\_\{\\varphi\}, as a test\-time verifier that selects among candidate completions ofpθp\_\{\\theta\}\. We do two experiments: a fixed\-budget best\-of\-nncomparison and an adaptive variant\([Aggarwal et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib7)\)where sampling stops early once a confidence condition is met\.
#### 4\.3\.1Best\-of\-nn
For baselines we take the following:
- •Majority vote\([Wang et al\., 2022](https://arxiv.org/html/2610.18878#bib.bib2)\)From thennsamples return the answer that appeared most often\.
- •Shortest CoTMany incorrect responses, especially for Countdown, have long CoT, as the model has to try a bunch of different ideas from scratch trying to get the correct answer\. As such, we consider the selector that from thennsamples returns the one with shortest CoT sequence\.
- •pθp\_\{\\theta\}’s self\-confidenceThe log probs ofpθp\_\{\\theta\}’s completion of the prompt\.
Our results are in Table[5](https://arxiv.org/html/2610.18878#S4.T5)\. We restrict to*mixed*prompts, i\.e\. those whosennsamples contain both a correct and an incorrect answer, since on the remaining prompts every selector scores identically and including them would only dilute the comparison\. On Countdown, selecting withqφq\_\{\\varphi\}cuts the selection error of majority vote from 24\.7% to 3\.1%\. On GSM8K, the aggregate difference betweenqφq\_\{\\varphi\}and majority vote \(0\.765 vs\. 0\.754 on 472 prompts\) is within noise at this sample size; Table[5](https://arxiv.org/html/2610.18878#S4.T5)shows where they differ\. Stratifying by whether the correct completions form a minority of the samples \(i\.e\. whetherpθp\_\{\\theta\}’s consensus is wrong\), we see thatqφq\_\{\\varphi\}’s advantage concentrates precisely on the correct\-minority prompts, where majority vote is by construction at or below random\. This is the same hard\-tail regime targeted by our distillation experiments, and exactly where a verifier is most valuable\. On correct\-majority prompts voting is near\-perfect andqφq\_\{\\varphi\}adds nothing \(slightly losing to it on GSM8K\)\.
Table 4:Selection accuracy of best\-of\-nntechniques forn=32n=32, on mixed prompts \(those with at least one correct and one incorrect sample; 3,483 for Countdown, 472 for GSM8K\)\.
Table 5:Best\-of\-nnselection accuracy stratified by whether correct completions are a minority among the prompt’s samples, i\.e\. whetherpθp\_\{\\theta\}’s consensus is wrong\.
#### 4\.3\.2Adaptive best\-of\-nn
Adaptive best\-of\-nntechniques\([Aggarwal et al\., 2023](https://arxiv.org/html/2610.18878#bib.bib7)\)involve the specification of a stopping criterion so that an answer may be returned before generating allnncompletions\. A thresholdτ\\tauon the verifier determines such a condition by returning the first generated answer withqφ\(C\|P,A\)\>τq\_\{\\varphi\}\(C\|P,A\)\>\\tau\. For baselines, we take adaptive analogs of majority vote and of the shortest\-CoT selector: for a positive integermm, we stop as soon as the most common answer appearsmmmore times than any other; and for a lengthLL, we return the first completion whose CoT has at mostLLtokens \(falling back to the shortest seen\)\. To make a fair comparison, we sweepτ\\tau,mm, andLLrespectively and, for each setting, measure both the selection accuracy and the expected number of completions drawn per prompt, using the same random completion orderings for all three\. Table[6](https://arxiv.org/html/2610.18878#S4.T6)compares the methods at matched expected budgets\.
On Countdown,qφq\_\{\\varphi\}at∼2\{\\sim\}2expected draws already exceeds adaptive majority vote at*any*budget\. The length rule is the stronger baseline on Countdown, beating the vote margin at every budget, yetqφq\_\{\\varphi\}leads it by 6–13 points throughout, so the advantage is not a length artifact; on GSM8K the length rule is weak, consistent with Table[5](https://arxiv.org/html/2610.18878#S4.T5)\. Against majority vote on GSM8K a significant advantage is confined to small budgets\. The structural reason is that a vote margin is a population signal, in particular it cannot trigger beforemmdraws and may never build up when answers scatter\. However a verifier score is a per\-sample signal that can trigger after a single draw\.
Table 6:Adaptive best\-of\-nn: selection accuracy at matched expected numbers of completions drawn per prompt, for theqφq\_\{\\varphi\}\-threshold stopping rule versus the adaptive majority\-vote \(margin\) and length\-threshold \(length\) baselines, on mixed prompts\.
## 5Conclusion and future work
In this work we demonstrated the utility of treating CoT tokens as latent variables and training an inverse modelqφq\_\{\\varphi\}that generates reasoning conditioned on both the prompt and groundtruth answer\. Our experiments show that after RL training has saturated, cheap SFT training usingqφq\_\{\\varphi\}\-generated traces recovers substantial performance on the hard tail of prompts\. These are problems that rejection sampling discards entirely because the forward model never solves them\. Further, the effectiveness of this SFT technique extends to fresh answer\-only data, making the approach suitable as an offline, continual\-learning update that avoids the per\-step rollout cost of policy\-gradient methods such as GRPO\. We have also shown thatqφq\_\{\\varphi\}is a suitable verifier for correctness, enabling efficient best\-of\-nnand adaptive test\-time selection precisely in the regime where majority voting fails\.
In terms of sample efficiency, our method occupies a middle ground between RL and SFT: the mixed correct/incorrect rollouts that GRPO needs at every training step are needed only once, up front, to trainqφq\_\{\\varphi\}via the contrastive objective\. All subsequent learning is supervised\. We also note that little in the method is specific to LLMs or to CoT: the same recipe applies to any latent\-variable sequence model where an observed outcome can be conditioned on to propose the latent\.
For future work, a natural extension is to iterate: refreshqφq\_\{\\varphi\}against the updatedpθp\_\{\\theta\}and alternate the two updates, which places the method in the expectation\-maximization/wake\-sleep family, with the single\-step version presented here as one outer step\. Relatedly, generating multiple CoT candidates per prompt would permit self\-normalized importance weighting in place of our hard forward\-consistency filter\. Finally, scaling the study to larger models and harder environments, and usingqφq\_\{\\varphi\}\-synthesized traces to re\-seed RL on the hard tail where GRPO currently receives no gradient signal, are both promising directions\.
## References
- Aggarwalet al\.\(2023\)P\. Aggarwal, A\. Madaan, Y\. Yang,et al\.Let’s sample step by step: adaptive\-consistency for efficient reasoning and coding with llms\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 12375–12396\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p4.1),[§2\.3\.2](https://arxiv.org/html/2610.18878#S2.SS3.SSS2.p1.1),[§4\.3\.2](https://arxiv.org/html/2610.18878#S4.SS3.SSS2.p1.1),[§4\.3](https://arxiv.org/html/2610.18878#S4.SS3.p1.1)\.
- Cappéet al\.\(2005\)O\. Cappé, E\. Moulines, and T\. RydénInference in hidden markov models\.Springer\.Cited by:[§2\.1](https://arxiv.org/html/2610.18878#S2.SS1.p3.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p4.1),[§1](https://arxiv.org/html/2610.18878#S1.p5.1),[§2\.3\.2](https://arxiv.org/html/2610.18878#S2.SS3.SSS2.p1.1),[§4](https://arxiv.org/html/2610.18878#S4.p1.1)\.
- Gandhiet al\.\(2024\)K\. Gandhi, D\. Lee, G\. Grand, M\. Liu, W\. Cheng, A\. Sharma, and N\. D\. GoodmanStream of search \(sos\): learning to search in language\.arXiv preprint arXiv:2404\.03683\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p5.1),[§4](https://arxiv.org/html/2610.18878#S4.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.Lora: low\-rank adaptation of large language models\.\.Iclr1\(2\),pp\. 3\.Cited by:[§4](https://arxiv.org/html/2610.18878#S4.p3.1)\.
- Lambertet al\.\(2024\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p1.1)\.
- Linet al\.\(2026\)T\. Lin, X\. Zhao, X\. Zhang, R\. Long, Y\. Xu, Z\. Jiang, W\. Su, and B\. ZhengRAVR: reference\-answer\-guided variational reasoning for large language models\.InFindings of the Association for Computational Linguistics: ACL 2026,M\. Liakata, V\. P\. Moreira, J\. Zhang, and D\. Jurgens \(Eds\.\),San Diego, California, United States,pp\. 18757–18775\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.findings-acl.936),ISBN 979\-8\-89176\-395\-1,[Link](https://aclanthology.org/2026.findings-acl.936/)Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p2.1),[§3](https://arxiv.org/html/2610.18878#S3.p1.1)\.
- Phanet al\.\(2023\)D\. Phan, M\. D\. Hoffman, D\. Dohan, S\. Douglas, T\. A\. Le, A\. Parisi, P\. Sountsov, C\. Sutton, S\. Vikram, and R\. A SaurousTraining chain\-of\-thought via latent\-variable inference\.Advances in Neural Information Processing Systems36,pp\. 72819–72841\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p2.1),[§3](https://arxiv.org/html/2610.18878#S3.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p1.1)\.
- Wanget al\.\(2022\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[1st item](https://arxiv.org/html/2610.18878#S4.I1.i1.p1.1)\.
- Weiet al\.\(2023\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. ZhouChain\-of\-thought prompting elicits reasoning in large language models\.External Links:2201\.11903,[Link](https://arxiv.org/abs/2201.11903)Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p1.1)\.
- Wenet al\.\(2025\)X\. Wen, J\. Lou, Y\. Liu, H\. Lin, B\. He, X\. Han, L\. Sun, Y\. Lu, and D\. ZhangCoupled variational reinforcement learning for language model general reasoning\.arXiv preprint arXiv:2512\.12576\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p2.1),[§3](https://arxiv.org/html/2610.18878#S3.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p5.1),[§4](https://arxiv.org/html/2610.18878#S4.p2.1)\.
- Yuanet al\.\(2023\)Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. ZhouScaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.Cited by:[§4\.2\.1](https://arxiv.org/html/2610.18878#S4.SS2.SSS1.p1.1)\.
- Zelikmanet al\.\(2022\)E\. Zelikman, Y\. Wu, J\. Mu, and N\. GoodmanStar: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[§1](https://arxiv.org/html/2610.18878#S1.p2.1),[§3](https://arxiv.org/html/2610.18878#S3.p2.1)\.
## Appendix ATraining details
### A\.1GRPO training ofpθp\_\{\\theta\}
Table[7](https://arxiv.org/html/2610.18878#A1.T7)summarizes the hyperparameters used for GRPO\. They are shared by the initial training ofpθp\_\{\\theta\}on each environment and by the continued\-GRPO baseline of the fresh\-data experiment; the latter is an exact continuation of the original run—same hyperparameters, with optimizer state restored—on the fresh dataset𝒟1\\mathcal\{D\}\_\{1\}\. Unlikeqφq\_\{\\varphi\}and the distillation SFT below, GRPO updates all model weights: in preliminary experiments we found GRPO training with LoRA to be unstable\.
Table 7:Hyperparameters used for GRPO training ofpθp\_\{\\theta\}\(both the initial training and the continued\-GRPO baseline\)\.
### A\.2Trainingqφq\_\{\\varphi\}
The hyperparameters used to trainqφq\_\{\\varphi\}are summarized in Table[8](https://arxiv.org/html/2610.18878#A1.T8)\. Heremmis the margin andλ\\lambdathe weight of the contrastive loss term \(see eq\.[3](https://arxiv.org/html/2610.18878#S2.E3)\)\.
HyperparameterValue*Contrastive loss*Marginmm1Loss weightλ\\lambda1Normalize by sequence lengthYes*LoRA*Rankrr32α\\alpha64*Optimization*OptimizerAdamLearning rate2×10−42\\times 10^\{\-4\}Batch size32Weight decay0\.01Gradient clipping10Precisionbf16*Sampling*Temperature1Max CoT tokens600Group sizegg32Table 8:Hyperparameters used for trainingqφq\_\{\\varphi\}\.We first trainqφq\_\{\\varphi\}for Countdown and saw FCR scores on the val split plateau around optimization step 25,000\. When training the smaller dataset GSM8K, which only has a train/test split, we therefore did not do any train\-time validation and simply took the checkpoint at step 25,000 forqφq\_\{\\varphi\}\.
### A\.3Distillingqφq\_\{\\varphi\}intopθp\_\{\\theta\}
The hyperparameters used for the distillation SFT are summarized in Table[9](https://arxiv.org/html/2610.18878#A1.T9), and are shared across both the Countdown and GSM8K experiments\. The only hyperparameter we vary is the mix ratioμ\\mu, which controls the fraction of CoTs sampled fromqφq\_\{\\varphi\}versuspθp\_\{\\theta\}\.
HyperparameterValue*LoRA*Rankrr16α\\alpha32*Optimization*OptimizerAdamLearning rate5×10−45\\times 10^\{\-4\}Batch size16Gradient clipping1Precisionbf16*Distillation*Mix ratioμ\\muvariedMax CoT tokens600Temperature1Table 9:Hyperparameters used for distillingqφq\_\{\\varphi\}intopθp\_\{\\theta\}\.
## Appendix BAdditional metrics
Figures[3](https://arxiv.org/html/2610.18878#A2.F3)and[4](https://arxiv.org/html/2610.18878#A2.F4)show the held\-out evaluation curves of the original\-data distillation experiment for the values ofkknot shown in the main text\.
\(a\)hard pass@1
\(b\)pass@1
\(c\)hard pass@32
\(d\)pass@32
Figure 3:Held\-out evaluation curves over training checkpoints on GSM8K atk∈\{1,32\}k\\in\\\{1,32\\\}, complementing Figure[2](https://arxiv.org/html/2610.18878#S4.F2)\. Shaded bands show±1\\pm 1standard deviation across the two seeds; the dashed line is the initial modelpθp\_\{\\theta\}\.\(a\)hard pass@1
\(b\)pass@1
\(c\)hard pass@32
\(d\)pass@32
Figure 4:Held\-out evaluation curves over training checkpoints on Countdown atk∈\{1,32\}k\\in\\\{1,32\\\}, complementing Figure[2](https://arxiv.org/html/2610.18878#S4.F2)\. Shaded bands show±1\\pm 1standard deviation across the two seeds; the dashed line is the initial modelpθp\_\{\\theta\}\.
## Appendix CQualitative examples
Figures[5](https://arxiv.org/html/2610.18878#A3.F5)and[6](https://arxiv.org/html/2610.18878#A3.F6)each trace a single hard prompt through our pipeline: a prompt the startingpθp\_\{\\theta\}never solves \(0 of 32 rollouts\), on whichqφq\_\{\\varphi\}nevertheless produces a forward\-consistent CoT, and which the distilledpθp\_\{\\theta\}then solves with greedy decoding\. The Countdown example \(Figure[5](https://arxiv.org/html/2610.18878#A3.F5)\) is a training prompt from the fresh\-data experiment \(Section[4\.2\.1](https://arxiv.org/html/2610.18878#S4.SS2.SSS1)\) and illustrates the transfer mechanism; the GSM8K example \(Figure[6](https://arxiv.org/html/2610.18878#A3.F6)\) is a*held\-out*prompt from the original\-data experiment \(Section[4\.2\.2](https://arxiv.org/html/2610.18878#S4.SS2.SSS2)\), so the distilled model’s success cannot be memorization\.
Prompt \(train\):Using the numbers \[75, 7, 7, 4\], create an equation that equals 244\. You can use basic arithmetic operations \(\+, \-, \*, /\) and each number exactly once\.*\(gold:4 \* \(\(75 \- 7\) \- 7\)\)*
- \(a\)pθp\_\{\\theta\}before distillation\(all 32 rollouts incorrect; the shortest is shown\)\. The arithmetic is correct, but the expression uses three 7s—a constraint violation that looks right: > First, 244 divided by 7 is about 34\.57, not helpful\. How about \(75 \* 4\) = 300, then minus 56\. How get 56 from 7 and 7: 7\*7=49, plus 7=56\. So \(75 \* 4\) \- \(7\*7 \+ 7\) = 300 \- 56 = 244\.
- \(b\)qφq\_\{\\varphi\}’s CoT, conditioned on the groundtruth answer4 \* \(\(75 \- 7\) \- 7\)\(forward\-consistent, and the actual SFT training datum\)\. It decomposes backward from the target: > First, 244 divided by 4 is 61\. How do I get 61 from the other numbers? 75 \- 14 = 61, but where do I get 14 from? 7 \+ 7 = 14\. So \(75 \- \(7 \+ 7\)\) = 75 \- 14 = 61, then multiply by 4: 61\*4=244\.
- \(c\)pθp\_\{\\theta\}after distillation, greedy decoding \(deterministic, and with no answer conditioning\), correct: > <think\> Okay, let’s see\. I need to get 244 using the numbers 75, 7, 7, and 4\. Hmm\. Let me think\. First, 244 divided by 4 is 61\. How can I get 61 from the other numbers: 75, 7, 7\. Let’s see\. 75 \- 7 = 68, minus 7 is 61\. So \(75 \- \(7 \+ 7\)\) \- 4 = 75 \- 14 = 61, then 61 \* 4 = 244\. </think\> <answer\> \(75 \- \(7 \+ 7\)\) \* 4 </answer\>
Figure 5:A qualitative example from the fresh\-data Countdown experiment\. The starting modelpθp\_\{\\theta\}fails on all 32 rollouts \(a\); given the answer,qφq\_\{\\varphi\}derives a plan backward from the target—divide by 4, then decompose 61 \(b\); after distillation,pθp\_\{\\theta\}executes the same divide\-the\-target strategy unaided \(c\)\. This is a training prompt, so \(b\) is the*actual SFT training datum*, letting us see the supervision and what was learned from it side by side\. Notably, even though the model trained on this very prompt, \(c\) is not a replay of \(b\): the student reaches 61 by a different route \(75−7−775\-7\-7rather than75−\(7\+7\)75\-\(7\+7\)\), slips and self\-corrects \(“\- 4” before “\* 4”\), and brackets the final expression differently—what transfers is the strategy, not the memorized sequence\. Held\-out generalization is established quantitatively in Table[2](https://arxiv.org/html/2610.18878#S4.T2)and qualitatively in Figure[6](https://arxiv.org/html/2610.18878#A3.F6)\. All model outputs are kept verbatim\.Prompt \(held out\):The local firefighters are doing a “fill the boot” fundraiser\. Their goal is to raise $6300\. After the first 3 hours, they have raised $2100\. For how many hours do they have to fundraise in total to reach their goal, assuming an equal amount raised in every hour?*\(gold: 9\)*
\(a\)pθp\_\{\\theta\}rollout*\(all 32 incorrect, all answering 6; shortest shown\)*:
> First, total amount needed:6300−2100=42006300\-2100=4200\. Each hour they raise an equal amount\. Time already used is 3 hours\. So, the amount per hour would be2100/3=$7002100/3=\\$700per hour\. Therefore, remaining hours needed would be4200/700=64200/700=6hours\. So answer is 6\.
*Correct arithmetic — but it answers “how many*more*hours,” not the total asked for\.*
\(b\)qφq\_\{\\varphi\}’s CoT*\(conditioned on the answer 9; the 1\-of\-8 sample passing the forward\-consistency check — shown for illustration: held\-out prompts contribute no training data\)*:
> First, total amount needed:6300−2100=42006300\-2100=4200\. Time already spent: 3 hours\. \[…\] So amount per hour would be4200/3=$14004200/3=\\$1400per hour\. \[…\] Wait, no\. Since they have already raised 2100 after 3 hours\. \[…\] amount per hour is same as2100/3=7002100/3=700per hour\. So4200/700=64200/700=6\. So number of hours needed is6\+3=96\+3=9\.
*Conditioning on 9 forces the derivation through the steppθp\_\{\\theta\}omits — via visible self\-correction\.*
\(c\) distilledpθp\_\{\\theta\}*\(greedy; correct\)*:
> Total needed: $6300\. Already raised: $2100\. So remaining amount needed:6300−2100=42006300\-2100=4200\. \[…\] Since after 3 hours they have raised 2100\. So, per hour, amount is2100/3=7002100/3=700\. \[…\] Therefore, remaining hours needed:4200/700=64200/700=6hours\. So total hours needed would be3\+6=93\+6=9hours\.⇒\\Rightarrow9
*Same rate computation, same intermediate 6 — plus the one question\-reading steppθp\_\{\\theta\}omits\.*
Figure 6:A held\-out GSM8K example from the original\-data experiment\. The startingpθp\_\{\\theta\}fails all 32 rollouts, unanimously answering 6—correct arithmetic for the number of*additional*hours, but not the total the question asks for \(a\)\. Conditioned on the gold answer 9,qφq\_\{\\varphi\}’s derivation is forced through the steppθp\_\{\\theta\}omits, with visible self\-correction \(b\); its forward\-consistency rate on this prompt \(1 of 8\) is in line with the hard\-prompt FCR@1 of 0\.15 in Table[1](https://arxiv.org/html/2610.18878#S4.T1)\. The distilledpθp\_\{\\theta\}solves the prompt greedily \(c\)\. Since the prompt is held out it contributed no training data—\(b\) is illustrative of the training signal on similar prompts—so memorization is impossible by construction\. Outputs abridged where marked \(\[…\]\)\.Similar Articles
Data-Centric Post-Training for Financial Reasoning: Mining, Distillation, and Verifiable Learning
This paper introduces a data-centric pipeline for post-training language models to enhance financial reasoning through mining reasoning traces, distilling instruction data, and generating verifiable QA pairs, demonstrating improvements in performance while preventing catastrophic forgetting.
Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning
Proposes BIRD, a two-stage self-reasoning distillation method that bootstraps concise reasoning trajectories before on-policy training, achieving stronger accuracy-efficiency trade-offs on MATH-500 and AIME benchmarks. On Qwen3-8B, accuracy improves from 86.2% to 92.0% while average response length drops from 3,099 to 1,115 tokens.
Answer-then-Edit: Reasoning Skeleton Editing for Anti-Distillation with Preserved Utility
This paper proposes SGRE, an Answer-then-Edit framework that modifies reasoning traces of LLMs to prevent unauthorized knowledge distillation while preserving answer accuracy and naturalness, drawing on cognitive load theory to increase extraneous load on student models.
Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline
Proposes Self-Verified Distillation, a method where LLMs generate and self-verify candidate solutions from unlabeled seed questions using prompt-based verification, then train on the filtered dataset, achieving significant gains on math, science, and coding benchmarks across Qwen3 models.
GUARD: Natural Forgetting in Large Reasoning Models via Guided Answer-Reasoning Distillation
The paper introduces GUARD, a method for natural forgetting in large reasoning models that uses guided answer-reasoning distillation to suppress unsafe or private content in chain-of-thought traces while preserving reasoning utility.