LeAct: Learning to Reason from Expert Actions

# LeAct: Learning to Reason from Expert Actions
Source: [https://arxiv.org/html/2607.21856](https://arxiv.org/html/2607.21856)
Ziran Yang Chengshuai Shi Raj Ghugare Benjamin Eysenbach Karthik Narasimhan Chi Jin Princeton University

###### Abstract

Modern reasoning models depend on reasoning data, today sourced from human annotations or distilled from stronger LLMs\. However, a rich and largely untapped source of supervision lies in expert systems \(e\.g\., game engines, classical planners, theorem provers\), which routinely produce near\-optimal actions across diverse domains\. But these experts are silent: they commit to an action without writing down the chain of thought \(CoT\) behind it\. Recovering that CoT as natural\-language reasoning would distill expert knowledge into a student that generalizes beyond the demonstrated actions\. We treat it as a latent variable and study how to recover it from the action alone\. Our approach,LeAct\(Learning to reason fromActions\), optimizes this latent variable: the student samples candidate CoTs for each expert action, and we retain those that measurably improve its own probability of recovering the action\. Across imperfect\-information games at multiple scales and a simulated robotics benchmark,LeActreaches the solver’s numerical floor on small enumerable games\. At larger scale, it is5×5\\timescloser to the solver than the strongest expert\-iteration baseline\. At Flop Hold’em \(∼109\\sim 10^\{9\}infosets\),LeActwins head\-to\-head by\+60\+60mbb/g, and on the robotics probe it is the only training recipe that improves on direct imitation\. We present a principled framework and the result: expert systems become a categorically new source of reasoning teachers for foundation models\.

## 1Introduction

Building chain\-of\-thought \(CoT\) data is now a crucial bottleneck for reasoning language models\[[33](https://arxiv.org/html/2607.21856#bib.bib49),[11](https://arxiv.org/html/2607.21856#bib.bib35)\]\. There is no shortage of work trying to scale beyond human annotation by automatically labeling CoT, either through distillation from stronger LLMs\[[14](https://arxiv.org/html/2607.21856#bib.bib31),[30](https://arxiv.org/html/2607.21856#bib.bib32),[11](https://arxiv.org/html/2607.21856#bib.bib35)\]or by bootstrapping a model’s own outputs\[[59](https://arxiv.org/html/2607.21856#bib.bib4),[10](https://arxiv.org/html/2607.21856#bib.bib5),[37](https://arxiv.org/html/2607.21856#bib.bib59)\]\. But these methods rarely close the loop directly on the CoT itself\. Filtering on synthetic CoT typically targets answer correctness or surface plausibility\. But whether the CoT actually contributes to the student’s prediction remains uncertain\.

We draw from a different source: silent action experts \(game solvers\[[62](https://arxiv.org/html/2607.21856#bib.bib24),[45](https://arxiv.org/html/2607.21856#bib.bib25),[5](https://arxiv.org/html/2607.21856#bib.bib26)\], classical planners, robotic planners\[[46](https://arxiv.org/html/2607.21856#bib.bib57)\], theorem provers\[[26](https://arxiv.org/html/2607.21856#bib.bib51)\]\)\. Each commits to a near\-optimal action at every state, but neither the action nor the reasoning behind it is expressed in natural language\. The naive approach, asking a student LLM to write a CoT for each expert action, runs into rationalization\[[50](https://arxiv.org/html/2607.21856#bib.bib65),[22](https://arxiv.org/html/2607.21856#bib.bib66)\]: the CoT can be plausible while bearing no causal link to the action it accompanies\. We close the loop instead\. For every state\-action pair, the student samples multiple candidate CoTs, and we retain only those that raise the student’s own probability of recovering the expert action\. The filter is the load\-bearing step: replacing it with random ranking erases the advantage in our ablations\.

LeAct\(Learning to reason fromActions\) realizes this idea as an iterative training pipeline \(Fig\.[1](https://arxiv.org/html/2607.21856#S1.F1)\)\. The conventional*reason\-then\-act*paradigm\[[56](https://arxiv.org/html/2607.21856#bib.bib16)\]takes reasoning as input and produces actions at inference time;LeActruns that loop in reverse at training time, conditioning on the expert action, sampling reasoning that recovers it, and training on the reasoning that survived the filter\. The recipe accepts any action\-only oracle: a CFR solver, a deep RL policy, or a frontier LLM committed to a single demonstration trajectory\. We test all three families:CFRsolvers on enumerable poker settings like Leduc Hold’em\[[44](https://arxiv.org/html/2607.21856#bib.bib33)\]; aDeepCFR\[[5](https://arxiv.org/html/2607.21856#bib.bib26)\]policy on Flop Hold’em \(∼109\\sim\\\!10^\{9\}infosets\); and frontier\-LLM successful trajectories on a simulated robotics benchmark\[[9](https://arxiv.org/html/2607.21856#bib.bib48)\]\.

![Refer to caption](https://arxiv.org/html/2607.21856v1/x1.png)Figure 1:LeActturns silent action experts into reasoning teachers\.For each statexx, the student samples candidate explanations conditioned on the oracle’s actionπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)\. We keep those that raise the student’s predicted probability of the action and use them as CoT to fine\-tune the student\.Contributions\.We make three: a new non\-LLM source of CoT supervision, a latent\-variable derivation of the algorithm, and empirical results on various games and a robotics benchmark\.

- •A non\-LLM data source for CoT supervision\.Rather than mining intermediate text\[[37](https://arxiv.org/html/2607.21856#bib.bib59),[59](https://arxiv.org/html/2607.21856#bib.bib4)\],LeActconverts a structured action policyπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)from any non\-LLM oracle into natural\-language reasoning data, a categorically distinct supervision modality from prior CoT pipelines\.
- •A principled algorithm\(§[3](https://arxiv.org/html/2607.21856#S3)\)\. We treat the CoT as a latent variable connecting stateXXto expert actionY∗∼π∗Y^\{\*\}\\sim\\pi^\{\*\}, and derive a learning procedure from this view\. The E\-step reduces to ranking candidate CoTs by how much each helps the student’s own decision\-making, and the M\-step prescribesπ∗\\pi^\{\*\}as the SFT target\.LeActsits in the recent line of latent\-EM self\-improvement\[[37](https://arxiv.org/html/2607.21856#bib.bib59)\], but anchors the latent to an external expert action rather than to text the model itself produced\.
- •Empirical results\(§[5](https://arxiv.org/html/2607.21856#S5)–[6](https://arxiv.org/html/2607.21856#S6)\)\. We testLeActon imperfect\-information games\[[20](https://arxiv.org/html/2607.21856#bib.bib43)\]at multiple scales and on a simulated robotics cube\-stacking benchmark\[[9](https://arxiv.org/html/2607.21856#bib.bib48)\]\. A student trained withLeActmatches the expert solver on small games, beats both behavior\-cloning and expert\-iteration baselines at the large games we test, and is the only recipe that improves on direct imitation in our robotics setting\. The advantage holds on unseen states, and we trace it to the explanation\-selection step\.

## 2Related Work

The closed\-loop feedback gap in CoT supervision\.There is no shortage of CoT\-supervision work: humans label rationales by hand\[[55](https://arxiv.org/html/2607.21856#bib.bib1),[17](https://arxiv.org/html/2607.21856#bib.bib2)\], stronger LLMs distill them into smaller students\[[14](https://arxiv.org/html/2607.21856#bib.bib31),[30](https://arxiv.org/html/2607.21856#bib.bib32),[11](https://arxiv.org/html/2607.21856#bib.bib35)\], and self\-improvement loops mine latent thoughts from text the model itself produces\[[59](https://arxiv.org/html/2607.21856#bib.bib4),[15](https://arxiv.org/html/2607.21856#bib.bib7),[58](https://arxiv.org/html/2607.21856#bib.bib8),[10](https://arxiv.org/html/2607.21856#bib.bib5),[42](https://arxiv.org/html/2607.21856#bib.bib6),[37](https://arxiv.org/html/2607.21856#bib.bib59)\]\. What none of these provide is a closed\-loop feedback signal that decides*which*CoTs are worth training on by checking them against an external authority\. Existing pipelines either skip the filter entirely \(annotation, distillation\), filter by binary correctness against a single verifiable answer\[[59](https://arxiv.org/html/2607.21856#bib.bib4),[10](https://arxiv.org/html/2607.21856#bib.bib5),[42](https://arxiv.org/html/2607.21856#bib.bib6)\], or filter by the model’s own likelihood\[[37](https://arxiv.org/html/2607.21856#bib.bib59),[7](https://arxiv.org/html/2607.21856#bib.bib58)\]\. In every case the supervision signal is either absent or LLM\-internal\.LeActfills this gap: each candidate CoT survives only if it measurably increases the student’s own probability of recovering an external oracle’s action, closing the loop with supervision that originates outside the LLM family\.

External verifiable feedback for training reasoning\.A second line of work uses an external oracle or verifier as the training signal, instead of \(or alongside\) text the LLM itself produced\. Pairing a generator with an external checker is a long\-running training recipe across game outcomes, formal verifiers, and theorem provers\[[41](https://arxiv.org/html/2607.21856#bib.bib10),[29](https://arxiv.org/html/2607.21856#bib.bib54),[40](https://arxiv.org/html/2607.21856#bib.bib55),[48](https://arxiv.org/html/2607.21856#bib.bib40),[16](https://arxiv.org/html/2607.21856#bib.bib41),[34](https://arxiv.org/html/2607.21856#bib.bib69),[25](https://arxiv.org/html/2607.21856#bib.bib50),[26](https://arxiv.org/html/2607.21856#bib.bib51),[61](https://arxiv.org/html/2607.21856#bib.bib56),[23](https://arxiv.org/html/2607.21856#bib.bib52),[31](https://arxiv.org/html/2607.21856#bib.bib14)\]\. In the LLM\-reasoning literature this signal takes three forms\. At the outcome level, reinforcement learning with verifiable rewards\[[39](https://arxiv.org/html/2607.21856#bib.bib34),[11](https://arxiv.org/html/2607.21856#bib.bib35),[19](https://arxiv.org/html/2607.21856#bib.bib36),[33](https://arxiv.org/html/2607.21856#bib.bib49),[13](https://arxiv.org/html/2607.21856#bib.bib11),[28](https://arxiv.org/html/2607.21856#bib.bib70),[18](https://arxiv.org/html/2607.21856#bib.bib71),[52](https://arxiv.org/html/2607.21856#bib.bib53)\]runs RL against a binary reward when the final answer matches ground truth\. Expert Iteration\[[2](https://arxiv.org/html/2607.21856#bib.bib9)\]and rejection sampling fine\-tuning\[[8](https://arxiv.org/html/2607.21856#bib.bib67),[57](https://arxiv.org/html/2607.21856#bib.bib37),[47](https://arxiv.org/html/2607.21856#bib.bib38)\]use the same outcome signal in an SFT loop: sample candidates, keep those the oracle endorses, fine\-tune on the survivors\. At the step level, process reward models\[[51](https://arxiv.org/html/2607.21856#bib.bib39),[24](https://arxiv.org/html/2607.21856#bib.bib19),[53](https://arxiv.org/html/2607.21856#bib.bib20),[60](https://arxiv.org/html/2607.21856#bib.bib21),[38](https://arxiv.org/html/2607.21856#bib.bib72)\]train a separate verifier and apply it to each reasoning step\.LeActuses a graded version of the Expert Iteration signal: each candidate CoT is scored by how much it raises the student’s probability of the oracle’s full action distribution, a continuous signal that works where the optimal action is mixed and no single token is the correct answer\. Conditioning the backward CoT on the expert action is a hindsight\-relabelling step\[[1](https://arxiv.org/html/2607.21856#bib.bib17),[27](https://arxiv.org/html/2607.21856#bib.bib18)\], re\-pairing each state with an outcome the model knows is correct\. A complementary line treats reasoning itself as inference\-time search\[[12](https://arxiv.org/html/2607.21856#bib.bib15),[43](https://arxiv.org/html/2607.21856#bib.bib68)\];LeActacts at training time, selecting which CoT samples become supervision rather than how to deploy them\.

## 3Method

The goal is to convert action\-only oracles into thinking traces for training LLMs\. Two obstacles: the reasoning tracezzis never observed, inherently a latent variable; and the observed actions are in structured game action space rather than text, modality misalignment\. We design a closed\-loop that adaptsRuanet al\.\[[37](https://arxiv.org/html/2607.21856#bib.bib59)\]’s proposal\-and\-score framework to our scope of action supervision instead of text backfilling\.

### 3\.1Problem Formulation

Setup\.We consider a decision\-making setting where statesx∈𝒳x\\in\\mathcal\{X\}are drawn from a task of interest, in which an action oracle \(aCFRsolver, a classical planner, a verifiable controller, or a trained policy\) supplies a per\-state expert action distributionπ∗\(⋅∣x\)\\pi^\{\*\}\(\\,\\cdot\\mid x\)\. The student LLM follows a reason\-then\-act paradigm, generating an intermediate chain\-of\-thoughtzzbefore committing to an actionyy; we model this explicitly by treatingzzas a latent reasoning variable and factoring the joint distribution aspθ​\(z,y∣x\)=pθ​\(z∣x\)​pθ​\(y∣x,z\)\.p\_\{\\theta\}\(z,\\,y\\mid x\)\\;=\\;p\_\{\\theta\}\(z\\mid x\)\\,p\_\{\\theta\}\(y\\mid x,\\,z\)\.Training data is\{\(xi,π∗\(⋅∣xi\)\)\}i\\\{\(x\_\{i\},\\pi^\{\*\}\(\\,\\cdot\\mid x\_\{i\}\)\)\\\}\_\{i\}: we observe states and expert policies but*no*reasoning traces\.

Objective\.Maximise the marginal log\-likelihood of expert actions under the student LLM,

𝒥​\(θ\)=𝔼x​𝔼y∼π∗\(⋅∣x\)​\[log⁡pθ​\(y∣x\)\],\\mathcal\{J\}\(\\theta\)\\;=\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\(\\cdot\\mid x\)\}\\\!\\bigl\[\\log p\_\{\\theta\}\(y\\mid x\)\\bigr\],\(1\)intractable becausepθ​\(y∣x\)=∑zpθ​\(z∣x\)​pθ​\(y∣x,z\)p\_\{\\theta\}\(y\\mid x\)=\\sum\_\{z\}p\_\{\\theta\}\(z\\mid x\)\\,p\_\{\\theta\}\(y\\mid x,z\)marginalises a latent natural\-language trace\. The outer expectation𝔼y∼π∗\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}is realised at the M\-step \(§[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\): closed\-form againstπ∗\\pi^\{\*\}when the oracle exposes a full distribution, and a Dirac mass when it commits to a single action\.

### 3\.2A View from Variational\-EM

Backward proposal and IWAE bound\.Introduce a backward proposalqbwd​\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\), instantiated by prompting the LLM to generate a CoT given\(x,y\)\(x,y\), and drawNNcandidatesz1,…,zN∼qbwd\(⋅∣x,y\)z\_\{1\},\\ldots,z\_\{N\}\\sim q\_\{\\mathrm\{bwd\}\}\(\\cdot\\mid x,y\)per state\. TheNN\-sample importance\-weighted bound\[[7](https://arxiv.org/html/2607.21856#bib.bib58)\], which we denoteℒN​\(θ\)\\mathcal\{L\}\_\{N\}\(\\theta\), is

𝒥​\(θ\)≥ℒN​\(θ\):=𝔼x​𝔼y∼π∗​𝔼z1\.\.N∼qbwd\(⋅∣x,y\)​\[log⁡1N​∑i=1Npθ​\(zi,y∣x\)qbwd​\(zi∣x,y\)\],\\mathcal\{J\}\(\\theta\)\\;\\geq\\;\\mathcal\{L\}\_\{N\}\(\\theta\)\\;:=\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\\,\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\(\\cdot\\mid x,y\)\}\\\!\\left\[\\log\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{p\_\{\\theta\}\(z\_\{i\},y\\mid x\)\}\{q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\}\\right\],\(2\)strictly tighter than the standard ELBO and exact asN→∞N\\\!\\to\\\!\\infty, holding for anyqbwdq\_\{\\mathrm\{bwd\}\}including a fixed LLM \(which sidesteps the variational gradient throughqbwdq\_\{\\mathrm\{bwd\}\}that theN=1N\{=\}1ELBO would require\)\.Burdaet al\.\[[7](https://arxiv.org/html/2607.21856#bib.bib58), Eq\. 8\]show that the gradient ofℒN​\(θ\)\\mathcal\{L\}\_\{N\}\(\\theta\)is a per\-sample weighted SFT update,

∇θℒN=𝔼x​𝔼y∼π∗​𝔼z1\.\.N∼qbwd​\[∑i=1Nw~i​∇θlog⁡pθ​\(zi,y∣x\)\],\\nabla\_\{\\theta\}\\,\\mathcal\{L\}\_\{N\}\\;=\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\\,\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\}\\\!\\left\[\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{i\}\\,\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(z\_\{i\},y\\mid x\)\\right\],\(3\)with normalised soft weightsw~i=wi/∑jwj\\widetilde\{w\}\_\{i\}=w\_\{i\}/\\sum\_\{j\}w\_\{j\}andwi=pθ​\(zi,y∣x\)/qbwd​\(zi∣x,y\)w\_\{i\}=p\_\{\\theta\}\(z\_\{i\},y\\mid x\)/q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\), andqbwdq\_\{\\mathrm\{bwd\}\}held fixed at the previous\-round parameters within each round, evolving across rounds through the shared M\-step updates\.

Factorising the IWAE weightwiw\_\{i\}into a task term and a proposal term\.Bayes’ rule factorises each per\-sample weightwiw\_\{i\}from Eq\. \([3](https://arxiv.org/html/2607.21856#S3.E3)\) as

wi=pθ​\(y∣x,zi\)pθ​\(y∣x\)⏟=exp⁡Δ​\(zi;x,y\)⋅pθ​\(zi∣x\)qbwd​\(zi∣x,y\)⏟ρ​\(zi;x,y\)⋅pθ​\(y∣x\),w\_\{i\}\\;=\\;\\underbrace\{\\frac\{p\_\{\\theta\}\(y\\mid x,z\_\{i\}\)\}\{p\_\{\\theta\}\(y\\mid x\)\}\}\_\{=\\;\\exp\\Delta\(z\_\{i\};\\,x,y\)\}\\;\\cdot\\;\\underbrace\{\\frac\{p\_\{\\theta\}\(z\_\{i\}\\mid x\)\}\{q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\}\}\_\{\\rho\(z\_\{i\};\\,x,y\)\}\\;\\cdot\\;p\_\{\\theta\}\(y\\mid x\),\(4\)withΔ​\(z;x,y\):=log⁡pθ​\(y∣x,z\)−log⁡pθ​\(y∣x\)\\Delta\(z;\\,x,y\):=\\log p\_\{\\theta\}\(y\\mid x,z\)\-\\log p\_\{\\theta\}\(y\\mid x\)\. The marginalpθ​\(y∣x\)p\_\{\\theta\}\(y\\mid x\)is constant iniiand cancels inw~i=wi/∑jwj\\widetilde\{w\}\_\{i\}=w\_\{i\}/\\sum\_\{j\}w\_\{j\}\. The two remaining factors live on opposite sides of the action–CoT modality boundary\.Δ\\Deltais a ratio over the structured expert actionyy\(a probability vector or discrete label whose task\-level meaning is independent of its language rendering\), and asks whetherziz\_\{i\}raises the student’s probability ofyy\(implementation in §[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\)\.ρ\\rhois a ratio over the free\-form CoTziz\_\{i\}, which has no such structured target; it measures whether the proposal exploitedyyin*generating*ziz\_\{i\}, not whetherziz\_\{i\}usefully*decides*yy\. We therefore approximatew~i∝exp⁡Δ​\(zi;x,y\)\\widetilde\{w\}\_\{i\}\\propto\\exp\\Delta\(z\_\{i\};\\,x,y\), retaining the task\-grounded factor; droppingρ\\rhois unavailable to latent\-text variants without an action anchor\[[37](https://arxiv.org/html/2607.21856#bib.bib59)\]\. §[6](https://arxiv.org/html/2607.21856#S6)verifies the resultingΔ\\Delta\-only filter is load\-bearing; App\.[A\.1](https://arxiv.org/html/2607.21856#A1.SS1)gives the bound\-level statement and the conditions under which theρ\\rho\-drop preserves ranking\.

LeAct objective\.Substitutingw~i∝eΔ​\(zi;x,y\)\\widetilde\{w\}\_\{i\}\\propto e^\{\\Delta\(z\_\{i\};\\,x,y\)\}into Eq\. \([3](https://arxiv.org/html/2607.21856#S3.E3)\) yields the \(approximate\) M\-step that defines theLeActtraining objective:

maxθ𝔼x𝔼y∼π∗\(⋅∣x\)𝔼z1\.\.N∼qbwd\[∑i=1Nw~ilog⁡pθ​\(zi,y∣x\)⏟SFT loss\],w~i=eΔ​\(zi;x,y\)∑j=1NeΔ​\(zj;x,y\)\.\\boxed\{\\;\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\(\\cdot\\mid x\)\}\\,\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\}\\\!\\Bigl\[\\,\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{i\}\\,\\underbrace\{\\log p\_\{\\theta\}\(z\_\{i\},\\,y\\mid x\)\}\_\{\\text\{SFT loss\}\}\\,\\Bigr\],\\quad\\widetilde\{w\}\_\{i\}\\;=\\;\\frac\{e^\{\\Delta\(z\_\{i\};\\,x,y\)\}\}\{\\sum\_\{j=1\}^\{N\}e^\{\\Delta\(z\_\{j\};\\,x,y\)\}\}\.\\;\}\(5\)In §[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)we approximate this softmax with a hard top\-KKfilter to recover a plain joint SFT loss\.

### 3\.3LeActin Practice

We now specify the three stages that produce the SFT dataset𝒟\\mathcal\{D\}and realise the M\-step: backward generation of CoT candidates, forward\-delta evaluation with top\-KKselection, and expert\-policy forcing combined with joint backward supervision\.

Backward generation \(E\-step proposal\)\.We instantiate the backward proposalqbwdq\_\{\\mathrm\{bwd\}\}by prompting the current student\. The prompt gives the statexxand a qualitative summaryπ~∗\\tilde\{\\pi\}^\{\*\}of the expert distributionπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)\(for example, “mostly fold with occasional calls”\), and asks the student to explain why this strategy is optimal\. We drawNNcandidatesz1,…,zNz\_\{1\},\\ldots,z\_\{N\}per state\. The qualitative redaction prevents the backward generator from copyingπ∗\\pi^\{\*\}verbatim from its prompt \(example traces in App\.[G](https://arxiv.org/html/2607.21856#A7)\)\. Conditioning on the expert action anchors the explanation in a known\-good answer rather than searching forward for one, applying the hindsight\-relabelling intuition \(§[2](https://arxiv.org/html/2607.21856#S2)\) to CoT supervision: the silent action becomes a teaching signal once paired with the explanation it licenses\.

Forward delta scoring and top\-KKselection\.For each candidateznz\_\{n\}we evaluateΔ​\(zn;x,y\)\\Delta\(z\_\{n\};x,y\)from Eq\. \([4](https://arxiv.org/html/2607.21856#S3.E4)\), with both terms read off the student’s parsed textual policy line over actions \(implementation and fidelity audit in App\.[A\.4](https://arxiv.org/html/2607.21856#A1.SS4)\)\. The conditionallog⁡pθ​\(y∣x,zn\)\\log p\_\{\\theta\}\(y\\mid x,z\_\{n\}\)uses the line decoded under the candidate trace; the baselinelog⁡pθ​\(y∣x\)=log⁡𝔼z∼pθ​\(z∣x\)​\[pθ​\(y∣x,z\)\]\\log p\_\{\\theta\}\(y\\mid x\)=\\log\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\(z\\mid x\)\}\\bigl\[p\_\{\\theta\}\(y\\mid x,z\)\\bigr\]is estimated by Monte Carlo fromMMself\-generated forward tracesz\(i\)∼pθ​\(z∣x\)z^\{\(i\)\}\\sim p\_\{\\theta\}\(z\\mid x\)\(no gold\-action conditioning\)\. We then form the per\-state averageΔ¯​\(z;x\)=𝔼y∼π∗​\[Δ​\(z;x,y\)\]\\bar\{\\Delta\}\(z;x\)=\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\[\\Delta\(z;x,y\)\]\. For oracles that commit to a single action rather than a distribution, the log\-likelihood terms inΔ​\(z;x,y\)\\Delta\(z;x,y\)are replaced with a heuristic action\-match reward againsty∗y^\{\*\}; the rest of the pipeline is unchanged\.

Carrying the softmax weightsw~i∝eΔi\\widetilde\{w\}\_\{i\}\\propto e^\{\\Delta\_\{i\}\}into the M\-step costsO​\(N⋅\|𝒳\|\)O\(N\\cdot\|\\mathcal\{X\}\|\)\. We sparsify in two complementary steps:

𝒵x=top​\-​K​\{zj∼qbwd:Δ¯​\(zj;x\)\>0\},𝒟=\{\(x,z\):x∈𝒳,z∈𝒵x\}\.\\mathcal\{Z\}\_\{x\}\\;=\\;\\mathrm\{top\}\\text\{\-\}K\\bigl\\\{z\_\{j\}\\sim q\_\{\\mathrm\{bwd\}\}:\\bar\{\\Delta\}\(z\_\{j\};\\,x\)\>0\\bigr\\\},\\qquad\\mathcal\{D\}\\;=\\;\\\{\(x,z\):x\\in\\mathcal\{X\},\\;z\\in\\mathcal\{Z\}\_\{x\}\\\}\.\(6\)The positive\-Δ¯\\bar\{\\Delta\}filter is a semantic admission criterion \(only baseline\-beating traces enter SFT\), not a soft\-weight approximation; it is what makes the round\-over\-round bound non\-decreasing\. The top\-KKindicator is the softmax truncation, cutting cost toO​\(K⋅\|𝒳\|\)O\(K\\cdot\|\\mathcal\{X\}\|\)\. The gate tightens across rounds: as the student improves, traces that merely match its current forward output drop out\.

Expert\-policy forcing and joint backward supervision \(M\-step\)\.Applying the top\-KKfilter from Eq\. \([6](https://arxiv.org/html/2607.21856#S3.E6)\) to the LeAct objective Eq\. \([5](https://arxiv.org/html/2607.21856#S3.E5)\) reduces the M\-step to a joint SFT lossℒ=ℒfwd\+ℒbwd\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{fwd\}\}\+\\mathcal\{L\}\_\{\\mathrm\{bwd\}\}on the selected traces𝒟\\mathcal\{D\}\.ℒfwd\\mathcal\{L\}\_\{\\mathrm\{fwd\}\}trains the forward student on inputsxxto emitzzfollowed by the oracle’s full action distributionπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)\. The analogy is teacher forcing in sequence modelling: there the ground\-truth token replaces the model’s own sample as the next\-step target; here the oracle’s distribution replaces any student\-decoded policy as the action target\. We call this*expert\-policy forcing*\(EPF\)\.ℒbwd\\mathcal\{L\}\_\{\\mathrm\{bwd\}\}retrains the backward proposal on inputs\(x,π~∗\)\(x,\\tilde\{\\pi\}^\{\*\}\)to emitzz, the reweighted\-wake\-sleep\[[4](https://arxiv.org/html/2607.21856#bib.bib60)\]update onqbwdq\_\{\\mathrm\{bwd\}\}\. EPF is closed\-form against a distributional oracle, and reduces to standard SFT on the single action when the oracle commits to one\. Across rounds,ℒbwd\\mathcal\{L\}\_\{\\mathrm\{bwd\}\}tracksqbwdq\_\{\\mathrm\{bwd\}\}to the posterior under the current student, monotonically tightening the IWAE bound\. Full formulae are in App\.[A\.2](https://arxiv.org/html/2607.21856#A1.SS2); iteration dynamics are in App\.[C](https://arxiv.org/html/2607.21856#A3)\.

Input :Expert oracleπ∗\\pi^\{\*\}, initial modelθ0\\theta\_\{0\}, states𝒳\\mathcal\{X\}, backward prompt templatePbwdP\_\{\\mathrm\{bwd\}\}, candidatesNN, selection countKK, roundsRR

Output :Trained model

θR\\theta\_\{R\}
for*roundr=1,2,…,Rr=1,2,\\ldots,R*do

//E\-step: backward generation

for*each statex∈𝒳x\\in\\mathcal\{X\}*do

Sample

z1,…,zN∼pθr−1\(⋅∣Pbwd\(x,π~∗\(x\)\)\)z\_\{1\},\\ldots,z\_\{N\}\\sim p\_\{\\theta\_\{r\-1\}\}\\\!\\bigl\(\\cdot\\mid P\_\{\\mathrm\{bwd\}\}\(x,\\tilde\{\\pi\}^\{\*\}\(x\)\)\\bigr\)
//E\-step: forward delta scoring \(impl\. App\.[A\.4](https://arxiv.org/html/2607.21856#A1.SS4)\)

for*each candidate\(x,zj\)\(x,z\_\{j\}\)*do

Δ¯j←Δ¯​\(zj;x\)\\bar\{\\Delta\}\_\{j\}\\leftarrow\\bar\{\\Delta\}\(z\_\{j\};\\,x\)
//averageΔ\\Deltafrom Eq\. \([4](https://arxiv.org/html/2607.21856#S3.E4)\),y∼π∗\(⋅∣x\)y\\sim\\pi^\{\*\}\(\\cdot\\mid x\)

//Hard top\-KKon positive deltas

for*each statexx*do

𝒵x←top​\-​K​\{zj:Δ¯j\>0\}\\mathcal\{Z\}\_\{x\}\\leftarrow\\mathrm\{top\\text\{\-\}\}K\\\{z\_\{j\}:\\bar\{\\Delta\}\_\{j\}\>0\\\}
//M\-step: forward \+ backward joint SFT

𝒟rfwd←\{\(x,\(z,π∗\(⋅∣x\)\)\):x∈𝒳,z∈𝒵x\}\\mathcal\{D\}\_\{r\}^\{\\mathrm\{fwd\}\}\\leftarrow\\\{\(x,\\;\(z,\\pi^\{\*\}\(\\cdot\\mid x\)\)\):x\\in\\mathcal\{X\},\\,z\\in\\mathcal\{Z\}\_\{x\}\\\}
𝒟rbwd←\{\(\(x,π~∗\),z\):x∈𝒳,z∈𝒵x\}\\mathcal\{D\}\_\{r\}^\{\\mathrm\{bwd\}\}\\leftarrow\\\{\(\(x,\\tilde\{\\pi\}^\{\*\}\),\\;z\):x\\in\\mathcal\{X\},\\,z\\in\\mathcal\{Z\}\_\{x\}\\\}
θr←SFT​\(θr−1,𝒟rfwd∪𝒟rbwd\)\\theta\_\{r\}\\leftarrow\\mathrm\{SFT\}\\bigl\(\\theta\_\{r\-1\},\\;\\mathcal\{D\}\_\{r\}^\{\\mathrm\{fwd\}\}\\cup\\mathcal\{D\}\_\{r\}^\{\\mathrm\{bwd\}\}\\bigr\)
return*θR\\theta\_\{R\}*

\-0\.1cm

Algorithm 1TheLeActalgorithm

## 4Experimental Setup

We evaluateLeActon six imperfect\-information games \(Leduc Hold’em at three scales, Liar’s Dice, 3\-Player Leduc, Flop Hold’em\) and the BuilderBench\[[9](https://arxiv.org/html/2607.21856#bib.bib48)\]robotics cube\-stacking benchmark, summarised in Table[1](https://arxiv.org/html/2607.21856#S4.T1); per\-domain rules and oracle construction are in App\.[H\.1](https://arxiv.org/html/2607.21856#A8.SS1),[F](https://arxiv.org/html/2607.21856#A6)\.

Recipe and baselines\.We use Qwen3\-8B\[[35](https://arxiv.org/html/2607.21856#bib.bib42)\]as the student LLM throughout\. Fixing the backbone is by design: it isolates supervision quality from model scale, Qwen3\-8B is a representative open\-weight student, and 8B fits within the compute budget required to fully train all seven settings\. The base model lacks the game\-specific forward/backward competence to seed reasoning traces directly, so the two iterative methods \(CoTExItandLeAct\) both initialise from a frontier\-LLM\-generated coldstart corpus and run 3 training rounds\. The coldstart is shared per\-setting between the two, so any gap between them is attributable to the iteration phase\. Full setup details are in App\.[H](https://arxiv.org/html/2607.21856#A8)\.

- •NoCoTBCskips the CoT\-candidate step: a single SFT pass on\(x,π∗\(⋅∣x\)\)\(x,\\pi^\{\*\}\(\\cdot\\mid x\)\)pairs without a reasoning channel\. Because it produces no CoT, it is not directly comparable on a data\-centric basis with the iterative methods; we report it as a memorisation anchor\.
- •CoTExItis standard expert iteration\[[2](https://arxiv.org/html/2607.21856#bib.bib9),[41](https://arxiv.org/html/2607.21856#bib.bib10)\]with a thinking model: each round samplesN=8N\{=\}8forward rollouts\(z,y\)\(z,y\)per state, ranks them by KL of the model’s parsed action distribution to the oracle, retains the top\-K=2K\{=\}2as next\-round SFT data, and uses the parsed action distribution itself \(notπ∗\\pi^\{\*\}\) as the action target \(the standard expert\-iteration practice; we ablate this in §[6](https://arxiv.org/html/2607.21856#S6)\)\.
- •LeActsamplesN=8N\{=\}8backward CoT candidates per state conditioned on the expert action, ranks them by forward delta against the student’s own self\-baseline \(M=4M\{=\}4parallel forward decodings, no gold\-action conditioning\), retains the top\-K=2K\{=\}2as next\-round SFT data, and usesπ∗\\pi^\{\*\}as the action target \(expert\-policy forcing\)\. This is not a design choice: the M\-step prescribes it \(§[3\.2](https://arxiv.org/html/2607.21856#S3.SS2)\), and empirically it is also the better target \(§[6](https://arxiv.org/html/2607.21856#S6)\)\.

Two reasons we do not implement RL/policy\-gradient baselines \(e\.g\.,GRPO\[[11](https://arxiv.org/html/2607.21856#bib.bib35)\]\): this work targets SFT / mid\-training, andCoTExItalready approximates a coarse\-grained REINFORCE through its rollout\-and\-bootstrap loop\. Each round retainsK=2K\{=\}2CoTs per state\.CoTExIttrains on the forward direction only \(K​\|𝒳\|K\|\\mathcal\{X\}\|examples per round\), whileLeAct’s M\-step jointly trains forward and backward directions \(2​K​\|𝒳\|2K\|\\mathcal\{X\}\|per round\)\. For each \(method, setting\) we report at the round \(out of 3\) with the best evaluation metric\. Per\-round dynamics, the stopping rule, and R1\-only separation evidence are in App\.[C](https://arxiv.org/html/2607.21856#A3)\.

Two output formats\.The seven settings split into two regimes by what the model emits at decoding time\. The six imperfect\-information games \(Leduc, Liar’s Dice, 3\-Player Leduc, FHP\) ask the model to output a mixed\-strategy*distribution*per state\. Each forward decode emits a parsed action distribution whose KL to the oracleπ∗\\pi^\{\*\}is the unit of supervision\. BuilderBench instead asks for a single JSON action per turn over a multi\-turn cube\-stacking rollout: success is multi\-step progress\. Absolute scores are not comparable across regimes; only within\-regime method ordering is\. Because BuilderBench’s per\-state KL is unavailable when the oracle commits to a single action, we replace forward\-delta scoring \(andCoTExIt’s ranking\) with a heuristic action\-match reward against the oracle’s actiony∗y^\{\*\}for both methods \(App\.[D\.2\.1](https://arxiv.org/html/2607.21856#A4.SS2.SSS1)\); the rest of the recipe is unchanged\.

Metrics\.For two\-player zero\-sum games \(Leduc, Liar’s Dice\) we report*exploitability*\(best\-response gap to Nash;0= Nash\)\. For 3\-Player Leduc we reportNashConv\[[21](https://arxiv.org/html/2607.21856#bib.bib44)\], annn\-player generalisation that reduces to2×2\\timesexploitability whenn=2n\{=\}2\. At FHP we report mean KL to theDeepCFRteacher and chip\-outcome win rate in*mbb/g*\(milli\-big\-blinds per game, the standard heads\-up play\-strength unit; App\.[D\.3](https://arxiv.org/html/2607.21856#A4.SS3)\)\. On BuilderBench we report*cube\-placement progress*\(per\-task fraction of cubes placed at target positions, averaged over6464inference rollouts per task and then across tasks\)\. All reasoning\-game metrics are reported undersingle\(one CoT→\\toone action\) andBoN\(per infoset, the candidate with smallest KL toπ∗\\pi^\{\*\}acrossNNsamples; oracle\-aware\), the standard inference\-time scaling axis\[[54](https://arxiv.org/html/2607.21856#bib.bib3)\]\. All are lower\-is\-better, except BuilderBench’s cube\-placement progress\.

Evaluation regime\.Imitation is evaluated on the full game tree at enumerable scales \(Leduc, Liar’s Dice, 3\-Player Leduc\) and on BuilderBench’s in\-domain2626\-task partition \(where the trajectory pool produced at least one fully\-successful episode\)\. Generalisation is evaluated on a held\-out infoset split at FHP, a feature\-disjoint rank\-split at Leduc 10r4s \(test ranks never appear in training\), and BuilderBench’s OOD2020\-task partition \(where the best\-of\-33frontier\-model trajectory pool produced no successful trajectory; App\.[H\.2](https://arxiv.org/html/2607.21856#A8.SS2)\)\.

Table 1:Evaluation spans four orders of magnitude in scale across game families and a robotics domain\.Per\-setting oracle, metric, and regime\.DomainScaleOracleMetricRegime \(split\)Leduc 6r2s4,032CFRExploitabilityImitation \(full tree\)Leduc 10r4s47,040CFRExploitabilityBoth \(full / rank\-split\)Leduc 13r4s79,872CFRExploitabilityImitation \(full tree\)Liar’s Dice24,576CFRExploitabilityImitation \(full tree\)3\-Player Leduc13,878CFRNashConvImitation \(full tree\)Flop Hold’em \(FHP\)∼109\\sim\\\!10^\{9\}DeepCFRKL \+ mbb/gGeneralisation \(20K hold\-out\)BuilderBench46 tasksFrontier\-LLM traj\.task progressBoth \(in\-domain / OOD\)

“Both” = the game is evaluated under both imitation \(full tree\) and generalisation \(held\-out split\) regimes\.

![Refer to caption](https://arxiv.org/html/2607.21856v1/x2.png)Figure 2:Imitation quality:LeActmatchesNoCoTBCwhere memorisation fits, and dominates where it doesn’t\.Five game settings \(4K–80K infosets, logyy\-axis, lower better,BoNusesN=8N\{=\}8\) and BuilderBench in\-domain 26 tasks \(linearyy\-axis, higher better;N=64N\{=\}64\)\. Numbers in Table[8](https://arxiv.org/html/2607.21856#A4.T8)\.
## 5Results

LeActtreats expert actions as imitation supervision while producing a latent CoT alongside each action; the empirical question is whether this latent helps relative to imitation without latent \(NoCoTBC\) or with self\-generated latent that bypasses our backward\-and\-score loop \(CoTExIt\)\. We organise the comparison along two axes\.Imitation quality\(§[5\.1](https://arxiv.org/html/2607.21856#S5.SS1)\): under matched coldstart and data budget, how closely does the trained forward policy track the expert? We measure across enumerable game scales \(Leduc44K–8080K\), player counts \(3\-player Leduc\), game families \(Liar’s Dice\), and domains \(BuilderBench cube\-stacking\)\.Generalisation\(§[5\.2](https://arxiv.org/html/2607.21856#S5.SS2)\): does the advantage persist on states the model never saw during training, where memorisation alone would not transfer? We test on the Flop Hold’em hold\-out infoset, the Leduc rank\-split, and BuilderBench’s OOD task partition\.

### 5\.1Imitation

LeActmatches or beats both baselines onBoNinference everywhere outside the 6r2s memorisation regime \(Fig\.[2](https://arxiv.org/html/2607.21856#S4.F2)\)\. TheBoNadvantage comes from generation diversity rather than higher single\-sample quality: at Leduc 10r4s,LeAct’s single\-to\-BoNratio is several times that ofNoCoT’s, and on BuilderBench in\-domainLeActstarts at the lowest single\-sample progress but leads atBoNacross all four recipes\. The CoT spreads the per\-state action distribution, soBoNcatches better samples than any single draw\.

Scaling on poker\.At 6r2s \(44K infosets\) the Nash table fits in model capacity andNoCoT’s tabular memorisation suffices\. At 13r4s \(8080K infosets\) memorisation breaks:NoCoTBCalso stalls \(App\.[D\.1](https://arxiv.org/html/2607.21856#A4.SS1)\), so the matchedCoTExItablation \(same coldstart, same supervision, same data \(game states\) for training\) stalling5\.4×5\.4\\timesworse on single\-sample and4×4\\timesworse onBoNthanLeActisolates backward\-delta selection as the load\-bearing component, not the presence of a CoT itself\. The trend is consistent: as scale grows, the value of selecting which CoTs to learn from grows\.

Cross\-domain results\.Beyond two\-player Leduc the same imitation gap appears in three further regimes \(different game family, multi\-player, robotics\)\. On Liar’s Dice \(different game family\),LeActlands within solver\-evaluation noise of theCFRoracle\. On 3\-player Leduc,LeActlands within solver noise whileExItstalls well above it\. Atn≥3n\\\!\\geq\\\!3players the 2P\-zero\-sum convergence guarantee for best\-response iteration fails \(Banach contraction\[[3](https://arxiv.org/html/2607.21856#bib.bib30)\]\), soExIt’s table\-fit policies have no monotone\-improvement guarantee across rounds; onlyLeAct’s reasoning\-based policy stays self\-consistent across roles\. On BuilderBench cube\-stacking in\-domain set \(using a frontier\-LLM successful trajectory pool as oracle\),LeActis the only training recipe whoseBoNprogress clears the joint coldstart anchor; the matchedCoTExItcomparator even regresses below it \(Table[8](https://arxiv.org/html/2607.21856#A4.T8)\)\.

Absolute KL and exploitability values are not directly comparable across these games: action\-space size and Nash\-policy concentration both differ, and large game trees like FHP have many infosets where the optimal move is near\-deterministic, pulling the average KL floor lower\. What is interpretable across rows is therefore the within\-game ordering rather than the absolute scale\.

Table 2:LeActgeneralises across three held\-out regimes\.\(a\) KL \(↓\\downarrow\) on a random infoset hold\-out \(FHP\) and a feature\-disjoint rank\-split \(Leduc 10r4s, test ranks never appear in training\); cube\-placement progress \(↑\\uparrow\) on the BuilderBench \(BB\) OOD partition\. Two frontier\-anchor rows: gpt\-5\.5 pass@11on FHP \(a comparator, not data source\); the best\-of\-33frontier\-LLM trajectory pool used as BuilderBench SFT data, evaluated as a strength baseline on the same OOD partition\. \(b\) FHP chip outcome in mbb/g over5050K paired hands \(head\-to\-head;±\\pmstandard error\)\. Details in App\.[D\.3](https://arxiv.org/html/2607.21856#A4.SS3),[D\.2\.2](https://arxiv.org/html/2607.21856#A4.SS2.SSS2)\.\(a\)Held\-out generalisation\.Leduc 10r4s \(↓\\downarrow\)FHP2020K \(↓\\downarrow\)BB OOD \(↑\\uparrow\)MethodSingleBoNSingleBoNProgressgpt\-5\.5——0\.7530——Frontier traj\.————0\.4923Coldstart2\.4620\.3320\.03210\.00180\.4815NoCoTBC2\.1100\.5740\.01450\.00490\.3980CoTExIt2\.0710\.4590\.00530\.00110\.4361LeAct0\.7160\.1680\.00190\.00050\.5777

\(b\)FHP chip outcome\.NoCoTCoTLeActNoCoT—\+41\+41±31\\pm 31−60\-60±13\\pm 13CoT−41\-41±31\\pm 31—−62\-62±28\\pm 28LeAct\+60\+60±13\\pm 13\+62\+62±28\\pm 28—

### 5\.2Generalisation

LeActcarries its advantage to states the model never saw during training, across three regimes: a random infoset hold\-out at million\-scale \(FHP\), a feature\-disjoint rank\-split at medium scale \(Leduc 10r4s\), and a task\-level OOD partition on BuilderBench\. Random hold\-outs test that the trained policy is not memorising the training infosets; feature\-disjoint splits stress transfer along an axis the model never saw; the BuilderBench OOD partition stresses generalisation beyond the supervised demonstration distribution\.

Random hold\-out at scale: FHP\.Flop Hold’em has on the order of10910^\{9\}playable infosets, several orders of magnitude beyond the8080K training infosets, so even random sampling exposes the policy to a state distribution it has never been trained on, making FHP a natural generalisation testbed even under the random hold\-out regime\. The absolute KL floor is small because most infosets have near\-trivial optimal strategy \(preflop fold lines on weak hands\)\. The within\-regime ordering is nonetheless sharp:LeActbeats both baselines, and a frontier zero\-shot anchor \(gpt\-5\.5; Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1)\) lands orders of magnitude worse\. Even a strong general reasoner is therefore severely under\-optimised at producing calibrated numerical mixed strategies for a fine\-grained game\. Chip\-outcome head\-to\-head \(Table[2\(b\)](https://arxiv.org/html/2607.21856#S5.T2.st2)\) corroborates the KL ordering\.

Feature\-disjoint shift: Leduc 10r4s rank\-split\.The rank\-split \(train ranks\{2,4,6,8,T\}\\\{2,4,6,8,\\text\{T\}\\\}, test\{3,5,7,9,J\}\\\{3,5,7,9,\\text\{J\}\\\}\) forces the model to transfer along a feature axis it never saw during training, a stronger distribution\-shift test than random hold\-out\. All three methods are retrained from scratch for this split: the joint coldstart corpus is regenerated on the train ranks alone, andCoTExItandLeActthen iterate from this rank\-restricted coldstart base\. The test ranks\{3,5,7,9,J\}\\\{3,5,7,9,\\text\{J\}\\\}are never seen during coldstart or any iteration round\.LeAct’s∼3×\{\\sim\}3\\timeslead overCoTExIton the held\-out ranks \(Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1)\) shows that the imitation gap is not memorisation residue, though absolute values remain an order of magnitude above the imitation regime \(single0\.7160\.716vs0\.0370\.037\): the gain reduces catastrophic generalisation failure rather than achieving clean transfer\.

Task\-level OOD on BuilderBench\.The OOD partition is2020tasks that no frontier model in the best\-of\-33trajectory pool fully solved, so the SFT data contains no golden trajectory for any of these\. We report*cube\-placement progress*, a partial\-solve metric \(fraction of cubes correctly placed\), so non\-zero scores correspond to placing some cubes correctly even when the task is not solved end\-to\-end\.LeActleads on this partition \(Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1)\), and the gain is broad\-based across tasks rather than concentrated on a few easy ones, so the model makes measurable progress on tasks it had no demonstration for\.

## 6Analysis

§[5\.1](https://arxiv.org/html/2607.21856#S5.SS1)establishes that backward\-delta selection drivesLeAct’s gap over the matchedCoTExItcomparator\. This section runs four probes that test whether the gap might instead come from one of four alternatives, and closes with a compute\-overhead analysis\. \(1\) Could any ranking rule have done equally well? \(2\) Does our parsed\-string proxy faithfully measure the formalΔ\\Delta? \(3\) Is the gain just memorising surface features \(e\.g\. Nash numbers\)? \(4\) Or is it the supervision\-target choice \(not backward delta\) doing the work?

Delta selection is informative, not random\.To test this, we replace forward\-delta ranking with random ranking on Leduc 10r4s, sharpened to top\-11\(so any gap comes from the ranking rule, not the data budget\)\. Forward\-delta beats random on both single andBoNexploitability \(Fig\.[3](https://arxiv.org/html/2607.21856#S6.F3)a\)\.LeAct’s gain therefore comes from*which*CoTs we keep, not just from keeping anyK=2K=2subset\.

The textual proxy faithfully reads the formalΔ\\Delta\.LeActscores CoT candidates by reading the model’s parsed policy distribution, not by computing the token\-level log\-likelihoods that the formalΔ\\Deltais defined over\. This is a proxy\. We check whether it ranks the same way the formalΔ\\Deltawould: on Leduc 13r4s, the parsed\-stringΔ¯\\bar\{\\Delta\}and a token\-level estimatorΔll\\Delta\_\{\\mathrm\{ll\}\}agree on the top\-11CoT in8989–98%98\\%of states \(App\.[A\.4](https://arxiv.org/html/2607.21856#A1.SS4), Table[5](https://arxiv.org/html/2607.21856#A1.T5)\)\.

The selection signal is not memorised Nash numbers\.IfLeAct’s scorer were just rewarding CoTs that quote oracle probabilities verbatim \(e\.g\., “fold0\.3760\.376”\), then CoTs without such verbatim numbers would score worse\. We split the backward pool by verbatim\-π∗\\pi^\{\*\}presence: at both Leduc 10r4s and 13r4s, the larger “no verbatim” stratum has a higher mean positive delta than the smaller “with verbatim” stratum \(Fig\.[3](https://arxiv.org/html/2607.21856#S6.F3)b\)\. The held\-out generalisation results \(§[5\.2](https://arxiv.org/html/2607.21856#S5.SS2)\) corroborate: a memorisation shortcut would not transfer to unseen states\.

The gap is backward delta, not the supervision target\.To separate target choice from method choice, we run a2×22\\times 2crossing method \(CoTExItvsLeAct\) with target \(*expert\-policy forcing*or EPF, vs the model’s own action\)\. Each cell is a separate run from the shared Leduc 10r4s coldstart, iterated for 2 rounds at matched data budget\. The principled pairs are the diagonals:LeAct\+EPF \(the M\-step prescribesπ∗\\pi^\{\*\}\) andCoTExIt\+no\-EPF \(standard ExIt does not relabel actions\)\. Only these diagonal cells beat coldstart \(Table[3](https://arxiv.org/html/2607.21856#S6.T3)\)\. The off\-diagonals collapse because in poker the CoT’s textual policy line and the model’s decoded action are essentially the same: supervising one with the other is self\-referential and provides no new signal \(App\.[E\.1](https://arxiv.org/html/2607.21856#A5.SS1)\)\. Within the diagonal,LeAct\+EPF dominatesCoTExIt\+no\-EPF on both Single \(0\.160\.16vs0\.440\.44\) andBoN\(0\.00190\.0019vs0\.060\.06\), so backward\-delta scoring carries the gap, not the EPF target\. The cross\-domain BuilderBench replication breaks this coupling and recovers a clean ordering on both axes \(App\.[D\.2\.2](https://arxiv.org/html/2607.21856#A4.SS2.SSS2)\)\.

![Refer to caption](https://arxiv.org/html/2607.21856v1/x3.png)
Figure 3:Two falsifiers of the shortcut hypothesis for backward\-delta selection\.\(a\) Forward\-delta vs\. random ranking on Single andBoN\(Leduc 10r4s\)\. \(b\) Mean\+Δ\+\\Deltaby verbatim\-π∗\\pi^\{\*\}status\.MethodRegimeSingle \(↓\\downarrow\)BoN\(↓\\downarrow\)Coldstart—1\.67010\.2824CoTExItEPF1\.02780\.6191CoTExItno\-EPF0\.44320\.0599LeActEPF0\.15660\.0019LeActno\-EPF1\.05650\.1940

Table 3:Method and supervision target are entangledWe cross method \(CoTExItvsLeAct\) with*expert\-policy forcing*\(use the oracle’sπ∗\\pi^\{\*\}vs the model’s own action\) on Leduc10r4s\. Matched pairings:LeAct\+EPF andCoTExIt\+no\-EPF\.
Compute overhead\.LeActandCoTExItboth generateNNCoTs per state during training \(backward forLeAct, forward forCoTExIt\);LeAct’s only extra cost isMMparallel short action generation for forward scoring, negligible relative to the shared long CoT generation pass\. The two methods therefore incur essentially the same training compute\.

## 7Conclusion

The key idea of this paper is that many domains already have a silent domain\-specific expert, and that the reasoning trace these experts cannot articulate can be recovered by the student itself and filtered against the expert’s own actions\. This opens a new data source for reasoning supervision that does not require a human annotator or a stronger LLM, and the oracle itself need not be exact\. At Flop Hold’em we score against aDeepCFRteacher, and on BuilderBench against a pool of frontier\-model trajectories, in both cases without modification to the recipe\.

Other silent experts \(protein structure predictors, formal proof checkers, robotic\-control libraries\) are natural candidates for the same recipe, and the most direct algorithmic extension is mergingLeActwith online RL, where the oracle’s action distribution can serve as a process\-level baseline for credit assignment\. HowLeActinteracts with frontier\-scale students, beyond the 8B fixed here, is another direct extension\.

## References

- \[1\]M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. Zaremba\(2017\)Hindsight experience replay\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[2\]T\. Anthony, Z\. Tian, and D\. Barber\(2017\)Thinking fast and slow with deep learning and tree search\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1),[2nd item](https://arxiv.org/html/2607.21856#S4.I1.i2.p1.4)\.
- \[3\]S\. Banach\(1922\)Sur les opérations dans les ensembles abstraits et leur application aux équations intégrales\.Fundamenta Mathematicae3\(1\),pp\. 133–181\.Cited by:[§5\.1](https://arxiv.org/html/2607.21856#S5.SS1.p3.1)\.
- \[4\]J\. Bornschein and Y\. Bengio\(2015\)Reweighted wake\-sleep\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:1406\.2751Cited by:[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p8.10),[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p8.25),[§3\.3](https://arxiv.org/html/2607.21856#S3.SS3.p5.13)\.
- \[5\]N\. Brown, A\. Lerer, S\. Gross, and T\. Sandholm\(2019\)Deep counterfactual regret minimization\.InProceedings of the 36th International Conference on Machine Learning \(ICML\),PMLR, Vol\.97,pp\. 793–802\.Cited by:[§F\.2](https://arxiv.org/html/2607.21856#A6.SS2.p2.2),[§F\.2](https://arxiv.org/html/2607.21856#A6.SS2.p4.7),[Table 10](https://arxiv.org/html/2607.21856#A8.T10),[Table 10](https://arxiv.org/html/2607.21856#A8.T10.16.8.8),[§1](https://arxiv.org/html/2607.21856#S1.p2.1),[§1](https://arxiv.org/html/2607.21856#S1.p3.1)\.
- \[6\]N\. Brown and T\. Sandholm\(2019\)Superhuman AI for multiplayer poker\.Science365\(6456\),pp\. 885–890\.Cited by:[§F\.2](https://arxiv.org/html/2607.21856#A6.SS2.p5.1)\.
- \[7\]Y\. Burda, R\. Grosse, and R\. Salakhutdinov\(2016\)Importance weighted autoencoders\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p1.9),[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p2.17),[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p2.7),[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p8.10),[§2](https://arxiv.org/html/2607.21856#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.21856#S3.SS2.p1.11),[§3\.2](https://arxiv.org/html/2607.21856#S3.SS2.p1.6)\.
- \[8\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[9\]R\. Ghugare, R\. C\. Castanyer, C\. Ji, K\. Wantlin, J\. Schofield, K\. Narasimhan, and B\. Eysenbach\(2025\)BuilderBench: the building blocks of intelligent agents\.External Links:2510\.06288,[Link](https://arxiv.org/abs/2510.06288)Cited by:[3rd item](https://arxiv.org/html/2607.21856#S1.I1.i3.p1.1),[§1](https://arxiv.org/html/2607.21856#S1.p3.1),[§4](https://arxiv.org/html/2607.21856#S4.p1.1)\.
- \[10\]C\. Gulcehre, T\. L\. Paine, S\. Srinivasan, K\. Konyushkova, L\. Weerts, A\. Sharma, A\. Siddhant, A\. Ahern, M\. Wang, C\. Gu, W\. Macherey, A\. Doucet, O\. Firat, and N\. de Freitas\(2023\)Reinforced self\-training \(ReST\) for language modeling\.arXiv preprint arXiv:2308\.08998\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[11\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)DeepSeek\-R1 incentivizes reasoning in LLMs through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Note:arXiv:2501\.12948External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09422-z)Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p2.1),[§4](https://arxiv.org/html/2607.21856#S4.p2.3)\.
- \[12\]S\. Hao, Y\. Gu, H\. Ma, J\. J\. Hong, Z\. Wang, D\. Z\. Wang, and Z\. Hu\(2023\)Reasoning with language model is planning with world model\.arXiv preprint arXiv:2305\.14992\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[13\]A\. Havrilla, Y\. Du, S\. C\. Raparthy, C\. Nalmpantis, J\. Dwivedi\-Yu, M\. Zhuravinskyi, E\. Hambro, S\. Sukhbaatar, and R\. Raileanu\(2024\)Teaching large language models to reason with reinforcement learning\.arXiv preprint arXiv:2403\.04642\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[14\]N\. Ho, L\. Schmid, and S\. Yun\(2023\)Large language models are reasoning teachers\.arXiv preprint arXiv:2212\.10071\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[15\]A\. Hosseini, X\. Yuan, N\. Malkin, A\. Courville, A\. Sordoni, and R\. Agarwal\(2024\)V\-STaR: training verifiers for self\-taught reasoners\.arXiv preprint arXiv:2402\.06457\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[16\]T\. Hubert, R\. Mehta, L\. Sartran,et al\.\(2025\)Olympiad\-level formal mathematical reasoning with reinforcement learning\.Nature651,pp\. 607–613\.External Links:[Document](https://dx.doi.org/10.1038/s41586-025-09833-y)Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[17\]T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. Iwasawa\(2022\)Large language models are zero\-shot reasoners\.Advances in Neural Information Processing Systems35,pp\. 22199–22213\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[18\]A\. Kumar, V\. Zhuang, R\. Agarwal, Y\. Su, J\. D\. Co\-Reyes, A\. Singh, K\. Baumli, S\. Iqbal, C\. Bishop, R\. Roelofs,et al\.\(2024\)Training language models to self\-correct via reinforcement learning\.arXiv preprint arXiv:2409\.12917\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[19\]N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.\(2024\)Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[20\]M\. Lanctot, E\. Lockhart, J\. Lespiau, V\. Zambaldi, S\. Upadhyay, J\. Pérolat, S\. Srinivasan, F\. Timbers, K\. Tuyls, S\. Omidshafiei,et al\.\(2019\)OpenSpiel: a framework for reinforcement learning in games\.arXiv preprint arXiv:1908\.09453\.Cited by:[3rd item](https://arxiv.org/html/2607.21856#S1.I1.i3.p1.1)\.
- \[21\]M\. Lanctot, V\. Zambaldi, A\. Gruslys, A\. Lazaridou, K\. Tuyls, J\. Pérolat, D\. Silver, and T\. Graepel\(2017\)A unified game\-theoretic approach to multiagent reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.30,pp\. 4191–4204\.Cited by:[§4](https://arxiv.org/html/2607.21856#S4.p4.8)\.
- \[22\]T\. Lanham, A\. Chen, A\. Radhakrishnan, B\. Steiner, C\. Denison, D\. Hernandez, D\. Li, E\. Durmus, E\. Hubinger, J\. Kernion,et al\.\(2023\)Measuring faithfulness in chain\-of\-thought reasoning\.arXiv preprint arXiv:2307\.13702\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p2.1)\.
- \[23\]Z\. Li, Z\. Yang, D\. He, H\. Zhao, A\. Zhao, S\. Tang, K\. Yang, A\. Gupta, Z\. Su, and C\. Jin\(2026\)Goedel\-Code\-Prover: hierarchical proof search for open state\-of\-the\-art code verification\.arXiv preprint arXiv:2603\.19329\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[24\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2024\)Let’s verify step by step\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2305\.20050Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[25\]Y\. Lin, S\. Tang, B\. Lyu, J\. Wu, H\. Lin, K\. Yang, J\. Li, M\. Xia, D\. Chen, S\. Arora, and C\. Jin\(2025\)Goedel\-Prover: a frontier model for open\-source automated theorem proving\.arXiv preprint arXiv:2502\.07640\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[26\]Y\. Lin, S\. Tang, B\. Lyu, Z\. Yang, J\. Chung, H\. Zhao, L\. Jiang, Y\. Geng, J\. Ge, J\. Sun, J\. Wu,et al\.\(2025\)Goedel\-Prover\-V2: scaling formal theorem proving with scaffolded data synthesis and self\-correction\.arXiv preprint arXiv:2508\.03613\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p2.1),[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[27\]H\. Liu, C\. Sferrazza, and P\. Abbeel\(2024\)Chain of hindsight aligns language models with feedback\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2302\.02676Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[28\]T\. Q\. Luong, X\. Zhang, Z\. Jie, P\. Sun, X\. Jin, and H\. Li\(2024\)ReFT: reasoning with reinforced fine\-tuning\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),Note:arXiv:2401\.08967Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[29\]C\. Ma, Z\. Yang, H\. Ci, J\. Gao, M\. Gao, X\. Pan, and Y\. Yang\(2026\)Evolving diverse red\-team language models in multi\-round multi\-agent games\.IEEE Transactions on Pattern Analysis and Machine Intelligence\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.2026.3694367)Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[30\]L\. C\. Magister, J\. Mallinson, J\. Adamek, E\. Malmi, and A\. Severyn\(2023\)Teaching small language models to reason\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 1773–1781\.Note:arXiv:2212\.08410Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[31\]Meta Fundamental AI Research Diplomacy Team \(FAIR\)\(2022\)Human\-level play in the game of Diplomacy by combining language models with strategic reasoning\.Science378\(6624\),pp\. 1067–1074\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[32\]M\. Moravčík, M\. Schmid, N\. Burch, V\. Lisý, D\. Morrill, N\. Bard, T\. Davis, K\. Waugh, M\. Johanson, and M\. Bowling\(2017\)DeepStack: expert\-level artificial intelligence in heads\-up no\-limit poker\.Science356\(6337\),pp\. 508–513\.Cited by:[§F\.2](https://arxiv.org/html/2607.21856#A6.SS2.p5.1)\.
- \[33\]OpenAI\(2024\)OpenAI o1 system card\.External Links:2412\.16720,[Link](https://arxiv.org/abs/2412.16720)Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[34\]S\. Polu and I\. Sutskever\(2020\)Generative language modeling for automated theorem proving\.arXiv preprint arXiv:2009\.03393\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[35\]Qwen Team\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4](https://arxiv.org/html/2607.21856#S4.p2.4)\.
- \[36\]G\. Roeder, Y\. Wu, and D\. K\. Duvenaud\(2017\)Sticking the landing: simple, lower\-variance gradient estimators for variational inference\.InAdvances in Neural Information Processing Systems,Cited by:[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p8.10)\.
- \[37\]Y\. Ruan, N\. Band, C\. J\. Maddison, and T\. Hashimoto\(2025\)Reasoning to learn from latent thoughts\.arXiv preprint arXiv:2503\.18866\.Cited by:[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p1.9),[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p5.12),[1st item](https://arxiv.org/html/2607.21856#S1.I1.i1.p1.1),[2nd item](https://arxiv.org/html/2607.21856#S1.I1.i2.p1.3),[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p1.1),[§3\.2](https://arxiv.org/html/2607.21856#S3.SS2.p2.20),[§3](https://arxiv.org/html/2607.21856#S3.p1.1)\.
- \[38\]A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. Kumar\(2024\)Rewarding progress: scaling automated process verifiers for LLM reasoning\.arXiv preprint arXiv:2410\.08146\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[39\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\.K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[40\]C\. Shi, W\. Li, X\. Liang, Y\. Lu, W\. Yang, R\. Feng, S\. Karten, Z\. Yang, Z\. Ding, G\. Sarch, D\. Chen, K\. Narasimhan, and C\. Jin\(2026\)Odysseus: scaling VLMs to 100\+ turn decision\-making in games via reinforcement learning\.arXiv preprint arXiv:2605\.00347\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[41\]D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel,et al\.\(2018\)A general reinforcement learning algorithm that masters chess, shogi, and Go through self\-play\.Science362\(6419\),pp\. 1140–1144\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1),[2nd item](https://arxiv.org/html/2607.21856#S4.I1.i2.p1.4)\.
- \[42\]A\. Singh, J\. D\. Co\-Reyes, R\. Agarwal, A\. Anand, P\. Patil, X\. Garcia, P\. J\. Liu, J\. Harrison, J\. Lee, K\. Xu,et al\.\(2024\)Beyond human data: scaling self\-training for problem\-solving with language models\.Transactions on Machine Learning Research\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[43\]C\. Snell, J\. Lee, K\. Xu, and A\. Kumar\(2024\)Scaling LLM test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[44\]F\. Southey, M\. Bowling, B\. Larson, C\. Piccione, N\. Burch, D\. Billings, and C\. Rayner\(2005\)Bayes’ bluff: opponent modelling in poker\.InProceedings of the Twenty\-First Conference on Uncertainty in Artificial Intelligence,pp\. 550–558\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p3.1)\.
- \[45\]O\. Tammelin\(2014\)Solving large imperfect information games using CFR\+\.arXiv preprint arXiv:1407\.5042\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p2.1)\.
- \[46\]M\. Toussaint\(2015\)Logic\-geometric programming: an optimization\-based approach to combined task and motion planning\.InProceedings of the 24th International Joint Conference on Artificial Intelligence \(IJCAI\),pp\. 1930–1936\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p2.1)\.
- \[47\]H\. Touvron, L\. Martin, K\. Stone,et al\.\(2023\)Llama 2: open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[48\]T\. H\. Trinh, Y\. Wu, Q\. V\. Le, H\. He, and T\. Luong\(2024\)Solving olympiad geometry without human demonstrations\.Nature625\(7995\),pp\. 476–482\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06747-5)Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[49\]G\. Tucker, D\. Lawson, S\. Gu, and C\. J\. Maddison\(2019\)Doubly reparameterized gradient estimators for Monte Carlo objectives\.InInternational Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2607.21856#A1.SS1.p8.10)\.
- \[50\]M\. Turpin, J\. Michael, E\. Perez, and S\. R\. Bowman\(2023\)Language models don’t always say what they think: unfaithful explanations in chain\-of\-thought prompting\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2305\.04388Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p2.1)\.
- \[51\]J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins\(2022\)Solving math word problems with process\- and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[52\]H\. Wang, S\. Hao, H\. Dong, S\. Zhang, Y\. Bao, Z\. Yang, and Y\. Wu\(2024\)Offline reinforcement learning for LLM multi\-step reasoning\.arXiv preprint arXiv:2412\.16145\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[53\]P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui\(2024\)Math\-Shepherd: verify and reinforce LLMs step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 9426–9439\.Note:arXiv:2312\.08935Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[54\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-consistency improves chain of thought reasoning in language models\.arXiv preprint arXiv:2203\.11171\.Cited by:[§4](https://arxiv.org/html/2607.21856#S4.p4.8)\.
- \[55\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[56\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao\(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2210\.03629Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p3.1)\.
- \[57\]Z\. Yuan, H\. Yuan, C\. Li, G\. Dong, K\. Lu, C\. Tan, C\. Zhou, and J\. Zhou\(2023\)Scaling relationship on learning mathematical reasoning with large language models\.arXiv preprint arXiv:2308\.01825\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[58\]E\. Zelikman, G\. Harik, Y\. Shao, V\. Jayasiri, N\. Haber, and N\. D\. Goodman\(2024\)Quiet\-STaR: language models can teach themselves to think before speaking\.arXiv preprint arXiv:2403\.09629\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[59\]E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman\(2022\)STaR: bootstrapping reasoning with reasoning\.Advances in Neural Information Processing Systems35,pp\. 15476–15488\.Cited by:[1st item](https://arxiv.org/html/2607.21856#S1.I1.i1.p1.1),[§1](https://arxiv.org/html/2607.21856#S1.p1.1),[§2](https://arxiv.org/html/2607.21856#S2.p1.1)\.
- \[60\]D\. Zhang, S\. Zhoubian, Z\. Hu, Y\. Yue, Y\. Dong, and J\. Tang\(2024\)ReST\-MCTS\*: LLM self\-training via process reward guided tree search\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2406\.03816Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[61\]H\. Zhao, Z\. Yang, J\. Li, D\. He, Z\. Li, C\. Jin, V\. V\. Veeravalli, A\. Gupta, and S\. Arora\(2026\)AlgoVeri: an aligned benchmark for verified code generation on classical algorithms\.arXiv preprint arXiv:2602\.09464\.Cited by:[§2](https://arxiv.org/html/2607.21856#S2.p2.1)\.
- \[62\]M\. Zinkevich, M\. Johanson, M\. Bowling, and C\. Piccione\(2007\)Regret minimization in games with incomplete information\.InAdvances in Neural Information Processing Systems,Vol\.20\.Cited by:[§1](https://arxiv.org/html/2607.21856#S1.p2.1)\.

## Appendix AFormulation: Derivations

This appendix gives the derivations summarised in §[3\.2](https://arxiv.org/html/2607.21856#S3.SS2): \(i\) the IWAE bound, per\-sample importance\-weight factorisation, and rho\-dropped soft\-weight approximation that define theLeActtraining objective \(App\.[A\.1](https://arxiv.org/html/2607.21856#A1.SS1)\); \(ii\) the M\-step target\-choice derivation for expert\-policy forcing \(App\.[A\.2](https://arxiv.org/html/2607.21856#A1.SS2)\); \(iii\) the correspondence between formal quantities andLeActalgorithm components \(App\.[A\.3](https://arxiv.org/html/2607.21856#A1.SS3)\); and \(iv\) the textual policy proxy that bridges the formalΔ\\Deltato its parsed\-policy implementation \(App\.[A\.4](https://arxiv.org/html/2607.21856#A1.SS4)\)\.

### A\.1From the ELBO to the LeAct Objective

This appendix expands §[3\.2](https://arxiv.org/html/2607.21856#S3.SS2)line by line\. The derivation specialises the importance\-weighted variational bound \(IWAE\) ofBurdaet al\.\[[7](https://arxiv.org/html/2607.21856#bib.bib58)\]to a setting in which the proposalqbwd​\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\)is a fixed \(within a round\) LLM that conditions on\(x,y\)\(x,y\), and in which two of the four log\-likelihood terms in the importance weight are intractable under a decoder\-only model\. The same IWAE bound was recently applied to latent reasoning data synthesis on unlabelled text corpora byRuanet al\.\[[37](https://arxiv.org/html/2607.21856#bib.bib59)\]; the present derivation differs in bounding the conditional likelihoodlog⁡pθ​\(y∣x\)\\log p\_\{\\theta\}\(y\\mid x\)for an expert\-anchored targety∼π∗y\\sim\\pi^\{\*\}rather than the corpus marginallog⁡p​\(X\)\\log p\(X\), which enables the additional reduction of the IWAE soft weight to a forward delta \(Step 3\)\. The argument proceeds in four steps: \(i\) introduce the backward proposal and obtain an ELBO valid for any choice ofqbwdq\_\{\\mathrm\{bwd\}\}, including a fixed LLM; \(ii\) factorise the importance ratio via Bayes’ rule; \(iii\) keep only the two computable terms, yieldingΔ\\Delta; \(iv\) re\-weight the ELBO byeΔe^\{\\Delta\}and replace the soft weighting by a hard top\-KKfilter for tractable SFT\.

Step 1: IWAE bound via a backward proposal\.The marginal log\-likelihood of an expert actiony∼π∗\(⋅∣x\)y\\sim\\pi^\{\*\}\(\\,\\cdot\\mid x\)under the student islog⁡pθ​\(y∣x\)=log​∑zpθ​\(z∣x\)​pθ​\(y∣x,z\)\\log p\_\{\\theta\}\(y\\mid x\)=\\log\\sum\_\{z\}p\_\{\\theta\}\(z\\mid x\)\\,p\_\{\\theta\}\(y\\mid x,z\), intractable over natural\-language traces\. Introduce a backward proposalqbwd​\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\)\(instantiated by prompting the LLM to generate a CoT given\(x,y\)\(x,y\)\) and drawNNcandidatesz1,…,zN∼qbwd\(⋅∣x,y\)z\_\{1\},\\ldots,z\_\{N\}\\sim q\_\{\\mathrm\{bwd\}\}\(\\cdot\\mid x,y\)i\.i\.d\. TheNN\-sample importance\-weighted bound ofBurdaet al\.\[[7](https://arxiv.org/html/2607.21856#bib.bib58)\]is

log⁡pθ​\(y∣x\)≥𝔼z1\.\.N∼qbwd​\[log⁡1N​∑i=1Npθ​\(zi∣x\)​pθ​\(y∣x,zi\)qbwd​\(zi∣x,y\)\],\\log p\_\{\\theta\}\(y\\mid x\)\\;\\geq\\;\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\}\\\!\\left\[\\log\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{p\_\{\\theta\}\(z\_\{i\}\\mid x\)\\,p\_\{\\theta\}\(y\\mid x,z\_\{i\}\)\}\{q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\}\\right\],\(7\)strictly tighter than the standard ELBO atN=1N\{=\}1and exact in the limitN→∞N\\\!\\to\\\!\\infty\. Taking the outer expectation overxxandy∼π∗y\\sim\\pi^\{\*\}gives the population IWAE bound,

𝒥​\(θ\)≥𝔼x​𝔼y∼π∗​𝔼z1\.\.N∼qbwd​\[log⁡1N​∑i=1Npθ​\(zi,y∣x\)qbwd​\(zi∣x,y\)\]\.\\mathcal\{J\}\(\\theta\)\\;\\geq\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\\,\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\}\\\!\\left\[\\log\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{p\_\{\\theta\}\(z\_\{i\},y\\mid x\)\}\{q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\}\\right\]\.\(8\)Burdaet al\.\[[7](https://arxiv.org/html/2607.21856#bib.bib58)\]show \(their Eq\. 8\) that the gradient of Eq\. \([8](https://arxiv.org/html/2607.21856#A1.E8)\) reduces to a per\-sample SFT update with normalised soft weights,

∇θℒN=𝔼z1\.\.N​\[∑i=1Nw~i​∇θlog⁡pθ​\(zi,y∣x\)\],w~i=wi∑j=1Nwj,wi=pθ​\(zi,y∣x\)qbwd​\(zi∣x,y\),\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{N\}\\;=\\;\\mathbb\{E\}\_\{z\_\{1\.\.N\}\}\\\!\\left\[\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}\_\{i\}\\,\\nabla\_\{\\theta\}\\log p\_\{\\theta\}\(z\_\{i\},y\\mid x\)\\right\],\\quad\\widetilde\{w\}\_\{i\}=\\frac\{w\_\{i\}\}\{\\sum\_\{j=1\}^\{N\}w\_\{j\}\},\\quad w\_\{i\}=\\frac\{p\_\{\\theta\}\(z\_\{i\},y\\mid x\)\}\{q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\},\(9\)withqbwdq\_\{\\mathrm\{bwd\}\}held fixed atθr−1\\theta\_\{r\-1\}within the round \(so∇θlog⁡qbwd=0\\nabla\_\{\\theta\}\\log q\_\{\\mathrm\{bwd\}\}=0for the per\-round bound\); the inner termlog⁡pθ​\(zi,y∣x\)\\log p\_\{\\theta\}\(z\_\{i\},y\\mid x\)is exactly the joint SFT loss on\(x,zi,y\)\(x,z\_\{i\},y\)\.

The lower bound in Eq\. \([8](https://arxiv.org/html/2607.21856#A1.E8)\) is valid for any choice ofqbwd​\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\), including a fixed LLM, andN\>1N\>1tightens it strictly over theN=1N\{=\}1ELBO without optimisingqbwdq\_\{\\mathrm\{bwd\}\}’s parameters\. This sidesteps the standard variational\-inference gradient onqbwdq\_\{\\mathrm\{bwd\}\}\(expensive whenqbwdq\_\{\\mathrm\{bwd\}\}is an LLM\) that would otherwise be required to tighten the ELBO further\. The per\-sample weightswi=pθ​\(zi,y∣x\)/qbwd​\(zi∣x,y\)w\_\{i\}=p\_\{\\theta\}\(z\_\{i\},y\\mid x\)/q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)inside the log\-mean are importance weights between the jointpθ​\(z,y∣x\)p\_\{\\theta\}\(z,y\\mid x\)and the proposalqbwd​\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\); Bayes’ rule factorises them in Step 2\. Within eachLeActround,qbwdq\_\{\\mathrm\{bwd\}\}is held fixed atθr−1\\theta\_\{r\-1\}; across rounds it evolves alongside the shared model parameters that the M\-step trains, recovering an approximate\-EM alternation \(the correspondence is summarised in App\.[A\.3](https://arxiv.org/html/2607.21856#A1.SS3)\)\. The practical pipeline additionally augments the M\-step with an explicit backward SFT term \(§[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\), which arises as the reverse\-KL update onqbwdq\_\{\\mathrm\{bwd\}\}in the reweighted\-wake\-sleep regime; Step 5 below derives this\.

Step 2: per\-sample importance\-weight factorisation\.For each sampleziz\_\{i\}withqbwd​\(zi∣x,y\)\>0q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\>0, Bayes’ rule factorises the unnormalised IWAE weightwi=pθ​\(zi,y∣x\)/qbwd​\(zi∣x,y\)w\_\{i\}=p\_\{\\theta\}\(z\_\{i\},y\\mid x\)/q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)as

wi=pθ​\(y∣x,zi\)pθ​\(y∣x\)⏟=exp⁡Δ​\(zi;x,y\)⋅pθ​\(zi∣x\)qbwd​\(zi∣x,y\)⏟ρ​\(zi;x,y\)⋅pθ​\(y∣x\),w\_\{i\}\\;=\\;\\underbrace\{\\frac\{p\_\{\\theta\}\(y\\mid x,z\_\{i\}\)\}\{p\_\{\\theta\}\(y\\mid x\)\}\}\_\{=\\,\\exp\\Delta\(z\_\{i\};\\,x,\\,y\)\}\\;\\cdot\\;\\underbrace\{\\frac\{p\_\{\\theta\}\(z\_\{i\}\\mid x\)\}\{q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\}\}\_\{\\rho\(z\_\{i\};\\,x,\\,y\)\}\\;\\cdot\\;p\_\{\\theta\}\(y\\mid x\),\(10\)withΔ​\(z;x,y\):=log⁡pθ​\(y∣x,z\)−log⁡pθ​\(y∣x\)\\Delta\(z;\\,x,y\):=\\log p\_\{\\theta\}\(y\\mid x,z\)\-\\log p\_\{\\theta\}\(y\\mid x\)\. The factorpθ​\(y∣x\)p\_\{\\theta\}\(y\\mid x\)does not depend oniiand so cancels in the normalised weightw~i=wi/∑jwj\\widetilde\{w\}\_\{i\}=w\_\{i\}/\\sum\_\{j\}w\_\{j\}; onlyexp⁡Δi\\exp\\Delta\_\{i\}andρi\\rho\_\{i\}remain\. The first factorexp⁡Δi\\exp\\Delta\_\{i\}is computable from the student’s own logits:log⁡pθ​\(y∣x,zi\)\\log p\_\{\\theta\}\(y\\mid x,z\_\{i\}\)is a teacher\-forced log\-likelihood of the action under the student conditioned onziz\_\{i\}, andlog⁡pθ​\(y∣x\)=log⁡𝔼z′∼pθ\(⋅∣x\)​\[pθ​\(y∣x,z′\)\]\\log p\_\{\\theta\}\(y\\mid x\)=\\log\\mathbb\{E\}\_\{z^\{\\prime\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\)\}\[p\_\{\\theta\}\(y\\mid x,z^\{\\prime\}\)\]is estimable by Monte Carlo from a small number of forward samples \(§[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\)\.

Step 3: keeping the computable terms in the soft weight\.Dropping theii\-independentpθ​\(y∣x\)p\_\{\\theta\}\(y\\mid x\)that cancels inw~i\\widetilde\{w\}\_\{i\}, the log per\-sample weight admits the additive decomposition

log⁡wi∝log⁡pθ​\(y∣x,zi\)−log⁡pθ​\(y∣x\)⏟Δ​\(zi;x,y\)\+log⁡pθ​\(zi∣x\)−log⁡qbwd​\(zi∣x,y\)⏟log⁡ρ​\(zi;x,y\)\.\\log w\_\{i\}\\;\\propto\\;\\underbrace\{\\log p\_\{\\theta\}\(y\\mid x,z\_\{i\}\)\-\\log p\_\{\\theta\}\(y\\mid x\)\}\_\{\\Delta\(z\_\{i\};\\,x,\\,y\)\}\\;\+\\;\\underbrace\{\\log p\_\{\\theta\}\(z\_\{i\}\\mid x\)\-\\log q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\}\_\{\\log\\rho\(z\_\{i\};\\,x,\\,y\)\}\.The first two terms \(formingΔ\\Delta\) are computable from the student’s logits:log⁡pθ​\(y∣x,zi\)\\log p\_\{\\theta\}\(y\\mid x,z\_\{i\}\)is a teacher\-forced action log\-likelihood, andlog⁡pθ​\(y∣x\)=log⁡𝔼z′∼pθ\(⋅∣x\)​\[pθ​\(y∣x,z′\)\]\\log p\_\{\\theta\}\(y\\mid x\)=\\log\\mathbb\{E\}\_\{z^\{\\prime\}\\sim p\_\{\\theta\}\(\\cdot\\mid x\)\}\[p\_\{\\theta\}\(y\\mid x,z^\{\\prime\}\)\]is estimable by Monte Carlo from a small number of forward samples \(§[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\)\. In practice the last two terms are difficult to compute and we omit them\.LeActtherefore approximates the unnormalised IWAE soft weight by

wLeAct​\(zi;x,y\):=exp⁡Δ​\(zi;x,y\),w\_\{\\mathrm\{LeAct\}\}\(z\_\{i\};\\,x,y\)\\;:=\\;\\exp\\Delta\(z\_\{i\};\\,x,y\),\(11\)i\.e\. replaces the soft weightw~i∝eΔi⋅ρi\\widetilde\{w\}\_\{i\}\\propto e^\{\\Delta\_\{i\}\}\\cdot\\rho\_\{i\}by the rho\-dropped surrogatew~iLeAct∝eΔi\\widetilde\{w\}^\{\\mathrm\{LeAct\}\}\_\{i\}\\propto e^\{\\Delta\_\{i\}\}\. Thisρ\\rho\-drop is the design choice the expert\-anchored setting enables: without an external action target \(e\.g\., latent\-text variational EM\[[37](https://arxiv.org/html/2607.21856#bib.bib59)\]\), the IWAE weight has no separableΔ\\Deltaandρ\\rhoremains entangled with the bound\. §[6](https://arxiv.org/html/2607.21856#S6)verifies empirically that the resultingΔ\\Delta\-only filter is load\-bearing\.

When droppingρ\\rhois safe\.The action anchor in the backward prompt fixes the conditioned targetyyacross all candidates, soρi\\rho\_\{i\}varies withziz\_\{i\}only through how well each candidate trace explains the same fixedyy\. For high\-probability oracle actions,ρi\\rho\_\{i\}concentrates and acts approximately as a constant scaling factor across candidates, so the ranking induced byeΔie^\{\\Delta\_\{i\}\}matches the ranking induced byeΔi​ρie^\{\\Delta\_\{i\}\}\\rho\_\{i\}up to ties\. The drop biases ranking only whenρi\\rho\_\{i\}correlates withΔi\\Delta\_\{i\}across candidates, which would require the backward proposal to systematically place mass on high\-Δ\\Deltatraces in a way that cancels with the forward likelihood\. We did not observe this regime in our experiments: §[6](https://arxiv.org/html/2607.21856#S6)confirms that theΔ\\Delta\-only filter is informative against random ranking, and App\.[A\.4](https://arxiv.org/html/2607.21856#A1.SS4)shows that the parsed\-policy proxy ranking agrees with a token\-levelΔll\\Delta\_\{\\mathrm\{ll\}\}estimator on8989–98%98\\%of states \(a domain whereρ\\rhocorrelation withΔ\\Deltawould surface as proxy\-vs\-token disagreement\)\. For oracles with low\-probability actions or backward proposals concentrated on rare reasoning paths, an empiricalρ\\rhoestimate or a token\-levelΔ\\Deltaproxy would be more conservative\.

Step 4: the LeAct objective and the hard top\-KKsparsification\.Substituting the rho\-dropped surrogatew~iLeAct∝eΔi\\widetilde\{w\}^\{\\mathrm\{LeAct\}\}\_\{i\}\\propto e^\{\\Delta\_\{i\}\}into the IWAE gradient \(Eq\. \([9](https://arxiv.org/html/2607.21856#A1.E9)\)\) gives the \(approximate\) M\-step

maxθ⁡𝔼x​𝔼y∼π∗​𝔼z1\.\.N∼qbwd​\[∑i=1Nw~iLeAct​log⁡pθ​\(zi,y∣x\)\],w~iLeAct=eΔi∑j=1NeΔj,\\max\_\{\\theta\}\\;\\mathbb\{E\}\_\{x\}\\,\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\\,\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\}\\\!\\left\[\\sum\_\{i=1\}^\{N\}\\widetilde\{w\}^\{\\mathrm\{LeAct\}\}\_\{i\}\\log p\_\{\\theta\}\(z\_\{i\},\\,y\\mid x\)\\right\],\\quad\\widetilde\{w\}^\{\\mathrm\{LeAct\}\}\_\{i\}=\\frac\{e^\{\\Delta\_\{i\}\}\}\{\\sum\_\{j=1\}^\{N\}e^\{\\Delta\_\{j\}\}\},\(12\)which is the form the paper takes as theLeActtraining objective \(Eq\. \([5](https://arxiv.org/html/2607.21856#S3.E5)\)\)\.LeActreplaces the softmax weightingw~iLeAct\\widetilde\{w\}^\{\\mathrm\{LeAct\}\}\_\{i\}with a hard top\-KKfilter on the average deltaΔ¯​\(z;x\):=𝔼y∼π∗​\[Δ​\(z;x,y\)\]\\bar\{\\Delta\}\(z;\\,x\):=\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\[\\Delta\(z;\\,x,y\)\]\(a sparse approximation that keeps only the high\-weight candidates\): train on\(x,z,π∗\(⋅∣x\)\)\(x,z,\\pi^\{\*\}\(\\cdot\\mid x\)\)tuples withz∈𝒵x=top​\-​K​\{zj∼qbwd:Δ¯​\(zj;x\)\>0\}z\\in\\mathcal\{Z\}\_\{x\}=\\mathrm\{top\}\\text\{\-\}K\\bigl\\\{z\_\{j\}\\sim q\_\{\\mathrm\{bwd\}\}:\\bar\{\\Delta\}\(z\_\{j\};x\)\>0\\bigr\\\}, and use the full oracle distributionπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)as the action target, the*expert\-policy\-forcing*M\-step derived in App\.[A\.2](https://arxiv.org/html/2607.21856#A1.SS2)\. The two\-step approximation \(dropping the intractableρ\\rhofactor from the soft weight, then sparsifying the soft weight to a hard top\-KKfilter\) is what we mean by “approximate IWAE M\-step”\.

Step 5: joint backward supervision as the reverse\-KL update on the proposal\.Steps 1–4 fixqbwdq\_\{\\mathrm\{bwd\}\}within a round and only update the shared model parameters inpθp\_\{\\theta\}\. The IWAE bound is tightest whenqbwdq\_\{\\mathrm\{bwd\}\}matches the true posteriorpθ​\(z∣x,y\)p\_\{\\theta\}\(z\\mid x,y\): atN=1N\{=\}1the gap is exactlyKL\(qbwd\(⋅∣x,y\)∥pθ\(⋅∣x,y\)\)\\mathrm\{KL\}\\bigl\(q\_\{\\mathrm\{bwd\}\}\(\\,\\cdot\\mid x,y\)\\,\\\|\\,p\_\{\\theta\}\(\\,\\cdot\\mid x,y\)\\bigr\), and atN\>1N\{\>\}1the gap is a soft generalisation of the same divergence\[[7](https://arxiv.org/html/2607.21856#bib.bib58)\]\. Reducing this gap requires a separate update step onqbwdq\_\{\\mathrm\{bwd\}\}\. The natural choice \(gradient ascent on the IWAE bound itself with respect toqbwdq\_\{\\mathrm\{bwd\}\}’s parameters\) is the “wake\-phase” update of variational EM, but its signal\-to\-noise ratio degrades asqbwdq\_\{\\mathrm\{bwd\}\}approaches the posterior, requiring the STL/DReG corrections ofRoederet al\.\[[36](https://arxiv.org/html/2607.21856#bib.bib61)\], Tuckeret al\.\[[49](https://arxiv.org/html/2607.21856#bib.bib62)\]; for an LLM\-parameterised proposal this is also expensive \(each gradient step requires backprop through the IWAE log\-mean\)\. Reweighted wake\-sleep\[[4](https://arxiv.org/html/2607.21856#bib.bib60)\]replaces the wake\-phase update with a reverse\-direction KL,

minqbwdKL\(pθ\(⋅∣x,y\)∥qbwd\(⋅∣x,y\)\)=minqbwd𝔼z∼pθ\(⋅∣x,y\)\[−logqbwd\(z∣x,y\)\]\+const,\\min\_\{q\_\{\\mathrm\{bwd\}\}\}\\;\\mathrm\{KL\}\\bigl\(p\_\{\\theta\}\(\\,\\cdot\\mid x,y\)\\,\\big\\\|\\,q\_\{\\mathrm\{bwd\}\}\(\\,\\cdot\\mid x,y\)\\bigr\)\\;=\\;\\min\_\{q\_\{\\mathrm\{bwd\}\}\}\\;\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\(\\,\\cdot\\mid x,y\)\}\\\!\\bigl\[\-\\log q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\)\\bigr\]\+\\mathrm\{const\},\(13\)which is a maximum\-likelihood objective onqbwdq\_\{\\mathrm\{bwd\}\}with samples drawn from the posterior\. The posterior is itself intractable, so we importance\-sample fromqbwdq\_\{\\mathrm\{bwd\}\}:𝔼z∼pθ\(⋅∣x,y\)​\[−log⁡qbwd​\(z∣x,y\)\]=𝔼z1\.\.N∼qbwd​\[∑iw~i​\(−log⁡qbwd​\(zi∣x,y\)\)\]\\mathbb\{E\}\_\{z\\sim p\_\{\\theta\}\(\\,\\cdot\\mid x,y\)\}\\\!\\bigl\[\-\\log q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\)\\bigr\]=\\mathbb\{E\}\_\{z\_\{1\.\.N\}\\sim q\_\{\\mathrm\{bwd\}\}\}\\\!\\bigl\[\\sum\_\{i\}\\widetilde\{w\}\_\{i\}\\,\(\-\\log q\_\{\\mathrm\{bwd\}\}\(z\_\{i\}\\mid x,y\)\)\\bigr\], recovering the same self\-normalised IWAE weightsw~i\\widetilde\{w\}\_\{i\}used in Step 4\. Substituting the rho\-dropped surrogatew~iLeAct∝eΔi\\widetilde\{w\}^\{\\mathrm\{LeAct\}\}\_\{i\}\\propto e^\{\\Delta\_\{i\}\}and then the hard top\-KKindicator on𝒵x\\mathcal\{Z\}\_\{x\}used by the M\-step yields the practical update

ℒbwd​\(θ\)=−𝔼\(x,z\)∈𝒟r​\[log⁡pθ​\(z∣x,π~∗\)\],\\mathcal\{L\}\_\{\\mathrm\{bwd\}\}\(\\theta\)\\;=\\;\-\\,\\mathbb\{E\}\_\{\(x,z\)\\in\\mathcal\{D\}\_\{r\}\}\\\!\\bigl\[\\log p\_\{\\theta\}\(z\\mid x,\\tilde\{\\pi\}^\{\*\}\)\\bigr\],\(14\)i\.e\., supervised cross\-entropy training ofqbwdq\_\{\\mathrm\{bwd\}\}on the same selected positive\-Δ\\Deltatraces𝒟r\\mathcal\{D\}\_\{r\}that drive the M\-step \(withπ~∗\\tilde\{\\pi\}^\{\*\}the redacted\-distribution prompt as defined in §[3\.2](https://arxiv.org/html/2607.21856#S3.SS2)\)\. Becauseqbwdq\_\{\\mathrm\{bwd\}\}shares parameters withpθp\_\{\\theta\}\(one decoder, two prompts\), the wake\-sleep update lands on the sameθ\\thetaand Eq\. \([14](https://arxiv.org/html/2607.21856#A1.E14)\) is the joint backward term in §[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\. Across rounds, this update tracksqbwdq\_\{\\mathrm\{bwd\}\}to the moving posterior under the current student, monotonically tightening the IWAE bound from one round to the next under the standard reweighted\-wake\-sleep guarantee\[[4](https://arxiv.org/html/2607.21856#bib.bib60)\]; the empirical effect of including this term is ablated in App\.[C](https://arxiv.org/html/2607.21856#A3)\.

EM as a special case\.SettingN=1N=1collapses Eq\. \([8](https://arxiv.org/html/2607.21856#A1.E8)\) back to the standard ELBO, and further settingqbwd\(⋅∣x,y\)=pθ\(t−1\)\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(\\,\\cdot\\mid x,y\)=p\_\{\\theta^\{\(t\-1\)\}\}\(z\\mid x,y\)gives the standard variational\-EM E\-step, in which caseρ≡1\\rho\\equiv 1, the soft weightexp⁡Δ\\exp\\Deltais also≡1\\equiv 1, and the bound is tight\.LeActreplaces this true posterior with an LLM\-defined backward proposal atN\>1N\>1, which is what introduces both theρ\\rhoresidual and theexp⁡Δ\\exp\\Deltacorrection; alternating Eq\. \([8](https://arxiv.org/html/2607.21856#A1.E8)\) maximisation with such a proposal recovers approximate\-EM\. TheLeAct–EM correspondence is summarised in App\.[A\.3](https://arxiv.org/html/2607.21856#A1.SS3)\.

### A\.2Expert\-Policy Forcing: the Exact M\-Step Target

The outer expectation𝔼y∼π∗​\[⋅\]\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\[\\,\\cdot\\,\]in the LeAct objective Eq\. \([5](https://arxiv.org/html/2607.21856#S3.E5)\) \(equivalently the second expectation in Eq\. \([8](https://arxiv.org/html/2607.21856#A1.E8)\)\) admits two implementations\.

Sampled\-action target\.Drawy~∼π∗\\tilde\{y\}\\sim\\pi^\{\*\}and minimise−log⁡pθ​\(y~∣x,z\)\-\\log p\_\{\\theta\}\(\\tilde\{y\}\\mid x,z\)\. This is the standard expert\-iteration choice and keeps the\(z,y~\)\(z,\\tilde\{y\}\)pair internally consistent because both were decoded together\.

Expert\-distribution target \(expert\-policy forcing\)\.Because the oracle supplies the*full*π∗\\pi^\{\*\}, the outer expectation can be evaluated exactly:

𝔼y∼π∗\[−logpθ\(y∣x,z\)\]=H\(π∗\)\+DKL\(π∗\(⋅∣x\)∥pθ\(⋅∣x,z\)\),\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\\\!\\bigl\[\-\\log p\_\{\\theta\}\(y\\mid x,z\)\\bigr\]\\;=\\;H\(\\pi^\{\*\}\)\+D\_\{\\mathrm\{KL\}\}\\bigl\(\\pi^\{\*\}\(\\,\\cdot\\mid x\)\\,\\\|\\,p\_\{\\theta\}\(\\,\\cdot\\mid x,z\)\\bigr\),\(15\)so minimising the left\-hand side with respect toθ\\thetais equivalent to minimisingDKL\(π∗∥pθ\(⋅∣x,z\)\)D\_\{\\mathrm\{KL\}\}\(\\pi^\{\*\}\\,\\\|\\,p\_\{\\theta\}\(\\,\\cdot\\mid x,z\)\), i\.e\., the SFT target is the exact oracle distribution, independent of whichever policy was decoded at generation time\.

When the two choices matter\.For mixed Nash equilibria the two gradients differ\. Delta scoring \(Eq\. \([4](https://arxiv.org/html/2607.21856#S3.E4)\)\) already evaluatesΔ\\Deltaagainst the action distribution underπ∗\\pi^\{\*\}, so takingπ∗\\pi^\{\*\}as the SFT target keeps the E\- and M\-steps aligned on the same distribution\. §[6](https://arxiv.org/html/2607.21856#S6)shows this alignment is load\-bearing: removing it reverses which algorithm wins at 47K information sets\.

The two\-part loss\.The selected traces𝒟r\\mathcal\{D\}\_\{r\}are reused in two prompt formats that share the sameθ\\theta\. The forward loss

ℒfwd​\(θ;𝒟r\)=−𝔼\(x,z\)∈𝒟r​\[log⁡pθ​\(z∣x\)\+𝔼y∼π∗\(⋅∣x\)​\[log⁡pθ​\(y∣x,z\)\]\]\\mathcal\{L\}\_\{\\mathrm\{fwd\}\}\(\\theta;\\,\\mathcal\{D\}\_\{r\}\)\\;=\\;\-\\,\\mathbb\{E\}\_\{\(x,\\,z\)\\in\\mathcal\{D\}\_\{r\}\}\\\!\\Bigl\[\\log p\_\{\\theta\}\(z\\mid x\)\\;\+\\;\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\(\\cdot\\mid x\)\}\\\!\\bigl\[\\log p\_\{\\theta\}\(y\\mid x,z\)\\bigr\]\\Bigr\]\(16\)trains the student to producezzfromxxand then the oracle’s full action distributionπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)given\(x,z\)\(x,z\)\. The backward loss

ℒbwd​\(θ;𝒟r\)=−𝔼\(x,z\)∈𝒟r​\[log⁡pθ​\(z∣x,π~∗\)\]\\mathcal\{L\}\_\{\\mathrm\{bwd\}\}\(\\theta;\\,\\mathcal\{D\}\_\{r\}\)\\;=\\;\-\\,\\mathbb\{E\}\_\{\(x,\\,z\)\\in\\mathcal\{D\}\_\{r\}\}\\\!\\bigl\[\\log p\_\{\\theta\}\(z\\mid x,\\,\\tilde\{\\pi\}^\{\*\}\)\\bigr\]\(17\)retrains the proposalqbwdq\_\{\\mathrm\{bwd\}\}on the same selectedzzgiven the\(x,π~∗\)\(x,\\tilde\{\\pi\}^\{\*\}\)prompt, the wake\-sleep update onqbwdq\_\{\\mathrm\{bwd\}\}derived in App\.[A\.1](https://arxiv.org/html/2607.21856#A1.SS1)Step 5\. The full M\-step minimisesℒ=ℒfwd\+ℒbwd\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{fwd\}\}\+\\mathcal\{L\}\_\{\\mathrm\{bwd\}\}, treating the forward and backward formats as two views of the same trace pool𝒟r\\mathcal\{D\}\_\{r\}\.

### A\.3Algorithm–Formulation Correspondence

Table[4](https://arxiv.org/html/2607.21856#A1.T4)maps each formal quantity from App\.[A\.1](https://arxiv.org/html/2607.21856#A1.SS1)to itsLeActcounterpart\.

Table 4:Mapping from the IWAE bound to theLeActalgorithm\.Each formal quantity \(App\.[A\.1](https://arxiv.org/html/2607.21856#A1.SS1)\) corresponds to a concrete pipeline component\.Formal quantityLeAct’s approximationAlgorithm componentProposalqbwd​\(z∣x,y\)q\_\{\\mathrm\{bwd\}\}\(z\\mid x,y\)Backward decode atθr−1\\theta\_\{r\-1\}, redactedπ~∗\\tilde\{\\pi\}^\{\*\}Backward generationPer\-sample IWAE weightwi=eΔi​ρi​pθ​\(y∣x\)w\_\{i\}=e^\{\\Delta\_\{i\}\}\\rho\_\{i\}\\,p\_\{\\theta\}\(y\\mid x\)wiLeAct=eΔiw^\{\\mathrm\{LeAct\}\}\_\{i\}=e^\{\\Delta\_\{i\}\}\(ρi\\rho\_\{i\}dropped\)Forward delta scoringSoftmax weightw~i∝eΔi\\widetilde\{w\}\_\{i\}\\propto e^\{\\Delta\_\{i\}\}Hard top\-KKon positiveΔ¯​\(z;x\)\\bar\{\\Delta\}\(z;x\)Top\-KKselectionJoint losslog⁡pθ​\(z,y∣x\)\\log p\_\{\\theta\}\(z,y\\mid x\)SFT on\(x,z,y\)\(x,z,y\)tuplesSupervised fine\-tuningOuter expectation𝔼y∼π∗​\[⋅\]\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\[\\,\\cdot\\,\]Target=π∗\(⋅∣x\)=\\pi^\{\*\}\(\\,\\cdot\\mid x\)Expert\-policy forcing

The full procedure is collected in Algorithm[1](https://arxiv.org/html/2607.21856#algorithm1)\(main text §[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\)\.

### A\.4Textual Policy Proxy: FromΔ\\Deltato Implementation

The formal definitionΔ​\(z;x,y\)=log⁡pθ​\(y∣x,z\)−log⁡pθ​\(y∣x\)\\Delta\(z;\\,x,y\)=\\log p\_\{\\theta\}\(y\\mid x,z\)\-\\log p\_\{\\theta\}\(y\\mid x\)in Eq\. \([4](https://arxiv.org/html/2607.21856#S3.E4)\) readslog⁡pθ​\(y∣x,z\)\\log p\_\{\\theta\}\(y\\mid x,z\)as the decoder’s next\-token log\-likelihood of the action under the student conditioned onzz\. Our implementation does not score the action token by token\. Instead, the model is supervised during SFT to emit a textual policy line of the form “Action:aa\\n Policy:\{ai:pi\}i\\\{a\_\{i\}\\\!:\\\!p\_\{i\}\\\}\_\{i\}” alongside its reasoning trace, and the scoring pipeline readsΔ\\Deltaoff this line\. This appendix bridges the formalΔ\\Deltaused in §[3\.2](https://arxiv.org/html/2607.21856#S3.SS2)and App\.[A\.1](https://arxiv.org/html/2607.21856#A1.SS1)to the parsed\-policy quantityΔ^\\widehat\{\\Delta\}that our pipeline actually evaluates\.

The implementation quantity\.At scoring time we parse the textual policy line into a distributionπ^θ\(⋅∣x,z\)\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x,z\)over legal actions and compute the average\-delta proxy

Δ^\(z;x\)=−KL\(π∗\(⋅∣x\)∥π^θ\(⋅∣x,z\)\)−\[−KL\(π∗\(⋅∣x\)∥π^θ\(⋅∣x\)\)\],\\widehat\{\\Delta\}\(z;\\,x\)\\;=\\;\-\\,\\mathrm\{KL\}\\\!\\bigl\(\\pi^\{\*\}\(\\cdot\\mid x\)\\;\\\|\\;\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x,z\)\\bigr\)\\;\-\\;\\bigl\[\-\\,\\mathrm\{KL\}\\\!\\bigl\(\\pi^\{\*\}\(\\cdot\\mid x\)\\;\\\|\\;\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x\)\\bigr\)\\bigr\],whereπ^θ\(⋅∣x\)\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x\)is averaged over Monte\-Carlo forward samples without conditioning on a backwardzz\.

Equivalence toΔ¯\\bar\{\\Delta\}under proxy correctness\.Up to the entropy termH​\(π∗\)H\(\\pi^\{\*\}\)that cancels between the two KLs,

Δ^​\(z;x\)=𝔼y∼π∗\(⋅∣x\)​\[log⁡π^θ​\(y∣x,z\)−log⁡π^θ​\(y∣x\)\],\\widehat\{\\Delta\}\(z;\\,x\)\\;=\\;\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\(\\cdot\\mid x\)\}\\\!\\bigl\[\\log\\hat\{\\pi\}\_\{\\theta\}\(y\\mid x,z\)\\;\-\\;\\log\\hat\{\\pi\}\_\{\\theta\}\(y\\mid x\)\\bigr\],which equals the average deltaΔ¯​\(z;x\):=𝔼y∼π∗​\[Δ​\(z;x,y\)\]\\bar\{\\Delta\}\(z;\\,x\):=\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\[\\Delta\(z;\\,x,y\)\]used by the hard top\-KKfilter \(footnote of Eq\. \([5](https://arxiv.org/html/2607.21856#S3.E5)\)\) wheneverπ^θ\(⋅∣x,z\)=pθ\(⋅∣x,z\)\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x,z\)=p\_\{\\theta\}\(\\cdot\\mid x,z\)on the action support\. So the implementation evaluates an𝔼y∼π∗\\mathbb\{E\}\_\{y\\sim\\pi^\{\*\}\}\-averaged version of the formalΔ\\Deltafor free, and the soft→\\tohard reduction is the only step left\.

When the proxy diverges from the decoder distribution\.The textual policy is a self\-report: the model is free to write rounded numbers, to disagree with the action token it just emitted, or to omit minor mass that the decoder would still place on rare actions\. A token\-level evaluationlog⁡pθ​\(y∣x,z\)\\log p\_\{\\theta\}\(y\\mid x,z\)via teacher forcing would instead read the next\-token logit at the action position and is the literal quantity in Eq\. \([4](https://arxiv.org/html/2607.21856#S3.E4)\)\. The two coincide only when the textual line faithfully matches the decoder’s softmax over the action vocabulary\.

Why the proxy suffices in our setting\.Three properties keep the substitution lossless in practice\. First, the SFT objective \(App\.[A\.2](https://arxiv.org/html/2607.21856#A1.SS2)\) supervises both the reasoning trace and the policy line, so training and scoring read off the same textual channel rather than two unaligned views of the model\. Second,Δ¯\\bar\{\\Delta\}is a difference between two quantities both computed from the same parser and the same continuation format; any systematic bias in the textual self\-report shiftsπ^θ\(⋅∣x,z\)\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x,z\)andπ^θ\(⋅∣x\)\\hat\{\\pi\}\_\{\\theta\}\(\\cdot\\mid x\)in the same direction and cancels in the delta\. Third, because the M\-step targets the oracleπ∗\\pi^\{\*\}rather than the parsed line, scoring noise propagates only through whichznz\_\{n\}are kept; the resulting𝒟r\\mathcal\{D\}\_\{r\}remains anchored toπ∗\\pi^\{\*\}\.

Selection\-Fidelity Audit\.We quantify the agreement betweenΔ¯\\bar\{\\Delta\}\(the parsed\-policy proxy used by the M\-step\) and a token\-level estimatorΔll​\(z;x\):=log⁡pθ​\(t∗​\(x\)∣x,z\)−1M​∑mlog⁡pθ​\(t∗​\(x\)∣x,zmfwd\)\\Delta\_\{\\mathrm\{ll\}\}\(z;\\,x\):=\\log p\_\{\\theta\}\(t^\{\*\}\(x\)\\mid x,z\)\-\\tfrac\{1\}\{M\}\\sum\_\{m\}\\log p\_\{\\theta\}\(t^\{\*\}\(x\)\\mid x,z^\{\\mathrm\{fwd\}\}\_\{m\}\), wheret∗​\(x\)t^\{\*\}\(x\)is the canonical policy linePolicy:​\{ai:pi∗\}i\\texttt\{Policy:\}\\\{a\_\{i\}\\\!:\\\!p^\{\*\}\_\{i\}\\\}\_\{i\}andzmfwdz^\{\\mathrm\{fwd\}\}\_\{m\}areM=4M\{=\}4student\-decoded forward CoTs scored without backward conditioning\. Because the hard top\-KKM\-step consumes only the within\-infoset ordering of positives, the load\-bearing audit metric is the within\-infoset top\-11agreement\. On a 100K\-candidate audit over16,97716\{,\}977multi\-candidate Leduc\-13r4s infosets, whenΔll\\Delta\_\{\\mathrm\{ll\}\}’s within\-infoset top\-11falls on an infoset containing some positive\-Δ¯\\bar\{\\Delta\}candidate, that selection is also positive underΔ¯\\bar\{\\Delta\}in89%89\\%of cases \(base coldstart\); on well\-trainedLeActiterates this conditional rate reaches98%98\\%\(Table[5](https://arxiv.org/html/2607.21856#A1.T5)\)\. Global rank correlation across all candidates is moderate \(ρ≈0\.65\\rho\\approx 0\.65\), reflecting scale disagreement on candidates the M\-step never sees; this is a secondary diagnostic rather than a load\-bearing fidelity number\.

Audit subsetNinfN\_\{\\mathrm\{inf\}\}top\-11hit∣∃\>​0\\mid\\exists\{\>\}0sign concord\.100100K candidates, base coldstart16,97716\{,\}9770\.890\.890\.800\.8055K candidates, base coldstart74740\.930\.930\.810\.8155K candidates, best LeAct iterate74740\.980\.980\.840\.84Table 5:Parsed\-policy proxyΔ¯\\bar\{\\Delta\}agrees with token\-levelΔll\\Delta\_\{\\mathrm\{ll\}\}on the load\-bearing top\-11selection \(89–98%\)\.Selection\-Fidelity Audit on Leduc\-13r4s, decisive\-Nash subset \(h​\(π∗\)/log⁡\|A\|≤0\.95h\(\\pi^\{\*\}\)/\\log\|A\|\\leq 0\.95\)\.NinfN\_\{\\mathrm\{inf\}\}counts multi\-candidate infosets\. top\-11hit∣∃\>​0\\mid\\exists\{\>\}0is the fraction ofΔll\\Delta\_\{\\mathrm\{ll\}\}\-top\-11picks with positiveΔ¯\\bar\{\\Delta\}, conditional on the infoset containing at least one positive\-Δ¯\\bar\{\\Delta\}candidate\. Sign concord\. isPr⁡\(Δ¯\>0​∣Δll\>​0\)\\Pr\(\\bar\{\\Delta\}\{\>\}0\\mid\\Delta\_\{\\mathrm\{ll\}\}\{\>\}0\)\.Boundary cases\.We clip parsed KL values at1010\(rare deterministic outputs against a mixedπ∗\\pi^\{\*\}would otherwise diverge\), and we treat candidates with malformed policy lines as having no information by routing them through a deterministic fallback on the parsed action\. Replacing the textual proxy with token\-levellog⁡pθ​\(y∣x,z\)\\log p\_\{\\theta\}\(y\\mid x,z\)scoring is a straightforward substitution that we leave to future work; we do not expect the qualitative findings of §[5](https://arxiv.org/html/2607.21856#S5)to change, since both estimators agree in expectation under a faithfully\-trained policy line\.

## Appendix BAblation Studies: Selection, Scoring, and Supervision

Top\-KKselection rate ablation \(Leduc 13r4s\)\.We ablate the selection countK∈\{1,2,4\}K\\in\\\{1,2,4\\\}in the Leduc 13r4s R2 pool withN=8N\{=\}8per infoset and matched hyperparameters \(coldstart base,lr=5×10−6\\text\{lr\}=5\\times 10^\{\-6\}, matched training budget\)\. Table[6](https://arxiv.org/html/2607.21856#A2.T6)reports per\-epoch results\.K=1K\{=\}1underperformsK≥2K\{\\geq\}2across all checkpoints, confirming that aggressive top\-1 selection discards useful gradient\. BetweenK=2K\{=\}2andK=4K\{=\}4the cells are close:K=2K\{=\}2wins at ep1 \(single0\.410\.41vs0\.610\.61,BoN0\.1070\.107vs0\.1100\.110\) and the gap is similar in the opposite direction at ep2–3, with neither setting dominating across all six cells\. Because the two are comparable on selection quality andK=2K\{=\}2halves the per\-round SFT data budget, we standardise onK=2K\{=\}2across the paper \(Table[11](https://arxiv.org/html/2607.21856#A8.T11)\) for compute efficiency\.

Table 6:K=1K\{=\}1underperformsK≥2K\{\\geq\}2;K=2K\{=\}2andK=4K\{=\}4are comparable\.Top\-KKselection ablation on the Leduc 13r4s R2 pool \(N=8N\{=\}8, matched HP\)\. Per\-epoch single andBoNexploitability;bold= best per metric across all cells\. All variants share the same R2 pool, base \(Leduc 10r4s joint coldstart\),lr=5×10−6\\text\{lr\}=5\{\\times\}10^\{\-6\}, 3 epochs\.Single↓\\downarrowBoN↓\\downarrowKKep1ep2ep3ep1ep2ep310\.78911\.09000\.48630\.12850\.11210\.139220\.41250\.39150\.40040\.10690\.07530\.128240\.60730\.32480\.39320\.10970\.05630\.0518Panel C: Within\-LeActcomponent ablation on Liar’s Dice\.A within\-LeActablation isolates the relative contribution of the joint forward–backward coldstart vs\. the backward delta scoring stage\. With the joint coldstart fixed at100%100\\%infoset coverage, removing the scoring stage stops the policy at exploitability0\.4390\.439\(coverage\-only, no scoring\); restoring the scoring stage brings the policy to0\.0190\.019\(R1\) and0\.0020\.002\(R2\)\. The219×219\\timesR2 gain is attributable to selection alone, once the coldstart contribution is held constant\.

Table 7:Backward scoring drives the219×219\\timesR2 gain on Liar’s Dice; coverage alone stops at0\.4390\.439\.Joint coldstart fixed at100%100\\%infoset coverage; only the backward delta scoring stage is toggled\.ComponentVariantScaleSingleBoNBackward scoringJoint coldstart, no scoringLD0\.4390\.251Joint coldstart\+\+scoring \(R2\)LD0\.0020\.001
## Appendix CIteration Dynamics

A distinctive feature ofLeActis its ability to iterate: the improved model both generates better backward reasoning and serves as a tighter forward scorer\. Per\-domain headline numbers appear in Table[8](https://arxiv.org/html/2607.21856#A4.T8); this appendix isolates two questions that table does not address: which E\-step variant to run, and when to stop\.

E\-step variant comparison\.We compare three E\-step variants from the same R1 base on Leduc 10r4s:*DAgger*\(expert backward prompts, model forward scorer\),*Joint*\(self\-generated backward traces, joint SFT\), and*REINFORCE*\(self\-generated backward traces, forward\-only SFT\)\. The rankingReinforce\>\>Joint\>\>DAggeris consistent across metrics\.Reinforcewins because the model’s own backward traces are more compatible with its forward reasoning than expert\-generated ones, and forward\-only SFT avoids introducing noise from the backward task during training\.

Iteration stopping rule\.LeActhalts at the round when the policy first matches the oracle’s distribution\. Three diagnostics indicate the floor has been reached: \(i\) close\-fraction \(KL<0\.1<\\,0\.1\) saturating at1\.01\.0\(Liar’s Dice R2 reaches NashConv0\.0020\.002and close\-frac1\.01\.0\); \(ii\) single\-sample exploitability falling below the prior round’sBoN\(Leduc 13r4s R2 single0\.0920\.092below R1BoN0\.0360\.036\); \(iii\) single\-sample KL falling below∼0\.002\{\\sim\}0\.002at10910^\{9\}\-scale \(FHP R2 single KL0\.00190\.0019\)\. The rule is prospective: once any of \(i\)–\(iii\) fires at roundrr, we report round\-rras the headline cell and do not extend tor\+1r\{\+\}1\. Applying it yields different per\-setting termination points: 3\-Player Leduc meets \(i\) at R1 \(close\-fraction=1=1at the solver floor\); Leduc 6r2s/10r4s/13r4s, Liar’s Dice, and FHP terminate at R2 once one of \(i\)–\(iii\) fires; BuilderBench is reported at R1 due to compute\.

LeAct’s contribution is the closed\-loop backward\-and\-score algorithm itself, not a depth claim about iteration count\. At matched coldstart and data budget, R1 alone already separatesLeActfrom forwardCoTExItacross every reasoning\-game setting \(for instance, Leduc 13r4s R1 single0\.0970\.097vs\.CoTExItR10\.5010\.501; Liar’s Dice R10\.0190\.019vs\.0\.6710\.671\); iteration is a means of tightening once the oracle’s distribution is matched, not the source of the imitation gap\.

## Appendix DAdditional Results

### D\.1Numerical Detail for Fig\.[2](https://arxiv.org/html/2607.21856#S4.F2)

Table[8](https://arxiv.org/html/2607.21856#A4.T8)reports the full numerical breakdown behind Fig\.[2](https://arxiv.org/html/2607.21856#S4.F2)of §[5\.1](https://arxiv.org/html/2607.21856#S5.SS1), including stds over multi\-seed evaluation and thegpt\-5\.5zero\-shot anchor at the smallest setting\.

Table 8:Numerical detail behind Fig\.[2](https://arxiv.org/html/2607.21856#S4.F2)\.Game rows: exploitability orNashConv\(lower better\); BuilderBench row: cube\-placement progress \(higher better\)\.BoNusesN=8N\{=\}8for games andN=64N\{=\}64for BuilderBench \(single max per task, no across\-seed std\)\.Bold= best per metric per row\.gpt\-5\.5is a frontier zero\-shot anchor \(Pass@1, medium reasoning effort\) reported on the smallest setting only; FHP lives in Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1)\(generalisation\-only at this scale\)\.gpt\-5\.5SingleBoNSetting \(\# infosets\)MetricCold\.NoCoTBCCoTExItLeActCold\.NoCoTBCCoTExItLeActLeduc 6r2s \(4K\)expl\. \(↓\\downarrow\)0\.7991\.351±\\pm0\.2260\.003±\\pm0\.0010\.087±\\pm0\.0180\.016±\\pm0\.0040\.107±\\pm0\.0260\.001±\\pm0\.00020\.029±\\pm0\.0070\.006±\\pm0\.002Leduc 10r4s \(47K\)expl\. \(↓\\downarrow\)—1\.670±\\pm0\.0630\.024±\\pm0\.0090\.319±\\pm0\.0890\.037±\\pm0\.0240\.282±\\pm0\.0110\.002±\\pm0\.00030\.058±\\pm0\.0160\.0008±\\pm0\.0002Leduc 13r4s \(80K\)expl\. \(↓\\downarrow\)—0\.687±\\pm0\.0680\.498±\\pm0\.0500\.501±\\pm0\.0160\.092±\\pm0\.0330\.402±\\pm0\.0180\.334±\\pm0\.0360\.087±\\pm0\.0050\.022±\\pm0\.004Liar’s Dice \(24K\)expl\. \(↓\\downarrow\)—0\.818±\\pm0\.0820\.018±\\pm0\.0030\.156±\\pm0\.0240\.002±\\pm0\.0010\.784±\\pm0\.0580\.009±\\pm0\.0020\.043±\\pm0\.0070\.001±\\pm0\.00023\-Player Leduc \(14K\)NashConv\(↓\\downarrow\)—1\.482±\\pm0\.1030\.773±\\pm0\.0410\.473±\\pm0\.0570\.005±\\pm0\.0010\.354±\\pm0\.0820\.770±\\pm0\.0240\.146±\\pm0\.0280\.003±\\pm0\.001BuilderBench \(26 tasks\)Progress \(↑\\uparrow\)—0\.5448±\\pm0\.028—0\.5017±\\pm0\.0380\.3527±\\pm0\.0680\.8229—0\.70060\.8853

##### Error Bar Calculation\.

For the five game\-setting rows, error bars are standard deviations across33inference\-rollout seeds of the single trained policy per cell \(each seed reseeds the vLLM sampler, producing a distinct sampled policy on the full game tree; std is taken across the33resulting exploitability orNashConvvalues\)\. BuilderBench is also single\-trained per cell; we substitute an inference\-side proxy with a larger seed count\. LetT=26T=26\(number of in\-domain tasks\) andSSbe the number of inference seeds \(S=64S=64for trained cells;S=16S=16for the joint coldstart eval\)\. Definext,s∈\[0,1\]x\_\{t,s\}\\in\[0,1\]as the cube\-placement ratio \(fraction of cubes placed at target\) on taskttin seedss\. The Single column reportsμ¯=1T​∑tμt\\bar\{\\mu\}=\\frac\{1\}\{T\}\\sum\_\{t\}\\mu\_\{t\}whereμt=1S​∑sxt,s\\mu\_\{t\}=\\frac\{1\}\{S\}\\sum\_\{s\}x\_\{t,s\}, with error barσsingle=std​\(x¯1,x¯2,…,x¯S\),x¯s=1T​∑txt,s,\\sigma\_\{\\mathrm\{single\}\}\\;=\\;\\mathrm\{std\}\\\!\\left\(\\,\\bar\{x\}\_\{1\},\\,\\bar\{x\}\_\{2\},\\,\\dots,\\,\\bar\{x\}\_\{S\}\\,\\right\),\\quad\\bar\{x\}\_\{s\}=\\tfrac\{1\}\{T\}\\sum\_\{t\}x\_\{t,s\},i\.e\. the per\-seed cross\-task mean is computed once per inference seed and the std is taken across theSSresulting values\. TheBoNcolumn uses best\-of\-N=64N\{=\}64,maxs⁡xt,s\\max\_\{s\}x\_\{t,s\}per task; this collapses the seed dimension to one value per task, so no across\-seed std is defined and we omit error bars there\.

### D\.2BuilderBench: Partition and SFT\-Headroom Diagnostic

BuilderBench serves as the cross\-domain probe and the low\-recall half of the EPF×\\timesmethod ablation \(§[6](https://arxiv.org/html/2607.21856#S6)\)\. The benchmark is a robotics cube\-manipulation task \(UR5e arm, MuJoCo, 51 tasks from cube\-1 to cube\-50\)\. The expert oracle is a pool of golden trajectories from three frontier models \(GPT\-5\.2 as a CoT agent, plus Claude Opus 4\.6 and Gemini 3 Flash as reflexion agents, first successful episode per model\)\. The model is Qwen3\-8B, and the pipeline matches the one used for game domains \(BuilderBench coldstart corpus details in App\.[H\.3](https://arxiv.org/html/2607.21856#A8.SS3)\): joint forward–backward coldstart SFT on the frontier\-model trajectories, backward generation on held\-out trajectories, forward delta scoring with top\-KKselection, then SFT\.

#### D\.2\.1Heuristic Action\-Match Reward

BuilderBench’s expert oracle commits to a single JSON actiony∗y^\{\*\}per state \(action type, target cube id, and target position\) rather than a distribution over actions, so the log\-likelihood terms inΔ​\(z;x,y\)\\Delta\(z;\\,x,y\)from Eq\. \([4](https://arxiv.org/html/2607.21856#S3.E4)\) are not directly defined\. We substitute a heuristic action\-match rewardr​\(y;y∗\)r\(y;\\,y^\{\*\}\)that grades partial overlap on the three action attributes:

r​\(y;y∗\)=\{1action type and cube id matched,​‖Δ​pos‖2≤1​cm,max⁡\(0,1−‖Δ​pos‖2\)action type and cube id matched, larger position offset,0\.1action type matched only,0otherwise\.r\(y;\\,y^\{\*\}\)\\;=\\;\\begin\{cases\}1&\\text\{action type and cube id matched, \}\\\|\\Delta\\mathrm\{pos\}\\\|\_\{2\}\\leq 1\\,\\text\{cm\},\\\\ \\max\\\!\\bigl\(0,\\,1\-\\\|\\Delta\\mathrm\{pos\}\\\|\_\{2\}\\bigr\)&\\text\{action type and cube id matched, larger position offset\},\\\\ 0\.1&\\text\{action type matched only\},\\\\ 0&\\text\{otherwise\}\.\\end\{cases\}\(18\)The candidate’s forward delta becomes the with\-CoT vs\. without\-CoT reward gapΔ​\(z;x,y∗\)=r​\(yz;y∗\)−r​\(y∅;y∗\)\\Delta\(z;\\,x,y^\{\*\}\)=r\(y\_\{z\};\\,y^\{\*\}\)\-r\(y\_\{\\emptyset\};\\,y^\{\*\}\), whereyzy\_\{z\}andy∅y\_\{\\emptyset\}are the actions the model decodes with and without the candidate CoTzzin context\. The rest of the M\-step pipeline \(positive\-delta filter, top\-KKselection\) is unchanged\.

#### D\.2\.2Cross\-Domain Replication of the Poker2×22\\times 2

Coldstart anchor\.After joint coldstart SFT \(3 epochs\), the model reaches aggregate pass@64=15/4664=15/46on the 46\-task BB\-51 denominator \(5 context\-overflow tasks excluded; §[6](https://arxiv.org/html/2607.21856#S6), Table[3](https://arxiv.org/html/2607.21856#S6.T3)\)\. The zero\-shot Qwen3\-8B base reaches6/466/46\.

Method×\\timesEPF interaction\.Running the fullLeActloop \(N=8N\{=\}8backward candidates per task, forward delta scoring, top\-KKselection, 3 SFT epochs\) and crossing with the EPF axis produces the four cells reported in Table[3](https://arxiv.org/html/2607.21856#S6.T3):\(LeAct,EPF\)=21/46\(\\texttt\{LeAct\}\{\},\\mathrm\{EPF\}\)=21/46,\(LeAct,no\-EPF\)=14/46\(\\texttt\{LeAct\}\{\},\\text\{no\-EPF\}\)=14/46,\(ExIt\+CoT,EPF\)=10/46\(\\textsc\{ExIt\}\{\}\+\\textsc\{CoT\}\{\},\\mathrm\{EPF\}\)=10/46, and\(ExIt\+CoT,no\-EPF\)=2/46\(\\textsc\{ExIt\}\{\}\+\\textsc\{CoT\}\{\},\\text\{no\-EPF\}\)=2/46\. Only the diagonal that combines backward generation with EPF clears the coldstart anchor of15/4615/46; the worst cell sits an order of magnitude below the best, separating the four configurations along both axes\.

Diagnosis\.Forward scoring picks up signal on this domain \(25%25\\%of backward candidates receive positive delta, well above the noise floor\) and translates into end\-task improvement, but*only*under the EPF supervision target\. Ablating either the backward pool \(CoTExItpipeline\) or the EPF target costs absolute pass@6464, in either combination\. This is the same dependence structure as poker \(§[6](https://arxiv.org/html/2607.21856#S6)\), now visible on a non\-game domain whose oracle and metric have nothing to do with Nash policies\. The non\-trivial positive\-delta rate shows the scoring signal is alive on this domain, so the failure of the three off\-diagonal cells cannot be attributed to delta\-scoring collapse—it has to be a supervision\-target effect, which is what EPF controls\.

Implication forLeAct’s scope\.BuilderBench establishes thatLeActadds aggregate end\-task value when the supervision target is the oracle distribution itself; without EPF the pipeline regresses on this domain\. The same domain doubles as a low\-recall mechanism probe: the cross\-domain2×22\\times 2on EPF×\\,\\times\\,method \(§[6](https://arxiv.org/html/2607.21856#S6)\) uses BuilderBench as the low\-recall half \(continuous\-valued action arguments rarely repeat across states\), and the pass@6464ordering \(LeAct\+EPF\>LeAct\+no\-EPF\>ExIt\+CoT\+EPF\>ExIt\+CoT\+no\-EPF\\texttt\{LeAct\}\{\}\+\\mathrm\{EPF\}\>\\texttt\{LeAct\}\{\}\+\\text\{no\-EPF\}\>\\textsc\{ExIt\}\{\}\+\\textsc\{CoT\}\{\}\+\\mathrm\{EPF\}\>\\textsc\{ExIt\}\{\}\+\\textsc\{CoT\}\{\}\+\\text\{no\-EPF\}\) rules out the “no\-EPF collapsesLeAct” explanation that high\-recall poker invites\. BuilderBench therefore contributes both generalisation evidence \(§[5\.2](https://arxiv.org/html/2607.21856#S5.SS2)\) and mechanism evidence \(§[6](https://arxiv.org/html/2607.21856#S6)\) to the paper’s claims\.

### D\.3Flop Hold’em: Win\-Rate Methodology

KL summary\.The hold\-out KL summary is in Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1)of the main text \(§[5\.2](https://arxiv.org/html/2607.21856#S5.SS2)\); this subsection focuses on the chip\-outcome \(win\-rate\) analysis that complements it\.

Setup\.Methodology recapped here for self\-containment: paired self\-play, model\-vs\-model \(not model\-vs\-teacher\), chip outcomes averaged overN=50,000N\{=\}50\{,\}000paired hands with two seatings \(25,00025\{,\}000each, identical card sequences, alternating Button vs Big Blind to remove positional bias\)\. Outcomes in milli\-big\-blinds per game \(mbb/g; one game==one hand in 2\-player poker\)\.95%95\\%confidence intervals from per\-hand chip\-delta variance \(CI95=1\.96⋅SEM\\mathrm\{CI\}\_\{95\}=1\.96\\cdot\\mathrm\{SEM\}\)\. Sampling: temperature1\.01\.0, max\-tokens=1024=1024\. Both methods compared use expert\-policy forcing \(matched supervision target\)\.

What the chip\-outcome adds beyond KL\.LeActandCoTExItuse matched supervision \(both targets are the oracle Nash policy\), so the chip\-outcome gap isolates the contribution of backward delta scoring on top of forward KL\-to\-Nash ranking; this is the same controlled ablation as the KL Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1), but on play\-strength rather than distribution\-faithfulness\.

Scope of the win\-rate matrix\.The matrix in Table[2\(b\)](https://arxiv.org/html/2607.21856#S5.T2.st2)is restricted to trained\-vs\-trained comparisons across the three trained methods \(NoCoTBC,CoTExIt,LeAct\) that share the same supervision target \(theDeepCFRteacher’s expert action\)\. The SFT coldstart is reported in the hold\-out KL summary \(Table[2\(a\)](https://arxiv.org/html/2607.21856#S5.T2.st1); coldstart single\-sample KL0\.03210\.0321vs\. trained\-method0\.00190\.0019–0\.01450\.0145\) but excluded from the chip\-outcome matrix because it is not the output of any iteration round\.

## Appendix EMechanism Validation: Nash\-Number Recall in CoT and Delta Scoring

The qualitative redaction in §[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)prevents the backward generator from copyingπ∗\\pi^\{\*\}from its prompt, but trained CoTs may still contain Nash\-plausible numbers as a side effect of expert\-policy supervision: the M\-step internalisesπ∗\\pi^\{\*\}, and the same model reused as the backward proposal can re\-emit Nash\-like values without ever seeing them as input\. This appendix records the CoT–policy coherence diagnostic that grounds the EPF×\\timesmethod2×22\\times 2pattern in §[6](https://arxiv.org/html/2607.21856#S6)\.

### E\.1CoT–Policy Coherence Diagnostic

§[6](https://arxiv.org/html/2607.21856#S6)identifies CoT–policy coherence and the resulting Nash\-number recall shortcut as the mechanism of the2×22\\times 2EPF×\\timesmethod interaction \(Table[3](https://arxiv.org/html/2607.21856#S6.T3)\)\. This subsection records the diagnostic that grounds that account\.

CoT–policy coherence\.ExIt\+CoT’s SFT target without EPF is an internally self\-consistent\(C​o​T,policy\)\(CoT,\\text\{policy\}\)trace: both components are produced by a single forward decode of the current model, so the gradient does not have to reconcile two independently produced signals\.LeAct’s SFT target without EPF, by contrast, is a hybrid: the CoT was produced by the backward model \(which sawπ∗\\pi^\{\*\}qualitatively\), while the policy line is the forward model’s decoded distribution\. The prediction is that per\-sample coherence \(measured as1−DKL​\(πC​o​T∥πpolicy line\)1\-D\_\{\\mathrm\{KL\}\}\(\\pi\_\{CoT\}\\,\\\|\\,\\pi\_\{\\text\{policy line\}\}\)\) should be higher forExIt\+CoT merge samples than forLeActmerge samples in the no\-EPF regime, and inspecting merged SFT data suffices to test it\.

Diagnostic \(R1 merge files,∼188\\sim\\\!188K lines per cell\)\.We measureDKL​\(πC​o​T∥πpolicy line\)D\_\{\\mathrm\{KL\}\}\(\\pi\_\{CoT\}\\,\\\|\\,\\pi\_\{\\text\{policy line\}\}\)on the R1 SFT merge data for all four cells\.

EPF row\.Cell A’s per\-sample KL distribution is right\-shifted relative to cell C across the bulk of the support, consistent with the prediction\.

No\-EPF row\.The surface ordering is reversed \(cell B is more coherent than cell D\), but inspection shows that the large majority \(93\.6%93\.6\\%\) ofLeAct’s no\-EPF backward CoTs restateπ∗\\pi^\{\*\}verbatim, because the backward generator saw the qualitative Nash context and the coldstart model was already near\-Nash, so the CoT and the model’s decoded policy line trivially agree\. This “Nash\-recall shortcut” is a degenerate form of coherence: the CoT does not analyse, it echoes, and because the SFT action target in the no\-EPF regime is the model’s own decode rather thanπ∗\\pi^\{\*\}, the gradient reinforces the model’s residual Nash errors instead of correcting them\. The symmetric flip is cell C \(ExIt\+CoT with EPF\), where replacing the self\-decoded\(C​o​T,policy\)\(CoT,\\text\{policy\}\)pair withπ∗\\pi^\{\*\}breaks the trace’s internal consistency\.

Synthesis\.The four cells share one mechanism: CoT–policy coherence plus supervision\-target choice together determine whether the gradient aligns with Nash \(winning cells A, D\) or with a memorised shortcut \(losing cells B, C\)\.

## Appendix FOracle Construction for Large Games

We document oracle construction per game scale: exactCFR/CFR\+viaOpenSpiel’s tabular enumeration for games up to∼\\sim80K infosets \(App\.[F\.1](https://arxiv.org/html/2607.21856#A6.SS1)\), and a streamingDeepCFRenumeration for Flop Hold’em where tabular enumeration exhausts memory \(App\.[F\.2](https://arxiv.org/html/2607.21856#A6.SS2)\)\.

### F\.1Exact CFR for Small\-to\-Medium Games

For all games reported in the main paper \(Leduc 6r2s through 13r4s, Liar’s Dice, 3\-player Leduc\), we use vanillaCFRorCFR\+to compute the exact Nash equilibrium policy table, enumerating the full information set space viaOpenSpiel’sto\_tabular\(\)function and storing the resulting\|𝒳\|×\|𝒜\|\|\\mathcal\{X\}\|\\times\|\\mathcal\{A\}\|policy matrix on disk\. This is feasible up to∼\\sim80,000 information sets on a standard 256 GB RAM node; larger games cause out\-of\-memory failures during tabular enumeration\.

### F\.2DeepCFR Streaming Enumeration for Flop Hold’em Poker

For Flop Hold’em Poker \(FHP\), the standard tabular enumeration viato\_tabular\(\)exhausts available RAM at 1\.5 TB MaxRSS before completing, making exact CFR table construction infeasible on current hardware\. We developed a trajectory\-based streaming enumeration method that avoids materialising the full game tree in memory\.

Method\.We run DeepCFR\[[5](https://arxiv.org/html/2607.21856#bib.bib26)\]on FHP using OpenSpiel’s PyTorch implementation \(open\_spiel\.python\.pytorch\.deep\_cfr\), which trains neural network value/strategy approximators from sampled game trajectories\. The game is specified via OpenSpiel’suniversal\_pokerconfiguration matching the Brown et al\. FHP benchmark \(2\-player limit hold’em, 2 betting rounds, blinds 50/100, raise size 100, 200 BB stacks\)\. Rather than tabularising the final strategy network by querying it at all information sets simultaneously, we enumerate infosets*incrementally*: during each CFR traversal we record the \(infoset, policy\) pair encountered in the trajectory, deduplicate across traversals via a hash table, and write policy entries to disk as they are first seen\. This streaming approach decouples memory consumption from game\-tree size, requiring onlyO​\(\|trajectory depth\|\)O\(\|\\text\{trajectory depth\}\|\)live memory per traversal rather thanO​\(\|𝒳\|\)O\(\|\\mathcal\{X\}\|\)\.

Configuration\.We run1,0001\{,\}000outer DeepCFR iterations on OpenSpiel’s PyTorch implementation, then draw10610^\{6\}on\-policy trajectories from the converged policy for streaming enumeration; end\-to\-end wall\-clock is approximately2222minutes on a single CPU node and produces a∼\\sim570 MB jsonl teacher table that we use throughout the paper\.

Results\.The streaming enumeration produced2,343,7322\{,\}343\{,\}732unique infoset policies, substantially more than any other game in our experiments \(see Table[9](https://arxiv.org/html/2607.21856#A6.T9)\)\. Policy coverage converges monotonically along policy\-induced trajectories: after the first 50K traversals, coverage stalls around 60% of newly\-encountered infosets;∼\\sim500K traversals already saturate at\>\>99% in\-support coverage, and we extend the run to10610^\{6\}trajectories for margin\. The full FHP game tree \(which contains∼109\\sim\\\!10^\{9\}unique infosets\[[5](https://arxiv.org/html/2607.21856#bib.bib26)\]including off\-policy subtrees not visited by external\-sampling MCCFR\) is therefore larger than the teacher’s2,343,7322\{,\}343\{,\}732\-infoset table\. This affects only out\-of\-table queries when the teacher would itself be queried as a self\-play opponent \(see App\.[D\.3](https://arxiv.org/html/2607.21856#A4.SS3)\); KL evaluation is computed exclusively on the held\-out subset of the teacher’s2,343,7322\{,\}343\{,\}732infosets and is therefore unaffected\.

Limitation\.DeepCFR provides an*approximate*Nash policy rather than an exact one; the approximation error decreases with more traversals but does not reach zero\. For the purposes ofLeActoracle supervision, we treat the converged DeepCFR policy as a high\-quality proxy forπ∗\\pi^\{\*\}, analogous to how approximate CFR\-family solvers \(continual re\-solving with neural counterfactual value networks\[[32](https://arxiv.org/html/2607.21856#bib.bib29)\], MCCFR\-blueprint\[[6](https://arxiv.org/html/2607.21856#bib.bib28)\]\) are used in large\-scale poker AI\.

Table 9:Tabular CFR scales to 80K infosets; FHP requires streamingDeepCFR\.Oracle exploitability is0for exact CFR by definition and approximate forDeepCFR\.Game\# InfosetsOracleMethodLeduc 6r2s4,032ExactCFRto\_tabular\(\)Leduc 10r4s47,040Exact CFRto\_tabular\(\)Leduc 13r4s79,872Exact CFRto\_tabular\(\)Liar’s Dice24,576Exact CFRto\_tabular\(\)3\-Player Leduc13,878Exact CFRto\_tabular\(\)Flop Hold’em Poker2,343,732DeepCFR \(approx\.\)Streaming enumeration

## Appendix GPrompt Format and Representative Examples

We show the prompt interface and one matched pair of backward\-generated traces\. The pair illustrates what the forward\-delta filter rewards and rejects; it is not used as quantitative evidence\.

### G\.1Prompt and Output Format

The forward prompt provides the information state and legal actions and asks the student for a reasoning trace followed by a policy\. The backward prompt uses the same state but reveals only a qualitative summary of the expert action and asks for reasoning alone\. Exact oracle probabilities are not shown to the backward generator\.

Forward prompt and output schemauser:Information state: \[Round 2\]\[Player: 0\]\[Private: 7h\] \[Public: 3d\]\[Sequences: cb\] Legal actions: \[Fold, Call, Raise\] What is your action? assistant:<think\>\.\.\.reasoning\.\.\.</think\> Policy: \{Fold:pfp\_\{f\}, Call:pcp\_\{c\}, Raise:prp\_\{r\}\}

Backward prompt and output schemauser:Information state: \[Round 2\]\[Player: 0\]\[Private: 7h\] \[Public: 3d\]\[Sequences: cb\] Legal actions: \[Fold, Call, Raise\] Optimal strategy description: Fold often; call sometimes; raise never\. Explain the reasoning without quoting probabilities\. assistant:<think\>\.\.\.reasoning only\.\.\.</think\>

The M\-step targetπ∗\(⋅∣x\)\\pi^\{\*\}\(\\cdot\\mid x\)is supplied separately and is never parsed from the backward output\. A candidate survives only when conditioning the forward model on it increases the per\-token log\-probability of the expert policy relative to the student’s self\-generated reasoning baseline \(App\.[A\.4](https://arxiv.org/html/2607.21856#A1.SS4)\)\.

### G\.2Matched Positive\- and Negative\-Delta Traces

For the state above, the expert qualitatively prefers folding, mixes in calls, and avoids raising\. The following positive\-delta trace connects observable features to that action structure:

Positive\-delta candidate<think\>The private card does not pair the board, and the opponent’s bet after a check is evidence of a stronger range\. Calling retains some value against bluffs and weaker unpaired hands, but raising is unattractive because stronger hands continue and weaker hands often fold\. The action should therefore lean toward folding, mix in calls, and rarely raise\.</think\>

By contrast, a circular restatement of the revealed action contains no state\-dependent explanation and receives negligible or negative delta:

Non\-positive\-delta candidate<think\>The optimal strategy is to fold most of the time, call occasionally, and avoid raising because this is the optimal strategy\.</think\>

These examples isolate the intended distinction: the filter rewards reasoning that makes the expert action more predictable from the state, not prose that merely repeats the action description\.

## Appendix HAdditional Experimental Details

### H\.1Domain Summary

Table 10:Domain summary with infoset count, oracle, and metric\.Leduc “NNrMMs” isNNranks×\\timesMMsuits per rank \(so 10r4s==40\-card deck\); Liar’s Dice uses the1d6fvariant; 3\-Player Leduc uses3r2sparameters\.†FHP’s infoset count is the full\-tree estimate\[[5](https://arxiv.org/html/2607.21856#bib.bib26)\]; the streamingDeepCFRteacher covers2,343,7322\{,\}343\{,\}732unique infosets along policy\-induced trajectories \(App\.[F](https://arxiv.org/html/2607.21856#A6)\)\.Domain\# Info SetsPlayersExpert OracleEvaluation MetricLeduc 6r2s4,0322CFR NashExploitabilityLeduc 10r4s47,0402CFR NashExploitabilityLeduc 13r4s79,8722CFR NashExploitabilityLiar’s Dice \(1d6f\)24,5762CFR NashExploitability3\-Player Leduc 3r2s13,8783CFR NashNashConvFlop Hold’em \(FHP\)†∼109\\sim\\\!10^\{9\}2DeepCFRKL to teacherBuilderBench51 tasks1Frontier\-model trajectoriesPass@KK, cube placement
### H\.2BuilderBench Task Partition

We evaluate on4646of the5151BuilderBench tasks \(55are excluded for exceeding the8,1928\{,\}192\-token context window\), split by trajectory\-pool success\.In\-domain\(2626tasks\): at least one of the three frontier oracles \(Claude Opus 4\.6, Gemini 3 Flash, GPT\-5\.2\) produced a fully\-successful episode in the trajectory pool, so the SFT data contains a golden trace\.OOD\(2020tasks\): the best\-of\-33frontier\-model trajectory pool produced no successful trajectory, so any progress on these tasks is net\-new training signal beyond the teacher pool\. Per\-oracle solve counts: Claude Opus 4\.624/5124/51; Gemini 3 Flash24/5124/51; GPT\-5\.215/5115/51; any\-oracle union26/5126/51\. Solve\-status lists are fixed by oracle data collection and do not change acrossLeActiterations\.

### H\.3Coldstart Construction

The coldstart phase fine\-tunes Qwen3\-8B on a domain\-specific format\-priming corpus\. Three coldstart variants are used across our experiments:NoCoT\(state→\\toaction only\),CoT\(state→\\toforward CoT→\\toaction\), and joint forward\+backward \(one SFT mix containing both forward training pairs and backward state, action→\\toCoT pairs\)\. The corpus source depends on what the oracle provides\.

Leduc, Liar’s Dice, 3\-player Leduc\.Frontier\-LLM sub\-agents \(Claude, Gemini, GPT\-5\) are prompted to produce candidate \(CoT, action\) pairs at every infoset; the action target is then replaced by theCFRsolver’s exact Nash distribution before SFT\. For joint variants, the backward direction is prompted with the oracle’s*qualitative*action summary \(e\.g\., “mostly fold with occasional calls”\) rather than exact probabilities, the same redaction protocol used inLeAct’s E\-step \(§[3\.3](https://arxiv.org/html/2607.21856#S3.SS3)\) to discourage Nash\-number recall\. Coldstart corpus sizes per setting:4,0324\{,\}032examples for Leduc 6r2s \(NoCoT, full coverage of all4,0324\{,\}032infosets\),2,8002\{,\}800for Leduc 10r4s joint \(1,4001\{,\}400forward \+1,4001\{,\}400backward, a sampled subset of the47,04047\{,\}040infosets to keep frontier\-LLM API generation cost bounded\),≈49,000\\approx\\\!49\{,\}000for Liar’s Dice joint \(full coverage of all24,57624\{,\}576infosets in both directions\), and27,75627\{,\}756for 3\-player Leduc joint \(13,87813\{,\}878forward \+13,87813\{,\}878backward, full coverage\)\. Leduc 13r4s reuses the Leduc 10r4s joint\-coldstart checkpoint rather than running a fresh coldstart, isolating the scaling effect from coldstart variance\.

Flop Hold’em\.ExactCFRis intractable, so coldstart CoTs are generated off\-GPU by frontier\-model sub\-agents \(Claude Opus 4\.6 / Sonnet 4\.6 / Haiku 4\.5\) prompted with the infoset description and theDeepCFRteacher’s policy\. Each sub\-agent emits a⟨\\langlereasoning, action, policy⟩\\rangletriple in a single forward pass; the reasoning block embeds both the forward justification and the backward explanation, so no separate backward direction is generated\. Outputs are validated for hole\-card faithfulness \(both hole cards named or referenced by exact\-card token\) and flop\-rank coverage before inclusion, guarding against Leduc\-style framing\. The resulting corpus contains1,5101\{,\}510NoCoTand1,5101\{,\}510CoTexamples \(3,0203\{,\}020merged for joint\)\.

BuilderBench\.The coldstart corpus is the frontier\-model trajectory pool itself: the first successful episode per \(task, model\) pair from three frontier agents \(GPT\-5\.2 as a CoT\-and\-act agent, Claude Opus 4\.6 and Gemini 3 Flash as reflexion agents\)\. The joint forward\+backward SFT mix contains14,14414\{,\}144examples across the 51 BuilderBench tasks; the resulting checkpoint is the “Coldstart” anchor in Table[3](https://arxiv.org/html/2607.21856#S6.T3)b\.

All coldstart SFT runs use 3 epochs at lr1×10−41\\\!\\times\\\!10^\{\-4\}for poker domains and BuilderBench,2×10−52\\\!\\times\\\!10^\{\-5\}for FHP; remaining hyperparameters follow Table[11](https://arxiv.org/html/2607.21856#A8.T11)\.

### H\.4Hyperparameters

Table[11](https://arxiv.org/html/2607.21856#A8.T11)summarises the training hyperparameters used across all domains and methods\. All experiments use Qwen3\-8B as the base model, LLaMA\-Factory for SFT with FSDPfull\_shardon 2–4 NVIDIA H200 GPUs, and vLLM for inference\. Sampling parameters: temperatureT=1\.0T=1\.0, top\-p=0\.95p=0\.95, top\-kkdisabled, frequency penalty0, repetition penalty11\(vLLM defaults\), max generation4,0964\{,\}096tokens for game domains and8,1928\{,\}192tokens for BuilderBench\.

Table 11:Training hyperparameters per experimental condition\.Learning rates and epoch counts are scale\-tuned: small Leduc variants tolerate aggressive schedules under full coverage, while 13r4s and FHP use longer warmup to avoid collapse on a wider distribution\. We standardise on top\-22selection across the paper;N=8N\{=\}8–1616for backward sampling\.SettingMethodEpochsLRSelectionKKGen\.NNGPUsLeduc 6r2sExIt /LeAct1–32e\-6 – 5e\-6top\-28–322Leduc 10r4sNoCoT BC12e\-6top\-28–322Leduc 10r4sLeAct1–32e\-6 – 5e\-6top\-282–4Leduc 13r4sNoCoT BC31e\-5top\-284Leduc 13r4sLeAct35e\-6top\-28–164–8Liar’s DiceLeAct32e\-6top\-2823\-Player LeducLeAct32e\-6top\-282FHP \(∼109\\sim\\\!10^\{9\}\)LeAct\(DeepCFR\)35e\-5top\-288BuilderBenchLeAct1–31e\-5top\-28–324

All experiments share the following defaults: AdamW \(β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, weight decay0\.010\.01\) with cosine LR decay and a3%3\\%linear warmup; max sequence length20482048tokens for game domains and81928192for BuilderBench; per\-device batch44–88, giving an effective batch of1616–6464depending on GPU count; bf16 mixed precision under PyTorch FSDPfull\_shardwithQwen3DecoderLayerwrapping\. Each headline cell is trained once per \(method, setting\) and evaluated under33inference seeds; reported single\-sample /BoNmetrics are best\-across\-epochs on the full game tree, with std across the33evaluations reported in Table[8](https://arxiv.org/html/2607.21856#A4.T8)\.

### H\.5E\-Step Load\-Bearing Ablation Study

Two ablations test whether backward\-delta selection contributes signal beyond what coverage alone provides; both share the production training recipe summarised above\.

Random vs forward\-delta ranking \(K=1K\{=\}1sharpening\)\.We compare random ranking and forward\-delta ranking on the sameLeActR2 backward pool at Leduc 10r4s, sharpened toK=1K\{=\}1per infoset \(within\-poolIoU≈0\.13\\mathrm\{IoU\}\\approx 0\.13instead of≈0\.50\\approx 0\.50atK=4K\{=\}4\) so that each arm commits to a single CoT per infoset and exposes the within\-positive ranking signal\. Both arms train from the same coldstart base on the same R2 backward pool with hyperparameters held matched; the comparison is between selection rules, not between training pipelines\. Reported metrics \(Fig\.[3](https://arxiv.org/html/2607.21856#S6.F3)a\) are full\-tree single\-sample andBoN\-of\-88exploitability on all4747K Leduc 10r4s infosets at the best epoch in each arm\.

Recall\-stratified delta on backward pools\.On theLeActR2 backward pools at Leduc 10r4s \(N=374,272N\{=\}374\{,\}272candidates\) and Leduc 13r4s \(N=638,976N\{=\}638\{,\}976candidates;N=393,512N\{=\}393\{,\}512after filtering trivial\-policy states\), we compute mean positive forward delta partitioned by whether each candidate CoT contains at least one verbatim Nash probability\. Stratification is on the full unfiltered candidate pool \(not on selected CoTs\), so the partition spans the entire pre\-selection distribution; per\-stratum positive\-delta means are reported in Fig\.[3](https://arxiv.org/html/2607.21856#S6.F3)b\. Verbatim detection uses exact substring matching against the per\-stateπ∗\\pi^\{\*\}probabilities at three\-decimal resolution\.

### H\.6Compute Budget

Table[12](https://arxiv.org/html/2607.21856#A8.T12)estimates the GPU\-hours for each phase of the experimental pipeline\. Generation and scoring are the dominant costs\.

Table 12:Per\-phase GPU\-hour estimates on NVIDIA H200 \(141GB\)\.Generation and scoring scale with infoset count and dominate at small to mid scales; SFT becomes the largest single phase only at FHP\. Units suppressed in cells; FHP figures are observed R1 wall\-times\.†\\daggerFHP coldstart CoT was generated off\-GPU by frontier\-model agents \(Claude Opus 4\.6 / Sonnet 4\.6 / Haiku 4\.5\); the<1<\\\!1figure is the action\-only forward pass over the coldstart trace pool\.PhaseLeduc 10r4sLeduc 13r4sLiar’s DiceFHPCFRsolver \(CPU\)0\.5h2h0\.5hDeepCFR\(pre\-trained\)Coldstart data gen484<1†\{<\}1^\{\\dagger\}Coldstart SFT24216ExIt generation \(N=8N\{=\}8\)816810ExIt SFT \(per round\)28216LeActbackward gen \(N=8N\{=\}8\)1632822LeActforward scoring163288LeActSFT \(per round\)416430Evaluation \(per checkpoint\)2424Total perLeActround∼\\sim38∼\\sim84∼\\sim22∼\\sim64

The total compute for the reported experiments \(including all iterations, ablations, and baselines\) is approximately 1,500 GPU\-hours on H200s\. The full research project, including failed experiments and hyperparameter searches, consumed approximately 5,000 GPU\-hours\.