FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

# FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience
Source: [https://arxiv.org/html/2609.03241](https://arxiv.org/html/2609.03241)
Kishan Panaganti\*†\\daggerHaitao MiLeowei LiangAffiliation:\[Affiliation:\[

September 3, 2026

###### Abstract

A reasoning model can improve from its own on\-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same\-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode\. We introduceFlowBalance, a verifier\-grounded self\-improvement method that learns a normalized distribution over complete responses\. For each on\-policy trajectory, a frozen training\-time view of the same policy uses privileged context to produce token\-level log\-probability gains, which are aggregated into a trajectory\-level self\-guidance score\. FlowBalance calibrates this score with the verifier\-derived group advantage: guidance is retained on positive\-advantage trajectories, reversed on negative\-advantage trajectories, and disabled when the rollout group provides no outcome preference\. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log\-partition estimate per rollout group\. This realizes*outcome\-calibrated self\-guidance via trajectory balance*, without a separate token\-level imitation loss\. Our analysis establishes within\-group contrast preservation, a minimum\-change reverse\-KL characterization, monotonic verifier control of target reward, and an exact correction against false\-positive self\-guidance on rejected responses\. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3\-4B and Qwen3\-8B, while also improving training speed and stability, avoiding direct OPSD’s response\-length collapse, and exhibiting higher correct\-strategy diversity in a controlled AIME24 diagnostic\.

## 1Introduction

A central promise of post\-training is that a reasoning model can improve from its own experience: sample several solutions, evaluate what worked, and update the policy so that the next round of experience is better\. We use*self\-improvement*in this operational sense—repeated policy improvement from the model’s own on\-policy trajectories under training\-time feedback—rather than to claim a fully autonomous or closed\-loop system\. For long\-horizon reasoning, this inner loop must learn more than a higher mean reward\. It must move probability toward correct reasoning, preserve useful alternative strategies, and avoid repeatedly sharpening one locally preferred trace\[[Guan et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib35),[Kujawa et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib36),[Ren and Sutherland, 2025](https://arxiv.org/html/2609.03241#bib.bib37)\]\.

Two failure modes make this loop fragile\. First, reinforcement learning with verifiable rewards \(RLVR\) provides dependable outcome grounding, but only through sparse terminal feedback\[[Shao et al\., 2024](https://arxiv.org/html/2609.03241#bib.bib13),[Yu et al\., 2025c](https://arxiv.org/html/2609.03241#bib.bib12)\]\. A response may contain hundreds or thousands of tokens while the verifier supplies one final reward; group\-based objectives improve comparison across responses but still attach a response\-level signal to every decision\. Distributional methods such as FlowRL improve how this terminal evidence is translated into probability over complete responses\[[Zhu et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib11)\], yet outcome\-only energies cannot exploit fine\-grained evidence along a sampled reasoning path\.

Second, the model can reassess its own sampled trajectories using a more informed training\-time view\. A frozen copy of the current policy can condition on a reference solution or task feedback and score every sampled token\[[Zhao et al\., 2026a](https://arxiv.org/html/2609.03241#bib.bib1),[Hübotter et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib2)\]\. This same\-model, privileged\-hindsight signal is dense and inexpensive: it requires neither a larger external model nor additional generated responses\. However, dense self\-guidance is not automatically trustworthy\. Because the scoring view observes information unavailable at inference, it may favor a plausible but ultimately rejected trajectory, shorten reasoning, suppress uncertainty\-driven exploration, or concentrate updates around a narrow local mode\[[Kim et al\., 2026b](https://arxiv.org/html/2609.03241#bib.bib4),[Zhu et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib6)\]\. Blindly following such confidence creates*self\-confirmation*: the model’s own erroneous preference becomes the next update’s supervision\.

We therefore ask a distributional self\-improvement question:

> *Given on\-policy reasoning experience, sparse verified outcomes, and dense but imperfect self\-guidance, what normalized response distribution should the next policy learn?*

We introduceFlowBalance, a verifier\-grounded self\-improvement operator that answers this question through*outcome\-calibrated self\-guidance via trajectory balance*\. The current policy generates a rollout group\. A verifier supplies stopped group\-relative advantages\. A frozen privileged\-hindsight view of the same policy then scores the already sampled tokens using training\-only context\. FlowBalance aggregates these scores into a trajectory\-level guidance gain, uses the verifier to determine its direction, and fits the resulting reference\-supported Gibbs target with profiled trajectory balance\. The policy is optimized only through this complete\-response distribution\-matching objective; there is no separate token\-level imitation loss\.

![Refer to caption](https://arxiv.org/html/2609.03241v1/example_flowbalance.png)Figure 1:FlowBalance as a verifier\-grounded self\-improvement cycle\.The current policy generates on\-policy reasoning experience \(1\); a verifier supplies sparse but reliable outcome feedback \(2\); a frozen privileged\-hindsight view of the same policy supplies dense self\-guidance on the sampled tokens \(3\); sign gating grounds that guidance in verified outcomes \(4\); and profiled trajectory balance internalizes the resulting normalized response distribution \(5\)\. Refreshing the frozen snapshot repeats the inner loop\.For a promptxx, let𝒢=\(y\(1\),…,y\(N\)\)\\mathcal\{G\}=\(y^\{\(1\)\},\\ldots,y^\{\(N\)\}\)be the on\-policy rollout group and letAi=A𝒢​\(y\(i\)\)A\_\{i\}=A\_\{\\mathcal\{G\}\}\(y^\{\(i\)\}\)be its stopped group\-relative verifier advantage\. The privileged\-hindsight scoring viewπH\\pi\_\{\\mathrm\{H\}\}conditions on training\-only contextccand yields clipped token\-level log\-probability gains relative to the reference policy\. Their trajectory average is the self\-guidance gainGH​\(y\(i\)∣x,c\)G\_\{\\mathrm\{H\}\}\(y^\{\(i\)\}\\mid x,c\)\. FlowBalance defines

EFlowBalance,𝒢​\(y\(i\)∣x,c\)=ηA​Ai\+βG​GH​\(y\(i\)∣x,c\)​sgn⁡\(Ai\)\.E\_\{\\mathrm\{FlowBalance\},\\mathcal\{G\}\}\(y^\{\(i\)\}\\mid x,c\)=\\eta\_\{A\}A\_\{i\}\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y^\{\(i\)\}\\mid x,c\)\\operatorname\{sgn\}\(A\_\{i\}\)\.\(1\)The verifier term anchors the direction of improvement\. WhenAi\>0A\_\{i\}\>0, positive self\-guidance can reinforce a verified response; whenAi<0A\_\{i\}<0, the same positive guidance is reversed rather than allowed to self\-confirm a failure; and whenAi=0A\_\{i\}=0, the dense branch is disabled\. The guidance therefore refines the verifier’s direction instead of overriding it\.

Conditioned on the realized group, FlowBalance defines the reference\-supported target

pFlowBalance,𝒢⋆​\(y∣x,c\)∝πref​\(y∣x\)​exp⁡\(EFlowBalance,𝒢​\(y∣x,c\)τ\),p^\{\\star\}\_\{\\mathrm\{FlowBalance\},\\mathcal\{G\}\}\(y\\mid x,c\)\\propto\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\!\\left\(\\frac\{E\_\{\\mathrm\{FlowBalance\},\\mathcal\{G\}\}\(y\\mid x,c\)\}\{\\tau\}\\right\),\(2\)and fits it through the profiled trajectory\-balance residual

ΔTB​\(y,x,c,𝒢\)=τ​log⁡ZFlowBalance,𝒢​\(x,c\)\+τ​log⁡πθ​\(y∣x\)πref​\(y∣x\)−EFlowBalance,𝒢​\(y∣x,c\)\.\\Delta\_\{\\mathrm\{TB\}\}\(y;x,c,\\mathcal\{G\}\)=\\tau\\log Z\_\{\\mathrm\{FlowBalance\},\\mathcal\{G\}\}\(x,c\)\+\\tau\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\-E\_\{\\mathrm\{FlowBalance\},\\mathcal\{G\}\}\(y\\mid x,c\)\.\(3\)One log\-partition estimate is profiled per rollout group\. The reference policy controls support and drift, the verifier determines outcome direction, privileged hindsight provides dense within\-trajectory evidence, and trajectory balance converts their composite energy into a normalized distribution over complete responses\.

This construction can also be viewed as a*distributional inner\-loop update*: a task source supplies prompts, the current policy supplies experience, FlowBalance maps\{\(yi,Ri,GH,i\)\}i=1N\\\{\(y\_\{i\},R\_\{i\},G\_\{\\mathrm\{H\},i\}\)\\\}\_\{i=1\}^\{N\}to a target distribution, and policy fitting produces the next policy\. Our experiments deliberately hold the task distribution fixed, isolating this experience\-to\-policy update from outer\-loop task generation or curriculum evolution\. The latter is complementary rather than assumed by the method\.

Our analysis studies the induced target directly\. We show that profiling the partition preserves every within\-group probability contrast; characterize FlowBalance as the unique minimum reverse\-KL displacement from the reference at its attained composite\-energy level; establish monotonic verifier control of the target reward statistic; and quantify exactly how outcome calibration converts false\-positive self\-guidance on a rejected response into a probability\-ratio correction favoring a verified response\. These are properties of the target and its local fitting objective, not a claim of global convergence for an arbitrary neural optimizer\.

On mathematical reasoning, FlowBalance obtains the strongest overall averages on both Qwen3\-4B and Qwen3\-8B among GRPO, OPSD, RLSD, FlowRL, and FlowBalance\. Relative to FlowRL, it improves the core four\-benchmark average over AIME24, HMMT25, MATH500, and OlympiadBench by1\.671\.67points on Qwen3\-4B and1\.981\.98points on Qwen3\-8B, while also improving the full aggregate average\. On Qwen3\-8B, FlowBalance reaches0\.50\.5AIME24 validation accuracy in about100100steps versus roughly143143for GRPO, remains stable over400400steps, and avoids the response\-length collapse observed under direct OPSD\. A controlled AIME24 diagnostic further finds higher correct\-only semantic strategy diversity than GRPO and RLSD\.

Our contributions are:

- •A distributional self\-improvement objective that improves over both verifier\-only RL and RL–self\-guidance hybrids\.GRPO improves reasoning from sparse verifier outcomes, while RLSD augments verifier\-based policy optimization with dense same\-model guidance\. FlowBalance addresses a different question:*what normalized complete\-response distribution should the next policy learn from these signals?*It combines verifier advantages and privileged\-hindsight guidance into one reference\-supported trajectory energy, then fits the induced distribution through profiled trajectory balance\. This makes the comparison to GRPO and RLSD especially informative: FlowBalance is not merely adding more supervision to GRPO or inserting another guidance term into an RL objective; it changes the policy\-update object from a local optimization signal into an explicitly normalized distribution over the model’s own reasoning trajectories\. *Concrete evidence\.*On the core four\-benchmark average over AIME24, HMMT25, MATH500, and OlympiadBench, FlowBalance improves over GRPO by2\.602\.60points on Qwen3\-4B \(67\.6967\.69versus65\.1065\.10\) and2\.442\.44points on Qwen3\-8B \(71\.0971\.09versus68\.6568\.65\)\. Relative to RLSD, the corresponding gains are5\.835\.83points on Qwen3\-4B \(67\.6967\.69versus61\.8761\.87\) and4\.184\.18points on Qwen3\-8B \(71\.0971\.09versus66\.9266\.92\)\. The same ordering holds on the full five\-benchmark aggregate: FlowBalance exceeds GRPO by1\.951\.95and2\.122\.12points and RLSD by4\.714\.71and3\.493\.49points on the 4B and 8B backbones, respectively\. On Qwen3\-8B, it also reaches0\.50\.5AIME24 validation accuracy in about100100steps rather than roughly143143for GRPO and remains near its peak over400400steps while GRPO degrades after about step180180\. These results isolate the practical contribution: learning a verifier\-grounded trajectory distribution improves over both outcome\-only RL and a direct RL–dense\-guidance combination across two model scales\.
- •Outcome\-calibrated self\-guidance that converts false confidence into a verifier\-grounded correction\.The privileged\-hindsight view is deliberately treated as*useful but imperfect*: it can recognize promising reasoning structure, yet it can also assign positive confidence to a trajectory that fails the verifier\. FlowBalance therefore uses\+GH\+G\_\{\\mathrm\{H\}\}when the group\-relative advantage is positive,−GH\-G\_\{\\mathrm\{H\}\}when it is negative, and zero guidance when the group provides no outcome preference\. This is more than a heuristic filter\. Proposition[4](https://arxiv.org/html/2609.03241#Thmproposition4)shows that, for a verified success and a rejected response, sign gating multiplies their target probability ratio by the exact factorexp⁡\(2​βG​GH​\(y−\)/τ\)\\exp\(2\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\)/\\tau\)relative to ungated guidance\. Dense same\-model evidence can therefore refine verified successes without allowing a confidently scored failure to become self\-reinforcing supervision\. *Concrete evidence\.*In the exact four\-mode diagnostic, reward\-only shaping reaches success mass0\.8180\.818, ungated self\-guidance reaches0\.8320\.832, and FlowBalance reaches0\.9000\.900while also increasing the robust\-success mode from0\.3270\.327under reward\-only shaping to0\.4400\.440\. In the binary false\-confidence sweep atG−=0\.5G\_\{\-\}=0\.5, FlowBalance attains target success probability0\.8940\.894, compared with0\.8170\.817for reward\-only shaping and0\.8070\.807for ungated shaping\. The language\-model results show the same qualitative distinction: direct OPSD rapidly collapses to short responses, whereas FlowBalance maintains long reasoning traces; moreover, increasing the guidance coefficient fromβG=1\\beta\_\{G\}=1to33lowers AIME24 from89\.3389\.33to86\.0086\.00and HMMT25 from34\.6734\.67to30\.0030\.00\. The benefit therefore comes from*calibrated*self\-guidance, not from simply making the dense signal stronger\.
- •A conservative and information\-efficient trajectory\-balance update with exact target\-level guarantees\.FlowBalance profiles one scalar log\-partition value per rollout group, but this nuisance parameter removes only the common offset: allN−1N\-1independent within\-group probability contrasts remain available to train the policy\. The induced target is also the unique minimum reverse\-KL displacement from the reference among group distributions reaching its expected composite\-energy level, and increasing the verifier coefficient monotonically increases the target’s expected verifier reward for fixed guidance\. Together, these results identify the update as a principled minimum\-change self\-improvement step that uses the full relative structure of the rollout group while keeping verified reward as an explicit control variable\. *Concrete evidence\.*The exact synthetic diagnostics quantify both effects\. A matched\-energy full\-support alternative requires reverse KL0\.9730\.973, whereas the FlowBalance exponential tilt requires only0\.2730\.273; the alternative is therefore3\.6×3\.6\\timesfarther from the reference\. At rollout\-group sizeN=32N=32, profiling all contrasts yields2\.92%2\.92\\%of the local Gaussian parameter risk of a one\-contrast\-per\-group estimator—about1/341/34as much risk\. Consistent with this conservative, all\-contrast update—without claiming that the target\-level results alone prove optimizer convergence—FlowBalance reaches0\.50\.5AIME24 validation accuracy in about100100steps rather than roughly143143for GRPO and remains near its peak over400400steps, while GRPO degrades sharply after approximately step180180\.
- •Performance and semantic\-diversity evidence that self\-improvement need not collapse to one reasoning mode\.FlowBalance improves mathematical reasoning while preserving a broader successful response distribution\. On Qwen3\-4B, it improves over FlowRL on all four core reported benchmarks: AIME24, HMMT25, MATH500, and OlympiadBench\. On Qwen3\-8B, it obtains the best mean on every benchmark in the main table and improves the core four\-benchmark average by1\.981\.98points over FlowRL\. These results matter for the paper’s central claim because the largest gains are not confined to a single metric: they span both multi\-sample competition reasoning and single\-sample accuracy, while the training traces show that the policy does not obtain them by collapsing to the short\-response behavior seen under direct OPSD\. *Concrete evidence\.*In the controlled AIME24 diagnostic, correct\-only Simpson strategy diversity is0\.21940\.2194for FlowBalance, compared with0\.10170\.1017for GRPO and0\.14560\.1456for RLSD\. The associated traces differ in their mathematical representation, not merely their wording: FlowBalance discovers a hidden××84\\\!\\times\\\!5\\\!\\times\\\!8box embedding where GRPO uses Cayley–Menger, and the appendix shows envelope versus multiple\-root reasoning, meridian\-section versus implicit\-normal geometry, and coordinate elimination versus secant–tangent hyperbola parameterization\. Although this is a one\-seed LLM\-judged diagnostic rather than a population\-level diversity guarantee, it supplies concrete qualitative evidence that the learned distribution places mass on genuinely different correct derivations instead of stylistic rewrites of one dominant template\.

## 2Preliminaries

### 2\.1On\-Policy Reasoning Experience

Letx∼𝒟x\\sim\\mathcal\{D\}be a reasoning prompt\. A response is a variable\-length token sequencey=\(y1,…,yT\)∈𝒴⁡\(x\)y=\(y\_\{1\},\\ldots,y\_\{T\}\)\\in\\mathcal\{Y\}\(x\)\. At tokentt, the state isst=\(x,y<t\)s\_\{t\}=\(x,y\_\{<t\}\), and the trainable policy factorizes as

πθ​\(y∣x\)=∏t=1Tπθ​\(yt∣st\)\.\\pi\_\{\\theta\}\(y\\mid x\)=\\prod\_\{t=1\}^\{T\}\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\)\.\(4\)At the start of each training iteration, we snapshot the current parameters asθ−←θ\\theta^\{\-\}\\leftarrow\\theta\. The frozen rollout policy isπθ−\\pi\_\{\\theta^\{\-\}\}, while the reference policyπref\\pi\_\{\\mathrm\{ref\}\}is a fixed copy of the initial checkpoint\.

For each prompt, we sample a group of responses\{y\(i\)\}i=1N∼πθ−\(⋅∣x\)\\\{y^\{\(i\)\}\\\}\_\{i=1\}^\{N\}\\sim\\pi\_\{\\theta^\{\-\}\}\(\\cdot\\mid x\)\. The verifier assigns the terminal rewardRi=R⁡\(y\(i\),x\)R\_\{i\}=R\(y^\{\(i\)\};x\)according to final\-answer correctness\. We compute the stopped group\-relative advantage

Ai=Ri−μR​\(x\)σR​\(x\)\+ϵ,A\_\{i\}=\\frac\{R\_\{i\}\-\\mu\_\{R\}\(x\)\}\{\\sigma\_\{R\}\(x\)\+\\epsilon\},\(5\)whereμR​\(x\)\\mu\_\{R\}\(x\)andσR​\(x\)\\sigma\_\{R\}\(x\)are the mean and standard deviation of the rewards in the response group\. Rewards, group statistics, sampled trajectories, and advantages receive no gradient\.

### 2\.2Privileged\-Hindsight Self\-Guidance

Each training promptxxis paired with training\-only contextcc, such as a reference solution or task feedback\. We evaluate the sampled tokenyty\_\{t\}through two views of the same frozen snapshot:

πroll\(⋅∣st\)\\displaystyle\\pi\_\{\\mathrm\{roll\}\}\(\\cdot\\mid s\_\{t\}\)=πθ−\(⋅∣x,y<t\),\\displaystyle=\\pi\_\{\\theta^\{\-\}\}\(\\cdot\\mid x,y\_\{<t\}\),\(6\)πH\(⋅∣st,c\)\\displaystyle\\pi\_\{\\mathrm\{H\}\}\(\\cdot\\mid s\_\{t\},c\)=πθ−\(⋅∣x,c,y<t\)\.\\displaystyle=\\pi\_\{\\theta^\{\-\}\}\(\\cdot\\mid x,c,y\_\{<t\}\)\.\(7\)The rollout view generates the response without observingcc\. The privileged\-hindsight view seescconly after the response has been sampled and scores the same tokens; it generates no replacement trajectory and receives no gradient\. At inference, the deployed policy observes neitherccnor the hindsight view\.

This same\-model scoring construction is related to on\-policy self\-distillation, but its role in FlowBalance is narrower: it supplies a stopped dense feature for defining a trajectory\-level target\. FlowBalance does not optimize a separate token\-level imitation loss\.

### 2\.3Reference\-Supported Target Distribution

We consider a normalized target over complete responses,

p⋆​\(y∣x,c\)=1Z⁡\(x,c\)​πref​\(y∣x\)​exp⁡\(E⁡\(y∣x,c\)τ\),p^\{\\star\}\(y\\mid x,c\)=\\frac\{1\}\{Z\(x,c\)\}\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\!\\left\(\\frac\{E\(y\\mid x,c\)\}\{\\tau\}\\right\),\(8\)whereE⁡\(y∣x,c\)E\(y\\mid x,c\)is a stopped trajectory energy,τ\>0\\tau\>0is its temperature, andZ⁡\(x,c\)Z\(x,c\)is the partition function\. The reference policy fixes support and controls drift; the energy specifies relative preference among complete responses\. Section[3](https://arxiv.org/html/2609.03241#S3)defines FlowBalance’s outcome\-calibrated self\-guidance energy and fits the target through trajectory balance\.

## 3FlowBalance: Verifier\-Grounded Self\-Improvement

FlowBalance maps a rollout group of self\-generated reasoning trajectories into a normalized next\-policy target\. The verifier supplies sparse but grounded outcome evidence, while a privileged\-hindsight view of the same frozen policy supplies dense self\-guidance on the sampled tokens\. FlowBalance first combines these stopped signals into an outcome\-calibrated trajectory energy and then fits the induced complete\-response distribution through trajectory balance\. The dense branch therefore shapes*which distribution is learned*; it is not optimized as a separate token\-level imitation objective\.

### 3\.1Training\-Time Self\-Guidance from Privileged Hindsight

For a sampled responseyy, define the clipped token\-level hindsight gain

δtH​\(y,x,c\)=clip⁡\(log⁡πH​\(yt∣st,c\)−log⁡πref​\(yt∣st\),−B,B\)\.\\delta\_\{t\}^\{\\mathrm\{H\}\}\(y;x,c\)=\\operatorname\{clip\}\\left\(\\log\\pi\_\{\\mathrm\{H\}\}\(y\_\{t\}\\mid s\_\{t\},c\)\-\\log\\pi\_\{\\mathrm\{ref\}\}\(y\_\{t\}\\mid s\_\{t\}\),\-B,B\\right\)\.\(9\)We aggregate these gains over the complete response:

GH​\(y∣x,c\)=1T​∑t=1TδtH​\(y,x,c\)\.G\_\{\\mathrm\{H\}\}\(y\\mid x,c\)=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}\\delta\_\{t\}^\{\\mathrm\{H\}\}\(y;x,c\)\.\(10\)GHG\_\{\\mathrm\{H\}\}measures how much the privileged\-hindsight view raises or lowers the average sampled\-token log probability relative to the fixed reference policy\. It is a trajectory feature computed on the policy’s own on\-policy experience\. No token is resampled fromπH\\pi\_\{\\mathrm\{H\}\}, and no gradient is propagated throughπH\\pi\_\{\\mathrm\{H\}\}orGHG\_\{\\mathrm\{H\}\}\.

### 3\.2Outcome\-Calibrated Self\-Improvement Target

Dense self\-guidance can be useful while still being wrong about the verified outcome\. FlowBalance therefore uses the group\-relative advantage to determine its direction:

EFlowBalance​\(y∣x,c\)=ηA​A​\(y\)\+βG​GH​\(y∣x,c\)​sgn⁡\(A⁡\(y\)\),E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)=\\eta\_\{A\}A\(y\)\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\\mid x,c\)\\operatorname\{sgn\}\(A\(y\)\),\(11\)whereηA,βG≥0\\eta\_\{A\},\\beta\_\{G\}\\geq 0\. IfA⁡\(y\)\>0A\(y\)\>0, positive guidance increases the trajectory energy\. IfA⁡\(y\)<0A\(y\)<0, positive guidance is reversed, preventing a confidently scored failure from becoming self\-reinforcing supervision\. IfA⁡\(y\)=0A\(y\)=0, the dense branch is disabled\. All quantities on the right\-hand side are stopped during the policy update\.

FlowBalance defines the unnormalized and normalized targets

p~FlowBalance​\(y∣x,c\)\\displaystyle\\widetilde\{p\}\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)=πref​\(y∣x\)​exp⁡\(EFlowBalance​\(y∣x,c\)τ\),\\displaystyle=\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\!\\left\(\\frac\{E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\}\{\\tau\}\\right\),\(12\)pFlowBalance⋆​\(y∣x,c\)\\displaystyle p^\{\\star\}\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)=p~FlowBalance​\(y∣x,c\)ZFlowBalance​\(x,c\),ZFlowBalance​\(x,c\)=∑y∈𝒴⁡\(x\)p~FlowBalance​\(y∣x,c\)\.\\displaystyle=\\frac\{\\widetilde\{p\}\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\}\{Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)\},\\quad Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)=\\sum\_\{y\\in\\mathcal\{Y\}\(x\)\}\\widetilde\{p\}\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\.\(13\)The three factors have distinct roles:πref\\pi\_\{\\mathrm\{ref\}\}retains reference support,AAsupplies verified outcome direction, andGHG\_\{\\mathrm\{H\}\}provides dense within\-trajectory evidence\. BecauseAAis group\-relative in implementation, the practical update uses the realized rollout group𝒢\\mathcal\{G\}:

pFlowBalance,𝒢⋆​\(y\(i\)∣x,c\)=πref​\(y\(i\)∣x\)​exp⁡\(EFlowBalance​\(y\(i\)∣x,c\)/τ\)∑j=1Nπref​\(y\(j\)∣x\)​exp⁡\(EFlowBalance​\(y\(j\)∣x,c\)/τ\)\.p^\{\\star\}\_\{\\mathrm\{FlowBalance\},\\mathcal\{G\}\}\(y^\{\(i\)\}\\mid x,c\)=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\\exp\(E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\(i\)\}\\mid x,c\)/\\tau\)\}\{\\sum\_\{j=1\}^\{N\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\\exp\(E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\(j\)\}\\mid x,c\)/\\tau\)\}\.\(14\)The response\-space notation describes the corresponding ideal Gibbs distribution; the objective and profiled partition are evaluated on sampled complete trajectories\.

##### Why a partition\-normalized target?

A local dense score specifies how individual sampled tokens look under privileged hindsight, but not where total probability mass should settle across complete responses\. The partition term converts relative trajectory energies into a probability\-conserving target\. Consequently, self\-guidance can refine the distribution within the verifier’s direction without becoming an independent local loss\. Appendix[A](https://arxiv.org/html/2609.03241#A1)gives the corresponding equilibrium interpretation\.

### 3\.3Trajectory Balance as a Normalized Self\-Update

The complete\-trajectory balance equation is

τ​log⁡ZFlowBalance​\(x,c\)\+τ​log⁡πθ​\(y∣x\)πref​\(y∣x\)−EFlowBalance​\(y∣x,c\)=0\.\\tau\\log Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)\+\\tau\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\-E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)=0\.\(15\)The trajectory\-balance residual is

ΔTB​\(y\(i\),x,c\)=τ​log⁡ZFlowBalance​\(x,c\)\+τ​log⁡πθ​\(y\(i\)∣x\)πref​\(y\(i\)∣x\)−EFlowBalance​\(y\(i\)∣x,c\)\.\\Delta\_\{\\mathrm\{TB\}\}\(y^\{\(i\)\};x,c\)=\\tau\\log Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)\+\\tau\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\(i\)\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\(i\)\}\\mid x,c\)\.\(16\)At zero residual, the partition cancels in pairwise probability ratios:

log⁡πθ​\(y\(a\)∣x\)πθ​\(y\(b\)∣x\)=log⁡πref​\(y\(a\)∣x\)πref​\(y\(b\)∣x\)\+EFlowBalance​\(y\(a\)∣x,c\)−EFlowBalance​\(y\(b\)∣x,c\)τ\.\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\(a\)\}\\mid x\)\}\{\\pi\_\{\\theta\}\(y^\{\(b\)\}\\mid x\)\}=\\log\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(a\)\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(b\)\}\\mid x\)\}\+\\frac\{E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\(a\)\}\\mid x,c\)\-E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\(b\)\}\\mid x,c\)\}\{\\tau\}\.\(17\)Thus the energy controls relative preference among the model’s own sampled experiences, whilelog⁡ZFlowBalance\\log Z\_\{\\mathrm\{FlowBalance\}\}absorbs only the common prompt\-level offset\.

The main loss is

ℒFlowBalance​\(θ\)=𝔼\(x,c\)∼𝒟​\[12​N​∑i=1NΔTB​\(y\(i\),x,c\)2\]\.\\mathcal\{L\}\_\{\\mathrm\{FlowBalance\}\}\(\\theta\)=\\mathbb\{E\}\_\{\(x,c\)\\sim\\mathcal\{D\}\}\\left\[\\frac\{1\}\{2N\}\\sum\_\{i=1\}^\{N\}\\Delta\_\{\\mathrm\{TB\}\}\(y^\{\(i\)\};x,c\)^\{2\}\\right\]\.\(18\)Rewards, advantages, self\-guidance scores, partition estimates, and sampled responses are stopped\. Gradients are taken only through the trainable\-policy log probabilitieslog⁡πθ​\(y\(i\)∣x\)\\log\\pi\_\{\\theta\}\(y^\{\(i\)\}\\mid x\)\.

##### Subtrajectory balance\.

The same principle can be applied between intermediate states\. LetZFlowBalance​\(s\)Z\_\{\\mathrm\{FlowBalance\}\}\(s\)denote the continuation partition from statess, withZFlowBalance​\(sterm\)=1Z\_\{\\mathrm\{FlowBalance\}\}\(s\_\{\\mathrm\{term\}\}\)=1\. Define the per\-token shaped increment

rtFlowBalance=τlogπref\(yt∣st\)\+βGTδtHsgn\(A\)\+ηAA1\{t=T\}\.r\_\{t\}^\{\\mathrm\{FlowBalance\}\}=\\tau\\log\\pi\_\{\\mathrm\{ref\}\}\(y\_\{t\}\\mid s\_\{t\}\)\+\\frac\{\\beta\_\{G\}\}\{T\}\\delta\_\{t\}^\{\\mathrm\{H\}\}\\operatorname\{sgn\}\(A\)\+\\eta\_\{A\}A\\,\\mathbf\{1\}\\\{t=T\\\}\.\(19\)For an intervali:ji\{:\}j, the subtrajectory residual is

Δi:j=τlogZFlowBalance\(si\)\+τ∑t=ij−1logπθ\(yt∣st\)−τlogZFlowBalance\(sj\)−∑t=ij−1rtFlowBalance\.\\Delta\_\{i:j\}=\\tau\\log Z\_\{\\mathrm\{FlowBalance\}\}\(s\_\{i\}\)\+\\tau\\sum\_\{t=i\}^\{j\-1\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid s\_\{t\}\)\-\\tau\\log Z\_\{\\mathrm\{FlowBalance\}\}\(s\_\{j\}\)\-\\sum\_\{t=i\}^\{j\-1\}r\_\{t\}^\{\\mathrm\{FlowBalance\}\}\.\(20\)The endpoint partitions account for different continuation distributions\. Sampling intervals and minimizing𝔼\[Δi:j2\]\\mathbb\{E\}\[\\Delta\_\{i:j\}^\{2\}\]provides a denser fitting objective along long responses; the experiments in this paper use the complete\-response implementation unless otherwise stated\.

### 3\.4Profiled Optimization over Rollout Groups

The partition can be represented by a learned prompt\-conditioned estimator or profiled directly from the rollout group\. We use the group estimator\. Each response implies

log⁡Z^i​\(x,c\)=EFlowBalance​\(y\(i\)∣x,c\)τ−log⁡πθ​\(y\(i\)∣x\)πref​\(y\(i\)∣x\),\\widehat\{\\log Z\}\_\{i\}\(x,c\)=\\frac\{E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\(i\)\}\\mid x,c\)\}\{\\tau\}\-\\log\\frac\{\\pi\_\{\\theta\}\(y^\{\(i\)\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\},\(21\)and we set

log⁡Z^FlowBalance​\(x,c\)=1N​∑i=1Nlog⁡Z^i​\(x,c\)\.\\widehat\{\\log Z\}\_\{\\mathrm\{FlowBalance\}\}\(x,c\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\widehat\{\\log Z\}\_\{i\}\(x,c\)\.\(22\)Gradients are stopped through this estimate\.

Algorithm 1FlowBalance: verifier\-grounded self\-improvement from on\-policy experience1:Prompt–context dataset

𝒟\\mathcal\{D\}; policy

πθ\\pi\_\{\\theta\}; fixed reference

πref\\pi\_\{\\mathrm\{ref\}\}; verifier

RR; coefficients

ηA,βG,τ\\eta\_\{A\},\\beta\_\{G\},\\tau\.

2:foreach training iterationdo

3:Snapshot the current policy:

θ−←θ\\theta^\{\-\}\\leftarrow\\theta\.

4:Sample a minibatch

\{\(xb,cb\)\}b=1B∼𝒟\\\{\(x\_\{b\},c\_\{b\}\)\\\}\_\{b=1\}^\{B\}\\sim\\mathcal\{D\}\.

5:Generate

NNresponses

yb\(i\)∼πθ−\(⋅∣xb\)y\_\{b\}^\{\(i\)\}\\sim\\pi\_\{\\theta^\{\-\}\}\(\\cdot\\mid x\_\{b\}\)for every prompt\.

6:Evaluate rewards and compute stopped group\-relative advantages

Ab,iA\_\{b,i\}using Eq\. \([5](https://arxiv.org/html/2609.03241#S2.E5)\)\.

7:Score sampled tokens under

πθ\\pi\_\{\\theta\},

πref\\pi\_\{\\mathrm\{ref\}\}, and the frozen hindsight view

πH=πθ−\(⋅∣xb,cb,y<t\)\\pi\_\{\\mathrm\{H\}\}=\\pi\_\{\\theta^\{\-\}\}\(\\cdot\\mid x\_\{b\},c\_\{b\},y\_\{<t\}\)\.

8:Compute

GH​\(yb\(i\)∣xb,cb\)G\_\{\\mathrm\{H\}\}\(y\_\{b\}^\{\(i\)\}\\mid x\_\{b\},c\_\{b\}\)and

EFlowBalance​\(yb\(i\)\)E\_\{\\mathrm\{FlowBalance\}\}\(y\_\{b\}^\{\(i\)\}\)using Eqs\. \([10](https://arxiv.org/html/2609.03241#S3.E10)\) and \([11](https://arxiv.org/html/2609.03241#S3.E11)\)\.

9:Profile

log⁡Z^FlowBalance​\(xb,cb\)\\widehat\{\\log Z\}\_\{\\mathrm\{FlowBalance\}\}\(x\_\{b\},c\_\{b\}\)by Eq\. \([22](https://arxiv.org/html/2609.03241#S3.E22)\)\.

10:Compute

ΔTB​\(yb\(i\),xb,cb\)\\Delta\_\{\\mathrm\{TB\}\}\(y\_\{b\}^\{\(i\)\};x\_\{b\},c\_\{b\}\)and update

θ\\thetawith Eq\. \([18](https://arxiv.org/html/2609.03241#S3.E18)\)\.

11:endfor

Algorithm[1](https://arxiv.org/html/2609.03241#alg1)makes the self\-improvement loop explicit\. The frozen snapshot first generates experience without privileged context, then provides a training\-only hindsight score on those same trajectories\. The verifier grounds the direction of the score, and trajectory balance internalizes the resulting normalized distribution into the next policy snapshot\.

## 4Why FlowBalance Supports Verifier\-Grounded Self\-Improvement

Section[3](https://arxiv.org/html/2609.03241#S3)defines FlowBalance as a partition\-normalized update over a realized group of the model’s own trajectories\. We analyze four properties that are useful for a self\-improvement inner loop\. First, profiling the group partition should retain the relative evidence contained in all sampled responses\. Second, the target should move conservatively from the reference\. Third, the verifier should remain an explicit control on target reward even in the presence of dense self\-guidance\. Fourth, false\-positive guidance on a rejected response should not become self\-reinforcement\. All statements below are conditional on the realized rollout group and treat verifier and guidance quantities as stopped\. Proofs are in Appendix[B](https://arxiv.org/html/2609.03241#A2)\.

### 4\.1Distributional Structure of the Self\-Update

For each sampled responsey\(i\)y^\{\(i\)\}, let

Ei=ηA​Ai\+βG​GH​\(y\(i\)∣x,c\)​sgn⁡\(Ai\)E\_\{i\}=\\eta\_\{A\}A\_\{i\}\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y^\{\(i\)\}\\mid x,c\)\\operatorname\{sgn\}\(A\_\{i\}\)\(23\)be its stopped FlowBalance energy\. The normalized group target is

pi⋆=πref​\(y\(i\)∣x\)​exp⁡\(Ei/τ\)∑j=1Nπref​\(y\(j\)∣x\)​exp⁡\(Ej/τ\)\.p\_\{i\}^\{\\star\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\\exp\(E\_\{i\}/\\tau\)\}\{\\sum\_\{j=1\}^\{N\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\\exp\(E\_\{j\}/\\tau\)\}\.\(24\)Letp⋆=\(p1⋆,…,pN⋆\)p^\{\\star\}=\(p\_\{1\}^\{\\star\},\\ldots,p\_\{N\}^\{\\star\}\)\. All results are conditional on the realized rollout group\.

###### Proposition 1\(Profiled balance uses all within\-group contrasts\)\.

For any candidate group distribution with positive mass on the realized responses, the profiled trajectory\-balance loss is zero if and only if every relative probability contrast matches

π⁡\(y\(i\)\)π⁡\(y\(j\)\)=πref​\(y\(i\)∣x\)πref​\(y\(j\)∣x\)​exp⁡\(Ei−Ejτ\),i,j∈\[N\]\.\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\(y^\{\(j\)\}\)\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\}\\exp\\\!\\left\(\\frac\{E\_\{i\}\-E\_\{j\}\}\{\\tau\}\\right\),\\qquad i,j\\in\[N\]\.\(25\)In particular, whenever the group target is representable, the global minimum is zero; the unknown prompt\-level normalizer removes exactly one common offset and preserves the remainingN−1N\-1contrast directions\.

The group partition therefore does not collapse the rollout group to one comparison\. It absorbs only the common energy shift while retaining all relative evidence supplied by the model’s own sampled experiences\.

###### Proposition 2\(Conservative minimum\-change self\-update\)\.

Let the reference be restricted and renormalized to the realized rollout group when computing KL\. For any group distributionp∈ΔNp\\in\\Delta\_\{N\},

KL\(p∥πref\)=KL\(p⋆∥πref\)\+KL\(p∥p⋆\)\+1τ\(∑i=1NpiEi−∑i=1Npi⋆Ei\)\.\\mathrm\{KL\}\(p\\\|\\pi\_\{\\mathrm\{ref\}\}\)=\\mathrm\{KL\}\(p^\{\\star\}\\\|\\pi\_\{\\mathrm\{ref\}\}\)\+\\mathrm\{KL\}\(p\\\|p^\{\\star\}\)\+\\frac\{1\}\{\\tau\}\\left\(\\sum\_\{i=1\}^\{N\}p\_\{i\}E\_\{i\}\-\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{\\star\}E\_\{i\}\\right\)\.\(26\)Consequently, among all group distributions that attain at least the expected FlowBalance energy ofp⋆p^\{\\star\}, the FlowBalance target is the unique minimum\-reverse\-KL displacement from the reference\.

FlowBalance is therefore the smallest reference\-supported distributional tilt that reaches its composite self\-improvement energy level\. This is a property of the target distribution, not a claim that an arbitrary finite neural\-network update exactly reaches the target in one optimizer step\.

### 4\.2Verifier Control against Self\-Confirmation

###### Proposition 3\(Verifier weight monotonically improves target reward\)\.

Hold the self\-guidance scores and the realized rollout group fixed\. Increasing the verifier coefficientηA\\eta\_\{A\}monotonically shifts the FlowBalance target toward higher\-reward responses\. In particular,

∂∂ηA​𝔼p⋆​\[R\]=Varp⋆​\(R\)τ​\(σR​\(x\)\+ϵ\)≥0\.\\frac\{\\partial\}\{\\partial\\eta\_\{A\}\}\\mathbb\{E\}\_\{p^\{\\star\}\}\[R\]=\\frac\{\\mathrm\{Var\}\_\{p^\{\\star\}\}\(R\)\}\{\\tau\(\\sigma\_\{R\}\(x\)\+\\epsilon\)\}\\geq 0\.\(27\)

The verifier remains an explicit control knob even when dense self\-guidance is present: for fixed guidance, increasing its weight cannot reduce the target’s expected verifier reward\.

###### Proposition 4\(Outcome calibration corrects false\-positive self\-guidance\)\.

Consider a verified successy\+y\_\{\+\}and a verifier\-rejected responsey−y\_\{\-\}in the same mixed\-outcome group\. Relative to otherwise identical*ungated*self\-guidance, sign gating changes their target probability ratio according to

pgated⋆​\(y\+\)pgated⋆​\(y−\)=pungated⋆​\(y\+\)pungated⋆​\(y−\)​exp⁡\(2​βG​GH​\(y−∣x,c\)τ\)\.\\frac\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\-\}\)\}=\\frac\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\-\}\)\}\\exp\\\!\\left\(\\frac\{2\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\)\}\{\\tau\}\\right\)\.\(28\)Hence, whenGH​\(y−∣x,c\)\>0G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\)\>0, positive self\-guidance on a rejected response is converted from self\-reinforcement into a probability\-ratio correction favoring the verified response\.

This proposition captures the intended useful\-but\-imperfect regime: privileged hindsight can assign positive local likelihood both to correct reasoning and to plausible failures\. FlowBalance preserves the ability of guidance to distinguish successful modes while reversing its false\-positive support where the verifier rejects the response\. Appendix[C\.1](https://arxiv.org/html/2609.03241#A3.SS1)provides exactly enumerable diagnostics for the same target, including a reliability sweep over imperfect self\-guidance\.

## 5Experiments

We evaluate whether FlowBalance supports reliable policy self\-improvement from on\-policy reasoning experience along four axes\. First, we compare final mathematical reasoning accuracy against a verifier\-only RL baseline, direct on\-policy distillation, a verifier–distillation hybrid, and outcome\-only trajectory balance\. Second, we inspect update efficiency, late\-training stability, and response\-length behavior\. Third, we isolate verifier grounding and self\-guidance strength\. Fourth, we test whether the learned response distribution retains multiple successful strategies through an LLM\-judged AIME24 diagnostic\. Appendix[D\.1](https://arxiv.org/html/2609.03241#A4.SS1)gives the training and decoding details, Appendix[D\.3](https://arxiv.org/html/2609.03241#A4.SS3)gives the diversity protocol, and Appendix[D\.2](https://arxiv.org/html/2609.03241#A4.SS2)gives the full ablation tables\.

### 5\.1Self\-Improvement Evaluation Setup

We evaluate Qwen3\-4B and Qwen3\-8B\[[Yang et al\., 2025a](https://arxiv.org/html/2609.03241#bib.bib16)\]\. Every deployed policy receives only the problem statement\. For FlowBalance, a frozen current\-policy snapshot additionally sees the training solution or task feedback only while scoring already sampled tokens; the resulting self\-guidance is unavailable at inference\. RLSD uses an analogous frozen current\-policy scoring path, whereas OPSD uses a fixed privileged teacher according to its original formulation\. All methods share the same training prompts, rollout group size, verifier, response\-length cap, checkpoint schedule, and evaluation script within each backbone\. We compare GRPO\[[Shao et al\., 2024](https://arxiv.org/html/2609.03241#bib.bib13),[Yu et al\., 2025c](https://arxiv.org/html/2609.03241#bib.bib12)\], OPSD\[[Zhao et al\., 2026a](https://arxiv.org/html/2609.03241#bib.bib1)\], RLSD\[[Yang et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib17)\], FlowRL\[[Zhu et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib11)\], and FlowBalance\. Table[1](https://arxiv.org/html/2609.03241#S5.T1)reports final benchmark accuracy, and Figure[2](https://arxiv.org/html/2609.03241#S5.F2)reports the intermediate traces used to diagnose update efficiency, stability, and response\-length behavior\. Appendix[D\.1](https://arxiv.org/html/2609.03241#A4.SS1), especially Table[5](https://arxiv.org/html/2609.03241#A4.T5), lists the concrete training and decoding fields used for reproducibility\.

### 5\.2Verifier\-Grounded Self\-Improvement on Mathematical Reasoning

##### Main Results\.

Table[1](https://arxiv.org/html/2609.03241#S5.T1)summarizes mathematical reasoning results across two model backbones\. All entries report step\-180 results over five seeds\. AIME24 is evaluated with Pass@16 to reflect competition\-style sampling, while HMMT25, Minerva, MATH500, and OlympiadBench use Pass@1\. The reported aggregate averages summarize broad performance rather than tuning to a single benchmark\.

Table 1:Mathematical reasoning results across Qwen3\-4B and Qwen3\-8B\. AIME24 uses Pass@16; all other benchmarks use Pass@1\. Entries are percentages reported as mean±\\pmsample standard deviation over five seeds at step 180\. “Avg\.” averages the five reported benchmark means\.ModelMethodAIME24@16HMMT25@1Minerva@1MATH500@1Olympiad@1Avg\.Qwen3\-4BGRPO78\.00±1\.8378\.00\\pm 1\.8326\.67±2\.3626\.67\\pm 2\.3651\.18±1\.3651\.18\\pm 1\.3692\.04±0\.9892\.04\\pm 0\.9863\.68±0\.5863\.68\\pm 0\.5862\.3162\.31OPSD65\.33±3\.8065\.33\\pm 3\.8014\.67±2\.9814\.67\\pm 2\.9847\.28±0\.5647\.28\\pm 0\.5687\.56±1\.3487\.56\\pm 1\.3455\.76±1\.4755\.76\\pm 1\.4754\.1254\.12RLSD73\.33±2\.3673\.33\\pm 2\.3621\.33±3\.8021\.33\\pm 3\.8050\.29±0\.8850\.29\\pm 0\.8891\.44±0\.5291\.44\\pm 0\.5261\.36±0\.6661\.36\\pm 0\.6659\.5559\.55FlowRL75\.33±1\.8375\.33\\pm 1\.8330\.67±4\.9430\.67\\pm 4\.9451\.99±1\.51\\mathbf\{51\.99\\pm 1\.51\}92\.84±0\.6292\.84\\pm 0\.6265\.25±0\.4965\.25\\pm 0\.4963\.2263\.22FlowBalance80\.00±0\.00\\mathbf\{80\.00\\pm 0\.00\}32\.00±2\.98\\mathbf\{32\.00\\pm 2\.98\}50\.51±0\.5650\.51\\pm 0\.5693\.28±0\.59\\mathbf\{93\.28\\pm 0\.59\}65\.49±0\.92\\mathbf\{65\.49\\pm 0\.92\}64\.26\\mathbf\{64\.26\}Qwen3\-8BGRPO85\.33±1\.8385\.33\\pm 1\.8331\.33±7\.6731\.33\\pm 7\.6752\.87±1\.0252\.87\\pm 1\.0293\.16±0\.8393\.16\\pm 0\.8364\.78±0\.6264\.78\\pm 0\.6265\.4965\.49OPSD48\.67±3\.8048\.67\\pm 3\.804\.00±3\.654\.00\\pm 3\.6538\.46±3\.8538\.46\\pm 3\.8574\.56±4\.8174\.56\\pm 4\.8140\.09±3\.5740\.09\\pm 3\.5741\.1641\.16RLSD82\.67±3\.6582\.67\\pm 3\.6528\.00±1\.8328\.00\\pm 1\.8352\.94±1\.3852\.94\\pm 1\.3893\.44±0\.1793\.44\\pm 0\.1763\.56±1\.1963\.56\\pm 1\.1964\.1264\.12FlowRL86\.67±0\.0086\.67\\pm 0\.0030\.67±4\.3530\.67\\pm 4\.3552\.79±1\.3752\.79\\pm 1\.3792\.92±0\.5092\.92\\pm 0\.5066\.20±1\.0566\.20\\pm 1\.0565\.8565\.85FlowBalance89\.33±1\.49\\mathbf\{89\.33\\pm 1\.49\}34\.67±9\.89\\mathbf\{34\.67\\pm 9\.89\}53\.68±0\.78\\mathbf\{53\.68\\pm 0\.78\}93\.52±0\.30\\mathbf\{93\.52\\pm 0\.30\}66\.85±0\.46\\mathbf\{66\.85\\pm 0\.46\}67\.61\\mathbf\{67\.61\}

On Qwen3\-4B, FlowBalance reaches a five\-benchmark average of64\.2664\.26, improving over GRPO by1\.951\.95points, OPSD by10\.1410\.14points, RLSD by4\.714\.71points, and FlowRL by1\.041\.04points\. It outperforms OPSD and RLSD on all five benchmarks and improves over GRPO on AIME24, HMMT25, MATH500, and OlympiadBench; FlowRL obtains the highest Minerva mean on this backbone\. On Qwen3\-8B, FlowBalance reaches67\.6167\.61and obtains the best mean on every reported benchmark, improving over GRPO by2\.122\.12points, OPSD by26\.4526\.45points, RLSD by3\.493\.49points, and FlowRL by1\.761\.76points\. The comparison to FlowRL isolates the value of self\-guidance inside the same trajectory\-distribution family, while the comparisons to GRPO, OPSD, and RLSD show that neither verifier\-only optimization nor direct local self\-imitation explains the aggregate gain\. A plausible explanation for OPSD’s poor reasoning performance is that imitating a solution\-conditioned privileged teacher can suppress epistemic verbalization and shorten reasoning, weakening uncertainty\-driven exploration and self\-correction\[[Kim et al\., 2026b](https://arxiv.org/html/2609.03241#bib.bib4)\]; this is consistent with Figure[2\(c\)](https://arxiv.org/html/2609.03241#S5.F2.sf3)\. The largest FlowBalance gain over stronger baselines appears on AIME24 Pass@16, where allocating mass across multiple correct trajectories is especially useful\. The smaller but consistent gains on MATH500 and OlympiadBench indicate that this distributional update does not trade broad Pass@1 accuracy for a sampling\-heavy metric\.

##### Training Dynamics\.

Figure[2](https://arxiv.org/html/2609.03241#S5.F2)summarizes the dynamics of repeated policy improvement on Qwen3\-8B\. The accuracy panels use matched rollout and evaluation budgets for FlowBalance and GRPO, while the response\-length panel compares FlowBalance with direct OPSD\. First, FlowBalance reaches0\.50\.5AIME24 validation accuracy in about100100steps, compared with roughly143143for GRPO, a1\.43×1\.43\\timesreduction in updates to the threshold\. Second, FlowBalance remains near its peak accuracy throughout400400steps, whereas GRPO degrades sharply after approximately step180180\. Third, FlowBalance maintains substantially longer reasoning trajectories than direct OPSD, which rapidly collapses to short responses\. These observations support a narrow but useful claim: in this fixed\-task inner loop, verifier\-grounded self\-guidance accelerates and stabilizes the update while avoiding the shortcut behavior of ungrounded local imitation\.

![Refer to caption](https://arxiv.org/html/2609.03241v1/training_acceleration.png)\(a\)Training acceleration\.
![Refer to caption](https://arxiv.org/html/2609.03241v1/training_dynamics.png)\(b\)Training stability\.
![Refer to caption](https://arxiv.org/html/2609.03241v1/response_length.png)\(c\)Response length\.

Figure 2:Training dynamics on mathematical reasoning with Qwen3\-8B\.\(a\) Training acceleration:FlowBalance reaches0\.50\.5AIME24 validation accuracy in about100100steps, compared with roughly143143steps for GRPO \(1\.43×1\.43\\timesfaster\)\.\(b\) Training stability:over400400training steps, FlowBalance remains near its peak performance, whereas GRPO degrades sharply after approximately step180180\.\(c\) Response length:direct OPSD rapidly collapses to substantially shorter responses, whereas FlowBalance maintains longer reasoning trajectories\. In the accuracy panels, solid curves show smoothed trends and lighter curves show the corresponding per\-step measurements\.
##### Ablation Study\.

We study the two coefficients in the FlowBalance energy:

EFlowBalance​\(y\)=ηA​A𝒢​\(y\)\+βG​GH​\(y\)​sgn⁡\(A𝒢​\(y\)\)\.E\_\{\\mathrm\{FlowBalance\}\}\(y\)=\\eta\_\{A\}A\_\{\\mathcal\{G\}\}\(y\)\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\)\\operatorname\{sgn\}\\\!\\big\(A\_\{\\mathcal\{G\}\}\(y\)\\big\)\.\(29\)We varyηA\\eta\_\{A\}to control verifier grounding andβG\\beta\_\{G\}to control self\-guidance strength\. The ablation uses one\-dimensional sweeps around the default setting rather than a full grid, so that each table isolates one mechanism while holding the other coefficient fixed\. Table[3](https://arxiv.org/html/2609.03241#S5.T3)sweepsηA∈\{5,10,15\}\\eta\_\{A\}\\in\\\{5,10,15\\\}, and Table[3](https://arxiv.org/html/2609.03241#S5.T3)sweeps the self\-guidance coefficientβG∈\{1,2,3\}\\beta\_\{G\}\\in\\\{1,2,3\\\}\. The default run corresponds toηA=15\\eta\_\{A\}=15andβG=1\\beta\_\{G\}=1; theηA=10\\eta\_\{A\}=10run and the fullβG\\beta\_\{G\}sweep are now complete\. Each ablation row uses the same backbone, training budget, seeds, and five\-benchmark average as Table[1](https://arxiv.org/html/2609.03241#S5.T1); in addition to accuracy, we track training stability to detect settings that improve one benchmark at the cost of unstable optimization\.

Table 2:Ablation over verifier coefficientηA\\eta\_\{A\}\.ηA\\eta\_\{A\}51015Avg\.65\.6565\.6565\.4165\.4167\.6167\.61
Table 3:Ablation over self\-guidance coefficientβG\\beta\_\{G\}\.βG\\beta\_\{G\}123Avg\.67\.6167\.6166\.4866\.4865\.9565\.95

### 5\.3Correct\-Strategy Diversity under Self\-Improvement

Figure 3:LLM\-judged strategy diversity\.Correct\-only Simpson diversity on AIME24; protocol and scope are given in Appendix[D\.3](https://arxiv.org/html/2609.03241#A4.SS3)\.A reliable self\-improvement update should not obtain accuracy only by sharpening one successful template\. FlowBalance also broadens the kinds of correct solutions represented by the policy\. We measure this with an LLM\-judged semantic strategy diagnostic on AIME24: a GPT\-5\.5 judge extracts each full trajectory’s mathematical representation and tools, clusters trajectories with the same core strategy, and we report Simpson strategy diversity,

DSimpson=1−∑kpk2,D\_\{\\mathrm\{Simpson\}\}=1\-\\sum\_\{k\}p\_\{k\}^\{2\},\(30\)wherepkp\_\{k\}is the fraction of correct trajectories in strategy clusterkk\. This metric is the probability that two sampled correct trajectories use different semantic strategies; Appendix[D\.3](https://arxiv.org/html/2609.03241#A4.SS3)gives the full protocol and scope\.

Figure[3](https://arxiv.org/html/2609.03241#S5.F3)reports correct\-only Simpson diversity\. FlowBalance achieves0\.21940\.2194, compared with0\.10170\.1017for GRPO and0\.14560\.1456for RLSD\. Thus, within this diagnostic setting, FlowBalance’s successful responses span a broader range of semantic solution strategies\.

Table 4:Case study on AIME24 Problem 23\.Boxed spans indicate key reasoning actions in representative correct trajectories; “⋯\\cdots” denotes omitted intermediate text\.Content \(boxed = reasoning actions; “⋯\\cdots” = omitted\)QuestionTetrahedronA​B​C​DABCDsatisfiesA​B=C​D=41AB=CD=\\sqrt\{41\},A​C=B​D=80AC=BD=\\sqrt\{80\}, andB​C=A​D=89BC=AD=\\sqrt\{89\}\. If the common distance from the incenter to the four faces ism​n/pm\\sqrt\{n\}/p, findm\+n\+pm\+n\+p\.GRPOCommon Cayley–Menger route\.“⋯\\cdotsall faces have sides41,80,89\\sqrt\{41\},\\sqrt\{80\},\\sqrt\{89\}⋯\\cdotsS=24​21S=24\\sqrt\{21\}⋯\\cdotsbuild Cayley–Menger matrixMMfrom41,80,8941,80,89⋯\\cdotsdiagonalize the distance block by symmetry⋯\\cdotsdetM=819200\\det M=819200⋯\\cdotsV=160/3V=160/3⋯\\cdotsr=3​V/S=20​21/63r=3V/S=20\\sqrt\{21\}/63⋯\\cdots20\+21\+63=10420\+21\+63=104⋯\\cdots”FlowBalanceHidden box\-embedding route\.“⋯\\cdotsobserve41=42\+5241=4^\{2\}\+5^\{2\},80=42\+8280=4^\{2\}\+8^\{2\},89=52\+8289=5^\{2\}\+8^\{2\}⋯\\cdotsembed in a4×5×84\\times 5\\times 8box⋯\\cdotsA=\(0,0,0\)A=\(0,0,0\),B=\(4,5,0\)B=\(4,5,0\),C=\(4,0,8\)C=\(4,0,8\),D=\(0,5,8\)D=\(0,5,8\)⋯\\cdotsverify all six edge lengths⋯\\cdotsscalar triple product givesV=160/3V=160/3⋯\\cdotsr=20​21/63r=20\\sqrt\{21\}/63⋯\\cdots20\+21\+63=10420\+21\+63=104⋯\\cdots”##### Case Study\.

Table[4](https://arxiv.org/html/2609.03241#S5.T4)illustrates what the LLM\-judged strategy clusters are intended to capture: differences in the mathematical objects and representations that make a solution work\. We choose this example for its structural contrast rather than for per\-problem win rate\. GRPO uses a standard and broadly applicable Cayley–Menger determinant solution: it treats the tetrahedron as six pairwise distances and obtains the volume from a distance invariant\. FlowBalance follows a different route by recognizing41=42\+5241=4^\{2\}\+5^\{2\},80=42\+8280=4^\{2\}\+8^\{2\}, and89=52\+8289=5^\{2\}\+8^\{2\}, which reveals a hidden4×5×84\\times 5\\times 8rectangular\-box embedding and reduces the volume computation to a scalar triple product\. This case study therefore grounds the aggregate LLM\-judged result: FlowBalance can place probability mass on correct trajectories that are not merely rephrasings of the dominant solution template, but use a different mathematical representation altogether\. Appendix[D\.5](https://arxiv.org/html/2609.03241#A4.SS5)gives three additional AIME24 examples in the same format, covering envelope versus multiple\-root reasoning, planar circle tangency versus implicit\-surface normals, and coordinate elimination versus trigonometric hyperbola parameterization\.

## 6Related Work

##### Reinforcement learning from verifiable outcomes\.

RLVR is a major route to improving LLM reasoning with automatically checkable answers\[[Shao et al\., 2024](https://arxiv.org/html/2609.03241#bib.bib13),[Yu et al\., 2025c](https://arxiv.org/html/2609.03241#bib.bib12),[Zhou et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib38),[Yu et al\., 2025a](https://arxiv.org/html/2609.03241#bib.bib39),[Liang et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib40),[Xia et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib41)\]\. Group\-based methods estimate advantages from multiple responses to one prompt without a separately learned critic\. Recent work improves this optimization through sequence\-level ratios, trust regions, variance\-aware weighting, dynamic clipping, and adaptive baselines or learning rates\[[Zheng et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib20),[Xie et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib21),[Zhang et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib22),[Yang et al\., 2025b](https://arxiv.org/html/2609.03241#bib.bib23),[Huang et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib24)\]\. These methods provide reliable outcome grounding, but their supervision remains response\-level\. FlowBalance addresses the complementary experience\-to\-policy question: how sparse outcomes and dense training\-time evidence should define a normalized distribution over complete responses\.

##### Privileged self\-guidance and on\-policy distillation\.

Knowledge distillation transfers behavior from a teacher to a student, while on\-policy distillation evaluates student\-generated trajectories under a teacher\[[Buciluǎ et al\., 2006](https://arxiv.org/html/2609.03241#bib.bib29),[Hinton et al\., 2015](https://arxiv.org/html/2609.03241#bib.bib30),[Agarwal et al\., 2024](https://arxiv.org/html/2609.03241#bib.bib31)\]\. Recent methods use a privileged view of the same model, conditioned on reference answers, demonstrations, correct rollouts, or environment feedback\[[Hübotter et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib2),[Zhao et al\., 2026a](https://arxiv.org/html/2609.03241#bib.bib1),[Penaloza et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib32),[Shenfeld et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib3),[Yang et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib17),[Ye et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib33)\]\. This produces dense token\-level information, but it can also shorten reasoning, suppress uncertainty, leak unavailable context into the learning signal, or plateau and degrade\[[Yang et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib17),[Kim et al\., 2026b](https://arxiv.org/html/2609.03241#bib.bib4),[Kim and Lee, 2026](https://arxiv.org/html/2609.03241#bib.bib9),[Zhu et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib6)\]\. Existing work modifies, selects, or contrasts local teacher signals\[[Pan et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib18),[Yu et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib5),[Zhao et al\., 2026b](https://arxiv.org/html/2609.03241#bib.bib7),[Zhang et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib8),[Jia et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib10),[Xu et al\., 2026b](https://arxiv.org/html/2609.03241#bib.bib15),[Kim et al\., 2026a](https://arxiv.org/html/2609.03241#bib.bib14)\]; concurrentβ\\beta\-OPSD derives an efficient KL\-regularized target with a tunable reference–teacher tradeoff\[[Xu et al\., 2026a](https://arxiv.org/html/2609.03241#bib.bib34)\]\. FlowBalance uses the same\-model privileged scorer only as a stopped*self\-guidance feature*\. The algorithmic object is instead a verifier\-grounded complete\-response distribution fitted by trajectory balance, with no separate token\-level imitation loss\.

##### Guided self\-improvement and self\-evolving systems\.

Self\-evolving systems add an outer loop that generates, selects, or schedules new learning experiences\. R\-Few, for example, stabilizes a Challenger–Solver loop with a small pool of human anchors and an online difficulty\-based curriculum\[[Yu et al\., 2025d](https://arxiv.org/html/2609.03241#bib.bib45)\]\. Its primary object is the evolving task and curriculum distribution\. FlowBalance studies a complementary inner\-loop object: given prompts and on\-policy solution trajectories, how should those experiences update the policy without amplifying false self\-confidence or collapsing successful\-strategy diversity? We do not combine the two systems experimentally; rather, R\-Few illustrates how an outer\-loop experience generator could supply tasks to a FlowBalance\-style distributional policy update\.

##### Distribution matching and trajectory balance\.

GFlowNets learn stochastic construction policies that sample objects in proportion to an unnormalized reward rather than concentrating only on maximizers\[[Bengio et al\., 2021](https://arxiv.org/html/2609.03241#bib.bib25),[Bengio et al\., 2023](https://arxiv.org/html/2609.03241#bib.bib26),[Xu et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib43),[Panaganti et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib42),[Panaganti et al\., 2022](https://arxiv.org/html/2609.03241#bib.bib44)\]\. Trajectory balance enforces this condition over complete trajectories and improves long\-horizon credit assignment\[[Malkin et al\., 2022](https://arxiv.org/html/2609.03241#bib.bib27)\]\. For LLM reasoning, Flow of Reasoning and FlowRL instantiate trajectory balance with a prompt\-conditioned partition term\[[Yu et al\., 2025b](https://arxiv.org/html/2609.03241#bib.bib28),[Zhu et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib11)\], while GFlowRL replaces the auxiliary partition network with an in\-batch rollout\-group estimate\[[Du et al\., 2026](https://arxiv.org/html/2609.03241#bib.bib19)\]\. FlowBalance builds on this distributional lineage but changes the target energy: verified outcome advantages are combined with outcome\-calibrated privileged\-hindsight guidance, and the resulting reference\-supported distribution is profiled over the rollout group\. The comparison to FlowRL in our experiments isolates this additional guidance inside the same broad trajectory\-balance family\.

## 7Conclusions and Discussions

##### Conclusion\.

FlowBalance treats policy self\-improvement as a distribution\-learning problem over the model’s own on\-policy reasoning experience\. A frozen privileged\-hindsight view of the same policy supplies dense self\-guidance, a verifier determines whether that guidance should be retained or reversed, and profiled trajectory balance fits the resulting reference\-supported target over complete responses\. This separation is central: dense guidance informs the target, but the policy is updated only through the normalized trajectory\-balance objective\. The theory shows that the profiled update preserves all within\-group contrasts, is the minimum reverse\-KL displacement at its attained energy level, retains monotonic verifier control of target reward, and converts false\-positive guidance on rejected trajectories into an anti\-self\-confirmation correction\. Across Qwen3\-4B and Qwen3\-8B, FlowBalance achieves the strongest five\-benchmark average among GRPO, OPSD, RLSD, FlowRL, and FlowBalance\. It also reaches the AIME24 validation threshold in fewer updates than GRPO, remains stable over extended training, avoids direct OPSD’s response\-length collapse, and exhibits higher correct\-only semantic strategy diversity in the controlled AIME24 diagnostic\.

##### Limitations and scope\.

Four limitations delimit these claims\. First, the large\-scale experiments focus on mathematical reasoning, so generalization to agentic, multimodal, or other long\-horizon domains remains open\. Second, the response\-length comparison diagnoses a correlated failure mode but does not establish that longer responses cause higher accuracy\. Third, the diversity analysis is an LLM\-judged AIME24 diagnostic at one checkpoint and seed rather than a multi\-seed human study\. Fourth, we isolate the experience\-to\-policy inner loop on a fixed prompt distribution; FlowBalance is not by itself a full self\-evolving system that generates or curates new tasks\. A natural next step is to combine the distributional update with an outer\-loop curriculum or guided task generator, including R\-Few\-style Challenger–Solver systems, while separately measuring task drift, policy stability, and strategy diversity\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Bengioet al\.\(2021\)E\. Bengio, M\. Jain, M\. Korablyov, D\. Precup, and Y\. BengioFlow network based generative models for non\-iterative diverse candidate generation\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 27381–27394\.Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Bengioet al\.\(2023\)Y\. Bengio, S\. Lahlou, T\. Deleu, E\. J\. Hu, M\. Tiwari, and E\. BengioGFlowNet foundations\.Journal of Machine Learning Research24\(210\),pp\. 1–55\.Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Buciluǎet al\.\(2006\)C\. Buciluǎ, R\. Caruana, and A\. Niculescu\-MizilModel compression\.InProceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 535–541\.External Links:[Document](https://dx.doi.org/10.1145/1150402.1150464)Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Duet al\.\(2026\)L\. Du, K\. Li, Y\. Cao, K\. Lin, Y\. Ma, Y\. Lu, X\. Chen, J\. Yang, Z\. Ma, Z\. Chu, X\. Huang, H\. Yu, J\. M\. Zhang, F\. Wei, B\. Yu, and L\. LiGFlowRL: scaling distribution\-matching RL to large language models\.External Links:2607\.13394Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Guanet al\.\(2026\)J\. Guan, Q\. Chen, L\. Qin, D\. Peng, J\. Liu, L\. Huo, J\. Xie, and W\. CheBeware of reasoning overconfidence: pitfalls in the reasoning process for multi\-solution tasks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 30843–30851\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v40i36.40342)Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p1.1)\.
- Hintonet al\.\(2015\)G\. Hinton, O\. Vinyals, and J\. DeanDistilling the knowledge in a neural network\.External Links:1503\.02531Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Huanget al\.\(2026\)Z\. Huang, J\. Sheng, and Z\. ZhengVariance\-aware baselines and adaptive learning rates for reinforcement learning with verifiable rewards\.External Links:2511\.23310Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Hübotteret al\.\(2026\)J\. Hübotter, F\. Lübeck, L\. Behric, A\. Baumann, M\. Bagatella, D\. Marta, I\. Hakimi, I\. Shenfeld, T\. Kleine Buening, C\. Guestrin, and A\. KrauseReinforcement learning via self\-distillation\.External Links:2601\.20802Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p3.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Jiaet al\.\(2026\)N\. Jia, H\. Yang, X\. Ma, J\. Lian, S\. Zhang, W\. Zhang, K\. Zeng, X\. Cai, and Z\. SunAsymmetric on\-policy distillation: bridging exploitation and imitation at the token level\.External Links:2605\.06387Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Kim and Lee \(2026\)J\. Kim and D\. LeeOPSD compresses what RLVR teaches: a post\-RL compaction stage for reasoning models\.External Links:2605\.06188Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Kimet al\.\(2026a\)J\. Kim, J\. Jeon, D\. Li, and Y\. YangRebellious student: reversing teacher signals for reasoning exploration with self\-distilled RLVR\.External Links:2605\.10781Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Kimet al\.\(2026b\)J\. Kim, X\. Luo, M\. Kim, S\. Lee, D\. Kim, J\. Jeon, D\. Li, and Y\. YangWhy does self\-distillation \(sometimes\) degrade the reasoning capability of LLMs?\.External Links:2603\.24472Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p3.1),[§5\.2](https://arxiv.org/html/2609.03241#S5.SS2.SSS0.Px1.p2.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Kujawaet al\.\(2025\)Z\. Kujawa, J\. Poole, D\. Georgiev, D\. Numeroso, H\. Fleischmann, and P\. LiòNeural algorithmic reasoning with multiple correct solutions\.InWorkshop on Machine Learning on Graphs in the Era of Generative Artificial Intelligence at KDD,External Links:2409\.06953Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p1.1)\.
- Lianget al\.\(2025\)Z\. Liang, S\. Lu, W\. Yu, K\. Panaganti, Y\. Zhou, H\. Mi, and D\. YuCan LLMs guide their own exploration? gradient\-guided reinforcement learning for LLM reasoning\.External Links:2512\.15687Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Malkinet al\.\(2022\)N\. Malkin, M\. Jain, E\. Bengio, C\. Sun, and Y\. BengioTrajectory balance: improved credit assignment in GFlowNets\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 5955–5967\.Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Panet al\.\(2026\)L\. Pan, S\. Tao, Y\. Zhai, S\. Zhang, W\. Li, X\. Liu, J\. Wang, J\. Shen, Z\. Tu, and K\. ZhaoRLCSD: reinforcement learning with contrastive on\-policy self\-distillation\.External Links:2606\.11709Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Panagantiet al\.\(2026\)K\. Panaganti, Z\. Liang, W\. Yu, H\. Mi, and D\. YuGroup distributionally robust optimization\-driven reinforcement learning for LLM reasoning\.External Links:2601\.19280Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Panagantiet al\.\(2022\)K\. Panaganti, Z\. Xu, D\. Kalathil, and M\. GhavamzadehRobust reinforcement learning using offline data\.InAdvances in Neural Information Processing Systems,External Links:2208\.05129Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Penalozaet al\.\(2026\)E\. Penaloza, D\. Vattikonda, N\. Gontier, A\. Lacoste, L\. Charlin, and M\. CacciaPrivileged information distillation for language models\.External Links:2602\.04942Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Ren and Sutherland \(2025\)Y\. Ren and D\. J\. SutherlandLearning dynamics of LLM finetuning\.InInternational Conference on Learning Representations,External Links:2407\.10490Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.External Links:2402\.03300Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.03241#S5.SS1.p1.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Shenfeldet al\.\(2026\)I\. Shenfeld, M\. Damani, J\. Hübotter, and P\. AgrawalSelf\-distillation enables continual learning\.External Links:2601\.19897Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Xiaet al\.\(2026\)J\. Xia, Y\. Shi, Y\. Yang, K\. Panaganti, Z\. Liang, and N\. LiuReasoning or memorization? direction\-aware diversity exploration in LLM reinforcement learning\.External Links:2606\.10346Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2025\)Z\. Xie, Q\. Zhang, F\. Yang, M\. Hutter, and R\. XuSimple policy optimization\.External Links:2401\.16025Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2026a\)H\. Xu, C\. Shi, C\. Yang, M\. Li, K\. Liu, W\. Zhang, and J\. Wangβ\\beta\-OPSD: Deriving with Policy Optimization, Training with Self\-Distillation\.External Links:2607\.23787Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2026b\)Y\. Xu, H\. Sang, Z\. Zhou, R\. He, Z\. Wang, and A\. GeramifardTIP: token importance in on\-policy distillation\.External Links:2604\.14084Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Xuet al\.\(2025\)Z\. Xu, S\. Vemuri, K\. Panaganti, D\. Kalathil, R\. Jain, and D\. RamachandranRobust LLM alignment via distributionally robust direct preference optimization\.InAdvances in Neural Information Processing Systems,External Links:2502\.01930Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Yanget al\.\(2025a\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, K\. Yang, L\. Yu, X\. Wang, J\. Zhou, and Q\. TeamQwen3 technical report\.External Links:2505\.09388Cited by:[§5\.1](https://arxiv.org/html/2609.03241#S5.SS1.p1.1)\.
- Yanget al\.\(2026\)C\. Yang, C\. Qin, Q\. Si, M\. Chen, N\. Gu, D\. Yao, Z\. Lin, W\. Wang, J\. Wang, and N\. DuanSelf\-distilled RLVR\.External Links:2604\.03128Cited by:[§5\.1](https://arxiv.org/html/2609.03241#S5.SS1.p1.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Yanget al\.\(2025b\)S\. Yang, C\. Dou, P\. Guo, K\. Lu, Q\. Ju, F\. Deng, and R\. XinDCPO: dynamic clipping policy optimization\.External Links:2509\.02333Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Yeet al\.\(2026\)T\. Ye, L\. Dong, X\. Wu, S\. Huang, and F\. WeiOn\-policy context distillation for language models\.External Links:2602\.12275Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2025a\)D\. Yu, Y\. Zhao, K\. Panaganti, L\. Song, H\. Mi, and D\. YuEvery question has its own value: reinforcement learning with explicit human values\.External Links:2510\.20187Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025b\)F\. Yu, L\. Jiang, H\. Kang, S\. Hao, and L\. QinFlow of reasoning: training LLMs for divergent reasoning with minimal examples\.Note:ICML 2025External Links:2406\.05673Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.
- Yuet al\.\(2025c\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Li, W\. Dai, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, Y\. Lin, M\. Qiao, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.External Links:2503\.14476Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.03241#S5.SS1.p1.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Yuet al\.\(2025d\)W\. Yu, Z\. Liang, C\. Huang, K\. Panaganti, T\. Fang, H\. Mi, and D\. YuGuided self\-evolving llms with minimal human supervision\.External Links:2512\.02472,[Link](https://arxiv.org/abs/2512.02472)Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px3.p1.1)\.
- Yuet al\.\(2026\)X\. Yu, G\. Li, Q\. Si, G\. Zhang, Y\. Xu, C\. Wang, S\. Dong, K\. Tuo, X\. Zeng, K\. Feng, Q\. Wang, Y\. Shi, X\. Hu, X\. Yue, J\. Wang, and S\. YanDOPD: dual on\-policy distillation\.External Links:2606\.30626Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2026\)H\. Zhang, C\. Wang, K\. Chen, Y\. Pan, Y\. Xiang, J\. Wang, and M\. ZhangTailoring teaching to aptitude: direction\-adaptive self\-distillation for llm reasoning\.External Links:2605\.22263Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Zhanget al\.\(2025\)K\. Zhang, Y\. Hong, J\. Bao, H\. Jiang, Y\. Song, D\. Hong, and H\. XiongGVPO: group variance policy optimization for large language model post\-training\.External Links:2504\.19599Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2026a\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. GroverSelf\-distilled reasoner: on\-policy self\-distillation for large language models\.External Links:2601\.18734Cited by:[§D\.1](https://arxiv.org/html/2609.03241#A4.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.03241#S1.p3.1),[§5\.1](https://arxiv.org/html/2609.03241#S5.SS1.p1.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026b\)Z\. Zhao, X\. Ma, L\. Yang, Y\. Feng, D\. Shi, J\. He, X\. Xin, Z\. Ren, and X\. WuReflective on\-policy self\-distillation for language model reasoning across domains\.External Links:2605\.28014Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.External Links:2507\.18071Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Zhouet al\.\(2025\)Y\. Zhou, Z\. Liang, H\. Liu, W\. Yu, K\. Panaganti, L\. Song, D\. Yu, X\. Zhang, H\. Mi, and D\. YuEvolving language models without labels: majority drives selection, novelty promotes variation\.External Links:2509\.15194Cited by:[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px1.p1.1)\.
- Zhuet al\.\(2026\)S\. Zhu, X\. Ye, H\. Lu, W\. Shi, and G\. LiuThe many faces of on\-policy distillation: pitfalls, mechanisms, and fixes\.External Links:2605\.11182Cited by:[§1](https://arxiv.org/html/2609.03241#S1.p3.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px2.p1.1)\.
- Zhuet al\.\(2025\)X\. Zhu, D\. Cheng, D\. Zhang, H\. Li, K\. Zhang, C\. Jiang, Y\. Sun, E\. Hua, Y\. Zuo, X\. Lv, Q\. Zhang, L\. Chen, F\. Shao, B\. Xue, Y\. Song, Z\. Yang, G\. Cui, N\. Ding, J\. Gao, X\. Liu, B\. Zhou, H\. Mei, and Z\. LinFlowRL: matching reward distributions for LLM reasoning\.External Links:2509\.15207Cited by:[§D\.1](https://arxiv.org/html/2609.03241#A4.SS1.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2609.03241#S1.p2.1),[§5\.1](https://arxiv.org/html/2609.03241#S5.SS1.p1.1),[§6](https://arxiv.org/html/2609.03241#S6.SS0.SSS0.Px4.p1.1)\.

## Appendix ADetailed\-Balance View of FlowBalance

##### Detailed balance\.

Letqqbe a probability distribution on the response space𝒴⁡\(x\)\\mathcal\{Y\}\(x\), and letK⁡\(y→y′\)K\(y\\to y^\{\\prime\}\)be a Markov transition kernel\. The kernel is reversible with respect toqqif, for every pairy,y′∈𝒴⁡\(x\)y,y^\{\\prime\}\\in\\mathcal\{Y\}\(x\),

q⁡\(y\)​K​\(y→y′\)=q⁡\(y′\)​K​\(y′→y\)\.q\(y\)K\(y\\to y^\{\\prime\}\)=q\(y^\{\\prime\}\)K\(y^\{\\prime\}\\to y\)\.\(31\)This is the detailed\-balance condition: at equilibrium, the probability flow fromyytoy′y^\{\\prime\}equals the reverse flow fromy′y^\{\\prime\}toyy\. Summing Eq\. \([31](https://arxiv.org/html/2609.03241#A1.E31)\) overyyshows thatqqis a stationary distribution ofKK\. Whenever both transition probabilities are positive, detailed balance also gives

K⁡\(y→y′\)K⁡\(y′→y\)=q⁡\(y′\)q⁡\(y\)\.\\frac\{K\(y\\to y^\{\\prime\}\)\}\{K\(y^\{\\prime\}\\to y\)\}=\\frac\{q\(y^\{\\prime\}\)\}\{q\(y\)\}\.\(32\)

##### FlowBalance as an equilibrium distribution\.

For a fixed prompt–context pair\(x,c\)\(x,c\), FlowBalance defines the Gibbs target

p⋆​\(y∣x,c\)=1ZFlowBalance​\(x,c\)​πref​\(y∣x\)​exp⁡\(EFlowBalance​\(y∣x,c\)τ\)\.p^\{\\star\}\(y\\mid x,c\)=\\frac\{1\}\{Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)\}\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\!\\left\(\\frac\{E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\}\{\\tau\}\\right\)\.\(33\)If a reversible Markov kernel were constructed withp⋆p^\{\\star\}as its equilibrium distribution, its forward–reverse transition ratio would satisfy

K⁡\(y→y′\)K⁡\(y′→y\)=πref​\(y′∣x\)πref​\(y∣x\)​exp⁡\(EFlowBalance​\(y′∣x,c\)−EFlowBalance​\(y∣x,c\)τ\)\.\\frac\{K\(y\\to y^\{\\prime\}\)\}\{K\(y^\{\\prime\}\\to y\)\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\\prime\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\\exp\\\!\\left\(\\frac\{E\_\{\\mathrm\{FlowBalance\}\}\(y^\{\\prime\}\\mid x,c\)\-E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\}\{\\tau\}\\right\)\.\(34\)Thus the reference policy supplies the baseline relative mass, while the FlowBalance energy exponentially reweights transitions toward responses preferred by the verifier\-grounded self\-guidance energy\. The partition function does not appear in this pairwise ratio, but it is required to turn these relative preferences into a normalized distribution\.

##### Trajectory\-balance interpretation\.

FlowBalance does not construct or simulate the Markov kernelKK\. Instead, trajectory balance directly imposes the global identity

ZFlowBalance​\(x,c\)​πθ​\(y∣x\)=πref​\(y∣x\)​exp⁡\(EFlowBalance​\(y∣x,c\)τ\)Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)\\pi\_\{\\theta\}\(y\\mid x\)=\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\\exp\\\!\\left\(\\frac\{E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\}\{\\tau\}\\right\)\(35\)for each complete response\. Taking logarithms and multiplying byτ\\taugives the trajectory\-balance residual

τ​log⁡ZFlowBalance​\(x,c\)\+τ​log⁡πθ​\(y∣x\)πref​\(y∣x\)−EFlowBalance​\(y∣x,c\)\.\\tau\\log Z\_\{\\mathrm\{FlowBalance\}\}\(x,c\)\+\\tau\\log\\frac\{\\pi\_\{\\theta\}\(y\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\\mid x\)\}\-E\_\{\\mathrm\{FlowBalance\}\}\(y\\mid x,c\)\.\(36\)When this residual is zero for all responses,πθ\\pi\_\{\\theta\}equals the normalized target in Eq\. \([33](https://arxiv.org/html/2609.03241#A1.E33)\) and therefore has the same pairwise probability ratios as the equilibrium distribution associated with Eq\. \([34](https://arxiv.org/html/2609.03241#A1.E34)\)\. In this sense, detailed balance provides an equilibrium interpretation, whereas trajectory balance is the learning constraint that fits the desired distribution without explicitly defining pairwise response transitions\.

## Appendix BProofs for Theoretical Analyses

Throughout this appendix we condition on the same fixed prompt–context pair and realized rollout group as in Section[4](https://arxiv.org/html/2609.03241#S4)\. We use the main\-text notation

Ei=ηA​Ai\+βG​GH​\(y\(i\)∣x,c\)​sgn⁡\(Ai\),pi⋆=πref​\(y\(i\)∣x\)​exp⁡\(Ei/τ\)∑j=1Nπref​\(y\(j\)∣x\)​exp⁡\(Ej/τ\)\.E\_\{i\}=\\eta\_\{A\}A\_\{i\}\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y^\{\(i\)\}\\mid x,c\)\\operatorname\{sgn\}\(A\_\{i\}\),\\qquad p\_\{i\}^\{\\star\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\\exp\(E\_\{i\}/\\tau\)\}\{\\sum\_\{j=1\}^\{N\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\\exp\(E\_\{j\}/\\tau\)\}\.\(37\)When KL is computed over the realized rollout group,πref\(⋅∣x\)\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid x\)denotes its restriction and renormalization to that group\.

### B\.1Distributional Properties

###### Proposition B\.1\(Profiled balance matches all within\-group contrasts\)\.

For a candidate policyπ\\piand scalar partition/interceptzz, consider the realized\-group trajectory\-balance loss

ℒN​\(π,z\)=12​N​∑i=1N\[τ​z\+τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\]2\.\\mathcal\{L\}\_\{N\}\(\\pi,z\)=\\frac\{1\}\{2N\}\\sum\_\{i=1\}^\{N\}\\left\[\\tau z\+\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}\\right\]^\{2\}\.\(38\)After profiling outzz, the trajectory\-balance loss is zero if and only if every within\-group relative probability contrast matches

π⁡\(y\(i\)\)π⁡\(y\(j\)\)=πref​\(y\(i\)∣x\)πref​\(y\(j\)∣x\)​exp⁡\(Ei−Ejτ\),i,j∈\[N\]\.\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\(y^\{\(j\)\}\)\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\}\\exp\\\!\\left\(\\frac\{E\_\{i\}\-E\_\{j\}\}\{\\tau\}\\right\),\\qquad i,j\\in\[N\]\.\(39\)Whenever the group target is representable, the global minimum is therefore zero\. Profiling removes only one common group\-level offset and preserves the remainingN−1N\-1contrast directions\.

###### Proof\.

For a candidate policyπ\\pi, the trajectory\-balance residual for responsey\(i\)y^\{\(i\)\}and scalar partition/interceptzzis

τ​z\+τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\.\\tau z\+\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}\.\(40\)The empirical trajectory\-balance loss over the realized group is therefore

ℒN​\(π,z\)=12​N​∑i=1N\[τ​z\+τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\]2\.\\mathcal\{L\}\_\{N\}\(\\pi,z\)=\\frac\{1\}\{2N\}\\sum\_\{i=1\}^\{N\}\\left\[\\tau z\+\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}\\right\]^\{2\}\.\(41\)For fixedπ\\pi, this is a convex quadratic function of the single scalarzz\. Differentiating Eq\. \([41](https://arxiv.org/html/2609.03241#A2.E41)\) with respect tozzgives

∂ℒN​\(π,z\)∂z=τN​∑i=1N\[τ​z\+τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\]\.\\frac\{\\partial\\mathcal\{L\}\_\{N\}\(\\pi,z\)\}\{\\partial z\}=\\frac\{\\tau\}\{N\}\\sum\_\{i=1\}^\{N\}\\left\[\\tau z\+\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}\\right\]\.\(42\)Thus the profiled interceptz^\\widehat\{z\}is the unique solution of

∑i=1N\[τ​z^\+τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\]=0\.\\sum\_\{i=1\}^\{N\}\\left\[\\tau\\widehat\{z\}\+\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}\\right\]=0\.\(43\)After profiling, the loss is zero if and only if every squared residual in Eq\. \([41](https://arxiv.org/html/2609.03241#A2.E41)\) is zero atz=z^z=\\widehat\{z\}\. Hence, for every pairi,ji,j,

τ​z^\+τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\\displaystyle\\tau\\widehat\{z\}\+\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}=0,\\displaystyle=0,\(44\)τ​z^\+τ​log⁡π⁡\(y\(j\)\)πref​\(y\(j\)∣x\)−Ej\\displaystyle\\tau\\widehat\{z\}\+\\tau\\log\\frac\{\\pi\(y^\{\(j\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\}\-E\_\{j\}=0\.\\displaystyle=0\.\(45\)Subtracting the second equality from the first cancels the profiled scalarτ​z^\\tau\\widehat\{z\}and yields

τ​log⁡π⁡\(y\(i\)\)π⁡\(y\(j\)\)−τ​log⁡πref​\(y\(i\)∣x\)πref​\(y\(j\)∣x\)=Ei−Ej\.\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\(y^\{\(j\)\}\)\}\-\\tau\\log\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\}=E\_\{i\}\-E\_\{j\}\.\(46\)Dividing byτ\\tau, exponentiating, and rearranging the reference\-policy ratio gives Eq\. \([39](https://arxiv.org/html/2609.03241#A2.E39)\)\.

Conversely, if Eq\. \([39](https://arxiv.org/html/2609.03241#A2.E39)\) holds for all pairs, then the quantities

τ​log⁡π⁡\(y\(i\)\)πref​\(y\(i\)∣x\)−Ei\\tau\\log\\frac\{\\pi\(y^\{\(i\)\}\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\-E\_\{i\}\(47\)are all equal to the same constant\. Choosingzzto be minus that constant divided byτ\\taumakes every trajectory\-balance residual zero\. Therefore zero profiled loss is equivalent to matching all within\-group target log probability\-ratio contrasts\. The scalarzzremoves exactly one common offset and cannot change any pairwise contrast\. ∎

###### Proposition B\.2\(Minimum reference displacement\)\.

Letπref\\pi\_\{\\mathrm\{ref\}\}denote the reference distribution restricted and renormalized to the realized rollout group\. Then, for any group distributionp∈ΔNp\\in\\Delta\_\{N\},

KL\(p∥πref\)=KL\(p⋆∥πref\)\+KL\(p∥p⋆\)\+1τ\(∑i=1NpiEi−∑i=1Npi⋆Ei\)\.\\mathrm\{KL\}\(p\\\|\\pi\_\{\\mathrm\{ref\}\}\)=\\mathrm\{KL\}\(p^\{\\star\}\\\|\\pi\_\{\\mathrm\{ref\}\}\)\+\\mathrm\{KL\}\(p\\\|p^\{\\star\}\)\+\\frac\{1\}\{\\tau\}\\left\(\\sum\_\{i=1\}^\{N\}p\_\{i\}E\_\{i\}\-\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{\\star\}E\_\{i\}\\right\)\.\(48\)Consequently, among all group distributions satisfying∑ipi​Ei≥∑ipi⋆​Ei\\sum\_\{i\}p\_\{i\}E\_\{i\}\\geq\\sum\_\{i\}p\_\{i\}^\{\\star\}E\_\{i\}, the FlowBalance targetp⋆p^\{\\star\}uniquely minimizes the reverse\-KL displacementKL\(p∥πref\)\\mathrm\{KL\}\(p\\\|\\pi\_\{\\mathrm\{ref\}\}\)\.

###### Proof\.

From Eq\. \([24](https://arxiv.org/html/2609.03241#S4.E24)\), for each group element,

logpi⋆πref​\(y\(i\)∣x\)=Eiτ−log∑j=1Nπref\(y\(j\)∣x\)exp\(Ej/τ\)\.\\log\\frac\{p\_\{i\}^\{\\star\}\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}=\\frac\{E\_\{i\}\}\{\\tau\}\-\\log\\sum\_\{j=1\}^\{N\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\\exp\(E\_\{j\}/\\tau\)\.\(49\)Letp∈ΔNp\\in\\Delta\_\{N\}be any group distribution\. Then

KL\(p∥πref\)\\displaystyle\\mathrm\{KL\}\(p\\\|\\pi\_\{\\mathrm\{ref\}\}\)=∑i=1Npi​log⁡pipi⋆\+∑i=1Npi​log⁡pi⋆πref​\(y\(i\)∣x\)\\displaystyle=\\sum\_\{i=1\}^\{N\}p\_\{i\}\\log\\frac\{p\_\{i\}\}\{p\_\{i\}^\{\\star\}\}\+\\sum\_\{i=1\}^\{N\}p\_\{i\}\\log\\frac\{p\_\{i\}^\{\\star\}\}\{\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(i\)\}\\mid x\)\}\(50\)=KL\(p∥p⋆\)\+1τ∑i=1NpiEi−log∑j=1Nπref\(y\(j\)∣x\)exp\(Ej/τ\)\.\\displaystyle=\\mathrm\{KL\}\(p\\\|p^\{\\star\}\)\+\\frac\{1\}\{\\tau\}\\sum\_\{i=1\}^\{N\}p\_\{i\}E\_\{i\}\-\\log\\sum\_\{j=1\}^\{N\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\\exp\(E\_\{j\}/\\tau\)\.\(51\)Applying the same identity withp=p⋆p=p^\{\\star\}gives

KL\(p⋆∥πref\)=1τ∑i=1Npi⋆Ei−log∑j=1Nπref\(y\(j\)∣x\)exp\(Ej/τ\)\.\\mathrm\{KL\}\(p^\{\\star\}\\\|\\pi\_\{\\mathrm\{ref\}\}\)=\\frac\{1\}\{\\tau\}\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{\\star\}E\_\{i\}\-\\log\\sum\_\{j=1\}^\{N\}\\pi\_\{\\mathrm\{ref\}\}\(y^\{\(j\)\}\\mid x\)\\exp\(E\_\{j\}/\\tau\)\.\(52\)Subtracting this expression from Eq\. \([51](https://arxiv.org/html/2609.03241#A2.E51)\) yields Eq\. \([48](https://arxiv.org/html/2609.03241#A2.E48)\):

KL\(p∥πref\)=KL\(p⋆∥πref\)\+KL\(p∥p⋆\)\+1τ\(∑i=1NpiEi−∑i=1Npi⋆Ei\)\.\\mathrm\{KL\}\(p\\\|\\pi\_\{\\mathrm\{ref\}\}\)=\\mathrm\{KL\}\(p^\{\\star\}\\\|\\pi\_\{\\mathrm\{ref\}\}\)\+\\mathrm\{KL\}\(p\\\|p^\{\\star\}\)\+\\frac\{1\}\{\\tau\}\\left\(\\sum\_\{i=1\}^\{N\}p\_\{i\}E\_\{i\}\-\\sum\_\{i=1\}^\{N\}p\_\{i\}^\{\\star\}E\_\{i\}\\right\)\.\(53\)Ifppattains at least the expected FlowBalance energy ofp⋆p^\{\\star\}, the last term is nonnegative;KL\(p∥p⋆\)≥0\\mathrm\{KL\}\(p\\\|p^\{\\star\}\)\\geq 0as well, with equality only whenp=p⋆p=p^\{\\star\}\. Hencep⋆p^\{\\star\}is the unique minimum\-reverse\-KL displacement from the reference among all such distributions\. ∎

### B\.2Effects of Verifier Weighting and Sign Gating

###### Proposition B\.3\(Verifier weight monotonically improves reward statistics\)\.

Hold the self\-guidance scores and the realized rollout group fixed\. For the GRPO\-normalized advantage

Ai=Ri−R¯σR​\(x\)\+ϵ,A\_\{i\}=\\frac\{R\_\{i\}\-\\bar\{R\}\}\{\\sigma\_\{R\}\(x\)\+\\epsilon\},\(54\)increasing the verifier coefficientηA\\eta\_\{A\}monotonically increases the expected reward under the FlowBalance target:

∂∂ηA​𝔼p⋆​\[R\]=Varp⋆​\(R\)τ​\(σR​\(x\)\+ϵ\)≥0\.\\frac\{\\partial\}\{\\partial\\eta\_\{A\}\}\\mathbb\{E\}\_\{p^\{\\star\}\}\[R\]=\\frac\{\\mathrm\{Var\}\_\{p^\{\\star\}\}\(R\)\}\{\\tau\(\\sigma\_\{R\}\(x\)\+\\epsilon\)\}\\geq 0\.\(55\)

###### Proof\.

Hold the self\-guidance scores fixed\. The only dependence ofpi⋆p\_\{i\}^\{\\star\}onηA\\eta\_\{A\}is through the termηA​Ai\\eta\_\{A\}A\_\{i\}\. Differentiating the normalized exponential\-family form gives

∂pi⋆∂ηA=pi⋆τ​\(Ai−∑j=1Npj⋆​Aj\)\.\\frac\{\\partial p\_\{i\}^\{\\star\}\}\{\\partial\\eta\_\{A\}\}=\\frac\{p\_\{i\}^\{\\star\}\}\{\\tau\}\\left\(A\_\{i\}\-\\sum\_\{j=1\}^\{N\}p\_\{j\}^\{\\star\}A\_\{j\}\\right\)\.\(56\)Therefore

∂∂ηA​𝔼p⋆​\[R\]=1τ​Covp⋆​\(R,A\)\.\\frac\{\\partial\}\{\\partial\\eta\_\{A\}\}\\mathbb\{E\}\_\{p^\{\\star\}\}\[R\]=\\frac\{1\}\{\\tau\}\\mathrm\{Cov\}\_\{p^\{\\star\}\}\(R,A\)\.\(57\)For the GRPO\-normalized advantage used in the main text,

Ai=Ri−R¯σR​\(x\)\+ϵ,A\_\{i\}=\\frac\{R\_\{i\}\-\\bar\{R\}\}\{\\sigma\_\{R\}\(x\)\+\\epsilon\},\(58\)whereR¯\\bar\{R\}andσR​\(x\)\+ϵ\\sigma\_\{R\}\(x\)\+\\epsilonare fixed for the realized group\. Hence

Covp⋆​\(R,A\)=Varp⋆​\(R\)σR​\(x\)\+ϵ,\\mathrm\{Cov\}\_\{p^\{\\star\}\}\(R,A\)=\\frac\{\\mathrm\{Var\}\_\{p^\{\\star\}\}\(R\)\}\{\\sigma\_\{R\}\(x\)\+\\epsilon\},\(59\)and thus

∂∂ηA​𝔼p⋆​\[R\]=Varp⋆​\(R\)τ​\(σR​\(x\)\+ϵ\)≥0,\\frac\{\\partial\}\{\\partial\\eta\_\{A\}\}\\mathbb\{E\}\_\{p^\{\\star\}\}\[R\]=\\frac\{\\mathrm\{Var\}\_\{p^\{\\star\}\}\(R\)\}\{\\tau\(\\sigma\_\{R\}\(x\)\+\\epsilon\)\}\\geq 0,\(60\)which proves Eq\. \([55](https://arxiv.org/html/2609.03241#A2.E55)\)\. ∎

###### Proposition B\.4\(Sign gating corrects self\-guidance support on failures\)\.

Consider a verified successy\+y\_\{\+\}with positive advantage and a verifier\-rejected responsey−y\_\{\-\}with negative advantage in the same rollout group\. Relative to otherwise identical ungated self\-guidance shaping, sign gating changes their target probability ratio according to

pgated⋆​\(y\+\)pgated⋆​\(y−\)=pungated⋆​\(y\+\)pungated⋆​\(y−\)​exp⁡\(2​βG​GH​\(y−∣x,c\)τ\)\.\\frac\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\-\}\)\}=\\frac\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\-\}\)\}\\exp\\\!\\left\(\\frac\{2\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\)\}\{\\tau\}\\right\)\.\(61\)Hence positive privileged support on a verifier\-rejected response is converted from self\-reinforcement pressure into a probability\-ratio correction favoring the verified response\.

###### Proof\.

Consider a verified successy\+y\_\{\+\}and a verifier\-rejected responsey−y\_\{\-\}in the same group\. Their probability ratio under the gated target is

pgated⋆​\(y\+\)pgated⋆​\(y−\)=πref​\(y\+∣x\)πref​\(y−∣x\)​exp⁡\(E\+gated−E−gatedτ\),\\frac\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\-\}\)\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{\+\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{\-\}\\mid x\)\}\\exp\\\!\\left\(\\frac\{E^\{\\mathrm\{gated\}\}\_\{\+\}\-E^\{\\mathrm\{gated\}\}\_\{\-\}\}\{\\tau\}\\right\),\(62\)where the group normalizer cancels\. The corresponding ungated target has

pungated⋆​\(y\+\)pungated⋆​\(y−\)=πref​\(y\+∣x\)πref​\(y−∣x\)​exp⁡\(E\+ungated−E−ungatedτ\)\.\\frac\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\-\}\)\}=\\frac\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{\+\}\\mid x\)\}\{\\pi\_\{\\mathrm\{ref\}\}\(y\_\{\-\}\\mid x\)\}\\exp\\\!\\left\(\\frac\{E^\{\\mathrm\{ungated\}\}\_\{\+\}\-E^\{\\mathrm\{ungated\}\}\_\{\-\}\}\{\\tau\}\\right\)\.\(63\)The reference ratio cancels when comparing the gated and ungated probability ratios\. Sincey\+y\_\{\+\}has positive advantage andy−y\_\{\-\}has negative advantage,

E\+gated−E−gated\\displaystyle E^\{\\mathrm\{gated\}\}\_\{\+\}\-E^\{\\mathrm\{gated\}\}\_\{\-\}=ηA​\(A𝒢​\(y\+\)−A𝒢​\(y−\)\)\+βG​GH​\(y\+∣x,c\)\+βG​GH​\(y−∣x,c\),\\displaystyle=\\eta\_\{A\}\\left\(A\_\{\\mathcal\{G\}\}\(y\_\{\+\}\)\-A\_\{\\mathcal\{G\}\}\(y\_\{\-\}\)\\right\)\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\+\}\\mid x,c\)\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\),\(64\)E\+ungated−E−ungated\\displaystyle E^\{\\mathrm\{ungated\}\}\_\{\+\}\-E^\{\\mathrm\{ungated\}\}\_\{\-\}=ηA​\(A𝒢​\(y\+\)−A𝒢​\(y−\)\)\+βG​GH​\(y\+∣x,c\)−βG​GH​\(y−∣x,c\)\.\\displaystyle=\\eta\_\{A\}\\left\(A\_\{\\mathcal\{G\}\}\(y\_\{\+\}\)\-A\_\{\\mathcal\{G\}\}\(y\_\{\-\}\)\\right\)\+\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\+\}\\mid x,c\)\-\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\)\.\(65\)The verifier differenceηA​\(A𝒢​\(y\+\)−A𝒢​\(y−\)\)\\eta\_\{A\}\\left\(A\_\{\\mathcal\{G\}\}\(y\_\{\+\}\)\-A\_\{\\mathcal\{G\}\}\(y\_\{\-\}\)\\right\)is identical in the two expressions, and the self\-guidance contribution differs by2​βG​GH​\(y−∣x,c\)2\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\)\. Therefore

pgated⋆​\(y\+\)pgated⋆​\(y−\)=pungated⋆​\(y\+\)pungated⋆​\(y−\)​exp⁡\(2​βG​GH​\(y−∣x,c\)τ\),\\frac\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{gated\}\}\(y\_\{\-\}\)\}=\\frac\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\+\}\)\}\{p^\{\\star\}\_\{\\mathrm\{ungated\}\}\(y\_\{\-\}\)\}\\exp\\\!\\left\(\\frac\{2\\beta\_\{G\}G\_\{\\mathrm\{H\}\}\(y\_\{\-\}\\mid x,c\)\}\{\\tau\}\\right\),\(66\)which proves Eq\. \([61](https://arxiv.org/html/2609.03241#A2.E61)\)\. ∎

## Appendix CNumerical Analysis Details

### C\.1Exact Numerical Diagnostics for Verifier\-Grounded Self\-Improvement

We instantiate the unchanged FlowBalance target on exactly enumerable response spaces\. Every distribution below is obtained by normalizing

q⁡\(y\)∝ρ⁡\(y\)​exp⁡\(η​A​\(y\)\+β​G​\(y\)​sgn⁡\(A⁡\(y\)\)τ\),q\(y\)\\propto\\rho\(y\)\\exp\\\!\\left\(\\frac\{\\eta A\(y\)\+\\beta G\(y\)\\operatorname\{sgn\}\(A\(y\)\)\}\{\\tau\}\\right\),\(67\)with no surrogate objective or alternative gate\. These are qualitative diagnostics of the target, not additional language\-model training runs\. We organize them as a short narrative\. The first plot gives the basic geometric intuition for verifier\-grounded self\-guidance\. The second isolates the exact anti\-self\-confirmation effect in the simplest mixed\-outcome group\. The third maps the reliability boundary of self\-guidance\. The fourth returns to the two structural properties emphasized by the theory: conservative reference change and efficient use of all within\-group contrasts\.

#### C\.1\.1Mechanism View: Grounded Self\-Guidance across Multiple Modes

Figure[4](https://arxiv.org/html/2609.03241#A3.F4)embeds four exact response modes as one\-dimensional Gaussians: two failures and two successes\. The verifier cannot distinguish the two successful modes, while privileged hindsight assigns its largest gain to the more robust success\. It also assigns positive local likelihood to plausible failures, which is exactly the false\-positive regime addressed by Proposition[4](https://arxiv.org/html/2609.03241#Thmproposition4)\. We use

ρ=\(0\.30,0\.20,0\.30,0\.20\),A=\(−1,−1,\+1,\+1\),G=\(0\.45,0\.25,0\.30,0\.75\),\\rho=\(0\.30,0\.20,0\.30,0\.20\),\\quad A=\(\-1,\-1,\+1,\+1\),\\quad G=\(0\.45,0\.25,0\.30,0\.75\),\(68\)with\(η,β,τ\)=\(0\.75,0\.80,1\)\(\\eta,\\beta,\\tau\)=\(0\.75,0\.80,1\)\. Reward\-only shaping raises total success mass to0\.8180\.818but cannot prefer the robust successful mode\. Ungated self\-guidance identifies that mode, yet also reinforces positive guidance on failures, reaching success mass0\.8320\.832\. FlowBalance preserves the useful ranking among successes while reversing the failed\-response contribution, reaching success mass0\.9000\.900and robust\-success mass0\.4400\.440\. The Gaussian curves visualize exact categorical masses; they do not approximate the target\.

Figure 4:Mechanism view of verifier\-grounded self\-guidance\.FlowBalance uses\+G\+Gon positive\-advantage responses and−G\-Gon negative\-advantage responses\. It retains useful guidance among successful modes without converting positive local confidence on failed responses into self\-reinforcement\.
#### C\.1\.2Pairwise View: Exact Anti\-Self\-Confirmation in a Mixed Group

Figure[5](https://arxiv.org/html/2609.03241#A3.F5)fixes positive self\-guidanceG\+\>0G\_\{\+\}\>0on a successful response and sweeps positive false\-supportG−\>0G\_\{\-\}\>0on a plausible failure\. Proposition[4](https://arxiv.org/html/2609.03241#Thmproposition4)gives the exact log\-odds advantages

logit⁡pFlowBalance−logit⁡preward=β⁡\(G\+\+G−\)τ\>0,logit⁡pFlowBalance−logit⁡pungated=2​β​G−τ\>0\.\\operatorname\{logit\}p\_\{\\mathrm\{FlowBalance\}\}\-\\operatorname\{logit\}p\_\{\\mathrm\{reward\}\}=\\frac\{\\beta\(G\_\{\+\}\+G\_\{\-\}\)\}\{\\tau\}\>0,\\qquad\\operatorname\{logit\}p\_\{\\mathrm\{FlowBalance\}\}\-\\operatorname\{logit\}p\_\{\\mathrm\{ungated\}\}=\\frac\{2\\beta G\_\{\-\}\}\{\\tau\}\>0\.\(69\)Thus FlowBalance dominates both baselines throughout this useful\-but\-imperfect quadrant\. As false\-positive support grows, ungated shaping becomes less reward\-aligned, whereas outcome calibration becomes more selective\. AtG−=0\.5G\_\{\-\}=0\.5, the exact target success probabilities are0\.8940\.894for FlowBalance,0\.8170\.817for reward\-only shaping, and0\.8070\.807for ungated shaping\.

Figure 5:Exact anti\-self\-confirmation advantage\.The left panel shows target success probability as positive self\-guidance on a failed response increases\. The right panel plots the closed\-form probability\-ratio margins in Eq\. \([69](https://arxiv.org/html/2609.03241#A3.E69)\); no fitted decision boundary is used\.
#### C\.1\.3Boundary View: When Self\-Guidance Helps and When It Stops Helping

Outcome calibration does not make arbitrary guidance safe\. Figure[6](https://arxiv.org/html/2609.03241#A3.F6)makes this boundary explicit on a five\-mode response space\. We interpolate the guidance vector asGq=q​Guseful\+\(1−q\)​GadvG\_\{q\}=qG\_\{\\mathrm\{useful\}\}\+\(1\-q\)G\_\{\\mathrm\{adv\}\}, whereq=1q=1is useful but still false\-positive on failures andq=0q=0is adversarial to the verifier\. We then sweep bothqqandβG/τ\\beta\_\{G\}/\\tau\. Above a reliability threshold, FlowBalance improves verified\-success mass over reward\-only shaping while retaining several successful modes\. Below that threshold, stronger guidance can reduce verified success\. The reverse\-KL panel quantifies the accompanying movement from the reference\. This diagnostic therefore supports a bounded claim: sign gating corrects false\-positive confidence on failures, but it does not justify arbitrarily strong guidance that is systematically wrong on successful responses\.

![Refer to caption](https://arxiv.org/html/2609.03241v1/flowbalance_self_guidance_reliability.png)Figure 6:Reliability–strength map for self\-guidance\.Left: gain in verified\-success mass over reward\-only shaping\. Middle: conditional Simpson diversity among successful modes; the white contour marks the reward\-only level\. Right: reverse KL to the reference\. The reliability parameter is a synthetic interpolation, not an empirical calibration estimate\.
#### C\.1\.4Structural View: Conservative Change and All\-Contrast Efficiency

Figure[7](https://arxiv.org/html/2609.03241#A3.F7)verifies two distributional consequences of the FlowBalance formulation\. First, the exponential\-tilt path traces the minimum reverse\-KL frontier as expected sign\-gated energy increases\. A constructed full\-support fixed\-data target is matched to the exact FlowBalance energy, but requires reverse KL0\.9730\.973instead of0\.2730\.273, a3\.6×3\.6\\timeslarger displacement from the reference\. Second, a local Gaussian calculation shows that profiling the group partition intercept retains allN−1N\-1within\-group contrast directions\. At group sizeN=32N=32, its exact parameter risk is2\.92%2\.92\\%of a one\-contrast\-per\-group estimator, approximately34×34\\timeslower risk\. Together, these diagnostics explain why the self\-improvement update is both conservative in where it moves probability mass and statistically efficient in how it uses each rollout group\.

Figure 7:Conservative change and all\-contrast fitting\.Left: FlowBalance is the minimum reverse\-KL target at its attained composite\-energy level\. Right: profiling one nuisance intercept per rollout group preserves allN−1N\-1contrast directions and yields the exact local Gaussian risk reduction\.

## Appendix DLLM Experiment Details

### D\.1Reasoning Self\-Improvement Setup and Protocol

##### Scope and fairness controls\.

The large\-scale experiments are designed to compare policy\-update objectives rather than infrastructure choices\. Within each backbone, all methods use the same training prompts, rollout group size, maximum response length, verifier, optimizer schedule, checkpoint cadence, and evaluation script\. For methods that use privileged training information \(OPSD, RLSD, and FlowBalance\), that information is available only to a frozen scoring path during training\. Rollout generation and evaluation never include the solution or feedback fieldcc\.

Table 5:Shared reporting fields for the reasoning experiments\. Entries marked “matched” are held identical across methods within a backbone\.FieldSettingBackbonesQwen3\-4B, Qwen3\-8BTraining supervisionfinal\-answer verifier; OPSD, RLSD, and FlowBalance also use privileged training solution/feedbackCompared methodsGRPO, OPSD, RLSD, FlowRL, FlowBalanceRollout group sizematched across methods; pending releaseTraining horizonstep\-180 final table; up to 400 steps for dynamics and response\-length curvesSeedsfive seeds for every main\-table entryMaximum response lengthmatched across methods; pending releaseReference policyfixed copy of the initial checkpointTraining\-time scorerOPSD: fixed privileged teacher; RLSD/FlowBalance: frozen current\-policy copy conditioned onccFlowBalance default coefficientsηA=15\\eta\_\{A\}=15,βG=1\\beta\_\{G\}=1for the completed default runEvaluation metricsAIME24 Pass@16; HMMT25/Minerva/MATH500/OlympiadBench Pass@1Aggregate scoreunweighted mean of the five benchmark means
##### Training signal construction\.

For every prompt in a minibatch, the frozen rollout snapshot samples a group of responses without seeingcc\. A rule\-based verifier assigns a terminal correctness reward, from which we compute the stopped group\-relative advantage in Eq\. \([5](https://arxiv.org/html/2609.03241#S2.E5)\)\. FlowBalance then reuses the same frozen snapshot as the privileged\-hindsight view by conditioning it onccand scoring the already sampled tokens\. The self\-guidance contribution is the clipped, length\-averaged log\-probability gain in Eq\. \([10](https://arxiv.org/html/2609.03241#S3.E10)\); no response is resampled from the guidance view and no guidance gradient is taken\. The sign\-gated trajectory energy is optimized through the trajectory\-balance residual with the rollout\-group partition estimate in Eq\. \([22](https://arxiv.org/html/2609.03241#S3.E22)\)\.

##### Baseline implementations\.

GRPO uses the same verifier rewards and group\-relative normalization as FlowBalance but optimizes the standard reward\-policy objective rather than a normalized trajectory target\. OPSD applies clipped forward\-KL self\-distillation from a fixed privileged teacher to the current policy’s on\-policy trajectories\[[Zhao et al\., 2026a](https://arxiv.org/html/2609.03241#bib.bib1)\]\. RLSD combines verifier and teacher signals in its policy\-optimization objective\. FlowRL follows the original outcome\-only trajectory\-balance algorithm\[[Zhu et al\., 2025](https://arxiv.org/html/2609.03241#bib.bib11)\]\. All baselines use the same sampled responses, answer verifier, and response\-length cap whenever their objectives permit this sharing\.

##### Evaluation protocol\.

For AIME24, each problem is decoded with 16 samples and is counted correct if any sample matches the final answer\. For HMMT25, Minerva, MATH500, and OlympiadBench, we report Pass@1\. The evaluator applies the same benchmark\-specific final\-answer extraction and normalization to every method\. All main\-table entries report the mean and sample standard deviation over five seeds at step 180\. The accuracy\-dynamics plots use matched evaluation checkpoints for FlowBalance and GRPO; the response\-length panel compares the logged training trajectories of FlowBalance and OPSD\. Solid curves denote smoothed trends and lighter curves denote the corresponding per\-step measurements\.

### D\.2Grounding and Guidance Ablations

##### Coefficient sweeps\.

The ablations in Tables[3](https://arxiv.org/html/2609.03241#S5.T3)–[3](https://arxiv.org/html/2609.03241#S5.T3)are one\-dimensional sweeps around the default setting\. In theηA∈\{5,10,15\}\\eta\_\{A\}\\in\\\{5,10,15\\\}sweep,βG\\beta\_\{G\}and all other training settings are held fixed\. In theβG∈\{1,2,3\}\\beta\_\{G\}\\in\\\{1,2,3\\\}sweep,ηA\\eta\_\{A\}and all other settings are held fixed\. Each sweep point is evaluated with the same five\-benchmark average used in Table[1](https://arxiv.org/html/2609.03241#S5.T1); all reported sweep points contain completed five\-seed runs\. Tables[6](https://arxiv.org/html/2609.03241#A4.T6)–[7](https://arxiv.org/html/2609.03241#A4.T7)provide the corresponding per\-benchmark breakdown\.

##### Coefficient ablations\.

Tables[6](https://arxiv.org/html/2609.03241#A4.T6)–[7](https://arxiv.org/html/2609.03241#A4.T7)expand the coefficient ablations in Section[5\.2](https://arxiv.org/html/2609.03241#S5.SS2.SSS0.Px3)to the same five benchmark columns used in the main result table\. All rows use the Qwen3\-8B backbone and report step\-180 evaluation\. AIME24 is evaluated with Pass@16, while HMMT25, Minerva, MATH500, and OlympiadBench are evaluated with Pass@1\. The average is the unweighted mean of the five benchmark means\.

Table 6:Full ablation over verifier coefficientηA\\eta\_\{A\}on Qwen3\-8B at step 180\. The self\-guidance coefficient is held fixed at the default valueβG=1\\beta\_\{G\}=1\. Entries are percentages reported as mean±\\pmsample standard deviation over five seeds\. “Avg\.” averages the five benchmark means\.ηA\\eta\_\{A\}AIME24@16HMMT25@1Minerva@1MATH500@1Olympiad@1Avg\.586\.00±1\.4986\.00\\pm 1\.4930\.00±8\.5030\.00\\pm 8\.5053\.46±1\.2453\.46\\pm 1\.2493\.32±0\.4193\.32\\pm 0\.4165\.46±0\.6965\.46\\pm 0\.6965\.6565\.651085\.33±1\.8385\.33\\pm 1\.8330\.00±3\.3330\.00\\pm 3\.3352\.43±0\.8152\.43\\pm 0\.8193\.20±0\.3293\.20\\pm 0\.3266\.11±0\.5166\.11\\pm 0\.5165\.4165\.411589\.33±1\.4989\.33\\pm 1\.4934\.67±9\.8934\.67\\pm 9\.8953\.68±0\.7853\.68\\pm 0\.7893\.52±0\.3093\.52\\pm 0\.3066\.85±0\.4666\.85\\pm 0\.4667\.6167\.61Table 7:Full ablation over self\-guidance coefficientβG\\beta\_\{G\}on Qwen3\-8B at step 180\. The verifier coefficient is held fixed at the default valueηA=15\\eta\_\{A\}=15\. Entries are percentages reported as mean±\\pmsample standard deviation over five seeds\. “Avg\.” averages the five benchmark means\.βG\\beta\_\{G\}AIME24@16HMMT25@1Minerva@1MATH500@1Olympiad@1Avg\.189\.33±1\.4989\.33\\pm 1\.4934\.67±9\.8934\.67\\pm 9\.8953\.68±0\.7853\.68\\pm 0\.7893\.52±0\.3093\.52\\pm 0\.3066\.85±0\.4666\.85\\pm 0\.4667\.6167\.61287\.33±1\.4987\.33\\pm 1\.4932\.00±5\.0632\.00\\pm 5\.0653\.68±1\.4053\.68\\pm 1\.4093\.64±0\.3393\.64\\pm 0\.3365\.73±1\.1665\.73\\pm 1\.1666\.4866\.48386\.00±1\.4986\.00\\pm 1\.4930\.00±4\.0830\.00\\pm 4\.0853\.46±1\.3953\.46\\pm 1\.3993\.56±0\.5593\.56\\pm 0\.5566\.74±0\.6666\.74\\pm 0\.6665\.9565\.95

### D\.3Semantic Diversity Diagnostic

The diversity study in Section[5\.3](https://arxiv.org/html/2609.03241#S5.SS3)evaluates semantic strategy variation on AIME24 using the step\-180 checkpoints of GRPO, RLSD, and FlowBalance\. We use seed00only, decode1616complete responses for each of the3030problems and each method, and submit all30×3×16=144030\\times 3\\times 16=1440full trajectories to a GPT\-5\.5 judge\. No response is truncated or heuristically compressed before judging\. These results are therefore a controlled diagnostic of strategy variation at one checkpoint and one seed, not a multi\-seed stability estimate\.

##### Two\-stage LLM\-judge protocol\.

The judge first performs strategy extraction independently for each full trajectory\. The extracted fields include the primary solution strategy, central mathematical representation, main tools or theorems, a short reasoning outline, a compact strategy signature, and, for incorrect responses, the attempted failure mode\. The instruction is to describe the strategy actually used by the trajectory, not to repair incorrect reasoning\. Differences in variable names, wording, notation, response length, arithmetic detail, or the order of equivalent calculations are not counted as different strategies\.

In the second stage, for each problem and method, the1616extracted strategy summaries are anonymized and randomly ordered\. The judge assigns every trajectory to exactly one semantic strategy cluster\. During clustering, the judge does not see the source algorithm, checkpoint name, correctness label, or original response order\. Correctness labels are applied only after clustering, and the reported diversity metric restricts the resulting cluster assignments to correct trajectories\.

##### Metric\.

For a fixed problem and method, restrict the judged cluster assignments to correct trajectories and letp1,…,pKp\_\{1\},\\ldots,p\_\{K\}denote the proportions of those trajectories in the resulting semantic strategy clusters\. We report correct\-only Simpson strategy diversity,

DSimpson=1−∑k=1Kpk2,D\_\{\\mathrm\{Simpson\}\}=1\-\\sum\_\{k=1\}^\{K\}p\_\{k\}^\{2\},\(70\)which is the probability that two randomly sampled correct trajectories use different semantic solution strategies\. A problem is included in the aggregate only when it has at least two correct trajectories\.

##### Scope\.

LLM\-based clustering is inherently partly subjective, even with anonymization and a two\-stage protocol\. The main\-text conclusion is therefore restricted to this AIME24 seed\-00diagnostic: FlowBalance exhibits higher judged semantic strategy diversity among correct trajectories at this checkpoint\.

### D\.4Prompt for Semantic Strategy Clustering

The LLM\-judged diagnostic uses two prompts: one extracts a strategy summary from each trajectory, and the other clusters anonymized summaries within the same problem and method\.

Prompt 1: Per\-trajectory strategy extractionGiven the problem statement and one complete reasoning trajectory, describe the mathematical strategy actually attempted by the trajectory\. Do not repair incorrect reasoning, replace the attempted method with a cleaner solution, or infer a method not present in the text\.Extract: \(i\) the primary solution strategy, \(ii\) the central mathematical representation, \(iii\) the main tools or theorems, \(iv\) a short reasoning outline, \(v\) a compact strategy signature, and \(vi\) for incorrect responses, the attempted failure mode\.Treat changes in notation, variable names, wording, response length, arithmetic detail, or the order of equivalent calculations as the same strategy\. Treat substantively different mathematical objects or tools—for example, rectangular\-box embedding versus Cayley–Menger determinant, coordinate versus synthetic geometry, generating functions versus direct counting, or roots\-of\-unity methods versus real polynomial methods—as different strategies\.

Prompt 2: Within\-problem strategy clusteringGiven 16 anonymized strategy summaries for the same problem and method, assign every trajectory to exactly one semantic strategy cluster\. The summaries are randomly ordered and do not reveal the source algorithm, checkpoint name, correctness label, or original response order\.Cluster by the core mathematical representation and main tools, not by surface wording, formatting, or response length\. Incorrect but coherent attempts should still be clustered by their attempted strategy; correctness labels are applied only after clustering when computing the correct\-only diversity metric\.Return the cluster assignment for each anonymous trajectory, a short cluster name, and a brief rationale for each cluster\.

### D\.5Why the FlowBalance Traces Are Interesting

The main text shows AIME24 Problem 23 as a compact illustration of the semantic distinctions used by the LLM\-judged diversity diagnostic\. Here we provide three additional FlowBalance cases from the same seed\-00, step\-180180,1616\-sample AIME24 run\. Each case contains two representative correct trajectories whose final answers agree but whose central mathematical objects differ; the boxes mark the strategy\-defining reasoning actions, following the format of Table[4](https://arxiv.org/html/2609.03241#S5.T4)\.

Together with the AIME24 Problem 23 example in the main text, these cases show that FlowBalance’s additional correct trajectories are not merely longer, more verbose, or differently worded variants of a standard template\. We regard a trace as*interesting*when the boxed steps change the mathematical object or criterion that drives the solution: for example, replacing a local algebraic test with a global envelope argument, replacing a three\-dimensional geometry problem with a two\-dimensional cross\-section reduction, or replacing direct elimination with a parameterized conic argument\. In other words, the diversity signal is intended to capture a semantic change in*how*the problem is solved, not a superficial change in phrasing\.

On Problem 5, FlowBalance finds both a global envelope view of the unit\-intercept line family and a local multiplicity view\. The first identifies the special point as the tangency point on the astroidx2/3\+y2/3=1x^\{2/3\}\+y^\{2/3\}=1, while the second detects uniqueness by forcing the known lineA​BABto be a double root of a one\-variable trigonometric equation\. On Problem 15, one FlowBalance trajectory reduces the three\-dimensional torus\-sphere contact to two circle tangencies in a meridian plane\. Another keeps the problem as an implicit surface calculation and derives the contact radii from collinear normals in cylindrical coordinates\. On Problem 27, FlowBalance again covers two different representations of the same constraint: direct coordinate elimination in squared variables, and a secant\-tangent hyperbola parameterization that turns the rhombus condition into the trigonometric productsinθsinϕ=−5/6\\sin\\theta\\sin\\phi=\-5/6\. These examples mirror the main\-text box\-embedding case and help explain why the FlowBalance traces are meaningful: the model is spreading probability mass across genuinely different correct derivations, not just stylistic rewrites of one template\.

##### How to read the traces\.

The boxed lines mark only the reasoning actions that define the route; standard algebra, routine simplifications, and repeated bookkeeping are suppressed with “⋯\\cdots”\. The goal is not to claim that every token sequence differs globally, but to show that the high\-level derivational backbone differs in a way that would matter to a human reader choosing between strategies\.

Table 8:Additional case study on AIME24 Problem 5\.Two correct FlowBalance trajectories solve the same unit\-intercept\-segment problem using different criteria for uniqueness\.Content \(boxed = reasoning actions; “⋯\\cdots” = omitted\)QuestionLetO=\(0,0\)O=\(0,0\),A=\(1/2,0\)A=\(1/2,0\), andB=\(0,3/2\)B=\(0,\\sqrt\{3\}/2\)\. Among all unit\-length segments with one endpoint on the positivexx\-axis and one endpoint on the positiveyy\-axis, there is a unique interior pointCCofA​BABthat lies on no other such segment\. IfO​C2=p/qOC^\{2\}=p/qin lowest terms, findp\+qp\+q\.Route 1Envelope / astroid route\.“⋯\\cdots”F⁡\(x,y,u\)=x/u\+y/1−u2−1F\(x,y,u\)=x/u\+y/\\sqrt\{1\-u^\{2\}\}\-1for the unit\-intercept line family\.SolveF=0F=0andFu=0F\_\{u\}=0to obtain the envelope\.x=u3x=u^\{3\},y=\(1−u2\)3/2y=\(1\-u^\{2\}\)^\{3/2\}, the first\-quadrant astroid\.A​BABcorresponds tou=1/2u=1/2, soC=\(1/8,3​3/8\)C=\(1/8,3\\sqrt\{3\}/8\)\.O​C2=7/16OC^\{2\}=7/16, hencep\+q=23p\+q=23\.Route 2Multiple\-root uniqueness route\.“⋯\\cdots”Write an interior point asC⁡\(t\)=\(\(1−t\)/2,3​t/2\)C\(t\)=\(\(1\-t\)/2,\\sqrt\{3\}t/2\)\.A unit\-intercept line throughC⁡\(t\)C\(t\)satisfies\(1−t\)/\(2​cos⁡θ\)\+3​t/\(2​sin⁡θ\)=1\(1\-t\)/\(2\\cos\\theta\)\+\\sqrt\{3\}t/\(2\\sin\\theta\)=1\.The segmentA​BABis the known solutionθ=π/3\\theta=\\pi/3\.Uniqueness at the boundary forces this solution to be a double root\.f′​\(π/3\)=3​\(1−t\)−3​t/3=0f^\{\\prime\}\(\\pi/3\)=\\sqrt\{3\}\(1\-t\)\-\\sqrt\{3\}t/3=0, sot=3/4t=3/4\.C=\(1/8,3​3/8\)C=\(1/8,3\\sqrt\{3\}/8\)andp\+q=23p\+q=23\.Table 9:Additional case study on AIME24 Problem 15\.Two correct FlowBalance trajectories use either a two\-dimensional meridian section or a three\-dimensional implicit\-normal condition\.Content \(boxed = reasoning actions; “⋯\\cdots” = omitted\)QuestionA torus is formed by rotating a circle of radius33about an axis in its plane at distance66from the circle’s center\. A sphere of radius1111is tangent to the torus in two rotationally symmetric positions, with corresponding contact\-circle radiirir\_\{i\}andror\_\{o\}\. Ifri−ro=m/nr\_\{i\}\-r\_\{o\}=m/nin lowest terms, findm\+nm\+n\.Route 1Meridian\-section circle\-tangency route\.“⋯\\cdots”Take a plane through the rotation axis; the torus cross\-section is a circle of radius33centered at\(6,0\)\(6,0\)\.The sphere cross\-section is a circle of radius1111centered at\(0,h\)\(0,h\)\.The two tangencies have center distance11−3=811\-3=8or11\+3=1411\+3=14\.The contact\-circle radius is the horizontal coordinate of the tangency point\.ri=11⋅6/8=33/4r\_\{i\}=11\\cdot 6/8=33/4andro=11⋅6/14=33/7r\_\{o\}=11\\cdot 6/14=33/7\.ri−ro=99/28r\_\{i\}\-r\_\{o\}=99/28, som\+n=127m\+n=127\.Route 2Implicit\-surface normal route\.“⋯\\cdots”Use cylindrical coordinates\(ρ,z\)\(\\rho,z\)\.Torus:\(ρ−6\)2\+z2=9\(\\rho\-6\)^\{2\}\+z^\{2\}=9; sphere:ρ2\+\(z−h\)2=121\\rho^\{2\}\+\(z\-h\)^\{2\}=121\.Tangency means normals are collinear:\(ρ−6,z\)=λ⁡\(ρ,z−h\)\(\\rho\-6,z\)=\\lambda\(\\rho,z\-h\)\.Eliminatez,hz,hto getρ/\|ρ−6\|=11/3\\rho/\|\\rho\-6\|=11/3\.ρ\>6\\rho\>6givesρ=33/4\\rho=33/4;ρ<6\\rho<6givesρ=33/7\\rho=33/7\.m\+n=127m\+n=127\.Table 10:Additional case study on AIME24 Problem 27\.Two correct FlowBalance trajectories optimize the same diagonal length using algebraic elimination or a trigonometric hyperbola parameterization\.Content \(boxed = reasoning actions; “⋯\\cdots” = omitted\)QuestionPointsA,B,C,DA,B,C,Dlie on the hyperbolax2/20−y2/24=1x^\{2\}/20\-y^\{2\}/24=1and form a rhombus whose diagonals intersect at the origin\. Find the greatest real number that is strictly less thanB​D2BD^\{2\}for every such rhombus\.Route 1Coordinate\-elimination route\.“⋯\\cdots”Write opposite vertices as\(a,b\),\(−a,−b\)\(a,b\),\(\-a,\-b\)and\(c,d\),\(−c,−d\)\(c,d\),\(\-c,\-d\)\.A centered parallelogram is a rhombus iff its diagonals are perpendicular, soa​c\+b​d=0ac\+bd=0\.Setx=a2x=a^\{2\}and useb2=6​x/5−24b^\{2\}=6x/5\-24\.From perpendicularity,c2=\(b2/a2\)​d2c^\{2\}=\(b^\{2\}/a^\{2\}\)d^\{2\}\.The hyperbola equation forBBgivesd2=600​x/\(11​x−720\)d^\{2\}=600x/\(11x\-720\)\.B​D2=4​\(c2\+d2\)=480\+288000/\(11​x−720\)\>480BD^\{2\}=4\(c^\{2\}\+d^\{2\}\)=480\+288000/\(11x\-720\)\>480, with limit480480from above\.Route 2sec\\sec–tan\\tanparameterization route\.“⋯\\cdots”Parameterize the hyperbola as\(2​5​sec⁡θ,2​6​tan⁡θ\)\(2\\sqrt\{5\}\\sec\\theta,2\\sqrt\{6\}\\tan\\theta\)\.TakeAAwith parameterθ\\thetaandBBwith parameterϕ\\phi\.Perpendicularity gives20​sec⁡θ​sec⁡ϕ\+24​tan⁡θ​tan⁡ϕ=020\\sec\\theta\\sec\\phi\+24\\tan\\theta\\tan\\phi=0\.Equivalently,sinθsinϕ=−5/6\\sin\\theta\\sin\\phi=\-5/6\.Since\|sin⁡θ\|<1\|\\sin\\theta\|<1, we have\|sin⁡ϕ\|\>5/6\|\\sin\\phi\|\>5/6, hencetan2⁡ϕ\>25/11\\tan^\{2\}\\phi\>25/11\.B​D2=80\+176​tan2⁡ϕ\>480BD^\{2\}=80\+176\\tan^\{2\}\\phi\>480, with limit480480attainable only at infinity\.