正则化得分中心化
摘要
该论文提出了正则化分数中心化(Regularized Score Centering,RSC),通过在分数中心化中加入反向 KL 惩罚项,限制异步/离策略强化学习中的策略失配,在包括 AppWorld 和 BabyAI 在内的六个智能体基准测试中,显著提升了样本效率和最佳性能。
查看缓存全文
缓存时间: 2026/11/19 03:34
# Regularized Score Centering Source: [https://arxiv.org/html/2611.21791](https://arxiv.org/html/2611.21791) ###### Abstract Asynchronous reinforcement learning increases training throughput, but data from stale policies and numerical mismatch between training and inference can destabilize learning\. To stabilize off\-policy learning, score centering \(SC\) removes the effect of reward baselines on the expected update, even without full importance sampling \(IS\)\. Our empirical study shows that full IS produces extreme gradients and that SC significantly reduces their magnitude\. However, SC can still fail when the learner moves far from the behavior policy that generated the data\. To limit this policy mismatch, we introduce Regularized Score Centering \(RSC\), which adds a reverse Kullback–Leibler \(KL\) penalty to encourage the learner’s next\-token distribution to stay close to the behavior policy’s next\-token distribution\. RSC works with or without truncated importance sampling \(TIS\), which caps token importance weights\. Our theoretical analysis supports this approach, showing that KL regularization improves convergence guarantees\. We evaluate dense and mixture\-of\-experts models on mathematical reasoning and agentic tasks, including online shopping, tool use, games, and fully asynchronous software engineering\. RSC with TIS substantially improves sample efficiency across all agentic benchmarks, attaining higher training rewards with the same number of sampled responses\. In final evaluations, it achieves top performance across all six agentic tasks, improving by14\.3%14\.3\\%on AppWorld and reaching9\.9×9\.9\\timesthe strongest baseline’s score on BabyAI\. 11footnotetext:Equal contribution\. Correspondence: Hao Bai \([haob2@illinois\.edu](mailto:[email protected])\)\.Figure 1:Regularization strengthens score centering\.\(a\) RSC adds reverse KL to SC or SC\+TIS, comparing distributions over next actions under the learnerπθ\\pi\_\{\\theta\}and behavior policyμ\\muat the same history\. \(b–d\) Training curves show mean±\\pmsample standard deviation across three seeds \(settings in Appendix[A\.2](https://arxiv.org/html/2611.21791#A1.SS2)\)\. Stronger methods are colored, and others are gray\.## 1Introduction Efficient reinforcement learning often uses data generated by a policy that differs from the current learner\. Asynchronous reinforcement learning \(async RL\) creates this difference by running generation and training concurrently: while a response is being generated, the learner can move on to a newer policy\. Reusing responses and refreshing the sampler less frequently further separate the learnerπθ\\pi\_\{\\theta\}from the behavior policyμ\\muthat generated the data\([Espeholt et al\., 2018](https://arxiv.org/html/2611.21791#bib.bib4);[Hilton et al\., 2022](https://arxiv.org/html/2611.21791#bib.bib7)\)\. A difference can also arise without any change in model weights\. When training and inference use different numerical precision or execution kernels, they can assign different probabilities to the same action, creating a mismatch between training and inference \(TIM\)\([Yao et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib20)\)\. In both cases, training is off\-policy: the data\-generating distribution differs from the current learner\. This matters because an update computed from behavior data may differ from the update the learner would obtain from its own responses\. In agentic tasks, the difference can be larger when actions lead to different future states and rewards\. Section[5\.3](https://arxiv.org/html/2611.21791#S5.SS3)analyzes how task structure and policy staleness contribute to this effect\. To account for this difference, importance sampling \(IS\) gives behavior samples more or less weight according to their probability under the learner\. Full sequence\-level IS uses the likelihood ratio of the complete response\. Whenμ\\muassigns positive probability to every response thatπθ\\pi\_\{\\theta\}can generate, weighting behavior samples by this ratio gives an unbiased estimate of the learner’s reward gradient, but multiplying token ratios can produce extremely high variance\([Owen, 2013](https://arxiv.org/html/2611.21791#bib.bib18)\)\. Practical sequence\-level methods such as GSPO stabilize the weights through length normalization and clipping\([Zheng et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib5)\)\. Another way to avoid multiplying ratios across the response is to use token\-level IS, which corrects only the distribution over the next action at a history sampled under the behavior policy\. Large token ratios can still cause high gradient variance\([Bai et al\., 2026b](https://arxiv.org/html/2611.21791#bib.bib9)\)\. To limit these extreme gradient contributions, truncated importance sampling \(TIS\) caps token weights at the cost of additional clipping bias\([Ionides, 2008](https://arxiv.org/html/2611.21791#bib.bib21);[MiniMax, 2025](https://arxiv.org/html/2611.21791#bib.bib6)\)\. Yet omitting or truncating the weights also leaves a specific problem: parts of the reward that do not distinguish between actions can still drive the update\. To see why, consider the learner’s score \(gradient of its log\-probability\)\. Its mean is zero under the learner’s own action distribution, so a reward baseline independent of the action makes no contribution to the gradient when sampling from the learner\. By contrast, under the behavior policy, that mean need not be zero, allowing the same baseline to contribute to an update without full importance weighting\. Score centering \(SC\) removes this contribution by subtracting the mean of the score under the behavior policy, or the mean of the weighted score under the behavior policy when combined with TIS\([Marek and Ryabinin, 2026](https://arxiv.org/html/2611.21791#bib.bib1)\)\. This mean is taken over actions at the same history, and exact centering therefore cancels the unwanted contribution in expectation\. In practice, Monte Carlo estimates approximate this conditional mean using auxiliary actions drawn at that history\. However, removing the baseline contribution does not remove every difference between behavior data and learner responses, because the remaining token advantages still depend on continuations generated byμ\\mu\. Even with a single terminal reward, a token can have different value when the learner completes the response differently\. Although TIS limits the weight of the current token, it does not correct the distribution of the preceding history or the subsequent continuation\. As the learner changes, improving the surrogate on behavior trajectories can therefore fail to improve the reward obtained from the learner’s own trajectories\. This dependence on behavior continuations motivates limiting how far the learner moves from the behavior policy\. We introduce Regularized Score Centering \(RSC\), a family of methods that add reverse KL regularization to SC:RSC = SC \+ KL \(\+TIS\)\. The family regularizes both SC without IS and SC\+TIS\([Marek and Ryabinin, 2026](https://arxiv.org/html/2611.21791#bib.bib1)\)\. At each history, the KL penalty compares the learner’s next\-token distribution with the behavior policy’s next\-token distribution and encourages them to stay close\. Because the behavior policy serves as the KL reference, refreshing it also updates the reference used by the penalty\. RSC thus preserves SC’s baseline correction while limiting movement away from the policy used to estimate token advantages\. The same token\-level regularizer applies to both choices of reward weights and requires no additional environment interactions\. To explain how regularization can help, our analysis connects accurate fitting of the regularized surrogate to improvement in expected reward\. The two RSC variants use different proof arguments, but both turn accurate fitting into reward progress\. Repeated fitting gives a final policy whose reward from a single response nearly matches the reward obtained by best\-of\-NNsampling from the initial policy\. Section[4\.2](https://arxiv.org/html/2611.21791#S4.SS2)explains these results, with proofs and sample bounds in Appendix[C](https://arxiv.org/html/2611.21791#A3)\. We then test whether this additional control helps when behavior data become less representative of the learner\. Both agentic interaction and sampler staleness can amplify the consequences of policy mismatch\. In agentic tasks, choosing a different action can change later observations, the actions available afterward, and the eventual reward, so a token’s advantage under behavior continuations may poorly represent its advantage under learner continuations\. Staleness introduces the same difficulty by allowing more learner updates while the behavior policy remains fixed, changing both action probabilities and the continuations they induce\. Section[5\.3](https://arxiv.org/html/2611.21791#S5.SS3)examines these two effects through task structure and sampler refresh intervals\. In these comparisons, RSC achieves substantially higher rewards than unregularized SC in agentic environments, while increasing the refresh interval weakens SC on Countdown and math tasks with mixture\-of\-experts \(MoE\) models\. On Countdown, these failures persist even with Monte Carlo centering\. Beyond these controlled comparisons, we evaluate RSC on mathematical reasoning with dense and MoE models, and on agentic tasks spanning online shopping, tool use, games, and fully asynchronous software engineering\. RSC with TIS attains substantially higher training rewards on Crafter and Craftax for the same number of sampled responses \(Figure[7](https://arxiv.org/html/2611.21791#S6.F7)\)\. In final evaluations, RSC with TIS achieves top performance across all six agentic tasks against baselines including IS\-PPOClip, which combines importance sampling with PPO\-style clipping\. It raises AppWorld’s mean evaluation score by14\.3%14\.3\\%over SC and SC\+TIS, and reaches9\.9×9\.9\\timesthe score of the strongest BabyAI baseline, IS\-PPOClip\. ## 2Related Work Score centering and off\-policy learning\.RL for language models creates a mismatch between the learner and behavior policy through asynchronous generation, response reuse and different numerical implementations for training and inference\([Fu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib11);[Yao et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib20)\)\. Engineering solutions align training and inference: kernels with batch invariance remove dependence on batch shape from normalization, matrix multiplication and attention\([He and Thinking Machines Lab, 2025](https://arxiv.org/html/2611.21791#bib.bib39)\), while consistent FP16 computation reduces rounding discrepancies\([Qi et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib40)\)\. Alongside these engineering solutions, algorithmic methods correct updates computed from stale data\. For example, IMPALA uses importance correction in asynchronous learning\([Espeholt et al\., 2018](https://arxiv.org/html/2611.21791#bib.bib4)\), and[Hilton et al\. \(2022\)](https://arxiv.org/html/2611.21791#bib.bib7)separate the sampling policy from the proximal policy used to control updates\. AReaL and AsyncWebRL adopt this separation, combining importance weights for data from earlier policies with PPO\-style clipping around a more recent policy\([Schulman et al\., 2017](https://arxiv.org/html/2611.21791#bib.bib3);[Fu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib11);[Bai et al\., 2026a](https://arxiv.org/html/2611.21791#bib.bib10)\)\. We include this combination as the IS\-PPOClip baseline\. TIS caps token importance weights\([Ionides, 2008](https://arxiv.org/html/2611.21791#bib.bib21);[Yao et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib20);[MiniMax, 2025](https://arxiv.org/html/2611.21791#bib.bib6)\), IcePop masks token corrections outside an interval\([Ling Team and others, 2025](https://arxiv.org/html/2611.21791#bib.bib41)\), Sequence MIS\([Liu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib42)\)combines sequence\-level importance weights with a sequence\-level mask, and GSPO clips a length\-normalized sequence ratio\([Zheng et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib5)\)\. Adaptive Layerwise Perturbation \(ALP\) introduces learnable perturbations into each layer’s input hidden states and uses the perturbed learner in the numerator of the importance ratio to address policy staleness and mismatch between training and inference\([Ye et al\., 2026](https://arxiv.org/html/2611.21791#bib.bib45)\)\. SC adds a different operation: subtracting the expectation of the learner score under the behavior policy, including the expectation of the weighted score when composed with TIS or MIS\([Marek and Ryabinin, 2026](https://arxiv.org/html/2611.21791#bib.bib1)\)\. Exact centering removes the expected contribution of a reward baseline independent of the action\. RSC preserves this correction and penalizes movement away from the behavior policy\. Policy optimization with KL regularization\.Alongside corrections to the sampled update, KL regularization limits policy movement relative to a reference distribution\. Relative Entropy Policy Search and TRPO constrain distributional change\([Peters et al\., 2010](https://arxiv.org/html/2611.21791#bib.bib30);[Schulman et al\., 2015](https://arxiv.org/html/2611.21791#bib.bib31)\), with regularized MDP theory linking such updates to Bellman operators and mirror descent\([Geist et al\., 2019](https://arxiv.org/html/2611.21791#bib.bib32)\)\. In language model alignment, RLHF penalizes deviation from a supervised reference\([Stiennon et al\., 2020](https://arxiv.org/html/2611.21791#bib.bib38);[Ouyang et al\., 2022](https://arxiv.org/html/2611.21791#bib.bib33)\), and DPO uses the closed\-form optimizer of a KL\-regularized objective to learn from preference pairs\([Rafailov et al\., 2023](https://arxiv.org/html/2611.21791#bib.bib34)\)\. Theory for preference learning provides guarantees for finite samples in offline, online and hybrid settings\([Xiong et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib35)\), including general preference oracles\([Ye et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib37)\)\. Divergence choice and data coverage also shape learning:χ2\\chi^\{2\}preference optimization addresses overoptimization of reward models\([Huang et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib36)\), and the coverage principle relates pretraining coverage to posttraining success\([Chen et al\., 2026](https://arxiv.org/html/2611.21791#bib.bib8)\)\.[Huang et al\. \(2026\)](https://arxiv.org/html/2611.21791#bib.bib44)study coverage collapse and establish convergence through KL regularization and resets of the reference policy, treating complete responses as actions and using a sequence\-level importance ratio\. At the token level, Future\-KL propagates subsequent tokens’ regularization costs into the update for the current token\([Yao et al\., 2026](https://arxiv.org/html/2611.21791#bib.bib2)\), while Proximal KL combines credit assignment for later tokens with token\-level importance sampling and a refreshed proximal reference policy\([Bai et al\., 2026b](https://arxiv.org/html/2611.21791#bib.bib9)\)\. RSC applies token\-level reverse KL directly to the score centering objective, using the behavior policy as its reference\. The same penalty applies to SC without IS and SC\+TIS, preserving their correction for the reward baseline’s contribution\. ## 3Methodology ### 3\.1Background: Score Centering We optimize the learnerπθ\\pi\_\{\\theta\}using responses generated by the behavior policyμ\\mu\. The gradient∇θ\\nabla\_\{\\theta\}differentiates throughπθ\\pi\_\{\\theta\}while holdingμ\\mufixed\. We draw a promptx∼𝒟x\\sim\\mathcal\{D\}, where𝒟\\mathcal\{D\}is the prompt distribution, and represent its response asY=\(a1,…,aH\)Y=\(a\_\{1\},\\ldots,a\_\{H\}\), with corresponding historyht=\(x,a<t\)h\_\{t\}=\(x,a\_\{<t\}\)\. The learner maximizes its expected reward,J\(πθ\)=𝔼x∼𝒟,Y∼πθ\(⋅∣x\)R\(x,Y\)J\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,Y\\sim\\pi\_\{\\theta\}\(\\cdot\\mid x\)\}R\(x,Y\)\. Hereμ\\mudenotes the policy that generated the training data\. With periodic sampler refreshes, it remains fixed between refreshes\. To express token\-level updates, we define the learner scoreψθ\\psi\_\{\\theta\}and token importance ratioρ\\rhoby ψθ\(h,a\):=∇θlogπθ\(a∣h\),ρ\(h,a\):=πθ\(a∣h\)μ\(a∣h\)\.\\psi\_\{\\theta\}\(h,a\):=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\\mid h\),\\qquad\\rho\(h,a\):=\\frac\{\\pi\_\{\\theta\}\(a\\mid h\)\}\{\\mu\(a\\mid h\)\}\.\(1\)The indexiilabels responses andttlabels token positions, soψi,t\\psi\_\{i,t\}andρi,t\\rho\_\{i,t\}are the score and ratio at historyhi,th\_\{i,t\}\. LetAiA\_\{i\}be the detached group\-centered reward advantage of responseii, computed as specified in Appendix[A](https://arxiv.org/html/2611.21791#A1), and writeg^i,t\\widehat\{g\}\_\{i,t\}for the token’s stochastic contribution to the ascent gradient\. Score centering removes the baseline contribution\.The learner score has mean zero under the learner’s own action distribution, but need not have mean zero under behaviorμ\\mu\. A reward baseline can therefore contribute to the expected update computed from behavior data\. At a fixed history, letQtμQ\_\{t\}^\{\\mu\}be the expected reward after choosing an action and followingμ\\mu, letVtμV\_\{t\}^\{\\mu\}average this value over actions fromμ\\mu, and letAtμ=Qtμ−VtμA\_\{t\}^\{\\mu\}=Q\_\{t\}^\{\\mu\}\-V\_\{t\}^\{\\mu\}be the token advantage\([Schulman et al\., 2015](https://arxiv.org/html/2611.21791#bib.bib31)\)\. SC\([Marek and Ryabinin, 2026](https://arxiv.org/html/2611.21791#bib.bib1)\)subtracts the behavior mean of the score to remove theVtμV\_\{t\}^\{\\mu\}contribution without learning a value function\. For SC without IS and exact centering, conditioning all expectations on that history gives 𝔼μ\[R\(ψθ−𝔼μ\[ψθ\]\)\]\\displaystyle\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[R\(\\psi\_\{\\theta\}\-\\mathbb\{E\}\_\{\\mu\}\[\\psi\_\{\\theta\}\]\)\\right\]=𝔼μ\[Atμψθ\]\\displaystyle=\\mathbb\{E\}\_\{\\mu\}\[A\_\{t\}^\{\\mu\}\\psi\_\{\\theta\}\]\(2\)=𝔼μ\[Qtμψθ\]−Vtμ𝔼μ\[ψθ\]⏟baseline\-induced drift\.\\displaystyle=\\mathbb\{E\}\_\{\\mu\}\[Q\_\{t\}^\{\\mu\}\\psi\_\{\\theta\}\]\-\\underbrace\{V\_\{t\}^\{\\mu\}\\mathbb\{E\}\_\{\\mu\}\[\\psi\_\{\\theta\}\]\}\_\{\\text\{baseline\-induced drift\}\}\.Thus centering removes the common\-value contribution from the expected update\. The removed term is a bias in the expected update, so collecting more data alone does not remove it\. SC and SC\+TIS fit a common TIS framework\.TIS uses the effective importance ratioρTIS:=clip\(ρ,clow,chigh\)\\rho^\{\\mathrm\{TIS\}\}:=\\operatorname\{clip\}\(\\rho,c\_\{\\mathrm\{low\}\},c\_\{\\mathrm\{high\}\}\), with0≤clow≤1≤chigh0\\leq c\_\{\\mathrm\{low\}\}\\leq 1\\leq c\_\{\\mathrm\{high\}\}, using the CISPO clipping form with both lower and upper bounds\([MiniMax, 2025](https://arxiv.org/html/2611.21791#bib.bib6)\)\. Upper truncation alone setsclow=0c\_\{\\mathrm\{low\}\}=0\([Ionides, 2008](https://arxiv.org/html/2611.21791#bib.bib21);[Yao et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib20)\)\. SC without IS is the special caseclow=chigh=1c\_\{\\mathrm\{low\}\}=c\_\{\\mathrm\{high\}\}=1, so every token receivesρTIS=1\\rho^\{\\mathrm\{TIS\}\}=1, whereas SC\+TIS uses a wider clipping interval and retains token\-dependent weights\. Both methods then subtract the conditional mean of the weighted score\. They use the same clipping rule for the sampled reward contribution and for the conditional mean used for centering\. Both variants therefore use the following sampled token update: g^i,tSC=Ai\(ρi,tTISψi,t−𝔼^μ\[ρTISψθ∣hi,t\]\)\.\\widehat\{g\}\_\{i,t\}^\{\\rm SC\}=A\_\{i\}\\\!\\left\(\\rho^\{\\mathrm\{TIS\}\}\_\{i,t\}\\psi\_\{i,t\}\-\\widehat\{\\mathbb\{E\}\}\_\{\\mu\}\[\\rho^\{\\mathrm\{TIS\}\}\\psi\_\{\\theta\}\\mid h\_\{i,t\}\]\\right\)\.\(3\)The conditional mean𝔼^μ\\widehat\{\\mathbb\{E\}\}\_\{\\mu\}is estimated using auxiliary actions sampled at the same history, each with its own weight\. Clipping an auxiliary action can therefore change the center even if the sampled action is unclipped\. When all action ratios are unclipped, however, the score weighted by full IS has zero conditional mean, so subtracting the exact center leaves the expected update unchanged\. We use Monte Carlo centers as described in Section[5\.2](https://arxiv.org/html/2611.21791#S5.SS2)\. Appendix[C\.3\.2](https://arxiv.org/html/2611.21791#A3.SS3.SSS2)gives the conditions on the group reward baseline\. ### 3\.2Introducing KL Regularization to Score Centering SC removes the baseline contribution but retains the behavior policy’s continuation values\.For updates weighted by response rewards, token\-level TIS usesQtμQ\_\{t\}^\{\\mu\}in its conditional expected update\. With exact centering, SC replaces this value byAtμ=Qtμ−VtμA\_\{t\}^\{\\mu\}=Q\_\{t\}^\{\\mu\}\-V\_\{t\}^\{\\mu\}\. The superscriptμ\\muidentifies the continuation policy: this token advantage compares the expected reward after choosing the token and continuing underμ\\muwith the average reward at that history\. The learner advantageAtπθA\_\{t\}^\{\\pi\_\{\\theta\}\}makes the same comparison under learner continuations\. The same token at the same history can therefore have different advantages under the two policies, even with a single terminal reward\. These are conditional expected values, whileAiA\_\{i\}is the sampled response advantage reused across tokens in the estimator\. Centering also leaves the distribution of sampled histories unchanged\. SC’s update usesAtμA\_\{t\}^\{\\mu\}on behavior histories, while the learner’s reward gradient usesAtπθA\_\{t\}^\{\\pi\_\{\\theta\}\}on learner histories, motivating reverse KL to control policy mismatch during fitting\. RSC uses reverse KLKL\(πθ∥μ\)\\operatorname\{KL\}\(\\pi\_\{\\theta\}\\\|\\mu\)to control this mismatch during fitting\.At each sampled history, the penalty discourages separation between the learner and behavior distributions over next tokens, withμ\\muheld fixed\. The regularized objective balances the improvement predicted byAtμA\_\{t\}^\{\\mu\}against the cost of policy movement\. It therefore penalizes the changes that can make these fixed continuation values less representative of learner outcomes\. In the outer loop of our analysis, refreshingμ\\mufrom the fitted learner aligns the policies and updates the histories and token advantages that define the next surrogate\. KL then controls the separation that develops during the next inner fit\. Section[4\.2](https://arxiv.org/html/2611.21791#S4.SS2)shows that sufficiently accurate fitting of the regularized objective yields improvement in true reward\. In experiments with periodic sampler refreshes, we replaceμ\\muwith a newer learner checkpoint at each refresh, so the KL reference changes along with the sampling policy\. In fully asynchronous training, a batch can contain data from different checkpoints, andμ\\mudenotes the corresponding behavior policy for each sample\. In both settings, we holdμ\\mufixed when differentiating the objective\. With regularization strengthβ≥0\\beta\\geq 0, subtractingβKL\(πθ∥μ\)\\beta\\operatorname\{KL\}\(\\pi\_\{\\theta\}\\\|\\mu\)encourages the two distributions over the next token to remain close\. Applying this penalty to the two choices of reward weights in Section[3\.1](https://arxiv.org/html/2611.21791#S3.SS1)gives RSC \(w/o IS\) and RSC \(TIS\)\. At a fixed historyhh, the penalty is KL\(πθ\(⋅∣h\)∥μ\(⋅∣h\)\)\\displaystyle\\operatorname\{KL\}\(\{\\pi\_\{\\theta\}\}\(\\cdot\\mid h\)\\\|\{\\mu\}\(\\cdot\\mid h\)\):=∑aπθ\(a∣h\)logπθ\(a∣h\)μ\(a∣h\)\\displaystyle:=\\sum\_\{a\}\{\\pi\_\{\\theta\}\}\(a\\mid h\)\\log\\frac\{\{\\pi\_\{\\theta\}\}\(a\\mid h\)\}\{\{\\mu\}\(a\\mid h\)\}\(4\)=𝔼a∼πθ\(⋅∣h\)\[logρ\(h,a\)\]=𝔼a∼μ\(⋅∣h\)\[ρ\(h,a\)logρ\(h,a\)\]\.\\displaystyle=\\mathbb\{E\}\_\{a\\sim\{\\pi\_\{\\theta\}\}\(\\cdot\\mid h\)\}\[\\log\\rho\(h,a\)\]=\\mathbb\{E\}\_\{a\\sim\{\\mu\}\(\\cdot\\mid h\)\}\[\\rho\(h,a\)\\log\\rho\(h,a\)\]\.The sum runs over next tokens, withμ\(a∣h\)\>0\\mu\(a\\mid h\)\>0whereverπθ\(a∣h\)\>0\\pi\_\{\\theta\}\(a\\mid h\)\>0\. The expectation under the behavior policy lets us estimate the penalty from the same data as the reward term\. To combine this penalty with centered reward updates, leta\(ρ\)=ρTISa\(\\rho\)=\\rho^\{\\mathrm\{TIS\}\}andFa\(ρ\)=∫1ρa\(u\)/u𝑑uF\_\{a\}\(\\rho\)=\\int\_\{1\}^\{\\rho\}a\(u\)/u\\,du\. This choice gives∇θFa\(ρ\)=a\(ρ\)ψθ\\nabla\_\{\\theta\}F\_\{a\}\(\\rho\)=a\(\\rho\)\\psi\_\{\\theta\}, withFa\(ρ\)=logρF\_\{a\}\(\\rho\)=\\log\\rhofor SC without IS\. Holding the behavior policyμ\\muand the token advantagesAtμA\_\{t\}^\{\\mu\}fixed, we writeρt=πθ\(at∣ht\)/μ\(at∣ht\)\\rho\_\{t\}=\\pi\_\{\\theta\}\(a\_\{t\}\\mid h\_\{t\}\)/\\mu\(a\_\{t\}\\mid h\_\{t\}\)and define the objective as 𝒲\(πθ\)=𝔼x∼𝒟,Y∼μ\(⋅∣x\)\[∑t=1HAtμ\(ht,at\)Fa\(ρt\)−β\(ρtlogρt−ρt\+1\)⏟wt\]\.\\mathcal\{W\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{x\\sim\\mathcal\{D\},\\,Y\\sim\\mu\(\\cdot\\mid x\)\}\\left\[\\sum\_\{t=1\}^\{H\}\\underbrace\{A\_\{t\}^\{\\mu\}\(h\_\{t\},a\_\{t\}\)F\_\{a\}\(\\rho\_\{t\}\)\-\\beta\(\\rho\_\{t\}\\log\\rho\_\{t\}\-\\rho\_\{t\}\+1\)\}\_\{w\_\{t\}\}\\right\]\.\(5\)Herewtw\_\{t\}is the token contribution to the objective\. Its first term rewards actions according toAtμA\_\{t\}^\{\\mu\}on behavior histories\. Its second term penalizes policy separation: since𝔼μ\[ρt∣ht\]=1\\mathbb\{E\}\_\{\\mu\}\[\\rho\_\{t\}\\mid h\_\{t\}\]=1, it has conditional expectationβKL\(πθ\(⋅∣ht\)∥μ\(⋅∣ht\)\)\\beta\\operatorname\{KL\}\(\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\\\|\\mu\(\\cdot\\mid h\_\{t\}\)\)\. Combining these terms defines the regularized surrogate whose population gradient is ∇θ𝒲\(πθ\)=𝔼μ\[∑t=1H\(Atμ\(ht,at\)ρtTIS−βρtlogρt\)ψθ\(ht,at\)\]\.\\nabla\_\{\\theta\}\\mathcal\{W\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[\\sum\_\{t=1\}^\{H\}\\left\(A\_\{t\}^\{\\mu\}\(h\_\{t\},a\_\{t\}\)\\rho\_\{t\}^\{\\mathrm\{TIS\}\}\-\\beta\\rho\_\{t\}\\log\\rho\_\{t\}\\right\)\\psi\_\{\\theta\}\(h\_\{t\},a\_\{t\}\)\\right\]\.\(6\)The expectation averages over prompts and responses generated by the behavior policy\. Appendix[C\.4](https://arxiv.org/html/2611.21791#A3.SS4)gives the derivative identities\. More generally, the KL reference can be chosen separately from the policy that generates the data\. Writingπref\\pi\_\{\\mathrm\{ref\}\}for this reference, the penalty and sampling ratio are KL\(πθ\(⋅∣h\)∥πref\(⋅∣h\)\),ρ\(h,a\)=πθ\(a∣h\)μ\(a∣h\)\.\\operatorname\{KL\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid h\)\\\|\\pi\_\{\\mathrm\{ref\}\}\(\\cdot\\mid h\)\\right\),\\qquad\\rho\(h,a\)=\\frac\{\\pi\_\{\\theta\}\(a\\mid h\)\}\{\\mu\(a\\mid h\)\}\.The reference is held fixed within each stage and refreshed from a learner checkpoint between stages\. It can be refreshed less often than the behavior policyμ\\mu, while importance weighting continues to use the ratio of learner to behavior probabilities\. We takeπref=μ\\pi\_\{\\mathrm\{ref\}\}=\\muthroughout this paper to keep the method and analysis simple\. Estimating the combined gradient\.The population gradient in Equation \([6](https://arxiv.org/html/2611.21791#S3.E6)\) uses expected continuation rewards throughAtμA\_\{t\}^\{\\mu\}, which are not observed directly\. RSC instead uses the rewards of sampled responses\. Score centering removes their common\-value contribution with exact centers or conditionally unbiased independent estimates, so the expected reward update is the same as if the token advantages were available\. The estimator below in Equation \([7](https://arxiv.org/html/2611.21791#S3.E7)\) combines the group\-centered rewardsAiA\_\{i\}with auxiliary actions sampled at the same history\. We also allow the penalty weight to be capped independently of the reward weight:ρKL=min\(ρ,cb\)\\rho^\{\\mathrm\{KL\}\}=\\min\(\\rho,c\_\{b\}\), withcb∈\[1,∞\]c\_\{b\}\\in\[1,\\infty\]\. Both effective ratios are computed from the originalρ\\rho\. An infinite cap gives standard reverse KL, whereas a finite cap limits contributions from large ratios\. The sampled token update is g^i,tRSC=Ai\(ρi,tTISψi,t−𝔼^μ\[ρTISψθ∣hi,t\]\)⏟centered reward gradient−β𝔼^μ\[ρKLlogρKLψθ∣hi,t\]⏟KL penalty gradient\.\\widehat\{g\}\_\{i,t\}^\{\\rm RSC\}=\\underbrace\{A\_\{i\}\\\!\\left\(\\rho^\{\\mathrm\{TIS\}\}\_\{i,t\}\\psi\_\{i,t\}\-\\widehat\{\\mathbb\{E\}\}\_\{\\mu\}\[\\rho^\{\\mathrm\{TIS\}\}\\psi\_\{\\theta\}\\mid h\_\{i,t\}\]\\right\)\}\_\{\\text\{centered reward gradient\}\}\-\\beta\\underbrace\{\\widehat\{\\mathbb\{E\}\}\_\{\\mu\}\\\!\\left\[\\rho^\{\\mathrm\{KL\}\}\\log\\rho^\{\\mathrm\{KL\}\}\\,\\psi\_\{\\theta\}\\mid h\_\{i,t\}\\right\]\}\_\{\\text\{KL penalty gradient\}\}\.\(7\)For an uncapped penalty \(cb=∞c\_\{b\}=\\infty\), draw a prompt andG≥2G\\geq 2independent behavior responses\. With the corrected group rewards and unbiased conditional centers of Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15), 𝔼\[1G∑i=1G∑t=1Hg^i,tRSC\]=∇θ𝒲\(πθ\)\.\\mathbb\{E\}\\\!\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{H\}\\widehat\{g\}\_\{i,t\}^\{\\rm RSC\}\\right\]=\\nabla\_\{\\theta\}\\mathcal\{W\}\(\\pi\_\{\\theta\}\)\.\(8\)The expectation includes the prompt, responses, and auxiliary action samples\. This identity shows that RSC performs stochastic gradient ascent on𝒲\\mathcal\{W\}in Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\)\. With exact conditional centers and the penalty estimated from generated actions, the proposition also gives a gradient identity for the fixed empirical objective when data are reused\. A finite penalty cap uses the modified objective in Equation \([C\.6](https://arxiv.org/html/2611.21791#A3.E6)\)\. Appendix[A](https://arxiv.org/html/2611.21791#A1)gives the sampling procedure and numerical settings\. ### 3\.3Other Compared Algorithms Token weighting: IS and TIS\.The uncentered baselines use the sampled advantageAiA\_\{i\}from Section[3\.1](https://arxiv.org/html/2611.21791#S3.SS1), with loss averaging specified for each task in Appendix[A](https://arxiv.org/html/2611.21791#A1)\. IS applies the uncapped token ratio to correct the distribution over the next action at a sampled history\([Owen, 2013](https://arxiv.org/html/2611.21791#bib.bib18)\): g^i,tIS=Aiρi,tψi,t\.\\widehat\{g\}\_\{i,t\}^\{\\rm IS\}=A\_\{i\}\\rho\_\{i,t\}\\psi\_\{i,t\}\.\(9\)To bound these importance weights, our TIS baseline uses the CISPO update\([MiniMax, 2025](https://arxiv.org/html/2611.21791#bib.bib6)\), which clips token importance weights\([Ionides, 2008](https://arxiv.org/html/2611.21791#bib.bib21);[Yao et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib20)\)\. This clipping introduces a bias relative to uncapped IS: g^i,tTIS=Aiρi,tTISψi,t\.\\widehat\{g\}\_\{i,t\}^\{\\rm TIS\}=A\_\{i\}\\rho\_\{i,t\}^\{\\mathrm\{TIS\}\}\\psi\_\{i,t\}\.\(10\)SC\+TIS adds conditional score centering to these weighted updates\. Treating the clipped weights as constants during differentiation preserves the score gradient\. Appendix[A](https://arxiv.org/html/2611.21791#A1)specifies the bounds\. IS\-PPOClip: importance sampling with PPO\-style clipping\.This baseline combines token\-level importance correction with PPO\-style loss clipping around a separately refreshed learner snapshotπprox\\pi\_\{\\mathrm\{prox\}\}\([Schulman et al\., 2017](https://arxiv.org/html/2611.21791#bib.bib3);[Hilton et al\., 2022](https://arxiv.org/html/2611.21791#bib.bib7);[Fu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib11);[Bai et al\., 2026a](https://arxiv.org/html/2611.21791#bib.bib10)\)\. Suppressing the shared token and history arguments, its token loss is −πproxμmin\{Aiπθπprox,Aiclip\(πθπprox,clow,chigh\)\}\.\-\\frac\{\\pi\_\{\\mathrm\{prox\}\}\}\{\\mu\}\\min\\\!\\left\\\{A\_\{i\}\\frac\{\\pi\_\{\\theta\}\}\{\\pi\_\{\\mathrm\{prox\}\}\},\\,A\_\{i\}\\operatorname\{clip\}\\\!\\left\(\\frac\{\\pi\_\{\\theta\}\}\{\\pi\_\{\\mathrm\{prox\}\}\},c\_\{\\mathrm\{low\}\},c\_\{\\mathrm\{high\}\}\\right\)\\right\\\}\.Bothμ\\muandπprox\\pi\_\{\\mathrm\{prox\}\}are held fixed during differentiation\. Without active clipping, the two ratios combine to give the IS update in Equation \([9](https://arxiv.org/html/2611.21791#S3.E9)\)\. PPO clipping acts onπθ/πprox\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{prox\}\}: once its clipped branch is active, the token’s reward gradient is zero\. TIS instead retains a score gradient weighted by the capped ratio\. Appendix[A\.2](https://arxiv.org/html/2611.21791#A1.SS2)specifies the clipping bounds and snapshot refresh interval\. Sequence MIS: masked sequence\-level importance sampling\.Sequence MIS\([Liu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib42)\)assigns one importance weight and one acceptance decision to the entire response\. The original rule accepts responses whose ratio is below an upper threshold, whereas our controlled experiments use an acceptance interval with lower and upper bounds\. For responseii, let𝒢i\\mathcal\{G\}\_\{i\}contain the positions of its generated tokens and writeρiseq\\rho\_\{i\}^\{\\mathrm\{seq\}\}for the ratio of the complete response: ρiseq=exp\(∑t∈𝒢ilogρi,t\),g^i,tSeq\-MIS=Aiρiseq𝟏\{clow≤ρiseq≤chigh\}ψi,t\.\\rho\_\{i\}^\{\\mathrm\{seq\}\}=\\exp\\\!\\left\(\\sum\_\{t\\in\\mathcal\{G\}\_\{i\}\}\\log\\rho\_\{i,t\}\\right\),\\qquad\\widehat\{g\}\_\{i,t\}^\{\\text\{Seq\-MIS\}\}=A\_\{i\}\\rho\_\{i\}^\{\\mathrm\{seq\}\}\\mathbf\{1\}\\\{c\_\{\\mathrm\{low\}\}\\leq\\rho\_\{i\}^\{\\mathrm\{seq\}\}\\leq c\_\{\\mathrm\{high\}\}\\\}\\psi\_\{i,t\}\.\(11\)Every generated token shares the same detached response weight, and accepted responses use the unmodified response ratio\. A ratio outside the acceptance interval removes the whole response’s contribution to the reward gradient, while its tokens remain in the loss normalization\. Appendix[A](https://arxiv.org/html/2611.21791#A1)gives the bounds\. ## 4Analysis of Regularized Score Centering ### 4\.1Surrogate Objectives and Empirical Updates Token contributions to the regularized surrogate\.Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\) defines the regularized objective𝒲\(πθ\)\\mathcal\{W\}\(\\pi\_\{\\theta\}\)\. We first examine its token contributions to understand how weighting and KL affect the update\. At a fixed history, writewwfor the token contributionwtw\_\{t\}in that objective\. Table[5](https://arxiv.org/html/2611.21791#A2.T5)lists these contributions for each method, and Figure[2](https://arxiv.org/html/2611.21791#S4.F2)shows the resulting logit derivatives\. Equation \([8](https://arxiv.org/html/2611.21791#S3.E8)\) shows how the sampled update optimizes𝒲\\mathcal\{W\}\. This objective uses histories and continuation values fromμ\\mu, whereas the rewardJ\(πθ\)J\(\\pi\_\{\\theta\}\)is evaluated on responses generated byπθ\\pi\_\{\\theta\}\. Section[4\.2](https://arxiv.org/html/2611.21791#S4.SS2)explains how accurate fitting of𝒲\\mathcal\{W\}can improveJ\(πθ\)J\(\\pi\_\{\\theta\}\)\. Figure 2:Weighting and KL shape logit gradients\.Curves show the derivative coefficient after factoring out1−πθ\(a∣h\)1\-\\pi\_\{\\theta\}\(a\\mid h\), withA=1A=1,c=2c=2andβ=0\.05\\beta=0\.05for RSC\. The expressions on the right give∇zaw\\nabla\_\{z\_\{a\}\}wfor fixedμ\\mu,A=Aμ\(h,a\)A=A^\{\\mu\}\(h,a\)and standard, uncapped reverse KL\.SC\+TIS limits gradient amplification by importance ratios, and KL damps large positive updates\.The integrands in Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\) let us examine how each method weights an individual token\. IS has∇zaw=Aρ\[1−πθ\(a∣h\)\]\\nabla\_\{z\_\{a\}\}w=A\\rho\[1\-\\pi\_\{\\theta\}\(a\\mid h\)\], so a large ratio between learner and behavior probabilities can amplify the gradient\. SC\+TIS replacesρ\\rhowithmin\(ρ,c\)\\min\(\\rho,c\), bounding the reward multiplier, while SC without IS fixes it at one\. Forρ\>1\\rho\>1, RSC subtractsβρlogρ\[1−πθ\(a∣h\)\]\\beta\\rho\\log\\rho\[1\-\\pi\_\{\\theta\}\(a\\mid h\)\], opposing further probability increases \(Figure[2](https://arxiv.org/html/2611.21791#S4.F2)\)\. Sinceρ=πθ/μ≤1/μ\\rho=\\pi\_\{\\theta\}/\\mu\\leq 1/\\mu, very large ratios require small behavior probabilities, motivatingμ\(a∣h\)\\mu\(a\\mid h\)as the horizontal axis in Figure[3](https://arxiv.org/html/2611.21791#S4.F3)\. The matched comparison in Figure[3](https://arxiv.org/html/2611.21791#S4.F3)shows smaller extreme gradient norms under SC and SC\+TIS in both logit and parameter space\. Adding KL further reduces the 99th percentile of logit gradient norms for tokens with low behavior probability under both variants\. Figure 3:Weighting, centering and KL shape the update\.We hold the learner and behavior checkpoints fixed and reuse the sampled actions, response advantages, and 128 auxiliary Monte Carlo draws across methods\. The panels therefore compare update rules on the same data, before any optimizer step\. Both rows use behavior probabilityμ\(a∣h\)\\mu\(a\\mid h\): panels \(a–e\) show logit gradient norms, and panels \(f–j\) show parameter gradient norms\. Black lines show the 99th percentile within each probability bin, and the top row reports this percentile forμ<0\.01\\mu<0\.01\. Appendix[A\.1](https://arxiv.org/html/2611.21791#A1.SS1)gives the diagnostic protocol\. ### 4\.2Why Regularization Makes Fitting More Informative Beyond its effect on individual updates, KL helps in two ways: it makes accurate fitting easier to certify from data, and it lets fitting accuracy translate into reward progress without shrinking the token radius as responses get longer\. Appendix[C\.1](https://arxiv.org/html/2611.21791#A3.SS1)gives the proofs and sample bounds\. The improving target\.Consider RSC \(TIS\) with deterministic transitions and binary terminal rewards\. Fix a prompt and behavior policyμ\\mu, and write𝒲μ\\mathcal\{W\}\_\{\\mu\}for the objective in Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\) conditional on this prompt\. In this target comparison,JJis also conditional on the prompt\. Forβ≥2\\beta\\geq 2, take the full policy family satisfying\|logρt\|≤1/β\|\\log\\rho\_\{t\}\|\\leq 1/\\betaat every history and action\. Reward and penalty caps of two are inactive, and the maximizing policy is π⋆\(a∣ht\)=μ\(a∣ht\)exp\(Atμ\(ht,a\)/β\)𝔼u∼μ\(⋅∣ht\)exp\(Atμ\(ht,u\)/β\)\.\\pi^\{\\star\}\(a\\mid h\_\{t\}\)=\\frac\{\\mu\(a\\mid h\_\{t\}\)\\exp\\\!\\left\(A\_\{t\}^\{\\mu\}\(h\_\{t\},a\)/\\beta\\right\)\}\{\\mathbb\{E\}\_\{u\\sim\\mu\(\\cdot\\mid h\_\{t\}\)\}\\exp\\\!\\left\(A\_\{t\}^\{\\mu\}\(h\_\{t\},u\)/\\beta\\right\)\}\.\(12\)The target assigns more weight to actions with higher advantage and belongs to the allowed policy family\. It multiplies success odds by at least1\+1/β1\+1/\\betafor0<J\(μ\)<10<J\(\\mu\)<1\(Lemmas[C\.10](https://arxiv.org/html/2611.21791#A3.Thmtheorem10)and[C\.11](https://arxiv.org/html/2611.21791#A3.Thmtheorem11)\)\. Inner fitting\.Without KL, changes in token probabilities can leave the surrogate unchanged at histories where all token advantages are zero\. A small objective gap then need not identify a particular target policy\. For the regularized TIS objective in Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\), KL adds strong concavity in the token probabilities, so moving away from the target incurs an objective loss\. Every candidateπθ\\pi\_\{\\theta\}in the family satisfies 𝒲μ\(π⋆\)−𝒲μ\(πθ\)=β∑t𝔼ht∼dtμKL\(πθ\(⋅∣ht\)∥π⋆\(⋅∣ht\)\)\.\\mathcal\{W\}\_\{\\mu\}\(\\pi^\{\\star\}\)\-\\mathcal\{W\}\_\{\\mu\}\(\\pi\_\{\\theta\}\)=\\beta\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\}\\operatorname\{KL\}\\\!\\left\(\\pi\_\{\\theta\}\(\\cdot\\mid h\_\{t\}\)\\\|\\pi^\{\\star\}\(\\cdot\\mid h\_\{t\}\)\\right\)\.\(13\)Heredtμd\_\{t\}^\{\\mu\}is the distribution of histories at positionttunderμ\\mu\. The identity shows that a small objective gap makes the fitted policy close to the improving target on behavior histories\. This closeness also controls sampling noise: the variance of the sampled objective difference between a candidate and the target is bounded by a constant timesHHtimes the population gap \(Lemma[C\.22](https://arxiv.org/html/2611.21791#A3.Thmtheorem22)\)\. As fitting improves, this comparison becomes less noisy, giving a sharper sample bound than treating the noise as constant\. This statistical use of KL curvature is analogous to the fast\-rate analysis of[Zhao et al\. \(2025\)](https://arxiv.org/html/2611.21791#bib.bib46)for KL\-regularized contextual bandits and RLHF\. Outer progress\.To relate this fitting guarantee to reward, we next examine changes in the history distribution\. Without KL, our analysis controls these changes by bounding each token log\-ratio\. For a response ofHHtokens, a bound\|logρt\|≤r\|\\log\\rho\_\{t\}\|\\leq rallows the response ratio to be as large aseHre^\{Hr\}\. At a fixed radius, this worst\-case bound grows exponentially withHH\. Keeping it bounded using the token constraint alone therefore requiresr=O\(1/H\)r=O\(1/H\)\(Lemma[C\.5](https://arxiv.org/html/2611.21791#A3.Thmtheorem5)\)\. For fixedβ\\beta, the KL\-regularized targetπ⋆\\pi^\{\\star\}instead has a worst\-case bound on every prefix and full\-response ratio that is independent ofHH, even at the constant token radiusr=O\(1\)r=O\(1\)\(Lemma[C\.12](https://arxiv.org/html/2611.21791#A3.Thmtheorem12)\)\. With this coverage bound, a policy that fits the surrogate accurately achieves reward close to the improving target \(Lemma[C\.13](https://arxiv.org/html/2611.21791#A3.Thmtheorem13)\)\. RSC without IS also turns accurate fitting into reward progress, as shown in Appendix[C\.2\.3](https://arxiv.org/html/2611.21791#A3.SS2.SSS3)\. Together, the inner fitting bounds and outer progress bounds give a comparison with best\-of\-NNsampling: one response from the final policy nearly matches the expected reward of the best ofNNindependent responses from the initial policy\. On the finite deterministic token tree with binary rewards, both RSC variants achieve reward error at mostϵ\\epsilonwith high probability afterO\(log\(N/ϵ\)\)O\(\\log\(N/\\epsilon\)\)outer rounds under the stated fitting and sampling conditions\. These bounds useβ=2\\beta=2, a token log\-ratio radius of1/21/2, and fresh data at each round\. Without KL, the sufficient radius is1/\(8H\)1/\(8H\)and the round bound isO\(Hlog\(N/ϵ\)\)O\(H\\log\(N/\\epsilon\)\), whereHHis the response horizon\. Theorems[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)and[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)give the regularized guarantees, and Theorem[C\.29](https://arxiv.org/html/2611.21791#A3.Thmtheorem29)gives the unregularized comparison\. In these guarantees, each outer round fits a policy on a fixed dataset before refreshing the behavior policy\. These counts therefore measure policy refreshes, while optimizer iterations and sampling costs remain separate\. The TIS theorem uses exact centers, while the theorem for unit weights also allows Monte Carlo centers\. Appendix[C\.1](https://arxiv.org/html/2611.21791#A3.SS1)gives the sample bounds that depend on the horizon and discusses the experimental KL coefficients\. These guarantees concern score\-centered updates\. KL regularization encourages the learner to remain close to the behavior policy, but does not ensure that every token ratio stays within the TIS bounds\. When clipping is active, the weighted score can have a nonzero conditional mean under the behavior policy, so a reward baseline can contribute to the expected update\. Score centering subtracts this mean and removes the baseline contribution even with clipped reward weights, as shown in Section[3\.1](https://arxiv.org/html/2611.21791#S3.SS1)\. This cancellation is essential to our analysis\. In other words, our analysis does not establish reward improvement for TIS\+KL without score centering when clipping is active\. ## 5Identifying Failure Modes of Score Centering The reward guarantees in Section[4\.2](https://arxiv.org/html/2611.21791#S4.SS2)rely on accurate fitting within a specified policy neighborhood\. In training, both the score center estimate and movement away from the behavior policy can affect learning\. We examine these two potential sources of SC failure in turn\. Section[5\.2](https://arxiv.org/html/2611.21791#S5.SS2)shows that Monte Carlo centering improves learning over approximate centers\. These improved estimates leave the dependence on behavior histories and continuations unchanged, motivating the task and staleness comparisons in Section[5\.3](https://arxiv.org/html/2611.21791#S5.SS3)\. We first contrast SC’s successful learning on math tasks using a dense model with its behavior on Crafter, then examine how SC can degrade as staleness increases on Countdown and math tasks with an MoE model\. ### 5\.1Experimental Setup In the controlled experiments in Figures[4](https://arxiv.org/html/2611.21791#S5.F4)through[7](https://arxiv.org/html/2611.21791#S6.F7), the sampler refresh interval controls how long the learner updates against a fixed behavior policy\. The learnerπθ\\pi\_\{\\theta\}updates whileμ\\muremains fixed forRRupdates\. The sampler then copies the learner weights to refreshμ\\mu\. In agentic experiments, rollout generation and learner training run concurrently: the learner consumes completed response groups from a queue, taking one update per new batch while generation continues under fixedμ\\mu\. Countdown alternates sampling and training within the same schedule withμ\\mufixed\. LargerRRtherefore permits greater policy separation\. Training curves measure reward underμ\\mu, while heldout scores evaluateπθ\\pi\_\{\\theta\}on separate tasks\. Appendix[A](https://arxiv.org/html/2611.21791#A1)specifies the models, datasets, optimization and evaluation settings\. ### 5\.2Score Centering Fails Under Top\-k Estimation Monte Carlo gives an unbiased estimate of the conditional center\.The original SC implementation uses the 128 highest behavior probabilities and a modeled tail\([Marek and Ryabinin, 2026](https://arxiv.org/html/2611.21791#bib.bib1)\), introducing approximation bias\. We replace this biased approximation with auxiliary draws fromμ\\mu\. At the historyhhpreceding each generated token, we draw auxiliary tokens independently with replacement fromμ\(⋅∣h\)\\mu\(\\cdot\\mid h\)\. These draws are independent of the generated action and the continuation of the response, and require no additional response generation or reward evaluation\. Backpropagation through the average learner log\-probability of the auxiliary tokens gives an unbiased estimate of𝔼μ\[ψθ∣h\]\\mathbb\{E\}\_\{\\mu\}\[\\psi\_\{\\theta\}\\mid h\]\. For SC\+TIS, each auxiliary log\-probability receives its own detachedρTIS\\rho^\{\\mathrm\{TIS\}\}weight, giving an unbiased estimate of𝔼μ\[ρTISψθ∣h\]\\mathbb\{E\}\_\{\\mu\}\[\\rho^\{\\mathrm\{TIS\}\}\\psi\_\{\\theta\}\\mid h\]\. If a token is sampled more than once, each draw contributes to the estimate\. Appendix[A](https://arxiv.org/html/2611.21791#A1)lists the sample counts\. Monte Carlo centering improves learning over top\-kkestimation\.Every tested Monte Carlo variant in Figure[4](https://arxiv.org/html/2611.21791#S5.F4)achieves a higher final evaluation score than the corresponding Top\-128 baseline\. The math comparisons show substantial evaluation gains with both BF16 and FP8 rollouts\. Countdown shows the same advantage across the tested Monte Carlo sample counts, supporting direct sampling from the behavior policy to estimate the center\. Figure 4:Monte Carlo centering improves on Top\-128\.Rows show training \(top\) and evaluation \(bottom\)\. Countdown uses three seeds, with bands indicating sample standard deviation\. The math experiments use one seed per method and rollout precision\. ### 5\.3Score Centering Fails under Agentic and Highly Stale Settings SC’s expected update weights actions byAtμA\_\{t\}^\{\\mu\}, which evaluates them using continuations under the behavior policy\. The learner’s advantageAtπθA\_\{t\}^\{\\pi\_\{\\theta\}\}instead uses continuations underπθ\\pi\_\{\\theta\}\. Atπθ=μ\\pi\_\{\\theta\}=\\mu, the advantages and history distributions agree\. When the relevant action and continuation distributions remain close, advantages under the behavior policy provide a local approximation to those under the learner policy\. As the policies separate, the value of the same action can change and learner trajectories can visit histories poorly represented in the behavior data\. The resulting error depends on how far the policies move apart and how much their different choices change the eventual reward\. Task structure can change the consequences of policy mismatch\.At the same sampler refresh interval, Figure[5](https://arxiv.org/html/2611.21791#S5.F5)compares math tasks using a dense model with Crafter, an agentic environment where the agent’s actions produce feedback that guides subsequent decisions\. SC and SC\+TIS sustain learning on the math tasks, and adding KL changes each final mean accuracy by less than1%1\\%in absolute accuracy\. On Crafter, however, RSC \(TIS\) achieves substantially higher training and evaluation scores than SC\+TIS\. Although the refresh interval is matched, the degree of separation between learner and behavior policies can still differ\. Figure[7](https://arxiv.org/html/2611.21791#S6.F7)compares all six agentic tasks, with final evaluations in Table[1](https://arxiv.org/html/2611.21791#S6.T1)\. Appendix[A\.1](https://arxiv.org/html/2611.21791#A1.SS1)gives the evaluation protocol for the math tasks\. Figure 5:Learning on math tasks and Crafter at the same refresh interval\.The left pair shows math tasks with a dense model and the right pair shows Crafter, each with training performance and signed meanlogπθ−logμ\\log\\pi\_\{\\theta\}\-\\log\\mu\. Curves and bands show mean±\\pmsample standard deviation across three seeds\. The policy\-gap metric for IS\-PPOClip is available only for Crafter\.To understand how task structure can amplify this mismatch, consider agentic actions that change the environment state and the choices available afterward\. In Crafter, gathering a resource or taking a route can open a different branch of future interactions\. Because the value of that action depends on how the policy uses the resulting opportunities, it may have little advantage underμ\\muif behavior continuations rarely exploit them, but a much larger advantage underπθ\\pi\_\{\\theta\}if learner continuations do\. Such branching can makeAtπθA\_\{t\}^\{\\pi\_\{\\theta\}\}andAtμA\_\{t\}^\{\\mu\}differ substantially even at the same sampler refresh interval\. The math and Crafter runs differ in both task and model, but the effect of actions on later choices may help explain the contrast in Figure[5](https://arxiv.org/html/2611.21791#S5.F5)\. Longer refresh intervals allow policy mismatch to accumulate\.Task structure can affect the consequences of different continuations, while the refresh interval controls how long the policies can move apart\. With more learner updates against a fixedμ\\mu,πθ\\pi\_\{\\theta\}can change both action probabilities and continuation values further before the sampler is refreshed\. Figure[6](https://arxiv.org/html/2611.21791#S5.F6)tests this source of error on Countdown and math tasks with an MoE model\. SC remains competitive at shorter intervals, while SC\+TIS and RSC retain stronger performance at the longest intervals\. At the largest tested Countdown interval, RSC \(TIS\) substantially outperforms SC in final heldout reward, with an absolute reward gain of approximately65%65\\%even with Monte Carlo centers\. In the experiments with the MoE model, SC exceeds SC\+TIS at shorter refresh intervals but falls behind at the longest interval, alongside greater policy separation\. Learner evaluations reveal degradation that rewards from the stale behavior policy can hide\. Settings for Countdown and the math experiments with the MoE model appear in Appendices[A\.3](https://arxiv.org/html/2611.21791#A1.SS3)and[A\.1](https://arxiv.org/html/2611.21791#A1.SS1), respectively\. Figure 6:Score centering under increasing staleness\.Columns show Countdown reward and signedlogμ−logπθ\\log\{\\mu\}\-\\log\{\\pi\_\{\\theta\}\}, then math training accuracy with the MoE model and\|logπθ−logμ\|\|\\log\{\\pi\_\{\\theta\}\}\-\\log\{\\mu\}\|\. Sampler refresh intervals increase down the rows\. Curves and bands show mean±\\pmsample standard deviation\. Countdown uses three seeds\. MoE uses two seeds at the two shorter intervals and three at the longest\. Countdown log\-probability panels cover the first three refresh cycles\. ## 6Scaling Experiments to Agentic Settings The controlled comparisons in Section[5](https://arxiv.org/html/2611.21791#S5)show why accurate centering can still benefit from KL regularization\. We now evaluate the resulting methods across a wider set of interactive tasks, using both periodic sampler refreshes and fully asynchronous data collection\. ### 6\.1Experimental Setup Controlled asynchronous environments\.Figure[7](https://arxiv.org/html/2611.21791#S6.F7)uses the same environment interaction and controlled asynchronous training setup described in Section[5\.1](https://arxiv.org/html/2611.21791#S5.SS1)for WebShop shopping\([Yao et al\., 2022](https://arxiv.org/html/2611.21791#bib.bib12)\), AppWorld application tool use\([Trivedi et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib13)\), BabyAI navigation and object manipulation guided by language\([Chevalier\-Boisvert et al\., 2019](https://arxiv.org/html/2611.21791#bib.bib14)\), Cooking games generated with TextWorld\([Côté et al\., 2018](https://arxiv.org/html/2611.21791#bib.bib15)\), and survival in Crafter\([Hafner, 2022](https://arxiv.org/html/2611.21791#bib.bib19)\)and Craftax\([Matthews et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib43)\)\. Appendix[A\.2](https://arxiv.org/html/2611.21791#A1.SS2)gives task rewards, training settings and episode limits \(Table[3](https://arxiv.org/html/2611.21791#A1.T3)\)\. Fully asynchronous software engineering environment\.Figure[8](https://arxiv.org/html/2611.21791#S6.F8)studies software engineering through a fully asynchronous pipeline\. Completed response groups enter a training queue as generation and tool interaction finish\. Whenever a batch is ready, the learner performs an optimizer update and publishes its new weights, while generation continues for other trajectories\. Policy lag therefore emerges from generation latency, environment interaction and queueing\. This resembles production RL workloads, where trajectories finish at different times and training consumes data as they arrive\. Appendix[A\.4](https://arxiv.org/html/2611.21791#A1.SS4)specifies the model, datasets, sampling and optimization settings\. Compared methods\.In Figures[7](https://arxiv.org/html/2611.21791#S6.F7)and[8](https://arxiv.org/html/2611.21791#S6.F8), our main baselines, SC and SC\+TIS, are among the strongest methods in the original score centering study\([Marek and Ryabinin, 2026](https://arxiv.org/html/2611.21791#bib.bib1)\)\. IS and TIS provide baselines without score centering\. Figure[7](https://arxiv.org/html/2611.21791#S6.F7)also includes IS\-PPOClip, which combines token\-level importance correction with PPO\-style loss clipping, and Sequence MIS, which combines sequence\-level importance weights with a sequence\-level mask\([Liu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib42)\)\. Section[3\.3](https://arxiv.org/html/2611.21791#S3.SS3)defines these methods, and Appendix[A](https://arxiv.org/html/2611.21791#A1)gives their settings and evaluation protocols\. ### 6\.2Controlled Asynchronous Agentic Tasks RSC with TIS substantially improves final performance\.RSC \(TIS\) achieves top final performance across all six tasks \(Table[1](https://arxiv.org/html/2611.21791#S6.T1)\)\. Compared with IS\-PPOClip, the strongest baseline outside the RSC family on Crafter, Craftax and BabyAI, it improves the mean score by52%52\\%on Crafter and16%16\\%on Craftax, and reaches9\.9×9\.9\\timesits score on BabyAI\. It also improves on SC\+TIS by43%43\\%on WebShop and on SC and SC\+TIS by14\.3%14\.3\\%on AppWorld, while scoring close to the maximum on TextWorld\. RSC with TIS substantially improves sample efficiency on Crafter and Craftax\.For the same number of sampled responses, RSC with TIS attains higher training rewards on both tasks, with the advantage persisting as training proceeds \(Figure[7](https://arxiv.org/html/2611.21791#S6.F7)\)\. Table[1](https://arxiv.org/html/2611.21791#S6.T1)complements these learning curves with final learner evaluations\. Table 1:Checkpoint evaluation scores \(mean±\\pmsample standard deviation across three seeds\)\. Crafter uses its geometric achievement score, Craftax uses total episode reward, and other scores are reported in %\. Bold marks the highest reported mean and the second highest mean if it is within 5% of the highest\.Figure 7:Training reward across six interaction tasks\.Curves show mean±\\pmsample standard deviation across three seeds\. Crafter and Craftax use total episode rewards\. Appendix[A\.2](https://arxiv.org/html/2611.21791#A1.SS2)specifies training settings\. ### 6\.3Fully Asynchronous RL on Software Engineering Tasks RSC leads both SWE configurations\.An RSC variant achieves the highest evaluation score in each configuration, A and B, where policy lag varies during training\. Configuration B shows substantial gains: RSC \(w/o IS\) and RSC \(TIS\) raise evaluation scores over SC and SC\+TIS by approximately127%127\\%and51%51\\%, respectively\. Figure[8](https://arxiv.org/html/2611.21791#S6.F8)shows the training curves, and Table[2](https://arxiv.org/html/2611.21791#S6.T2)reports the SWE\-bench Verified scores\. SWE training is computationally expensive and slow, so we evaluate a subset of baselines\. Larger KL gains coincide with greater policy mismatch in this comparison\.To examine the difference between configurations A and B, we compare their policy lag and probability mismatch\. The configurations have similar measured policy lag, defined as the number of learner updates between the policy version used for sampling and the policy version used for training\. SC\+TIS in B has approximately2×2\\timesthe mean\|logπθ−logμ\|\|\\log\\pi\_\{\\theta\}\-\\log\\mu\|of SC\+TIS in A over the same range of training updates\. Thus, similar delays can correspond to different amounts of policy mismatch\. The KL gains follow this contrast: with or without TIS, RSC’s improvement over its corresponding SC baseline is substantially larger in configuration B than in A, where the gain is small without TIS and negative with TIS\. These are observational comparisons of single runs at different learning rates\. These results suggest that a larger learning rate can increase off\-policyness and that RSC can sustain improvements at higher learning rates than SC\. Figure 8:Fully asynchronous SWE training\.Each method and configuration uses one seed\. The left pair shows configuration A and the right pair configuration B, each with training reward followed by the absolute log\-probability difference\. Axes use restricted ranges\. Appendix[A\.4](https://arxiv.org/html/2611.21791#A1.SS4)specifies the configurations and smoothing\. Table 2:SWE\-bench Verified evaluation \(Average@8, %\)\.Learning rates match the training panels\. Appendix[A\.4](https://arxiv.org/html/2611.21791#A1.SS4)gives the evaluation protocol\. Best scores are bold\. ## 7Conclusion and Future Work Regularized Score Centering removes the reward baseline contribution and adds reverse KL to penalize movement away from the behavior policy\. With this control of policy movement, our analysis connects accurate surrogate fitting to reward improvement\. Consistent with this motivation, the experiments show benefits when data come from stale policies and in agentic tasks where actions change subsequent states and rewards\. RSC with TIS substantially improves sample efficiency on Crafter and Craftax, attaining higher training rewards for the same number of sampled responses\. It also achieves top final performance across all six agentic tasks\. Its gains over the strongest baselines include a14\.3%14\.3\\%improvement on AppWorld and9\.9×9\.9\\timesthe score on BabyAI\. The software engineering results extend these gains to fully asynchronous training, where policy lag varies over time\. The theoretical guarantee also allows a token log\-ratio radius that does not shrink with the response horizon and, for unit weights, accommodates Monte Carlo estimates of the center\. Building on these guarantees, a useful direction for future work is to study how greater policy staleness affects RSC empirically, including how many optimizer updates are needed to maintain reward improvement as the learner moves further from the policies that generated the data\. ## Acknowledgements This work was supported in part by NSF grants 2416897, 2505932, and 2601666 and ONR grant N000142512318\. Computing resources were provided by the National Center for Supercomputing Applications \(NCSA\) through Delta \(NSF OAC\-2005572\) and DeltaAI \(NSF OAC\-2320345\), as well as Illinois Computes and the NAIRR Pilot \(NAIRR250157\)\. ## References - Baiet al\.\(2026a\)H\. Bai, R\. Yang, C\. Ye, S\. Whitehead, A\. Kumar, and T\. ZhangAsyncWebRL: efficient asynchronous reinforcement learning for multi\-step visual web agents\.InFindings of the Association for Computational Linguistics: EMNLP 2026,pp\. 28704–28722\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p2.1)\. - Baiet al\.\(2026b\)H\. Bai, J\. Yao, and T\. ZhangEnabling token\-wise importance sampling in off\-policy RLVR with proximal KL regularization\.arXiv preprint arXiv:2610\.18738\.Cited by:[§C\.2\.1](https://arxiv.org/html/2611.21791#A3.SS2.SSS1.p2.1.1.1),[Lemma C\.3](https://arxiv.org/html/2611.21791#A3.Thmtheorem3),[§1](https://arxiv.org/html/2611.21791#S1.p2.1),[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Chenet al\.\(2026\)F\. Chen, A\. Huang, N\. Golowich, S\. Malladi, A\. Block, J\. T\. Ash, A\. Krishnamurthy, and D\. J\. FosterThe Coverage Principle: How Pre\-Training Enables Post\-Training\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Chevalier\-Boisvertet al\.\(2019\)M\. Chevalier\-Boisvert, D\. Bahdanau, S\. Lahlou, L\. Willems, C\. Saharia, T\. H\. Nguyen, and Y\. BengioBabyAI: a platform to study the sample efficiency of grounded language learning\.InInternational Conference on Learning Representations,Cited by:[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p1.1)\. - Chowdhuryet al\.\(2024\)N\. Chowdhury, J\. Aung, C\. J\. Shern, O\. Jaffe, D\. Sherburn, G\. Starace, E\. Mays, R\. Dias, M\. Aljubeh, M\. Glaese, C\. E\. Jimenez, J\. Yang, L\. Ho, T\. Patwardhan, K\. Liu, and A\. MadryIntroducing SWE\-bench Verified\.Note:OpenAIPublished August 13, 2024, updated February 24, 2025External Links:[Link](https://openai.com/index/introducing-swe-bench-verified/)Cited by:[§A\.4](https://arxiv.org/html/2611.21791#A1.SS4.p3.1)\. - Côtéet al\.\(2018\)M\. Côté, Á\. Kádár, X\. Yuan, B\. Kybartas, T\. Barnes, E\. Fine, J\. Moore, M\. Hausknecht, R\. Y\. Tao, L\. El Asri, M\. Adada, W\. Tay, and A\. TrischlerTextWorld: a learning environment for text\-based games\.arXiv preprint arXiv:1806\.11532\.Cited by:[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p1.1)\. - Espeholtet al\.\(2018\)L\. Espeholt, H\. Soyer, R\. Munos, K\. Simonyan, V\. Mnih, T\. Ward, Y\. Doron, V\. Firoiu, T\. Harley, I\. Dunning, S\. Legg, and K\. KavukcuogluIMPALA: scalable distributed deep\-RL with importance weighted actor\-learner architectures\.InProceedings of the 35th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.80,pp\. 1407–1416\.Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p1.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1)\. - Fuet al\.\(2025\)W\. Fu, J\. Gao, X\. Shen, C\. Zhu, Z\. Mei, C\. He, S\. Xu, G\. Wei, J\. Mei, J\. Wang, T\. Yang, B\. Yuan, and Y\. WuAReaL: a large\-scale asynchronous reinforcement learning system for language reasoning\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p2.1)\. - Geistet al\.\(2019\)M\. Geist, B\. Scherrer, and O\. PietquinA theory of regularized Markov decision processes\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 2160–2169\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Hafner \(2022\)D\. HafnerBenchmarking the spectrum of agent capabilities\.InInternational Conference on Learning Representations,Cited by:[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p1.1)\. - Heet al\.\(2024\)C\. He, R\. Luo, Y\. Bai, S\. Hu, Z\. Thai, J\. Shen, J\. Hu, X\. Han, Y\. Huang, Y\. Zhang, J\. Liu, L\. Qi, Z\. Liu, and M\. SunOlympiadBench: a challenging benchmark for promoting AGI with olympiad\-level bilingual multimodal scientific problems\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3828–3850\.Cited by:[§A\.1](https://arxiv.org/html/2611.21791#A1.SS1.p2.1)\. - He and Thinking Machines Lab \(2025\)H\. He and Thinking Machines LabDefeating nondeterminism in LLM inference\.Thinking Machines Lab: Connectionism\.External Links:[Document](https://dx.doi.org/10.64434/tml.20250910),[Link](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/)Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1)\. - Hiltonet al\.\(2022\)J\. Hilton, K\. Cobbe, and J\. SchulmanBatch size\-invariance for policy optimization\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 17086–17098\.Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p1.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p2.1)\. - Huanget al\.\(2026\)A\. Huang, N\. Jiang, S\. M\. Kakade, S\. Malladi, A\. Krishnamurthy, and D\. J\. FosterA theory of policy optimization in post\-training: coverage collapse and reference resets\.arXiv preprint arXiv:2611\.14467\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Huanget al\.\(2025\)A\. Huang, W\. Zhan, T\. Xie, J\. D\. Lee, W\. Sun, A\. Krishnamurthy, and D\. J\. FosterCorrecting the mythos of KL\-regularization: direct alignment without overoptimization via chi\-squared preference optimization\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Ionides \(2008\)E\. L\. IonidesTruncated importance sampling\.Journal of Computational and Graphical Statistics17\(2\),pp\. 295–311\.Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p2.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.1](https://arxiv.org/html/2611.21791#S3.SS1.p3.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p1.2)\. - Jimenezet al\.\(2024\)C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. NarasimhanSWE\-bench: can language models resolve real\-world GitHub issues?\.InInternational Conference on Learning Representations,Cited by:[§A\.4](https://arxiv.org/html/2611.21791#A1.SS4.p3.1)\. - Lewkowyczet al\.\(2022\)A\. Lewkowycz, A\. Andreassen, D\. Dohan, E\. Dyer, H\. Michalewski, V\. Ramasesh, A\. Slone, C\. Anil, I\. Schlag, T\. Gutman\-Solo, Y\. Wu, B\. Neyshabur, G\. Gur\-Ari, and V\. MisraSolving quantitative reasoning problems with language models\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§A\.1](https://arxiv.org/html/2611.21791#A1.SS1.p2.1)\. - Lightmanet al\.\(2024\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe Twelfth International Conference on Learning Representations,Cited by:[§A\.1](https://arxiv.org/html/2611.21791#A1.SS1.p2.1)\. - Ling Teamet al\.\(2025\)Ling Teamet al\.Every step evolves: scaling reinforcement learning for trillion\-scale thinking model\.External Links:2510\.18855Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1)\. - Liuet al\.\(2025\)J\. Liu, Y\. Li, Y\. Fu, J\. Wang, Q\. Liu, and Y\. ShenWhen speed kills stability: demystifying RL collapse from the training\-inference mismatch\.Note:Technical articleExternal Links:[Link](https://richardli.xyz/post/rl-collapse-training-inference/)Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p3.1),[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p3.1)\. - Marek and Ryabinin \(2026\)M\. Marek and M\. RyabininScore centering stabilizes off\-policy reinforcement learning\.External Links:2609\.20807Cited by:[§A\.1](https://arxiv.org/html/2611.21791#A1.SS1.p1.1),[§A\.3](https://arxiv.org/html/2611.21791#A1.SS3.p1.1),[§1](https://arxiv.org/html/2611.21791#S1.p3.1),[§1](https://arxiv.org/html/2611.21791#S1.p4.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.1](https://arxiv.org/html/2611.21791#S3.SS1.p2.2),[§5\.2](https://arxiv.org/html/2611.21791#S5.SS2.p1.1),[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p3.1)\. - Matthewset al\.\(2024\)M\. Matthews, M\. Beukman, B\. Ellis, M\. Samvelyan, M\. T\. Jackson, S\. Coward, and J\. N\. FoersterCraftax: a lightning\-fast benchmark for open\-ended reinforcement learning\.InProceedings of the 41st International Conference on Machine Learning,Vol\.235,pp\. 35104–35137\.Cited by:[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p1.1)\. - MiniMax \(2025\)MiniMaxMiniMax\-M1: scaling test\-time compute efficiently with lightning attention\.arXiv preprint arXiv:2506\.13585\.Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p2.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.1](https://arxiv.org/html/2611.21791#S3.SS1.p3.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p1.2)\. - Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. F\. Christiano, J\. Leike, and R\. LoweTraining language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Owen \(2013\)A\. B\. OwenMonte carlo theory, methods and examples\.Note:Chapter 9: Importance SamplingExternal Links:[Link](https://artowen.su.domains/mc/Ch-var-is.pdf)Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p2.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p1.1)\. - Panet al\.\(2025\)J\. Pan, X\. Wang, G\. Neubig, N\. Jaitly, H\. Ji, A\. Suhr, and Y\. ZhangTraining software engineering agents and verifiers with SWE\-Gym\.InInternational Conference on Machine Learning,Cited by:[§A\.4](https://arxiv.org/html/2611.21791#A1.SS4.p1.1)\. - Peterset al\.\(2010\)J\. Peters, K\. Mülling, and Y\. AltunRelative entropy policy search\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.24,pp\. 1607–1612\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Prime Intellect Teamet al\.\(2025\)Prime Intellect Team, S\. Jaghouar, J\. Mattern, J\. M\. Ong, J\. Straube, M\. Basra, A\. Pazdera, K\. Thaman, M\. Di Ferrante, F\. Gabriel, F\. Obeid, K\. Erdem, M\. Keiblinger, and J\. HagemannINTELLECT\-2: a reasoning model trained through globally decentralized reinforcement learning\.External Links:2505\.07291Cited by:[§A\.1](https://arxiv.org/html/2611.21791#A1.SS1.p1.1)\. - Qiet al\.\(2025\)P\. Qi, Z\. Liu, X\. Zhou, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinDefeating the training\-inference mismatch via FP16\.External Links:2510\.26788Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1)\. - Qwen Team \(2025\)Qwen TeamQwen3 technical report\.External Links:2505\.09388Cited by:[§A\.3](https://arxiv.org/html/2611.21791#A1.SS3.p1.1)\. - Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.External Links:[Link](https://qwen.ai/blog?id=qwen3.5)Cited by:[§A\.4](https://arxiv.org/html/2611.21791#A1.SS4.p1.1)\. - Rafailovet al\.\(2023\)R\. Rafailov, A\. Sharma, E\. Mitchell, C\. D\. Manning, S\. Ermon, and C\. FinnDirect preference optimization: your language model is secretly a reward model\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 53728–53741\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Schulmanet al\.\(2015\)J\. Schulman, S\. Levine, P\. Abbeel, M\. Jordan, and P\. MoritzTrust region policy optimization\.InProceedings of the 32nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.37,pp\. 1889–1897\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1),[§3\.1](https://arxiv.org/html/2611.21791#S3.SS1.p2.2)\. - Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.External Links:1707\.06347Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p2.1)\. - Stiennonet al\.\(2020\)N\. Stiennon, L\. Ouyang, J\. Wu, D\. Ziegler, R\. Lowe, C\. Voss, A\. Radford, D\. Amodei, and P\. F\. ChristianoLearning to summarize with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 3008–3021\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Trivediet al\.\(2024\)H\. Trivedi, T\. Khot, M\. Hartmann, R\. Manku, V\. Dong, E\. Li, S\. Gupta, A\. Sabharwal, and N\. BalasubramanianAppWorld: a controllable world of apps and people for benchmarking interactive coding agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,Cited by:[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p1.1)\. - Xionget al\.\(2024\)W\. Xiong, H\. Dong, C\. Ye, Z\. Wang, H\. Zhong, H\. Ji, N\. Jiang, and T\. ZhangIterative preference learning from human feedback: bridging theory and practice for RLHF under KL\-constraint\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 54715–54754\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Yaoet al\.\(2025\)F\. Yao, L\. Liu, D\. Zhang, C\. Dong, J\. Shang, and J\. GaoOn the rollout\-training mismatch in modern RL systems\.InNeurIPS 2025 Workshop on Efficient Reasoning,Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p1.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1),[§3\.1](https://arxiv.org/html/2611.21791#S3.SS1.p3.1),[§3\.3](https://arxiv.org/html/2611.21791#S3.SS3.p1.2)\. - Yaoet al\.\(2026\)J\. Yao, R\. Wang, H\. Bai, and T\. ZhangFuture\-KL regularized GRPO: process\-level credit assignment fromff\-divergence regularization\.arXiv preprint arXiv:2601\.10201\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Yaoet al\.\(2022\)S\. Yao, H\. Chen, J\. Yang, and K\. NarasimhanWebShop: towards scalable real\-world web interaction with grounded language agents\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 20744–20757\.Cited by:[§6\.1](https://arxiv.org/html/2611.21791#S6.SS1.p1.1)\. - Yeet al\.\(2024\)C\. Ye, W\. Xiong, Y\. Zhang, H\. Dong, N\. Jiang, and T\. ZhangOnline iterative reinforcement learning from human feedback with general preference model\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 81773–81807\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p2.1)\. - Yeet al\.\(2026\)C\. Ye, X\. Zhang, Y\. Hao, Z\. Yu, Z\. Zhang, A\. Gullapalli, H\. Chen, J\. Huang, and T\. ZhangAdaptive layerwise perturbation: unifying off\-policy corrections for LLM RL\.arXiv preprint arXiv:2603\.19470\.Cited by:[§2](https://arxiv.org/html/2611.21791#S2.p1.1)\. - Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, J\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, R\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, Y\. Wu, and M\. WangDAPO: an open\-source LLM reinforcement learning system at scale\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 113222–113244\.Cited by:[§A\.1](https://arxiv.org/html/2611.21791#A1.SS1.p1.1)\. - Zhaoet al\.\(2025\)H\. Zhao, C\. Ye, Q\. Gu, and T\. ZhangSharp analysis for KL\-regularized contextual bandits and RLHF\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/9b04b2d8c488ed1e943401dcdd927216-Abstract-Conference.html)Cited by:[§4\.2](https://arxiv.org/html/2611.21791#S4.SS2.p3.2)\. - Zhenget al\.\(2025\)C\. Zheng, S\. Liu, M\. Li, X\. Chen, B\. Yu, C\. Gao, K\. Dang, Y\. Liu, R\. Men, A\. Yang, J\. Zhou, and J\. LinGroup sequence policy optimization\.arXiv preprint arXiv:2507\.18071\.Cited by:[§1](https://arxiv.org/html/2611.21791#S1.p2.1),[§2](https://arxiv.org/html/2611.21791#S2.p1.1)\. ## Appendix AExperimental Settings Figures[5](https://arxiv.org/html/2611.21791#S5.F5)through[7](https://arxiv.org/html/2611.21791#S6.F7)compare methods on math and agentic tasks with periodic behavior policy updates, Figure[4](https://arxiv.org/html/2611.21791#S5.F4)compares estimators of the conditional mean score, and Figure[8](https://arxiv.org/html/2611.21791#S6.F8)studies fully asynchronous training for software engineering\. Section[5\.1](https://arxiv.org/html/2611.21791#S5.SS1)describes controlled sampling, and Section[6\.1](https://arxiv.org/html/2611.21791#S6.SS1)describes fully asynchronous data collection\. Final scores use each run’s last evaluation\. Shared settings for controlled experiments\.The controlled experiments in Figures[5](https://arxiv.org/html/2611.21791#S5.F5)through[7](https://arxiv.org/html/2611.21791#S6.F7)use eight responses per prompt and one optimizer update per batch\. The main SC comparisons use 128 independent Monte Carlo samples to estimate the conditional mean score at the history preceding each generated token\. SC sets the importance weight to one, SC\+TIS caps it at two, and RSC adds reverse KL regularization withβ=0\.01\\beta=0\.01and a separate cap of two on the penalty ratio\. SC and SC\+TIS subtract the group mean reward, and RSC additionally appliesG/\(G−1\)G/\(G\-1\)withG=8G=8to match the theoretical normalization\. The paired gradient diagnostics in Figure[3](https://arxiv.org/html/2611.21791#S4.F3)apply this correction to every method\. Appendix[B\.2](https://arxiv.org/html/2611.21791#A2.SS2)tests its effect\. Each batch loss averages over the generated tokens\. Rewards are not divided by their standard deviation\. Entropy bonuses and KL regularization to a fixed reference model are disabled\. IS uses an uncapped token ratio, TIS caps it at five, and Sequence MIS accepts responses whose ratio of response probabilities lies in\[0\.1,10\]\[0\.1,10\]\. These runs and the Countdown estimator comparison in Figure[4](https://arxiv.org/html/2611.21791#S5.F4)sample at temperature one with top\-kkand top\-ppfiltering disabled\. Section[5\.2](https://arxiv.org/html/2611.21791#S5.SS2)describes SC sampling\. Appendix[A\.4](https://arxiv.org/html/2611.21791#A1.SS4)specifies reward normalization and the uncapped penalty for the software engineering experiments\. Sampling the KL penalty\.The score center and KL penalty can both be expressed as conditional expectations over behavior actions\. The experiments use two sampling procedures for the penalty\.\(a\)For Figures[5](https://arxiv.org/html/2611.21791#S5.F5)through[7](https://arxiv.org/html/2611.21791#S6.F7), controlled runs estimate the KL gradient in Equation \([7](https://arxiv.org/html/2611.21791#S3.E7)\) with the generated action, a sample fromμ\(⋅∣h\)\\mu\(\\cdot\\mid h\)\. We multiply the learner’s log\-probability byρKLlogρKL\\rho^\{\\mathrm\{KL\}\}\\log\\rho^\{\\mathrm\{KL\}\}, treating this coefficient as constant during differentiation\. Independent auxiliary samples estimate the SC center\.\(b\)For Figure[8](https://arxiv.org/html/2611.21791#S6.F8), the software engineering experiments use the same 32 auxiliary samples fromμ\\mufor the SC center and the uncapped reverse KL penalty, differentiating the sample mean ofρlogρ\\rho\\log\\rhothrough the learner probabilities withμ\\mufixed\. Both implementations use behavior policy token samples and require no additional rollouts\. Hyperparameter selection\.RSC hyperparameters vary across settings to examine different design choices motivated by our theoretical analysis\. We select these hyperparameters without tuning them for higher benchmark scores\. The empirical improvements reported in this paper are achieved with these untuned configurations\. ### A\.1Math Training dense models and mixtures of experts\.\(a\)For Figure[5](https://arxiv.org/html/2611.21791#S5.F5), Qwen3\-8B\-Base trains on DAPO\-Math\-17K\([Yu et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib23)\)withR=64R=64\.\(b\)For the MoE panels in Figure[6](https://arxiv.org/html/2611.21791#S5.F6), we use the model and dataset studied by[Marek and Ryabinin \(2026\)](https://arxiv.org/html/2611.21791#bib.bib1): Qwen3\-30B\-A3B\-Base and an INTELLECT\-2 math subset\([Prime Intellect Team et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib24)\)\. We select problems whose solve rates recorded in the dataset lie in\[0,0\.375\]\[0,0\.375\]\. On this subset, we vary staleness usingR∈\{64,256,512\}R\\in\\\{64,256,512\\\}\. Both models use 32 prompts per update, responses of at most 16,384 tokens, Adam at10−610^\{\-6\}and gradient norm clipping at one\. Training uses BF16, and rollouts use FP8 weights and activations with BF16 key/value caches\. Verified answers receive\+1\+1and other answers receive−1\-1, giving training accuracy50\(1\+R¯\)%50\(1\+\\overline\{R\}\)\\%\. For each sampler refresh interval, the MoE curves end at the last update reached by every compared run\. We smooth each training run with an exponential moving average \(EMA\) before computing the mean and sample standard deviation across runs, usingα=0\.05\\alpha=0\.05for dense models andα=0\.1\\alpha=0\.1for MoE models\. Each curve averages the same runs throughout\. Math evaluation\.To evaluate the dense models used in Figure[5](https://arxiv.org/html/2611.21791#S5.F5), we measure accuracy on heldout math tasks in BF16, with thinking enabled and at most 16,384 new tokens per response\. MATH500\([Lightman et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib26)\), OlympiadBench\([He et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib27)\), Minerva Math\([Lewkowycz et al\., 2022](https://arxiv.org/html/2611.21791#bib.bib28)\)and Gaokao2023 Math English use one greedy response per question\. AIME24/25 and AMC23/24 use 16 responses at temperature0\.60\.6and topp=0\.95p=0\.95\. The task set contains 1,974 questions and 4,149 responses\. Accuracy is the fraction of correct responses across the task set\. We report its mean and sample standard deviation across training runs\. Comparing estimators of the conditional mean score\.For the Math panels in Figure[4](https://arxiv.org/html/2611.21791#S5.F4), we train Qwen3\-32B with thinking mode disabled on DAPO\-17K withR=64R=64and 32 prompts per update, using the Adam settings and response limit specified in this subsection, BF16 training, and separate BF16 and FP8 rollout configurations\. We compare Top\-128 score estimates with Monte Carlo estimates using 32, 64 or 128 samples, with KL and entropy penalties disabled\. Evaluation uses BF16\. For each benchmark, Average@32 averages correctness over 32 sampled responses per question\. We give equal weight to these benchmark scores on MATH500 and AIME24/25/26\. Training curves use a trailing moving average over 50 updates\. Token\-level gradient measurements\.Figure[3](https://arxiv.org/html/2611.21791#S4.F3)compares a Qwen3\-8B\-Base learner at update 1,500 with a behavior checkpoint at update 1,400, giving a gap of 100 updates independently of the training refresh schedule\. The learner runs in BF16, and behavior sampling uses FP8 weights and activations with BF16 key/value caches\. We sample eight responses for each of 32 DAPO prompts and hold sampled actions, advantages and 128 auxiliary Monte Carlo draws fixed across methods\. Response advantages follow the normalization specified in Appendix[A](https://arxiv.org/html/2611.21791#A1)\. RSC usesβ=0\.1\\beta=0\.1, with both the TIS cap and the cap on the KL ratio set to two\. Token\-levelℓ2\\ell\_\{2\}norms include reward and KL contributions and are measured before optimizer updates and division by the batch token count\. The output logit norm differentiates with respect to the token’s vocabulary logits, and the parameter norm differentiates with respect to model parameters\. Both rows use 25 probability bins of equal width on\[0,1\]\[0,1\]\. Percentiles use linear interpolation and include zero norms and values beyond the displayed axes\. ### A\.2Agentic Tasks Training\.The agentic experiments in Figures[1](https://arxiv.org/html/2611.21791#S0.F1),[5](https://arxiv.org/html/2611.21791#S5.F5)and[7](https://arxiv.org/html/2611.21791#S6.F7)use Qwen3\-4B\-Base, Adam at10−610^\{\-6\}, and responses to 16 tasks per update\. The sampler refresh interval isR=64R=64\. For these agentic comparisons, IS\-PPOClip clipsπθ/πprox\\pi\_\{\\theta\}/\\pi\_\{\\mathrm\{prox\}\}to\[0\.8,1\.2\]\[0\.8,1\.2\]in the PPO objective and refreshesπprox\\pi\_\{\\mathrm\{prox\}\}from the learner every 32 successful optimizer updates\. Its outer importance weightπprox/μ\\pi\_\{\\mathrm\{prox\}\}/\\muis uncapped and held fixed during differentiation\. Training uses BF16, and rollouts use FP8 with BF16 key/value caches\. The loss covers generated action tokens, while observations remain in context, and each episode supplies one scalar reward\. All methods on AppWorld start from a model fine\-tuned for 32 supervised updates on 539 action examples from 49 official training tasks\. We construct these examples from the official training solutions, pairing each action with the preceding visible interaction history\. We use this shared SFT initialization to obtain reasonable task performance before RL\. The other tasks start from the base model\. Table 3:Agentic episode limits\. Token limits for responses and contexts include environment observations\.Tasks and rewards\.For the tasks in Figure[7](https://arxiv.org/html/2611.21791#S6.F7), WebShop uses the full product catalog and 10,587 training goals, with the environment’s partial credit reward at the end of each episode\. AppWorld uses 90 official training tasks and binary task success\. BabyAI uses GoToSeq, OpenDoorsOrderN2, PutNextLocal and UnlockPickup, with partial environment observations rendered as text and seven primitive actions\. The terminal reward is discounted by the number of steps taken\. The training set contains 16,384 combinations of levels and seeds for environment generation\. TextWorld uses 1,000 generated Cooking games with three ingredients, three objects to collect and six rooms\. Reward is the final environment score divided by the maximum score\. Crafter and the full Craftax environment use textual observations and sum the environment’s rewards for achievements and changes in health over each episode\. All tasks allow 4,096 tokens in the initial prompt\. Token budgets may end an episode before its action limit\. BabyAI and Crafter additionally cap the number of generated action attempts at2×2\\timesthe environment’s step limit\. Evaluation and reporting\.Table[1](https://arxiv.org/html/2611.21791#S6.T1)evaluates the policies compared in Figure[7](https://arxiv.org/html/2611.21791#S6.F7)\. Each policy generates one evaluation episode per heldout instance, using greedy decoding in BF16\. We pool the development and test episodes for WebShop, BabyAI, Crafter and Craftax, and the validation and test episodes for TextWorld\. AppWorld uses its development split\. BabyAI, Crafter and Craftax use separate seeds to generate training and evaluation environments\. We report the mean and sample standard deviation of the per\-model scores across training runs\. For Crafter, we first combine achievement counts across evaluation episodes, then compute its geometric achievement score\. Craftax final scores average the total environment reward per episode\. All episodes enter the score, including failures and truncations\. For runs stopped because of low training scores, we evaluate the last available checkpoint\. ### A\.3Countdown Training\.For the Countdown experiments in Figures[4](https://arxiv.org/html/2611.21791#S5.F4)and[6](https://arxiv.org/html/2611.21791#S5.F6), we use the Countdown task studied by[Marek and Ryabinin \(2026\)](https://arxiv.org/html/2611.21791#bib.bib1)and initialize the models from Qwen3\-0\.6B\([Qwen Team, 2025](https://arxiv.org/html/2611.21791#bib.bib22)\)\. From a shuffled dataset of 20,000 problems, we reserve 64 prompts for evaluation\. Each update uses 64 training prompts and SGD without momentum at0\.010\.01, with zero weight decay, BF16 computation and an FP32 copy of the weights for optimizer updates\. Thinking mode is disabled\. The limit of 512 tokens includes the prompt\. The binary reward checks whether the response reaches the target using the supplied numbers\. We scale the group\-centered reward advantage by two\. The comparison of SC, SC\+TIS and the two RSC variants usesR∈\{16,32,65\}R\\in\\\{16,32,65\\\}, with automatic early stopping for low training scores disabled\. Evaluation\.For the Countdown evaluation in Figure[4](https://arxiv.org/html/2611.21791#S5.F4)and the staleness comparison discussed alongside Figure[6](https://arxiv.org/html/2611.21791#S5.F6), we sample eight responses per heldout prompt every 25 updates and before the final update\. The estimator comparison uses Top\-128 with a modeled tail, MC128 and MC512 withR=65R=65\. ### A\.4Fully Asynchronous Software Engineering For Figure[8](https://arxiv.org/html/2611.21791#S6.F8), Qwen3\.5\-35B\-A3B\-Base\([Qwen Team, 2026](https://arxiv.org/html/2611.21791#bib.bib25)\)trains on 2,394 SWE\-Gym tasks\([Pan et al\., 2025](https://arxiv.org/html/2611.21791#bib.bib16)\)with BF16 training and generation\. We exclude a small number of tasks because of environment setup failures or failure of the provided gold solution to pass validation: 44 of the 2,438 SWE\-Gym tasks and two of the 500 SWE\-bench Verified tasks\. Each batch contains 16 tasks with eight trajectories each and receives one Adam update\. Up to 1,024 trajectories may be in progress concurrently, increasing policy staleness\. Table[4](https://arxiv.org/html/2611.21791#A1.T4)defines the two configurations and their learning rates\. Table 4:Asynchronous SWE configurations\. Trajectory budgets include tool observations\.Both configurations in Figure[8](https://arxiv.org/html/2611.21791#S6.F8)allow 262,144 total context tokens and 32,768 generated tokens per model call, with up to 131,072 tokens after the initial prompt and an agent time limit of 4,800 seconds\. We compute advantages by subtracting the group mean from each reward and dividing by the sample standard deviation plus10−610^\{\-6\}\. SWE does not apply theG/\(G−1\)G/\(G\-1\)correction\. We average the loss over generated action tokens within each trajectory, then average over trajectories\. RSC uses uncapped reverse KL withβ=0\.001\\beta=0\.001, while SC\+TIS and RSC \(TIS\) cap the reward importance ratio at two\. KL regularization to a fixed reference model and entropy bonuses are disabled\. Table[2](https://arxiv.org/html/2611.21791#S6.T2)evaluates the Figure[8](https://arxiv.org/html/2611.21791#S6.F8)models on 498 SWE\-bench Verified tasks\([Jimenez et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib17);[Chowdhury et al\., 2024](https://arxiv.org/html/2611.21791#bib.bib29)\), with eight trajectories per task at temperature one\. Average@8 is the mean success rate across these trajectories and tasks\. Figure[8](https://arxiv.org/html/2611.21791#S6.F8)smooths training rewards and log\-probability differences with EMA \(α=0\.05\\alpha=0\.05\)\. ## Appendix BAdditional Analysis Figure 9:Clipping the KL penalty on BabyAI\.Mean±\\pmsample standard deviation across three seeds\.The controlled experiments use a fixed cap on the KL penalty and a correction for the group reward baseline\. We examine sensitivity to these choices before listing the surrogate formulas used in the gradient analysis\. ### B\.1Sensitivity to Clipping the KL Penalty Clipping the KL penalty has a modest effect\.Figure[9](https://arxiv.org/html/2611.21791#A2.F9)compares capped and uncapped KL on BabyAI\. Both RSC variants learn with either choice, with similar training curves and small differences in mean reward\. The cap used in the main experiments was fixed before this ablation and was not tuned against the uncapped results\. ### B\.2Testing Sensitivity to Group\-Size Correction Group\-size correction has no consistent advantage across tasks and variants\.Figure[10](https://arxiv.org/html/2611.21791#A2.F10)compares RSC with and without the correction defined in Appendix[A](https://arxiv.org/html/2611.21791#A1), keeping the group size and KL coefficient fixed\. Results do not support a uniform conclusion across tasks: the effect of group\-size correction differs between AppWorld and BabyAI\. Figure 10:Sensitivity to group\-size correction\.Solid and dashed curves compare each RSC variant with and without the group\-size correction at fixed group size\. Curves and bands show mean training reward±\\pmsample standard deviation across three seeds\. ### B\.3Surrogate Formulas Table[5](https://arxiv.org/html/2611.21791#A2.T5)expands the surrogate integrands and logit derivatives in Figure[2](https://arxiv.org/html/2611.21791#S4.F2)\. Table 5:Surrogate integrands and logit derivatives\.Hereρ=πθ\(a∣h\)/μ\(a∣h\)\\rho=\\pi\_\{\\theta\}\(a\\mid h\)/\\mu\(a\\mid h\),A=Aμ\(h,a\)A=A^\{\\mu\}\(h,a\)andc≥1c\\geq 1\. The last column factors out1−πθ\(a∣h\)1\-\\pi\_\{\\theta\}\(a\\mid h\)from the logit derivative, withμ\\mu,AAand the other logits fixed\. RSC usesβ\>0\\beta\>0and standard, uncapped reverse KL\. ## Appendix CConvergence with and without KL regularization We analyze policy updates with no regularization \(β=0\\beta=0\) and with KL regularization \(β\>0\\beta\>0\), for unit or token importance sampling \(TIS\) reward weights\. Theorems[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)and[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)state the main regularized convergence results after the setup below\. The analysis has three parts\. Section[C\.1](https://arxiv.org/html/2611.21791#A3.SS1)defines the objectives and policy updates and explains the benefit of KL regularization\. Section[C\.2](https://arxiv.org/html/2611.21791#A3.SS2)proves how population fitting accuracy implies reward progress\. Section[C\.3](https://arxiv.org/html/2611.21791#A3.SS3)bounds the population error of a policy fitted to a fixed dataset and concludes with the combined reward and sample guarantees\. ### C\.1Setup and main convergence results A promptx∼𝒟x\\sim\\mathcal\{D\}generates a responseY=\(a1,…,aH\)Y=\(a\_\{1\},\\ldots,a\_\{H\}\)on a finite deterministic token tree, with binary terminal rewardR\(x,Y\)∈\{0,1\}R\(x,Y\)\\in\\\{0,1\\\}\. HereHHis the horizon andht=\(x,a<t\)h\_\{t\}=\(x,a\_\{<t\}\)is the history at token positiontt\. Policies have common full support; terminal histories are padded with deterministic actions\. Start from a full\-support policyπ0\\pi\_\{0\}\. At outer roundk=0,…,K−1k=0,\\ldots,K\-1, the behavior policy is fixed atμ=πk\\mu=\\pi\_\{k\}, andπ\\pidenotes a candidate next policy\. HereKKis the total number of outer rounds\. Define its token ratio and the behavior continuation values by ρt\(h,a\)\\displaystyle\\rho\_\{t\}\(h,a\)=π\(a∣h\)μ\(a∣h\),\\displaystyle=\\frac\{\\pi\(a\\mid h\)\}\{\\mu\(a\\mid h\)\},Qtμ\(h,a\)\\displaystyle Q\_\{t\}^\{\\mu\}\(h,a\)=𝔼μ\[R∣ht=h,at=a\],\\displaystyle=\\mathbb\{E\}\_\{\\mu\}\[R\\mid h\_\{t\}=h,a\_\{t\}=a\],Vtμ\(h\)\\displaystyle V\_\{t\}^\{\\mu\}\(h\)=𝔼a∼μ\(⋅∣h\)Qtμ\(h,a\),\\displaystyle=\\mathbb\{E\}\_\{a\\sim\\mu\(\\cdot\\mid h\)\}Q\_\{t\}^\{\\mu\}\(h,a\),Atμ\(h,a\)\\displaystyle A\_\{t\}^\{\\mu\}\(h,a\)=Qtμ\(h,a\)−Vtμ\(h\)\.\\displaystyle=Q\_\{t\}^\{\\mu\}\(h,a\)\-V\_\{t\}^\{\\mu\}\(h\)\.\(C\.1\)Writedtμd\_\{t\}^\{\\mu\}for the distribution of histories underμ\\mu, including the prompt distribution, and𝔼μ\\mathbb\{E\}\_\{\\mu\}for expectation over prompts and responses sampled fromμ\\mu\. The reward of a policy isJ\(π\)=𝔼πRJ\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}R, withJx\(π\)=𝔼π\[R∣x\]J\_\{x\}\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}\[R\\mid x\]for a fixed prompt\. Writesk\(x\)=Jx\(πk\)s\_\{k\}\(x\)=J\_\{x\}\(\\pi\_\{k\}\)andμt=μ\(⋅∣ht\)\\mu\_\{t\}=\\mu\(\\cdot\\mid h\_\{t\}\)\. For any other policyπ\\pi, defineQtπQ\_\{t\}^\{\\pi\},VtπV\_\{t\}^\{\\pi\}, andAtπA\_\{t\}^\{\\pi\}by replacingμ\\muin Equation \([C\.1](https://arxiv.org/html/2611.21791#A3.E1)\); at a terminal history,VH\+1π=RV\_\{H\+1\}^\{\\pi\}=R\. Whenπ=πθ\\pi=\\pi\_\{\\theta\}, its score isψθ\(h,u\)=∇θlogπθ\(u∣h\)\\psi\_\{\\theta\}\(h,u\)=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(u\\mid h\), as in Equation \([1](https://arxiv.org/html/2611.21791#S3.E1)\)\. ##### Population objective\. Use the effective importance ratio of Section[3\.1](https://arxiv.org/html/2611.21791#S3.SS1):a\(ρ\)=ρTISa\(\\rho\)=\\rho^\{\\mathrm\{TIS\}\}\. For the one\-sided TIS rule analyzed here,a\(ρ\)=min\(ρ,c\)a\(\\rho\)=\\min\(\\rho,c\)with capc≥1c\\geq 1; unit\-weight SC setsa\(ρ\)=1a\(\\rho\)=1\. Integrating this weight gives a potentialFaF\_\{a\}and the centered reward objective: Fa\(ρ\)=∫1ρa\(u\)u𝑑u,ρFa′\(ρ\)=a\(ρ\)\.F\_\{a\}\(\\rho\)=\\int\_\{1\}^\{\\rho\}\\frac\{a\(u\)\}\{u\}\\,du,\\qquad\\rho F\_\{a\}^\{\\prime\}\(\\rho\)=a\(\\rho\)\.\(C\.2\)𝒞a\(π\)=𝔼μ∑t=1HAtμ\(ht,at\)Fa\(π\(at∣ht\)μ\(at∣ht\)\)\.\\mathcal\{C\}\_\{a\}\(\\pi\)=\\mathbb\{E\}\_\{\\mu\}\\sum\_\{t=1\}^\{H\}A\_\{t\}^\{\\mu\}\(h\_\{t\},a\_\{t\}\)F\_\{a\}\\\!\\left\(\\frac\{\\pi\(a\_\{t\}\\mid h\_\{t\}\)\}\{\{\\mu\}\(a\_\{t\}\\mid h\_\{t\}\)\}\\right\)\.\(C\.3\)The behavior policyμ\\mu, its history distribution, and its advantages stay fixed during fitting\. In particular,Fa\(ρ\)=logρF\_\{a\}\(\\rho\)=\\log\\rhofor SC andFa\(ρ\)=ρ−1F\_\{a\}\(\\rho\)=\\rho\-1for TIS whenever the cap is inactive\. Use the independent penalty ratio from Section[3\.2](https://arxiv.org/html/2611.21791#S3.SS2),b\(ρ\)=ρKL=min\(ρ,cb\)b\(\\rho\)=\\rho^\{\\mathrm\{KL\}\}=\\min\(\\rho,c\_\{b\}\)withcb∈\[1,∞\]c\_\{b\}\\in\[1,\\infty\]\. An infinite cap gives standard reverse KL\. Section[C\.4](https://arxiv.org/html/2611.21791#A3.SS4)gives the explicit integrands and derivatives used in the main text\. The penalty potential is Kb\(ρ\)=∫1ρb\(v\)logb\(v\)v𝑑v\.K\_\{b\}\(\\rho\)=\\int\_\{1\}^\{\\rho\}\\frac\{b\(v\)\\log b\(v\)\}\{v\}\\,dv\.\(C\.4\)At roundkk, write𝒞k\\mathcal\{C\}\_\{k\}for the centered reward objective withμ=πk\\mu=\\pi\_\{k\}\. The weightsaa,bb, and coefficientβ\\betaare fixed within each argument, so we omit them from the subscripts of𝒞k\\mathcal\{C\}\_\{k\}and the regularized objective𝒲k\\mathcal\{W\}\_\{k\}\. For a token log\-ratio radiusr\>0r\>0, the full feasible family is Γk\(r\)=\{π:\|logπ\(u∣h\)πk\(u∣h\)\|≤rfor every reachable historyhand available tokenu\}\.\\Gamma\_\{k\}\(r\)=\\left\\\{\\pi:\\left\|\\log\\frac\{\\pi\(u\\mid h\)\}\{\\pi\_\{k\}\(u\\mid h\)\}\\right\|\\leq r\\ \\text\{for every reachable history $h$ and available token $u$\}\\right\\\}\.\(C\.5\)Each member is a normalized token policy\. WriteΓk=Γk\(r\)\\Gamma\_\{k\}=\\Gamma\_\{k\}\(r\)when the radius is fixed\. With regularization coefficientβ≥0\\beta\\geq 0, the population fitting problem is supπ∈Γk𝒲k\(π\),𝒲k\(π\)=𝒞k\(π\)−β𝔼μ∑tKb\(ρt\)\.\\sup\_\{\\pi\\in\\Gamma\_\{k\}\}\\mathcal\{W\}\_\{k\}\(\\pi\),\\qquad\\mathcal\{W\}\_\{k\}\(\\pi\)=\\mathcal\{C\}\_\{k\}\(\\pi\)\-\\beta\\mathbb\{E\}\_\{\\mu\}\\sum\_\{t\}K\_\{b\}\(\\rho\_\{t\}\)\.\(C\.6\)Hereρt=π\(at∣ht\)/μ\(at∣ht\)\\rho\_\{t\}=\\pi\(a\_\{t\}\\mid h\_\{t\}\)/\\mu\(a\_\{t\}\\mid h\_\{t\}\)in the expectation\. Forβ=0\\beta=0, this is just𝒞k\\mathcal\{C\}\_\{k\}\. When the penalty cap is inactive,Kb\(ρ\)=ρlogρ−ρ\+1K\_\{b\}\(\\rho\)=\\rho\\log\\rho\-\\rho\+1, so the same objective is 𝒲k\(π\)=𝒞k\(π\)−β∑t=1H𝔼ht∼dtμKL\(π\(⋅∣ht\)∥μ\(⋅∣ht\)\)\.\\mathcal\{W\}\_\{k\}\(\\pi\)=\\mathcal\{C\}\_\{k\}\(\\pi\)\-\\beta\\sum\_\{t=1\}^\{H\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\}\\operatorname\{KL\}\\bigl\(\\pi\(\\cdot\\mid h\_\{t\}\)\\\|\\mu\(\\cdot\\mid h\_\{t\}\)\\bigr\)\.\(C\.7\)Equation \([C\.7](https://arxiv.org/html/2611.21791#A3.E7)\) is the main\-text surrogate in Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\), with the round index made explicit\. Equation \([C\.6](https://arxiv.org/html/2611.21791#A3.E6)\) also covers an active penalty cap throughKbK\_\{b\}\. Thus the penalty compares current\-token distributions on behavior histories\. The objective is defined for everyβ≥0\\beta\\geq 0\. The unregularized analysis usesr=1/\(8H\)r=1/\(8H\), while the regularized theorems below specify a sufficient coefficient and user=1/2r=1/2\. At this constant radius, reward and penalty caps of two are inactive throughout the feasible family\. ##### Empirical objective and the update loop\. The inner fit estimates this objective using response rewards and token probabilities\. For a behavior response, define Zπ\(Y\)\\displaystyle Z\_\{\\pi\}\(Y\)=∑t=1H\[Fa\(ρt\(ht,at\)\)−𝔼u∼μ\(⋅∣ht\)Fa\(ρt\(ht,u\)\)\],\\displaystyle=\\sum\_\{t=1\}^\{H\}\\left\[F\_\{a\}\(\\rho\_\{t\}\(h\_\{t\},a\_\{t\}\)\)\-\\mathbb\{E\}\_\{u\\sim\\mu\(\\cdot\\mid h\_\{t\}\)\}F\_\{a\}\(\\rho\_\{t\}\(h\_\{t\},u\)\)\\right\],Bπ\(Y\)\\displaystyle B\_\{\\pi\}\(Y\)=∑t=1HKb\(ρt\(ht,at\)\)\.\\displaystyle=\\sum\_\{t=1\}^\{H\}K\_\{b\}\(\\rho\_\{t\}\(h\_\{t\},a\_\{t\}\)\)\.\(C\.8\)At roundkk, drawmmindependent prompts andG≥2G\\geq 2independent responsesYjgY\_\{jg\}fromμ\\mufor each promptxjx\_\{j\}\. LetRjg=R\(xj,Yjg\)R\_\{jg\}=R\(x\_\{j\},Y\_\{jg\}\)andR¯j=G−1∑gRjg\\overline\{R\}\_\{j\}=G^\{\-1\}\\sum\_\{g\}R\_\{jg\}\. Use 𝒲^k\(π\)=1m∑j=1m\[1G−1∑g=1G\(Rjg−R¯j\)Zπ\(Yjg\)−βG∑g=1GBπ\(Yjg\)\]\.\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\)=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\left\[\\frac\{1\}\{G\-1\}\\sum\_\{g=1\}^\{G\}\(R\_\{jg\}\-\\overline\{R\}\_\{j\}\)Z\_\{\\pi\}\(Y\_\{jg\}\)\-\\frac\{\\beta\}\{G\}\\sum\_\{g=1\}^\{G\}B\_\{\\pi\}\(Y\_\{jg\}\)\\right\]\.\(C\.9\)The reward term is the unbiased sample covariance of reward and centered potential; the penalty is a response average\. Hence its expectation is𝒲k\(π\)\\mathcal\{W\}\_\{k\}\(\\pi\)for each fixed candidate\. Whenβ=0\\beta=0, denote this empirical objective by𝒞^k\\widehat\{\\mathcal\{C\}\}\_\{k\}\. Differentiating Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) gives the grouped gradient in Equation \([C\.38](https://arxiv.org/html/2611.21791#A3.E38)\)\. It is the sum of the main\-text contributions in Equation \([7](https://arxiv.org/html/2611.21791#S3.E7)\), with the corrected group rewardAi=G\(Ri−R¯\)/\(G−1\)A\_\{i\}=G\(R\_\{i\}\-\\overline\{R\}\)/\(G\-1\), exact conditional centers, and the generated action used to sample the penalty\. Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15)establishes both the exact empirical gradient identity and its population expectation\. Independent Monte Carlo centers give the corresponding sampled\-potential objective; Section[A](https://arxiv.org/html/2611.21791#A1)describes the implementation choices for sampling and reducing token contributions\. The inner solver holdsμ\\muand this dataset fixed and returnsπk\+1∈Γk\\pi\_\{k\+1\}\\in\\Gamma\_\{k\}satisfying 𝒲^k\(πk\+1\)≥supπ∈Γk𝒲^k\(π\)−ek\.\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\_\{k\+1\}\)\\geq\\sup\_\{\\pi\\in\\Gamma\_\{k\}\}\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\)\-e\_\{k\}\.\(C\.10\)The toleranceeke\_\{k\}includes optimization error and representation error relative to the full family\. The outer loop then setsμ=πk\+1\\mu=\\pi\_\{k\+1\}and collects a fresh dataset\. AfterKKrounds, the budget isKmKmprompt draws andKmGKmGresponses\. ##### Main convergence results\. We compare the reward of one response from the fitted policy with best\-of\-NNsampling, whereNNis a positive integer counting independent responses from the initial policy\. The benchmark reward is VN\(π0\)=𝔼x\[1−\(1−Jx\(π0\)\)N\]\.V\_\{N\}\(\\pi\_\{0\}\)=\\mathbb\{E\}\_\{x\}\\left\[1\-\(1\-J\_\{x\}\(\\pi\_\{0\}\)\)^\{N\}\\right\]\.\(C\.11\)To state the sample bounds, measure the distance between policies by d∞\(π,π′\)=supt,h,u\|logπ\(u∣h\)−logπ′\(u∣h\)\|\.d\_\{\\infty\}\(\\pi,\\pi^\{\\prime\}\)=\\sup\_\{t,h,u\}\|\\log\\pi\(u\\mid h\)\-\\log\\pi^\{\\prime\}\(u\\mid h\)\|\.\(C\.12\)A radius\-vvcover approximates every feasible policy within distancevv, with cover centers in the feasible family\. The covering number is the minimum number of such centers\. Let𝒩\(v\)\\mathcal\{N\}\(v\)be a common finite nonincreasing upper bound on these covering numbers across rounds\. This bound holds conditional on the history before each round; the cover may depend on that history but not on the fresh data\. The first result concerns SC\+TIS\+KL with exact conditional centers\. The second concerns unit\-weight SC\+KL and also allows Monte Carlo centers\. For a positive integerMM, the number of auxiliary tokens per visited history, replace the conditional expectation inZπZ\_\{\\pi\}from Equation \([C\.8](https://arxiv.org/html/2611.21791#A3.E8)\) by 𝔼^μ\[Fa\(ρt\(ht,u\)\)∣ht\]=1M∑i=1MFa\(ρt\(ht,Ut,i\)\),Ut,i∼μ\(⋅∣ht\)\.\\widehat\{\\mathbb\{E\}\}\_\{\\mu\}\[F\_\{a\}\(\\rho\_\{t\}\(h\_\{t\},u\)\)\\mid h\_\{t\}\]=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}F\_\{a\}\(\\rho\_\{t\}\(h\_\{t\},U\_\{t,i\}\)\),\\qquad U\_\{t,i\}\\sim\\mu\(\\cdot\\mid h\_\{t\}\)\.\(C\.13\)Conditional on each visited history, these auxiliary draws are independent of the response’s current and future actions\. Use independent randomness at different positions and for different responses, and hold all responses and auxiliary draws fixed during fitting\. WriteM=∞M=\\inftyfor exact conditional expectations, with1/M=01/M=0\. Both theorems use the finite deterministic token tree, binary reward, and common full support specified at the start of this subsection\. The target reward error isϵ\\epsilon, andδ\\deltais the allowed failure probability over the sampled prompts, responses, and auxiliary tokens when present\. The fitted policy is evaluated by a single fresh response, throughJ\(πK\)J\(\\pi\_\{K\}\)\. ###### Theorem C\.1\(Horizon\-independent outer rounds with regularization\)\. Usea\(ρ\)=b\(ρ\)=min\(ρ,2\)a\(\\rho\)=b\(\\rho\)=\\min\(\\rho,2\)andβ=2\\beta=2\. At each round, use the radius\-1/21/2familyΓk\\Gamma\_\{k\}in Equation \([C\.5](https://arxiv.org/html/2611.21791#A3.E5)\)\. ForN≥1N\\geq 1,0<ϵ<10<\\epsilon<1, and0<δ<10<\\delta<1, choose K=⌈4log6Nϵ⌉,ek≤ϵ160π2NK2\.K=\\left\\lceil 4\\log\\frac\{6N\}\{\\epsilon\}\\right\\rceil,\\qquad e\_\{k\}\\leq\\frac\{\\epsilon\}\{160\\pi^\{2\}NK^\{2\}\}\.At every round, drawmmfresh independent prompts from𝒟\\mathcal\{D\}andG≥2G\\geq 2independent responses per prompt fromμ=πk\\mu=\\pi\_\{k\}\. Use exact conditional centers in Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) and suppose the returned policy satisfies Equation \([C\.10](https://arxiv.org/html/2611.21791#A3.E10)\)\. If m≥⌈32K\+1760π2NK2H/⌊G/2⌋ϵlog\[4Kδ𝒩\(ϵ2160π2NK2H\)\]⌉,m\\geq\\left\\lceil\\frac\{32K\+1760\\pi^\{2\}NK^\{2\}H/\\lfloor G/2\\rfloor\}\{\\epsilon\}\\log\\\!\\left\[\\frac\{4K\}\{\\delta\}\\,\\mathcal\{N\}\\\!\\left\(\\frac\{\\epsilon\}\{2160\\pi^\{2\}NK^\{2\}H\}\\right\)\\right\]\\right\\rceil,\(C\.14\)thenVN\(π0\)−J\(πK\)≤ϵV\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilonwith probability at least1−δ1\-\\delta\. ###### Theorem C\.2\(RSC \(w/o IS\) with finite samples and Monte Carlo centers\)\. LetN≥1N\\geq 1,0<ϵ<10<\\epsilon<1,0<δ<10<\\delta<1,G≥2G\\geq 2, andM∈ℕ∪\{∞\}M\\in\\mathbb\{N\}\\cup\\\{\\infty\\\}\. Usea\(ρ\)=1a\(\\rho\)=1,b\(ρ\)=min\(ρ,2\)b\(\\rho\)=\\min\(\\rho,2\),β=2\\beta=2, and the radius\-1/21/2family in Equation \([C\.5](https://arxiv.org/html/2611.21791#A3.E5)\)\. Use Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) with the conditional centers in Equation \([C\.13](https://arxiv.org/html/2611.21791#A3.E13)\)\. Set K\\displaystyle K=⌈log\(6N/ϵ\)log\(67/60\)⌉,\\displaystyle=\\left\\lceil\\frac\{\\log\(6N/\\epsilon\)\}\{\\log\(67/60\)\}\\right\\rceil,κ\\displaystyle\\kappa=4π2NK,\\displaystyle=4\\pi^\{2\}NK,BM\\displaystyle B\_\{M\}=24\+8e\(1\+1/M\),\\displaystyle=24\+8e\(1\+1/M\),ν\\displaystyle\\nu=ϵ288κHK\.\\displaystyle=\\frac\{\\epsilon\}\{288\\kappa HK\}\.\(C\.15\)At each round, drawmmfresh independent prompts from𝒟\\mathcal\{D\}andGGindependent responses per prompt fromμ=πk\\mu=\\pi\_\{k\}, with auxiliary tokens sampled as specified above\. Suppose the fitting condition in Equation \([C\.10](https://arxiv.org/html/2611.21791#A3.E10)\) holds withek≤ϵ/\(32κK\)e\_\{k\}\\leq\\epsilon/\(32\\kappa K\)\. If the number of prompt draws per round satisfies m≥⌈log\[4K𝒩\(ν\)/δ\]ϵ\(32K\+16κBMHK⌊G/2⌋\)⌉,m\\geq\\left\\lceil\\frac\{\\log\[4K\\mathcal\{N\}\(\\nu\)/\\delta\]\}\{\\epsilon\}\\left\(32K\+\\frac\{16\\kappa B\_\{M\}HK\}\{\\lfloor G/2\\rfloor\}\\right\)\\right\\rceil,\(C\.16\)then, with probability at least1−δ1\-\\delta, VN\(π0\)−J\(πK\)≤ϵ,VN\(π0\)=𝔼x\[1−\(1−Jx\(π0\)\)N\]\.V\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilon,\\qquad V\_\{N\}\(\\pi\_\{0\}\)=\\mathbb\{E\}\_\{x\}\\\!\\left\[1\-\(1\-J\_\{x\}\(\\pi\_\{0\}\)\)^\{N\}\\right\]\.\(C\.17\) Theorems[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)and[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)give sufficient outer\-round counts independent of the horizon, under their stated fitting conditions\. Sections[C\.3\.6](https://arxiv.org/html/2611.21791#A3.SS3.SSS6)and[C\.3\.7](https://arxiv.org/html/2611.21791#A3.SS3.SSS7)combine the outer and inner lemmas to prove these results\. The unregularized comparison appears in Theorem[C\.29](https://arxiv.org/html/2611.21791#A3.Thmtheorem29)\. The guarantees concern the accuracy achieved by fitting a fixed dataset; they do not specify how many optimizer iterations are needed\. Each outer round uses fresh data from the current policy\. Horizon\-independent round counts therefore still have horizon\-dependent sampling costs\. ##### Relation to the experimental coefficient\. The coefficient fixed in these theorems is a sufficient analytical choice\. The controlled training experiments useβ=0\.01\\beta=0\.01, and asynchronous SWE usesβ=0\.001\\beta=0\.001\(Section[A](https://arxiv.org/html/2611.21791#A1)\)\. The objectives and gradient identities apply at these smaller coefficients, but the stated convergence and sample bounds use the theorem assumptions\. Extending those bounds to smaller positive coefficients requires tracking their effect on curvature, target feasibility, and reward transfer\. ##### Benefit of KL regularization\. Without KL regularization, we control the change in visited histories using the token log\-ratio bound\. A response ratio is a product ofHHtoken ratios and can be as large aseHre^\{Hr\}\. This bound is sharp, so to keep sequence ratios uniformly bounded using the token constraint alone, we must chooser=O\(1/H\)r=O\(1/H\)\(Lemma[C\.5](https://arxiv.org/html/2611.21791#A3.Thmtheorem5)\)\. We user=1/\(8H\)r=1/\(8H\), which also controls the local objective\-to\-reward approximation error\. The resulting sufficient round count isO\(Hlog\(N/ϵ\)\)O\(H\\log\(N/\\epsilon\)\)for reward accuracyϵ\\epsilonagainst the best\-of\-NNbenchmark in Equation \([C\.11](https://arxiv.org/html/2611.21791#A3.E11)\)\. Forβ\>0\\beta\>0, KL adds a penalty for policy movement\. The following two mechanisms yield the improved bounds under the assumptions of Theorems[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)and[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)\. 1. 1\.Inner fitting: objective accuracy controls distance to the target\.For TIS with inactive caps, the population maximizer has token probabilities proportional toμ\(u∣h\)exp\(Qtμ\(h,u\)/β\)\\mu\(u\\mid h\)\\exp\(Q\_\{t\}^\{\\mu\}\(h,u\)/\\beta\)\. The objective gap equalsβ\\betatimes the conditional KL from the candidate to this target, averaged over behavior histories \(Lemma[C\.10](https://arxiv.org/html/2611.21791#A3.Thmtheorem10)\)\. For unit\-weight SC, curvature also makes the gap control squared token\-ratio error \(Lemma[C\.26](https://arxiv.org/html/2611.21791#A3.Thmtheorem26)\)\. In both cases, the second moment of the sampled objective difference is bounded by a constant timesHHtimes the population gap \(Lemmas[C\.22](https://arxiv.org/html/2611.21791#A3.Thmtheorem22)and[C\.27](https://arxiv.org/html/2611.21791#A3.Thmtheorem27)\)\. Thus the noise in comparing a candidate with the target decreases as the fit improves, which reduces the samples needed to certify an accurate fit\. 2. 2\.Outer transfer: fitting accuracy controls reward at a constant radius\.For the regularized TIS target, continuation\-value differences telescope along each response\. Consequently, its target\-to\-behavior ratio is at moste1/βe^\{1/\\beta\}for every prefix and full response at token radius1/β1/\\betawhen the caps are inactive and the target is feasible \(Lemma[C\.12](https://arxiv.org/html/2611.21791#A3.Thmtheorem12)\)\. For fixedβ\\beta, this bound is independent ofHH, whereas the token constraint alone permitseH/βe^\{H/\\beta\}\. This coverage bound converts the objective gap into response\-level reward control \(Lemma[C\.13](https://arxiv.org/html/2611.21791#A3.Thmtheorem13)\)\. The ratio bound applies to the target; an approximate fit is controlled through its objective gap\. Unit\-weight SC uses a different transfer argument: under the stated coefficient and radius conditions, its regularized objective directly lower\-boundsJ\(π\)−J\(μ\)J\(\\pi\)\-J\(\\mu\)\(Lemma[C\.8](https://arxiv.org/html/2611.21791#A3.Thmtheorem8)\), so it does not require a sequence\-ratio bound for its optimizer\. For the settings in the main theorems, the TIS target increases success odds by a constant factor, and a feasible unit\-weight SC perturbation gains a constant fraction of reward variance \(Lemmas[C\.11](https://arxiv.org/html/2611.21791#A3.Thmtheorem11)and[C\.9](https://arxiv.org/html/2611.21791#A3.Thmtheorem9)\)\. Together, the inner fitting certificates and outer progress bounds giveO\(log\(N/ϵ\)\)O\(\\log\(N/\\epsilon\)\)sufficient outer rounds, compared withO\(Hlog\(N/ϵ\)\)O\(H\\log\(N/\\epsilon\)\)for the unregularized analysis\. Up to logarithmic and covering\-number factors, the sufficient total prompt bounds change from\(H2\+NH3/G\)/ϵ\(H^\{2\}\+NH^\{3\}/G\)/\\epsilonto\(1\+NH/G\)/ϵ\(1\+NH/G\)/\\epsilon, whereGGis the number of responses per prompt; the latter statement uses a fixed number of auxiliary center samples for unit\-weight SC\. The precise budgets are in Equations \([C\.57](https://arxiv.org/html/2611.21791#A3.E57)\), \([C\.58](https://arxiv.org/html/2611.21791#A3.E58)\), and \([C\.60](https://arxiv.org/html/2611.21791#A3.E60)\)\. ### C\.2Outer\-loop reward improvement We first state the common lemma that accumulates one\-round reward errors\. We then prove the progress bounds for unregularized SC and TIS, unit\-weight KL regularization, and TIS with KL regularization\. In each case, the fitting error is measured by the corresponding population objective defined in Section[C\.1](https://arxiv.org/html/2611.21791#A3.SS1)\. #### C\.2\.1Angular transfer for reward improvement This lemma accumulates one\-round errors relative to a fixed increase in success odds\. Its two deficit bounds allow the later proofs to use either a KL error or a direct reward shortfall\. ###### Lemma C\.3\(Relative angular transfer, adapted from[Bai et al\. \(2026b\)](https://arxiv.org/html/2611.21791#bib.bib9)\)\. Letsk\(x\)=Jx\(πk\)s\_\{k\}\(x\)=J\_\{x\}\(\\pi\_\{k\}\)be success probabilities and fix a log\-odds incrementλ\>0\\lambda\>0\. Writeαk=arcsinsk\\alpha\_\{k\}=\\arcsin\\sqrt\{s\_\{k\}\},T\(α\)=arctan\(eλ/2tanα\)T\(\\alpha\)=\\arctan\(e^\{\\lambda/2\}\\tan\\alpha\), so thatsin2T\(αk\)\\sin^\{2\}T\(\\alpha\_\{k\}\)has log\-oddslogitsk\+λ\\operatorname\{logit\}s\_\{k\}\+\\lambda, and define the one\-sided angular deficit ξk\(x\)=\[T\(αk\(x\)\)−αk\+1\(x\)\]\+\.\\xi\_\{k\}\(x\)=\[T\(\\alpha\_\{k\}\(x\)\)\-\\alpha\_\{k\+1\}\(x\)\]\_\{\+\}\.Puts\(x\)=1−\(1−s0\(x\)\)Ns\(x\)=1\-\(1\-s\_\{0\}\(x\)\)^\{N\}\. ForN≥1N\\geq 1,0<ϵ<10<\\epsilon<1, andK≥⌈λ−1log\(3N/ϵ\)⌉K\\geq\\lceil\\lambda^\{\-1\}\\log\(3N/\\epsilon\)\\rceil, at every prompt, s\(x\)−sK\(x\)≤ϵ\+min\{1,16N\(∑k<Kξk\(x\)\)2\}≤ϵ\+min\{1,16NK∑k<Kξk\(x\)2\}\.s\(x\)\-s\_\{K\}\(x\)\\leq\\epsilon\+\\min\\left\\\{1,16N\\left\(\\sum\_\{k<K\}\\xi\_\{k\}\(x\)\\right\)^\{2\}\\right\\\}\\leq\\epsilon\+\\min\\left\\\{1,16NK\\sum\_\{k<K\}\\xi\_\{k\}\(x\)^\{2\}\\right\\\}\.\(C\.18\)Each deficit admits a bound in terms of Bernoulli KL or probability shortfall: 1. 1\.For target success probabilitiessk∗\(x\)s\_\{k\}^\{\*\}\(x\), iflogitsk∗\(x\)≥logitsk\(x\)\+λ\\operatorname\{logit\}s\_\{k\}^\{\*\}\(x\)\\geq\\operatorname\{logit\}s\_\{k\}\(x\)\+\\lambda, thenξk2≤\(π2/8\)DKL\(Bern\(sk\+1\)∥Bern\(sk∗\)\)\\xi\_\{k\}^\{2\}\\leq\(\\pi^\{2\}/8\)D\_\{\\mathrm\{KL\}\}\(\\operatorname\{Bern\}\(s\_\{k\+1\}\)\\\|\\operatorname\{Bern\}\(s\_\{k\}^\{\*\}\)\), and the same holds with the Bernoulli arguments reversed\. 2. 2\.ξk2≤\(π2/4\)\[sin2T\(αk\)−sk\+1\]\+\\xi\_\{k\}^\{2\}\\leq\(\\pi^\{2\}/4\)\[\\sin^\{2\}T\(\\alpha\_\{k\}\)\-s\_\{k\+1\}\]\_\{\+\}, that is,\(π2/4\)×\(\\pi^\{2\}/4\)\\timesthe shortfall from the ideal step in probability\. In particular, withEk=𝔼xDKL\(Bern\(sk\+1\(x\)\)∥Bern\(sk∗\(x\)\)\)E\_\{k\}=\\mathbb\{E\}\_\{x\}D\_\{\\mathrm\{KL\}\}\(\\operatorname\{Bern\}\(s\_\{k\+1\}\(x\)\)\\\|\\operatorname\{Bern\}\(s\_\{k\}^\{\*\}\(x\)\)\)and targets as in the first case,𝔼x\[s\(x\)−sK\(x\)\]≤ϵ\+2π2NK∑k<KEk\\mathbb\{E\}\_\{x\}\[s\(x\)\-s\_\{K\}\(x\)\]\\leq\\epsilon\+2\\pi^\{2\}NK\\sum\_\{k<K\}E\_\{k\}\. ###### Proof\. Step 1: compare the actual and ideal sequences\.The relative angular\-transfer lemma of[Bai et al\. \(2026b\)](https://arxiv.org/html/2611.21791#bib.bib9)uses comparator coverages⋆\(x\)≤C⋆s0\(x\)s\_\{\\star\}\(x\)\\leq C\_\{\\star\}s\_\{0\}\(x\)\. Heres⋆\(x\)=1−\(1−s0\(x\)\)N≤Ns0\(x\)s\_\{\\star\}\(x\)=1\-\(1\-s\_\{0\}\(x\)\)^\{N\}\\leq Ns\_\{0\}\(x\), soC⋆=NC\_\{\\star\}=N\. We give the specialization explicitly\. Fix a prompt and puts=1−\(1−s0\)Ns=1\-\(1\-s\_\{0\}\)^\{N\}\. ExtendTTcontinuously to the endpoints\. This map is increasing and concave on\[0,π/2\]\[0,\\pi/2\], withT\(0\)=0T\(0\)=0andT\(α\)≥αT\(\\alpha\)\\geq\\alpha\. Indeed, T′\(α\)=eλ/21\+\(eλ−1\)sin2αT^\{\\prime\}\(\\alpha\)=\\frac\{e^\{\\lambda/2\}\}\{1\+\(e^\{\\lambda\}\-1\)\\sin^\{2\}\\alpha\}is positive and nonincreasing, establishing monotonicity and concavity\. Define its ideal sequenceαk∗=Tk\(α0\)\\alpha\_\{k\}^\{\*\}=T^\{k\}\(\\alpha\_\{0\}\)\. Ifs0=0s\_\{0\}=0, the comparator tail is zero and contributes no deficit\. Otherwise concavity implies, withak=\[1−αk/αk∗\]\+a\_\{k\}=\[1\-\\alpha\_\{k\}/\\alpha\_\{k\}^\{\*\}\]\_\{\+\}, ak\+1≤ak\+ξkαk\+1∗,aK≤∑k<Kξkα0\.a\_\{k\+1\}\\leq a\_\{k\}\+\\frac\{\\xi\_\{k\}\}\{\\alpha\_\{k\+1\}^\{\*\}\},\\qquad a\_\{K\}\\leq\\frac\{\\sum\_\{k<K\}\\xi\_\{k\}\}\{\\alpha\_\{0\}\}\.The ideal success mass isPK=sin2αK∗=eKλs0/\(1−s0\+eKλs0\)P\_\{K\}=\\sin^\{2\}\\alpha\_\{K\}^\{\*\}=e^\{K\\lambda\}s\_\{0\}/\(1\-s\_\{0\}\+e^\{K\\lambda\}s\_\{0\}\)\. Sinces≤Ns0s\\leq Ns\_\{0\}andeKλ≥3N/ϵe^\{K\\lambda\}\\geq 3N/\\epsilon,PK≥3s/\(ϵ\+3s\)P\_\{K\}\\geq 3s/\(\\epsilon\+3s\)\. Ifs<ϵs<\\epsilon, the reward deficit is already at mostϵ\\epsilon\. Fors≥ϵs\\geq\\epsilon,PK≥3/4P\_\{K\}\\geq 3/4, soαK∗≥π/3\\alpha\_\{K\}^\{\*\}\\geq\\pi/3\. Ifs≤1/2s\\leq 1/2andsK<ss\_\{K\}<s, thenαK<π/4\\alpha\_\{K\}<\\pi/4, henceaK\>1/4a\_\{K\}\>1/4ands−sK≤16saK2s\-s\_\{K\}\\leq 16sa\_\{K\}^\{2\}\. Ifs\>1/2s\>1/2andsK≤PKs\_\{K\}\\leq P\_\{K\}, the Lipschitz bound for cosine gives 1−sK≤2\(1−PK\)\+2\(αK∗−αK\)2\.1\-s\_\{K\}\\leq 2\(1\-P\_\{K\}\)\+2\(\\alpha\_\{K\}^\{\*\}\-\\alpha\_\{K\}\)^\{2\}\.The convex functions−1\+2ϵ/\(ϵ\+3s\)s\-1\+2\\epsilon/\(\\epsilon\+3s\)is at mostϵ\\epsilonons∈\[ϵ,1\]s\\in\[\\epsilon,1\], by checking its endpoints\. Also2\(αK∗aK\)2≤16saK22\(\\alpha\_\{K\}^\{\*\}a\_\{K\}\)^\{2\}\\leq 16sa\_\{K\}^\{2\}whens\>1/2s\>1/2\. IfsK\>PKs\_\{K\}\>P\_\{K\}, the ideal bound givess−sK≤s−PK≤s−1\+ϵ/\(ϵ\+3s\)≤ϵs\-s\_\{K\}\\leq s\-P\_\{K\}\\leq s\-1\+\\epsilon/\(\\epsilon\+3s\)\\leq\\epsilon, using the same endpoint bound\. Thus in every case s−sK≤ϵ\+16saK2≤ϵ\+16N\(∑k<Kξk\)2,s\-s\_\{K\}\\leq\\epsilon\+16sa\_\{K\}^\{2\}\\leq\\epsilon\+16N\\left\(\\sum\_\{k<K\}\\xi\_\{k\}\\right\)^\{2\},usings≤Ns0≤Nα02s\\leq Ns\_\{0\}\\leq N\\alpha\_\{0\}^\{2\}\. Since alsos−sK≤s≤1s\-s\_\{K\}\\leq s\\leq 1, andmin\{1,ϵ\+y\}≤ϵ\+min\{1,y\}\\min\\\{1,\\epsilon\+y\\\}\\leq\\epsilon\+\\min\\\{1,y\\\}fory≥0y\\geq 0, this proves the first inequality in \([C\.18](https://arxiv.org/html/2611.21791#A3.E18)\)\. Cauchy\-Schwarz proves the second\. Step 2: bound each angular deficit\.For the first deficit bound, the target angleαk†=arcsinsk∗\\alpha\_\{k\}^\{\\dagger\}=\\arcsin\\sqrt\{s\_\{k\}^\{\*\}\}is at leastT\(αk\)T\(\\alpha\_\{k\}\), becauseTTadds exactlyλ\\lambdato the log\-odds\. Thusξk≤\|αk\+1−αk†\|\\xi\_\{k\}\\leq\|\\alpha\_\{k\+1\}\-\\alpha\_\{k\}^\{\\dagger\}\|\. Use the unnormalized squared Hellinger distance∑u\(P\(u\)−Q\(u\)\)2\\sum\_\{u\}\(\\sqrt\{P\(u\)\}\-\\sqrt\{Q\(u\)\}\)^\{2\}\. For Bernoulli probabilities with angular differenced≤π/2d\\leq\\pi/2, this distance is4sin2\(d/2\)≥8d2/π24\\sin^\{2\}\(d/2\)\\geq 8d^\{2\}/\\pi^\{2\}and is at most their KL divergence in either direction\. This gives the stated bound in both directions\. For the second, let0≤ϕ≤θ≤π/20\\leq\\phi\\leq\\theta\\leq\\pi/2\. Thenθ\+ϕ∈\[θ−ϕ,π−\(θ−ϕ\)\]\\theta\+\\phi\\in\[\\theta\-\\phi,\\pi\-\(\\theta\-\\phi\)\], so sin2θ−sin2ϕ=sin\(θ\+ϕ\)sin\(θ−ϕ\)≥sin2\(θ−ϕ\)≥4π2\(θ−ϕ\)2\.\\sin^\{2\}\\theta\-\\sin^\{2\}\\phi=\\sin\(\\theta\+\\phi\)\\sin\(\\theta\-\\phi\)\\geq\\sin^\{2\}\(\\theta\-\\phi\)\\geq\\frac\{4\}\{\\pi^\{2\}\}\(\\theta\-\\phi\)^\{2\}\.Apply this withθ=T\(αk\)\\theta=T\(\\alpha\_\{k\}\)andϕ=αk\+1\\phi=\\alpha\_\{k\+1\}whenαk\+1<T\(αk\)\\alpha\_\{k\+1\}<T\(\\alpha\_\{k\}\)\. Otherwiseξk=0\\xi\_\{k\}=0\. The averaged statement follows from the first case and \([C\.18](https://arxiv.org/html/2611.21791#A3.E18)\) after averaging prompts, since16NK⋅π2/8=2π2NK16NK\\cdot\\pi^\{2\}/8=2\\pi^\{2\}NK\. The argument applies to every realized sequence and hence also in expectation for random fitted policies\. ∎ #### C\.2\.2Unregularized reward progress ###### Assumption C\.4\(Optimization oracle accuracy\)\. At roundkk, fix behaviorμ=πk\\mu=\\pi\_\{k\}and use the radius\-rrfamilyΓk\(r\)\\Gamma\_\{k\}\(r\)from Equation \([C\.5](https://arxiv.org/html/2611.21791#A3.E5)\)\. The fitted policy is parameterized by the nonlinear model and must belong to this neighborhood\. Let𝒞k,x\\mathcal\{C\}\_\{k,x\}be \([C\.3](https://arxiv.org/html/2611.21791#A3.E3)\) conditional on promptxx, and write𝒞k=𝔼x𝒞k,x\\mathcal\{C\}\_\{k\}=\\mathbb\{E\}\_\{x\}\\mathcal\{C\}\_\{k,x\}\. Givenζk≥0\\zeta\_\{k\}\\geq 0, the optimization routine returns one policyπk\+1∈Γk\\pi\_\{k\+1\}\\in\\Gamma\_\{k\}with 𝒞k\(πk\+1\)≥supπ∈Γk𝒞k\(π\)−ζk\.\\mathcal\{C\}\_\{k\}\(\\pi\_\{k\+1\}\)\\geq\\sup\_\{\\pi\\in\\Gamma\_\{k\}\}\\mathcal\{C\}\_\{k\}\(\\pi\)\-\\zeta\_\{k\}\.\(C\.19\) The next lemma isolates the history\-distribution change caused by token updates\. It explains why a token\-radius argument alone requires the radius to decrease with the horizon\. ###### Lemma C\.5\(Token\-radius control of prefix and response ratios\)\. Letπ,μ\\pi,\\muhave common full support and satisfy\|log\(π\(u∣h\)/μ\(u∣h\)\)\|≤r\|\\log\(\\pi\(u\\mid h\)/\\mu\(u\\mid h\)\)\|\\leq rat every history and token\. For every promptxxand prefixht=\(x,a<t\)h\_\{t\}=\(x,a\_\{<t\}\), e−\(t−1\)r≤dtπ\(ht∣x\)dtμ\(ht∣x\)≤e\(t−1\)r,1≤t≤H\+1\.e^\{\-\(t\-1\)r\}\\leq\\frac\{d\_\{t\}^\{\\pi\}\(h\_\{t\}\\mid x\)\}\{d\_\{t\}^\{\\mu\}\(h\_\{t\}\\mid x\)\}\\leq e^\{\(t\-1\)r\},\\qquad 1\\leq t\\leq H\+1\.\(C\.20\)In particular,e−Hr≤π\(Y∣x\)/μ\(Y∣x\)≤eHre^\{\-Hr\}\\leq\\pi\(Y\\mid x\)/\\mu\(Y\\mid x\)\\leq e^\{Hr\}\. The response bounds are attained by full\-support binary token policies\. Thus a horizon\-independent uniform ratio bound based only on the token constraint requiresHr=O\(1\)Hr=O\(1\)\. The choicer=1/\(8H\)r=1/\(8H\)bounds all these ratios betweene−1/8e^\{\-1/8\}ande1/8e^\{1/8\}\. ###### Proof\. At a fixed prompt, the prefix ratio is the product of the firstt−1t\-1conditional token ratios\. Taking logarithms and summing their bounds proves Equation \([C\.20](https://arxiv.org/html/2611.21791#A3.E20)\), including the full response whent=H\+1t=H\+1\. For sharpness, let tokens be independent and binary at every position, withμ\(1\)=1/\(1\+er\)\\mu\(1\)=1/\(1\+e^\{r\}\)andπ\(1\)=er/\(1\+er\)\\pi\(1\)=e^\{r\}/\(1\+e^\{r\}\)\. The ratios for tokens11and00areere^\{r\}ande−re^\{\-r\}, respectively\. The all\-one and all\-zero responses therefore attaineHre^\{Hr\}ande−Hre^\{\-Hr\}\. Keeping a uniform upper bound independent ofHHthus requiresHrHrto remain bounded\. ∎ ###### Lemma C\.6\(Feasible token policy with positive objective gain\)\. Forr\>0r\>0and behaviorπk\\pi\_\{k\}, define the token policy π~k\(b∣ht\)=πk\(b∣ht\)1\+\(er−1\)Qtπk\(ht,b\)1\+\(er−1\)Vtπk\(ht\)\.\\widetilde\{\\pi\}\_\{k\}\(b\\mid h\_\{t\}\)=\\pi\_\{k\}\(b\\mid h\_\{t\}\)\\frac\{1\+\(e^\{r\}\-1\)Q\_\{t\}^\{\\pi\_\{k\}\}\(h\_\{t\},b\)\}\{1\+\(e^\{r\}\-1\)V\_\{t\}^\{\\pi\_\{k\}\}\(h\_\{t\}\)\}\.\(C\.21\)It is normalized, has token log\-ratios in\[−r,r\]\[\-r,r\], and, for SC or SC\+TIS withc≥erc\\geq e^\{r\}, satisfies 𝒞k,x\(π~k\)≥\(1−e−r\)sk\(x\)\(1−sk\(x\)\)\.\\mathcal\{C\}\_\{k,x\}\(\\widetilde\{\\pi\}\_\{k\}\)\\geq\(1\-e^\{\-r\}\)s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\.For the symmetric neighborhoodΓk\\Gamma\_\{k\}in \([C\.5](https://arxiv.org/html/2611.21791#A3.E5)\), this policy belongs toΓk\\Gamma\_\{k\}, so the same lower bound holds for its conditional optimum\. ###### Proof\. Normalization follows from𝔼πk\(⋅∣ht\)Qtπk=Vtπk\\mathbb\{E\}\_\{\\pi\_\{k\}\(\\cdot\\mid h\_\{t\}\)\}Q\_\{t\}^\{\\pi\_\{k\}\}=V\_\{t\}^\{\\pi\_\{k\}\}\. Both numerator and denominator of the ratio lie in\[1,er\]\[1,e^\{r\}\]\. Putu=er−1u=e^\{r\}\-1\. For SC, the conditional gain isCov\(Q,log\(1\+uQ\)\)\\operatorname\{Cov\}\(Q,\\log\(1\+uQ\)\)\. The derivative ofz↦log\(1\+uz\)z\\mapsto\\log\(1\+uz\)on\[0,1\]\[0,1\]is at leastu/\(1\+u\)u/\(1\+u\)\. Using an independent copy ofQQin the covariance identity therefore bounds this gain below by\(1−e−r\)Var\(Q\)\(1\-e^\{\-r\}\)\\operatorname\{Var\}\(Q\)\. For inactive TIS, the gain isuVar\(Q\)/\(1\+uV\)u\\operatorname\{Var\}\(Q\)/\(1\+uV\), which has the same lower bound becauseV≤1V\\leq 1\. The reward\-martingale variance identity sums these contributions to\(1−e−r\)sk\(x\)\(1−sk\(x\)\)\(1\-e^\{\-r\}\)s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\. ∎ ###### Lemma C\.7\(Population optimization gives best\-of\-N improvement\)\. Use either SC,a\(ρ\)=1a\(\\rho\)=1, or SC\+TIS,a\(ρ\)=min\(ρ,c\)a\(\\rho\)=\\min\(\\rho,c\), and choose 0<r≤18H,c≥erfor TIS\.0<r\\leq\\frac\{1\}\{8H\},\\qquad c\\geq e^\{r\}\\quad\\text\{for TIS\}\.\(C\.22\)LetΓk\\Gamma\_\{k\}be the neighborhood in \([C\.5](https://arxiv.org/html/2611.21791#A3.E5)\)\. At every round, its prompt\-averaged optimum satisfies supπ∈Γk𝔼x𝒞k,x\(π\)≥\(1−e−r\)𝔼x\[sk\(x\)\(1−sk\(x\)\)\]\.\\sup\_\{\\pi\\in\\Gamma\_\{k\}\}\\mathbb\{E\}\_\{x\}\\mathcal\{C\}\_\{k,x\}\(\\pi\)\\geq\(1\-e^\{\-r\}\)\\mathbb\{E\}\_\{x\}\[s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\]\.\(C\.23\)ForN≥1N\\geq 1and0<ϵ<10<\\epsilon<1, use the averaged oracle \([C\.19](https://arxiv.org/html/2611.21791#A3.E19)\) at each round and choose K≥⌈8rlog6Nϵ⌉,∑k=0K−1ζk≤ϵ16π2NK\.K\\geq\\left\\lceil\\frac\{8\}\{r\}\\log\\frac\{6N\}\{\\epsilon\}\\right\\rceil,\\qquad\\sum\_\{k=0\}^\{K\-1\}\\zeta\_\{k\}\\leq\\frac\{\\epsilon\}\{16\\pi^\{2\}NK\}\.\(C\.24\)Then VN\(π0\)−J\(πK\)≤ϵ\.V\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilon\.\(C\.25\)The same conclusion holds in expectation if the oracle gaps and their accumulated bound hold in expectation for random fitted policies\. ###### Proof of Lemma[C\.7](https://arxiv.org/html/2611.21791#A3.Thmtheorem7)\. Step 1: attainable objective gain\.The witness in Lemma[C\.6](https://arxiv.org/html/2611.21791#A3.Thmtheorem6)attains\(1−e−r\)sk\(x\)\(1−sk\(x\)\)\(1\-e^\{\-r\}\)s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)at every prompt, proving \([C\.23](https://arxiv.org/html/2611.21791#A3.E23)\) after averaging\. For progress, we use the full optimization\-oracle condition in \([C\.19](https://arxiv.org/html/2611.21791#A3.E19)\)\. Step 2: local objective\-to\-reward comparison\.Fix a round and a history, and writeμ=πk\(⋅∣h\)\{\\mu\}=\\pi\_\{k\}\(\\cdot\\mid h\),A=Qμ\(h,⋅\)−Vμ\(h\)A=Q^\{\\mu\}\(h,\\cdot\)\-V^\{\\mu\}\(h\), andah=𝔼μ\|A\|a\_\{h\}=\\mathbb\{E\}\_\{\\mu\}\|A\|\. For a feasible token policyπ\\pi, let ch\(π\)=𝔼μ\[AFa\(π/μ\)\],lh\(π\)=𝔼πA,δh=maxv:\|log\(v/μ\)\|≤rch\(v\)−ch\(π\)\.c\_\{h\}\(\\pi\)=\\mathbb\{E\}\_\{\\mu\}\[AF\_\{a\}\(\\pi/\{\\mu\}\)\],\\qquad l\_\{h\}\(\\pi\)=\\mathbb\{E\}\_\{\\pi\}A,\\qquad\\delta\_\{h\}=\\max\_\{v:\\,\|\\log\(v/\{\\mu\}\)\|\\leq r\}c\_\{h\}\(v\)\-c\_\{h\}\(\\pi\)\.The maximum exists on the compact feasible token simplex, andδh≥0\\delta\_\{h\}\\geq 0\. For SC or SC\+TIS with cap at least one, the scalar remainder bound gives \|lh\(π\)−ch\(π\)\|≤B\(r\)r2ah,B\(r\)=\(er−1−r\)/r2\.\|l\_\{h\}\(\\pi\)\-c\_\{h\}\(\\pi\)\|\\leq B\(r\)r^\{2\}a\_\{h\},\\qquad B\(r\)=\(e^\{r\}\-1\-r\)/r^\{2\}\.Indeed, withρ=π/μ\\rho=\\pi/\{\\mu\}andg\(ρ\)=ρ−1−Fa\(ρ\)g\(\\rho\)=\\rho\-1\-F\_\{a\}\(\\rho\), centering𝔼μA=0\\mathbb\{E\}\_\{\\mu\}A=0giveslh\(π\)−ch\(π\)=𝔼μ\[Ag\(ρ\)\]l\_\{h\}\(\\pi\)\-c\_\{h\}\(\\pi\)=\\mathbb\{E\}\_\{\\mu\}\[A\\,g\(\\rho\)\]\. For SC,g\(ρ\)=ρ−1−logρ∈\[0,er−1−r\]g\(\\rho\)=\\rho\-1\-\\log\\rho\\in\[0,e^\{r\}\-1\-r\]on\[e−r,er\]\[e^\{\-r\},e^\{r\}\]\. For SC\+TIS with capc≥1c\\geq 1,g=0g=0forρ≤c\\rho\\leq c, and0≤g\(ρ\)=ρ−c−clog\(ρ/c\)≤ρ−1−logρ0\\leq g\(\\rho\)=\\rho\-c\-c\\log\(\\rho/c\)\\leq\\rho\-1\-\\log\\rhoforρ\>c\\rho\>c\. Hence\|𝔼μ\[Ag\]\|≤\(er−1−r\)max\{𝔼μA\+,𝔼μA−\}=\(er−1−r\)ah/2\|\\mathbb\{E\}\_\{\\mu\}\[Ag\]\|\\leq\(e^\{r\}\-1\-r\)\\max\\\{\\mathbb\{E\}\_\{\\mu\}A\_\{\+\},\\mathbb\{E\}\_\{\\mu\}A\_\{\-\}\\\}=\(e^\{r\}\-1\-r\)a\_\{h\}/2\. To lower\-bound the local optimum, putbh=𝔼μsign\(A\)b\_\{h\}=\\mathbb\{E\}\_\{\\mu\}\\operatorname\{sign\}\(A\), and choose v\(a\)=μ\(a\)\[1\+12\(1−e−r\)\(sign\(A\(a\)\)−bh\)\]\.v\(a\)=\{\\mu\}\(a\)\[1\+\\tfrac\{1\}\{2\}\(1\-e^\{\-r\}\)\(\\operatorname\{sign\}\(A\(a\)\)\-b\_\{h\}\)\]\.This policy is normalized and its ratios lie in\[e−r,2−e−r\]⊆\[e−r,er\]\[e^\{\-r\},2\-e^\{\-r\}\]\\subseteq\[e^\{\-r\},e^\{r\}\]\. Since𝔼μA=0\\mathbb\{E\}\_\{\\mu\}A=0, it satisfieslh\(v\)=12\(1−e−r\)ahl\_\{h\}\(v\)=\\tfrac\{1\}\{2\}\(1\-e^\{\-r\}\)a\_\{h\}\. Applying the remainder bound tovvand toπ\\piyields lh\(π\)≥\[12\(1−e−r\)−2B\(r\)r2\]ah−δh≥r4ah−δh\.l\_\{h\}\(\\pi\)\\geq\[\\tfrac\{1\}\{2\}\(1\-e^\{\-r\}\)\-2B\(r\)r^\{2\}\]a\_\{h\}\-\\delta\_\{h\}\\geq\\frac\{r\}\{4\}a\_\{h\}\-\\delta\_\{h\}\.For the last inequality, user≤1/8r\\leq 1/8,12\(1−e−r\)≥r/2−r2/4\\tfrac\{1\}\{2\}\(1\-e^\{\-r\}\)\\geq r/2\-r^\{2\}/4, andB\(r\)≤3/4B\(r\)\\leq 3/4\. Step 3: transfer along learner histories\.The global objective is a sum of these local objectives weighted by behavior\-history probabilities\. The full feasible neighborhood allows each token policy to be chosen independently, so its optimization gap is the corresponding sum of the nonnegative local gaps\. Define Δk\(x\)=∑t𝔼ht∼dtμ\(⋅∣x\)δht\.\\Delta\_\{k\}\(x\)=\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\(\\cdot\\mid x\)\}\\delta\_\{h\_\{t\}\}\.The averaged oracle gives𝔼xΔk\(x\)≤ζk\\mathbb\{E\}\_\{x\}\\Delta\_\{k\}\(x\)\\leq\\zeta\_\{k\}\. The token\-wise performance\-difference identity, obtained by telescopingVtμV\_\{t\}^\{\\mu\}along learner histories, is sk\+1\(x\)−sk\(x\)=∑t𝔼ht∼dtπk\+1\(⋅∣x\)lht\(πk\+1\(⋅∣ht\)\)\.s\_\{k\+1\}\(x\)\-s\_\{k\}\(x\)=\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\pi\_\{k\+1\}\}\(\\cdot\\mid x\)\}l\_\{h\_\{t\}\}\(\\pi\_\{k\+1\}\(\\cdot\\mid h\_\{t\}\)\)\.Lemma[C\.5](https://arxiv.org/html/2611.21791#A3.Thmtheorem5)bounds the prefix density ratios between4/54/5and5/45/4, sincee\(H−1\)r≤e1/8<5/4e^\{\(H\-1\)r\}\\leq e^\{1/8\}<5/4\. Therefore sk\+1\(x\)−sk\(x\)≥r5∑t𝔼dtμ\(⋅∣x\)aht−54Δk\(x\)≥r5sk\(x\)\(1−sk\(x\)\)−54Δk\(x\)\.s\_\{k\+1\}\(x\)\-s\_\{k\}\(x\)\\geq\\frac\{r\}\{5\}\\sum\_\{t\}\\mathbb\{E\}\_\{d\_\{t\}^\{\\mu\}\(\\cdot\\mid x\)\}a\_\{h\_\{t\}\}\-\\frac\{5\}\{4\}\\Delta\_\{k\}\(x\)\\geq\\frac\{r\}\{5\}s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\-\\frac\{5\}\{4\}\\Delta\_\{k\}\(x\)\.Here𝔼μ\|A\|≥𝔼μA2\\mathbb\{E\}\_\{\\mu\}\|A\|\\geq\\mathbb\{E\}\_\{\\mu\}A^\{2\}, and the reward\-martingale total variance is∑t𝔼μAt2=sk\(x\)\(1−sk\(x\)\)\\sum\_\{t\}\\mathbb\{E\}\_\{\\mu\}A\_\{t\}^\{2\}=s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\. This step uses ordinary behavior histories, without conditioning on the terminal reward\. Step 4: accumulate the reward shortfalls\.Sincer≤1/8r\\leq 1/8,log\(1\+r/5\)≥\(r/5\)/\(1\+r/5\)≥r/8\\log\(1\+r/5\)\\geq\(r/5\)/\(1\+r/5\)\\geq r/8\. The ideal log\-odds incrementλ=r/8\\lambda=r/8increases probability by at most\(r/5\)sk\(1−sk\)\(r/5\)s\_\{k\}\(1\-s\_\{k\}\)\. Its one\-sided probability deficit is thus at most5Δk\(x\)/45\\Delta\_\{k\}\(x\)/4\. By the second deficit bound of Lemma[C\.3](https://arxiv.org/html/2611.21791#A3.Thmtheorem3), the squared angular deficit is at most5π2Δk\(x\)/165\\pi^\{2\}\\Delta\_\{k\}\(x\)/16\. Applying \([C\.18](https://arxiv.org/html/2611.21791#A3.E18)\) with accuracyϵ/2\\epsilon/2, which the stated round count permits, and averaging prompts gives VN\(π0\)−J\(πK\)≤ϵ/2\+5π2NK∑k<Kζk≤ϵ/2\+8π2NK∑k<Kζk≤ϵ\.V\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilon/2\+5\\pi^\{2\}NK\\sum\_\{k<K\}\\zeta\_\{k\}\\leq\\epsilon/2\+8\\pi^\{2\}NK\\sum\_\{k<K\}\\zeta\_\{k\}\\leq\\epsilon\.The argument holds for every fitted policy sequence\. Taking expectations proves the expected\-oracle version\. ∎ #### C\.2\.3One\-round progress for unit\-weight KL regularization For RSC \(w/o IS\), seta\(ρ\)=1a\(\\rho\)=1,β=2\\beta=2, andΓk=Γk\(1/2\)\\Gamma\_\{k\}=\\Gamma\_\{k\}\(1/2\)from Equation \([C\.5](https://arxiv.org/html/2611.21791#A3.E5)\)\. Its proof uses a direct reward lower bound and objective curvature, rather than an explicit target formula\. Theorem[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)allows finite Monte Carlo centers; their fitting analysis is given in Section[C\.3\.4](https://arxiv.org/html/2611.21791#A3.SS3.SSS4)\. At roundkk, holdμ=πk\\mu=\\pi\_\{k\}fixed\. Writeρtπ\(h,u\)=π\(u∣h\)/μ\(u∣h\)\\rho\_\{t\}^\{\\pi\}\(h,u\)=\\pi\(u\\mid h\)/\{\\mu\}\(u\\mid h\),k\(v\)=vlogv−v\+1k\(v\)=v\\log v\-v\+1, andAtμ=Qtμ−VtμA\_\{t\}^\{\\mu\}=Q\_\{t\}^\{\\mu\}\-V\_\{t\}^\{\\mu\}\. For each prompt define Ck,x\(π\)\\displaystyle C\_\{k,x\}\(\\pi\)=𝔼μ\[∑t=1HAtμ\(ht,at\)logρtπ\(ht,at\)\|x\],\\displaystyle=\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[\\sum\_\{t=1\}^\{H\}A\_\{t\}^\{\\mu\}\(h\_\{t\},a\_\{t\}\)\\log\\rho\_\{t\}^\{\\pi\}\(h\_\{t\},a\_\{t\}\)\\,\\middle\|\\,x\\right\],Kk,x\(π\)\\displaystyle K\_\{k,x\}\(\\pi\)=∑t=1H𝔼ht∼dtμ\(⋅∣x\)KL\(π\(⋅∣ht\)∥μ\(⋅∣ht\)\),\\displaystyle=\\sum\_\{t=1\}^\{H\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\(\\cdot\\mid x\)\}\\operatorname\{KL\}\\\!\\left\(\\pi\(\\cdot\\mid h\_\{t\}\)\\\|\{\\mu\}\(\\cdot\\mid h\_\{t\}\)\\right\),Wk,x\(π\)\\displaystyle W\_\{k,x\}\(\\pi\)=Ck,x\(π\)−2Kk,x\(π\),𝒲k\(π\)=𝔼xWk,x\(π\)\.\\displaystyle=C\_\{k,x\}\(\\pi\)\-2K\_\{k,x\}\(\\pi\),\\qquad\\mathcal\{W\}\_\{k\}\(\\pi\)=\\mathbb\{E\}\_\{x\}W\_\{k,x\}\(\\pi\)\.\(C\.26\)Thus the KL coefficient isβ=2\\beta=2\. Equivalently, useρKL=b\(ρ\)=min\(ρ,2\)\\rho^\{\\mathrm\{KL\}\}=b\(\\rho\)=\\min\(\\rho,2\)in Equation \([C\.6](https://arxiv.org/html/2611.21791#A3.E6)\): its cap is inactive throughoutΓk\\Gamma\_\{k\}\. The penalty is conditional reverse KL averaged over behavior histories, not the KL between whole response distributions\. Population progress\.UseWk,xW\_\{k,x\}andΓk\\Gamma\_\{k\}from Section[C\.2\.3](https://arxiv.org/html/2611.21791#A3.SS2.SSS3), in particular Equation \([C\.26](https://arxiv.org/html/2611.21791#A3.E26)\)\. Fix a round and letπk∗\\pi\_\{k\}^\{\*\}maximizeWk,xW\_\{k,x\}at every prompt overΓk\\Gamma\_\{k\}\. Such an optimizer exists on each finite tree\. The full family separates across prompts and histories, so the conditional gap Dk\(x,π\)=Wk,x\(πk∗\)−Wk,x\(π\)D\_\{k\}\(x,\\pi\)=W\_\{k,x\}\(\\pi\_\{k\}^\{\*\}\)\-W\_\{k,x\}\(\\pi\)\(C\.27\)is nonnegative\. We do not require a closed form forπk∗\\pi\_\{k\}^\{\*\}\. The first lemma converts the regularized objective directly into reward gain\. ###### Lemma C\.8\(RSC \(w/o IS\) lower\-bounds reward gain\)\. For terminal rewards in\[0,1\]\[0,1\], token ratios at leaste−re^\{\-r\}, andβ≥er\\beta\\geq e^\{r\}, Jx\(π\)−Jx\(μ\)≥Ck,x\(π\)−βKk,x\(π\)\.J\_\{x\}\(\\pi\)\-J\_\{x\}\(\{\\mu\}\)\\geq C\_\{k,x\}\(\\pi\)\-\\beta K\_\{k,x\}\(\\pi\)\.In particular,Jx\(π\)−Jx\(μ\)≥Wk,x\(π\)J\_\{x\}\(\\pi\)\-J\_\{x\}\(\{\\mu\}\)\\geq W\_\{k,x\}\(\\pi\)onΓk\\Gamma\_\{k\}\. ###### Proof\. LetL\(Y\)=π\(Y∣x\)/μ\(Y∣x\)L\(Y\)=\\pi\(Y\\mid x\)/\{\\mu\}\(Y\\mid x\)\. Nonnegative reward andL−1≥logLL\-1\\geq\\log Limply Jx\(π\)−Jx\(μ\)\\displaystyle J\_\{x\}\(\\pi\)\-J\_\{x\}\(\{\\mu\}\)≥𝔼μ\[RlogL∣x\]\\displaystyle\\geq\\mathbb\{E\}\_\{\\mu\}\[R\\log L\\mid x\]=Ck,x\(π\)−∑t𝔼ht∼dtμ\(⋅∣x\)Vtμ\(ht\)KL\(μ\(⋅∣ht\)∥π\(⋅∣ht\)\)\.\\displaystyle=C\_\{k,x\}\(\\pi\)\-\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\(\\cdot\\mid x\)\}V\_\{t\}^\{\\mu\}\(h\_\{t\}\)\\operatorname\{KL\}\(\{\\mu\}\(\\cdot\\mid h\_\{t\}\)\\\|\\pi\(\\cdot\\mid h\_\{t\}\)\)\.Here we condition each summand on its history and current action and useQtμ=Atμ\+VtμQ\_\{t\}^\{\\mu\}=A\_\{t\}^\{\\mu\}\+V\_\{t\}^\{\\mu\}\. Since0≤Vtμ≤10\\leq V\_\{t\}^\{\\mu\}\\leq 1, it remains to compare the two conditional KL divergences\. The functionsf\(u\)=u−1−loguf\(u\)=u\-1\-\\log uandk\(u\)=ulogu−u\+1k\(u\)=u\\log u\-u\+1satisfyf\(1\)=f′\(1\)=k\(1\)=k′\(1\)=0f\(1\)=f^\{\\prime\}\(1\)=k\(1\)=k^\{\\prime\}\(1\)=0andf′′\(u\)≤erk′′\(u\)f^\{\\prime\\prime\}\(u\)\\leq e^\{r\}k^\{\\prime\\prime\}\(u\)foru≥e−ru\\geq e^\{\-r\}\. Taylor’s integral remainder and normalization give KL\(μ∥π\)=𝔼μf\(ρ\)≤er𝔼μk\(ρ\)=erKL\(π∥μ\)\.\\operatorname\{KL\}\(\{\\mu\}\\\|\\pi\)=\\mathbb\{E\}\_\{\\mu\}f\(\\rho\)\\leq e^\{r\}\\mathbb\{E\}\_\{\\mu\}k\(\\rho\)=e^\{r\}\\operatorname\{KL\}\(\\pi\\\|\{\\mu\}\)\.Apply this at every history\. Forr=1/2r=1/2,β=2≥e1/2\\beta=2\\geq e^\{1/2\}\. ∎ The next lemma constructs a feasible policy with positive objective gain\. Together with Lemma[C\.8](https://arxiv.org/html/2611.21791#A3.Thmtheorem8), this makes the objective gap a bound on lost reward progress\. ###### Lemma C\.9\(Constant reward progress from accurate fitting\)\. Writesk\(x\)=Jx\(μ\)s\_\{k\}\(x\)=J\_\{x\}\(\\mu\)\. Everyπ∈Γk\\pi\\in\\Gamma\_\{k\}satisfies Jx\(π\)−sk\(x\)≥760sk\(x\)\(1−sk\(x\)\)−Dk\(x,π\)\.J\_\{x\}\(\\pi\)\-s\_\{k\}\(x\)\\geq\\tfrac\{7\}\{60\}s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\-D\_\{k\}\(x,\\pi\)\.\(C\.28\) ###### Proof\. Considerπ~\(u∣ht\)=μ\(u∣ht\)\(1\+Atμ\(ht,u\)/4\)\\widetilde\{\\pi\}\(u\\mid h\_\{t\}\)=\{\\mu\}\(u\\mid h\_\{t\}\)\(1\+A\_\{t\}^\{\\mu\}\(h\_\{t\},u\)/4\)\. It is normalized because𝔼μtAtμ=0\\mathbb\{E\}\_\{\{\{\\mu\_\{t\}\}\}\}A\_\{t\}^\{\\mu\}=0, and feasible because its ratios lie in\[3/4,5/4\]⊂\[e−1/2,e1/2\]\[3/4,5/4\]\\subset\[e^\{\-1/2\},e^\{1/2\}\]\. For\|A\|≤1\|A\|\\leq 1, Alog\(1\+A/4\)≥A2/5,k\(1\+A/4\)≤A2/24\.A\\log\(1\+A/4\)\\geq A^\{2\}/5,\\qquad k\(1\+A/4\)\\leq A^\{2\}/24\.The first inequality follows by integrating1/\(1\+s\)1/\(1\+s\)between00andA/4A/4\. The second usesk′′\(u\)=1/u≤4/3k^\{\\prime\\prime\}\(u\)=1/u\\leq 4/3on\[3/4,5/4\]\[3/4,5/4\]\. On the deterministic response tree,Atμ\(ht,at\)=Vt\+1μ\(ht\+1\)−Vtμ\(ht\)A\_\{t\}^\{\\mu\}\(h\_\{t\},a\_\{t\}\)=V\_\{t\+1\}^\{\\mu\}\(h\_\{t\+1\}\)\-V\_\{t\}^\{\\mu\}\(h\_\{t\}\)are martingale differences ending atRR\. Hence Wk,x\(π~\)≥\(1/5−1/12\)𝔼μ\[∑t\(Atμ\)2\|x\]=760Varμ\(R∣x\)=760sk\(x\)\(1−sk\(x\)\)\.W\_\{k,x\}\(\\widetilde\{\\pi\}\)\\geq\(1/5\-1/12\)\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[\\sum\_\{t\}\(A\_\{t\}^\{\\mu\}\)^\{2\}\\,\\middle\|\\,x\\right\]=\\tfrac\{7\}\{60\}\\operatorname\{Var\}\_\{\\mu\}\(R\\mid x\)=\\tfrac\{7\}\{60\}s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\.The conditional objective ofπ\\piis at least that ofπ~\\widetilde\{\\pi\}minusDk\(x,π\)D\_\{k\}\(x,\\pi\)\. Apply Lemma[C\.8](https://arxiv.org/html/2611.21791#A3.Thmtheorem8)\. ∎ #### C\.2\.4Target progress and reward transfer for TIS with KL regularization We now specialize the objective in Equation \([C\.6](https://arxiv.org/html/2611.21791#A3.E6)\) to cap\-two reward and penalty weights,a\(ρ\)=b\(ρ\)=min\(ρ,2\)a\(\\rho\)=b\(\\rho\)=\\min\(\\rho,2\), as used by RSC \(TIS\)\. Theorem[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)gives its reward and sample bounds\. The unit\-weight variant usesa\(ρ\)=1a\(\\rho\)=1and is treated in Theorem[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)\. The improving target\.For the reward guarantee, chooseβ=2\\beta=2and token log\-ratio radius1/21/2\. Both caps are inactive throughout this family\. The regularized objective then balances the token advantage against conditional KL\. Its maximizing token policy is πk∗\(u∣ht\)=πk\(u∣ht\)exp\(Qtπk\(ht,u\)/β\)𝔼v∼πk\(⋅∣ht\)exp\(Qtπk\(ht,v\)/β\)\.\\pi\_\{k\}^\{\*\}\(u\\mid h\_\{t\}\)=\\frac\{\\pi\_\{k\}\(u\\mid h\_\{t\}\)\\exp\(Q\_\{t\}^\{\\pi\_\{k\}\}\(h\_\{t\},u\)/\\beta\)\}\{\\mathbb\{E\}\_\{v\\sim\\pi\_\{k\}\(\\cdot\\mid h\_\{t\}\)\}\\exp\(Q\_\{t\}^\{\\pi\_\{k\}\}\(h\_\{t\},v\)/\\beta\)\}\.\(C\.29\)The target gives more probability to tokens with higher reference continuation value\. It belongs to the feasible family because its log\-ratios are bounded by1/β1/\\beta\. Fitting uses response rewards, so the algorithm need not evaluate these continuation values\. The target improves success odds by at least a factor1\+1/β1\+1/\\betaper round\. To follow that improvement with an approximate fit, we also need to control which histories the target visits\. We show that it can increase the probability of any prefix by at moste1/βe^\{1/\\beta\}, at every horizon\. Jensen’s inequality bounds each token normalizer, and the reference continuation values telescope along the prefix\. The objective gap is characterized by conditional reverse KLKL\(π\(⋅∣ht\)∥πk∗\(⋅∣ht\)\)\\operatorname\{KL\}\(\\pi\(\\cdot\\mid h\_\{t\}\)\\\|\\pi\_\{k\}^\{\*\}\(\\cdot\\mid h\_\{t\}\)\), averaged over reference histories as specified in Equation \([C\.30](https://arxiv.org/html/2611.21791#A3.E30)\)\. The prefix bound lets us convert this quantity to a divergence between full responses from the two policies\. Passing from responses to their binary rewards then bounds the reward error\. Lemmas[C\.10](https://arxiv.org/html/2611.21791#A3.Thmtheorem10)–[C\.13](https://arxiv.org/html/2611.21791#A3.Thmtheorem13)prove the target, coverage, and reward\-transfer statements\. The Q\-function target and angular transfer\.The first lemma identifies the population optimizer and expresses its objective gap as conditional KL\. The second proves that this optimizer improves reward, providing the ideal update for the outer analysis\. Writeμ=πk\{\\mu\}=\\pi\_\{k\},η=1/β\\eta=1/\\beta, and initially allow anyβ≥2\\beta\\geq 2\. LetΓk\(η\)\\Gamma\_\{k\}\(\\eta\)be the full family with token log\-ratio radiusr=ηr=\\eta\. Atβ=2\\beta=2it is the familyΓk\\Gamma\_\{k\}of Section[C\.2\.4](https://arxiv.org/html/2611.21791#A3.SS2.SSS4)withr=1/2r=1/2\. Let𝒲k,x\\mathcal\{W\}\_\{k,x\}denote𝒲k\\mathcal\{W\}\_\{k\}conditional on the promptxx, so that𝒲k=𝔼x𝒲k,x\\mathcal\{W\}\_\{k\}=\\mathbb\{E\}\_\{x\}\\mathcal\{W\}\_\{k,x\}\. ###### Lemma C\.10\(Feasible target and exact objective gap\)\. Forβ≥2\\beta\\geq 2, cap\-two reward and penalty weights, and the full radius\-1/β1/\\betafamily, the target in \([C\.29](https://arxiv.org/html/2611.21791#A3.E29)\) is feasible and maximizes the regularized objective at each prompt\. For every candidateπ\\piin this family and every prompt, Dk\(x\):=𝒲k,x\(πk∗\)−𝒲k,x\(π\)=β∑t𝔼ht∼dtμ\(⋅∣x\)KL\(π\(⋅∣ht\)∥πk∗\(⋅∣ht\)\)\.D\_\{k\}\(x\):=\\mathcal\{W\}\_\{k,x\}\(\\pi\_\{k\}^\{\*\}\)\-\\mathcal\{W\}\_\{k,x\}\(\\pi\)=\\beta\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\(\\cdot\\mid x\)\}\\mathrm\{KL\}\(\\pi\(\\cdot\\mid h\_\{t\}\)\\\|\\pi\_\{k\}^\{\*\}\(\\cdot\\mid h\_\{t\}\)\)\.\(C\.30\) ###### Proof\. OnΓk\(η\)\\Gamma\_\{k\}\(\\eta\)both caps are inactive\. The local objective is𝔼μ\[A\(ρ−1\)−βk\(ρ\)\]\\mathbb\{E\}\_\{\\mu\}\[A\(\\rho\-1\)\-\\beta k\(\\rho\)\], withk\(ρ\)=ρlogρ−ρ\+1k\(\\rho\)=\\rho\\log\\rho\-\\rho\+1\. Strict concavity and normalization give \([C\.29](https://arxiv.org/html/2611.21791#A3.E29)\)\. SinceQ∈\[0,1\]Q\\in\[0,1\], its ratios lie in\[e−η,eη\]\[e^\{\-\\eta\},e^\{\\eta\}\], so the unconstrained optimum is feasible\. For any fitted policyπ\\piin this family, the Gibbs variational identity at each history gives 𝔼πk∗A−βKL\(πk∗∥μ\)−𝔼πA\+βKL\(π∥μ\)=βKL\(π∥πk∗\)\.\\mathbb\{E\}\_\{\\pi\_\{k\}^\{\*\}\}A\-\\beta\\operatorname\{KL\}\(\\pi\_\{k\}^\{\*\}\\\|\\mu\)\-\\mathbb\{E\}\_\{\\pi\}A\+\\beta\\operatorname\{KL\}\(\\pi\\\|\\mu\)=\\beta\\operatorname\{KL\}\(\\pi\\\|\\pi\_\{k\}^\{\*\}\)\.Averaging over behavior histories conditional on the prompt proves Equation \([C\.30](https://arxiv.org/html/2611.21791#A3.E30)\)\. The gap is nonnegative for each prompt\. ∎ ###### Lemma C\.11\(Reward progress of the target\)\. Writesk∗\(x\)=Jx\(πk∗\)s\_\{k\}^\{\*\}\(x\)=J\_\{x\}\(\\pi\_\{k\}^\{\*\}\)for the target success probability, withπk∗\\pi\_\{k\}^\{\*\}defined in Equation \([C\.29](https://arxiv.org/html/2611.21791#A3.E29)\) andsk\(x\)=Jx\(πk\)s\_\{k\}\(x\)=J\_\{x\}\(\\pi\_\{k\}\)as in Section[C\.1](https://arxiv.org/html/2611.21791#A3.SS1)\. For deterministic binary terminal rewards, the target satisfieslogitsk∗\(x\)≥logitsk\(x\)\+log\(1\+1/β\)\\operatorname\{logit\}s\_\{k\}^\{\*\}\(x\)\\geq\\operatorname\{logit\}s\_\{k\}\(x\)\+\\log\(1\+1/\\beta\)\. ###### Proof\. Putf\(v\)=\(1\+η\)v/\(1\+ηv\)f\(v\)=\(1\+\\eta\)v/\(1\+\\eta v\)\. For any random variableQ∈\[0,1\]Q\\in\[0,1\]with meanQ¯\\overline\{Q\}, 𝔼\[eηQ\(f\(Q\)−f\(Q¯\)\)\]=1\+η1\+ηQ¯𝔼\[\(Q−Q¯\)eηQ1\+ηQ\]≥0\.\\mathbb\{E\}\[e^\{\\eta Q\}\(f\(Q\)\-f\(\\overline\{Q\}\)\)\]=\\frac\{1\+\\eta\}\{1\+\\eta\\overline\{Q\}\}\\mathbb\{E\}\\\!\\left\[\(Q\-\\overline\{Q\}\)\\frac\{e^\{\\eta Q\}\}\{1\+\\eta Q\}\\right\]\\geq 0\.The last expectation is nonnegative becauseeηv/\(1\+ηv\)e^\{\\eta v\}/\(1\+\\eta v\)is nondecreasing, so its covariance withvvis nonnegative\. At terminal historiesf\(R\)=Rf\(R\)=Rfor binary rewards\. Backward induction using the target’s token probabilities therefore provesVπk∗\(h\)≥f\(Vμ\(h\)\)V^\{\\pi\_\{k\}^\{\*\}\}\(h\)\\geq f\(V^\{\\mu\}\(h\)\)at every history\. In particular, logitsk∗\(x\)≥logitsk\(x\)\+log\(1\+1/β\)\.\\operatorname\{logit\}s\_\{k\}^\{\*\}\(x\)\\geq\\operatorname\{logit\}s\_\{k\}\(x\)\+\\log\(1\+1/\\beta\)\.Endpoint probabilities are interpreted by continuity\. ∎ The target in Equation \([C\.29](https://arxiv.org/html/2611.21791#A3.E29)\) has a prefix\-probability bound independent of the horizon\. This lets us compare errors measured under behavior histories with errors along target responses\. ###### Lemma C\.12\(Horizon\-independent prefix and response coverage\)\. Letμ=πk\\mu=\\pi\_\{k\}and letπk∗\\pi\_\{k\}^\{\*\}be the regularized TIS target in Equation \([C\.29](https://arxiv.org/html/2611.21791#A3.E29)\), withβ≥2\\beta\\geq 2\. For every prompt and prefix, including terminal histories att=H\+1t=H\+1, dtπk∗\(ht∣x\)dtμ\(ht∣x\)≤exp\(Vtμ\(ht\)−V1μ\(x\)β\)≤e1/β\.\\frac\{d\_\{t\}^\{\\pi\_\{k\}^\{\*\}\}\(h\_\{t\}\\mid x\)\}\{d\_\{t\}^\{\\mu\}\(h\_\{t\}\\mid x\)\}\\leq\\exp\\\!\\left\(\\frac\{V\_\{t\}^\{\\mu\}\(h\_\{t\}\)\-V\_\{1\}^\{\\mu\}\(x\)\}\{\\beta\}\\right\)\\leq e^\{1/\\beta\}\.\(C\.31\)At a terminal history,VH\+1μ=RV\_\{H\+1\}^\{\\mu\}=RandV1μ\(x\)=Jx\(μ\)V\_\{1\}^\{\\mu\}\(x\)=J\_\{x\}\(\\mu\), so πk∗\(Y∣x\)μ\(Y∣x\)≤exp\(R\(x,Y\)−Jx\(μ\)β\)≤e1/β\.\\frac\{\\pi\_\{k\}^\{\*\}\(Y\\mid x\)\}\{\\mu\(Y\\mid x\)\}\\leq\\exp\\\!\\left\(\\frac\{R\(x,Y\)\-J\_\{x\}\(\\mu\)\}\{\\beta\}\\right\)\\leq e^\{1/\\beta\}\.\(C\.32\)Atβ=2\\beta=2, both prefix and response ratios are therefore at moste1/2e^\{1/2\}, even though the target’s token log\-ratio radius is1/21/2\. ###### Proof\. Writeη=1/β\\eta=1/\\beta\. Jensen’s inequality gives logπk∗\(a∣h\)μ\(a∣h\)=ηQμ\(h,a\)−log𝔼u∼μ\(⋅∣h\)eηQμ\(h,u\)≤η\(Qμ\(h,a\)−Vμ\(h\)\)\.\\log\\frac\{\\pi\_\{k\}^\{\*\}\(a\\mid h\)\}\{\{\\mu\}\(a\\mid h\)\}=\\eta Q^\{\\mu\}\(h,a\)\-\\log\\mathbb\{E\}\_\{u\\sim\{\\mu\}\(\\cdot\\mid h\)\}e^\{\\eta Q^\{\\mu\}\(h,u\)\}\\leq\\eta\\bigl\(Q^\{\\mu\}\(h,a\)\-V^\{\\mu\}\(h\)\\bigr\)\.On the token tree,Qtμ\(ht,at\)=Vt\+1μ\(ht\+1\)Q\_\{t\}^\{\\mu\}\(h\_\{t\},a\_\{t\}\)=V\_\{t\+1\}^\{\\mu\}\(h\_\{t\+1\}\)\. Sum along the prefix\. The value differences telescope, proving the first inequality in \([C\.31](https://arxiv.org/html/2611.21791#A3.E31)\)\. Values lie in\[0,1\]\[0,1\], which proves its second inequality\. Att=H\+1t=H\+1, the prefix is the full response and the terminal value isRR, giving Equation \([C\.32](https://arxiv.org/html/2611.21791#A3.E32)\)\. ∎ Combining this coverage bound with the exact gap in Equation \([C\.30](https://arxiv.org/html/2611.21791#A3.E30)\) controls the reward loss accumulated across rounds\. ###### Lemma C\.13\(Objective gap controls accumulated reward error\)\. Forβ≥2\\beta\\geq 2and candidates in the radius\-1/β1/\\betafamily, KL\(πk∗\(Y∣x\)∥π\(Y∣x\)\)≤e3/βDk\(x\)/β≤5Dk\(x\)/β\.\\mathrm\{KL\}\(\\pi\_\{k\}^\{\*\}\(Y\\mid x\)\\\|\\pi\(Y\\mid x\)\)\\leq e^\{3/\\beta\}D\_\{k\}\(x\)/\\beta\\leq 5D\_\{k\}\(x\)/\\beta\.Consequently, the one\-round reward shortfall satisfies Jx\(πk∗\)−Jx\(π\)≤e3/βDk\(x\)2β\.J\_\{x\}\(\\pi\_\{k\}^\{\*\}\)\-J\_\{x\}\(\\pi\)\\leq\\sqrt\{\\frac\{e^\{3/\\beta\}D\_\{k\}\(x\)\}\{2\\beta\}\}\.IfK≥⌈2βlog\(6N/ϵ\)⌉K\\geq\\lceil 2\\beta\\log\(6N/\\epsilon\)\\rceil, the fitted sequence satisfies VN\(π0\)−J\(πK\)≤ϵ/2\+∑k<K𝔼xℒk\(x\),ℒk\(x\)=min\{1,κDk\(x\)\},κ=10π2NKβ\.V\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilon/2\+\\sum\_\{k<K\}\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}\(x\),\\qquad\\mathcal\{L\}\_\{k\}\(x\)=\\min\\\{1,\\kappa D\_\{k\}\(x\)\\\},\\quad\\kappa=\\frac\{10\\pi^\{2\}NK\}\{\\beta\}\.\(C\.33\) ###### Proof\. Bothπ\\piandπk∗\\pi\_\{k\}^\{\*\}have ratios toμ\{\\mu\}in\[e−η,eη\]\[e^\{\-\\eta\},e^\{\\eta\}\]\. Comparing the second derivatives of the two KL integrands on\[e−2η,e2η\]\[e^\{\-2\\eta\},e^\{2\\eta\}\]givesKL\(πk∗∥π\)≤e2ηKL\(π∥πk∗\)\\mathrm\{KL\}\(\\pi\_\{k\}^\{\*\}\\\|\\pi\)\\leq e^\{2\\eta\}\\mathrm\{KL\}\(\\pi\\\|\\pi\_\{k\}^\{\*\}\)at every history\. The chain rule and \([C\.31](https://arxiv.org/html/2611.21791#A3.E31)\) imply KL\(πk∗\(Y∣x\)∥π\(Y∣x\)\)≤e3ηβDk\(x\)\.\\mathrm\{KL\}\(\\pi\_\{k\}^\{\*\}\(Y\\mid x\)\\\|\\pi\(Y\\mid x\)\)\\leq\\frac\{e^\{3\\eta\}\}\{\\beta\}D\_\{k\}\(x\)\.Pinsker’s inequality bounds the reward difference by the square root of half the full\-response KL, proving the one\-round statement\. Forβ≥2\\beta\\geq 2,e3/β≤e3/2<5e^\{3/\\beta\}\\leq e^\{3/2\}<5\. Takeλ=log\(1\+1/β\)\\lambda=\\log\(1\+1/\\beta\)in Lemma[C\.3](https://arxiv.org/html/2611.21791#A3.Thmtheorem3)\. The target satisfieslogitsk∗≥logitsk\+λ\\operatorname\{logit\}s\_\{k\}^\{\*\}\\geq\\operatorname\{logit\}s\_\{k\}\+\\lambda\. Data processing to the binary reward and the Bernoulli\-KL deficit bound in Lemma[C\.3](https://arxiv.org/html/2611.21791#A3.Thmtheorem3)bound the squared one\-sided angular deficitξk\(x\)2\\xi\_\{k\}\(x\)^\{2\}by5π2Dk\(x\)/\(8β\)5\\pi^\{2\}D\_\{k\}\(x\)/\(8\\beta\)\. Withs\(x\)=1−\(1−s0\(x\)\)Ns\(x\)=1\-\(1\-s\_\{0\}\(x\)\)^\{N\}, the pointwise bound \([C\.18](https://arxiv.org/html/2611.21791#A3.E18)\) at accuracyϵ/2\\epsilon/2gives s\(x\)−sK\(x\)≤ϵ/2\+min\{1,16NK∑k<Kξk\(x\)2\}\.s\(x\)\-s\_\{K\}\(x\)\\leq\\epsilon/2\+\\min\\left\\\{1,16NK\\sum\_\{k<K\}\\xi\_\{k\}\(x\)^\{2\}\\right\\\}\.Sincemin\{1,∑kyk\}≤∑kmin\{1,yk\}\\min\\\{1,\\sum\_\{k\}y\_\{k\}\\\}\\leq\\sum\_\{k\}\\min\\\{1,y\_\{k\}\\\}foryk≥0y\_\{k\}\\geq 0, this implies Equation \([C\.33](https://arxiv.org/html/2611.21791#A3.E33)\)\. The chosenKKsuffices becauselog\(1\+1/β\)≥1/\(2β\)\\log\(1\+1/\\beta\)\\geq 1/\(2\\beta\)\. ∎ ### C\.3Inner fitting and statistical accuracy The inner solver uses a fixed round dataset\. The following lemmas connect its empirical accuracy to population accuracy by controlling response noise within each prompt and variation between prompts\. The fitting certificates hold uniformly over candidates, so they also apply to a policy chosen using that dataset\. #### C\.3\.1Empirical objectives and gradient identities SC and SC\+TIS optimize different objectives\.Fix the behavior policyμ\{\\mu\}\. Writeψθ\(h,a\)\\psi\_\{\\theta\}\(h,a\)for the learner’s log\-policy gradient, and fix the reward weighta\(ρ\)a\(\\rho\)as defined in Section[C\.1](https://arxiv.org/html/2611.21791#A3.SS1)\. The weighted score is centered underμ\{\\mu\}before it is multiplied by the response reward\. The resulting sampled update is Ga\(θ,y\)=R\(y\)∑t=1H\[a\(ρθ,t\(ht,at\)\)ψθ,t\(ht,at\)−𝔼b∼μ\(⋅∣ht\)a\(ρθ,t\(ht,b\)\)ψθ,t\(ht,b\)\]\.G\_\{a\}\(\\theta,y\)=R\(y\)\\sum\_\{t=1\}^\{H\}\\left\[a\(\\rho\_\{\\theta,t\}\(h\_\{t\},a\_\{t\}\)\)\\psi\_\{\\theta,t\}\(h\_\{t\},a\_\{t\}\)\-\\mathbb\{E\}\_\{b\\sim\{\\mu\}\(\\cdot\\mid h\_\{t\}\)\}a\(\\rho\_\{\\theta,t\}\(h\_\{t\},b\)\)\\psi\_\{\\theta,t\}\(h\_\{t\},b\)\\right\]\.\(C\.34\)Hereρθ,t\(h,b\)=πθ\(b∣h\)/μ\(b∣h\)\\rho\_\{\\theta,t\}\(h,b\)=\{\\pi\_\{\\theta\}\}\(b\\mid h\)/\{\\mu\}\(b\\mid h\)andψθ,t=∇θlogπθ\\psi\_\{\\theta,t\}=\\nabla\_\{\\theta\}\\log\{\\pi\_\{\\theta\}\}\. The effective reward ratio isρTIS=a\(ρ\)\\rho^\{\\mathrm\{TIS\}\}=a\(\\rho\): SC usesρTIS=1\\rho^\{\\mathrm\{TIS\}\}=1, and SC\+TIS usesρTIS=min\(ρ,c\)\\rho^\{\\mathrm\{TIS\}\}=\\min\(\\rho,c\)with capc≥1c\\geq 1\. Each token in the conditional mean receives its own weight\. The potentialFaF\_\{a\}and population objective𝒞a\\mathcal\{C\}\_\{a\}are defined in Equations \([C\.2](https://arxiv.org/html/2611.21791#A3.E2)\) and \([C\.3](https://arxiv.org/html/2611.21791#A3.E3)\)\. In that objective, the behavior policy, its histories, and its continuation advantages remain fixed\. The expected centered update is the gradient of this objective, including for shared nonlinear logits\. Proposition[C\.14](https://arxiv.org/html/2611.21791#A3.Thmtheorem14)proves the identity\. The objective can be estimated from rewards and token probabilities, without evaluating continuation values\. For a group ofG≥2G\\geq 2independent responses to the same prompt, subtracting their mean reward uses the correctionG/\(G−1\)G/\(G\-1\)because those responses also estimate their own baseline\. We write𝒞^k\\widehat\{\\mathcal\{C\}\}\_\{k\}for the resulting empirical objective at roundkk\. Equations \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) and \([C\.36](https://arxiv.org/html/2611.21791#A3.E36)\) give its exact form and gradient\. ###### Proposition C\.14\(The population gradient of score centering\)\. Fix the behavior policyμ\\mu, reward weightaa, and a differentiable full\-support candidateπθ\\pi\_\{\\theta\}\. With exact conditional centers, the response update in Equation \([C\.34](https://arxiv.org/html/2611.21791#A3.E34)\) satisfies 𝔼μGa\(θ,Y\)=𝔼μ∑tAtμa\(ρθ,t\)ψθ,t=∇θ𝒞a\(πθ\),\\mathbb\{E\}\_\{\\mu\}G\_\{a\}\(\\theta,Y\)=\\mathbb\{E\}\_\{\\mu\}\\sum\_\{t\}A\_\{t\}^\{\\mu\}a\(\\rho\_\{\\theta,t\}\)\\psi\_\{\\theta,t\}=\\nabla\_\{\\theta\}\\mathcal\{C\}\_\{a\}\(\\pi\_\{\\theta\}\),whenever differentiation and expectation can be interchanged\. The expectation is for a fixed candidate, or one independent of the sampled responses\. On a fixed dataset, the grouped update instead gives the gradient of the empirical objective, as stated in Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15)withβ=0\\beta=0\. ###### Proof\. Condition the sampled reward on\(ht,at\)\(h\_\{t\},a\_\{t\}\)in the first term of Equation \([C\.34](https://arxiv.org/html/2611.21791#A3.E34)\), and onhth\_\{t\}in the centered term\. Their difference in conditional expectation isAtμa\(ρt\)ψθ,tA\_\{t\}^\{\\mu\}a\(\\rho\_\{t\}\)\\psi\_\{\\theta,t\}\. Since behavior and its values remain fixed,ρFa′\(ρ\)=a\(ρ\)\\rho F\_\{a\}^\{\\prime\}\(\\rho\)=a\(\\rho\)and∇θρ=ρψθ,t\\nabla\_\{\\theta\}\\rho=\\rho\\psi\_\{\\theta,t\}identify this expectation with∇θ𝒞a\\nabla\_\{\\theta\}\\mathcal\{C\}\_\{a\}\. The group\-centering identity in Appendix[C\.3\.2](https://arxiv.org/html/2611.21791#A3.SS3.SSS2)gives the same population expectation for grouped responses\. ∎ ##### Sampling and group centering\. Exact conditional score centering and same\-prompt reward centering have separate roles: the first centers each token score, while the second estimates a reward baseline from independent responses\. Conditional centering is exact in this analysis\. An independently sampled, conditionally unbiased Monte Carlo center adds variance without introducing gradient bias\. Systematic error in the center can bias the gradient\. For a fixed candidate, letg^\\widehat\{g\}average Equation \([C\.34](https://arxiv.org/html/2611.21791#A3.E34)\) overnnindependent responses\. If‖ψθ,t‖≤B\\\|\\psi\_\{\\theta,t\}\\\|\\leq Band𝔼b∼μ\(⋅∣h\)a\(ρθ,t\(h,b\)\)2≤Ma\\mathbb\{E\}\_\{b\\sim\\mu\(\\cdot\\mid h\)\}a\(\\rho\_\{\\theta,t\}\(h,b\)\)^\{2\}\\leq M\_\{a\}uniformly, then 𝔼‖g^−∇𝒞a‖2≤4B2H2Man\.\\mathbb\{E\}\\\|\\widehat\{g\}\-\\nabla\\mathcal\{C\}\_\{a\}\\\|^\{2\}\\leq\\frac\{4B^\{2\}H^\{2\}M\_\{a\}\}\{n\}\.\(C\.35\)One may takeMa=1M\_\{a\}=1for SC andMa=cM\_\{a\}=cfor TIS, sincemin\(ρ,c\)2≤cρ\\min\(\\rho,c\)^\{2\}\\leq c\\rhoand𝔼μρ=1\\mathbb\{E\}\_\{\\mu\}\\rho=1\. For the variance, putXt=a\(ρt\)ψtX\_\{t\}=a\(\\rho\_\{t\}\)\\psi\_\{t\}\. Conditional centering gives𝔼‖Xt−𝔼\[Xt∣ht\]‖2≤B2Ma\\mathbb\{E\}\\\|X\_\{t\}\-\\mathbb\{E\}\[X\_\{t\}\\mid h\_\{t\}\]\\\|^\{2\}\\leq B^\{2\}M\_\{a\}\. Since0≤R≤10\\leq R\\leq 1and‖∑tvt‖2≤H∑t‖vt‖2\\\|\\sum\_\{t\}v\_\{t\}\\\|^\{2\}\\leq H\\sum\_\{t\}\\\|v\_\{t\}\\\|^\{2\}, the second moment of a response update is at mostB2H2MaB^\{2\}H^\{2\}M\_\{a\}\. Independent averaging gives \([C\.35](https://arxiv.org/html/2611.21791#A3.E35)\), with its stated conservative factor four\. Group centering\.We can also subtract a reward baseline estimated from independent responses to the same prompt\. This group centering is in addition to the conditional token\-score centering in \([C\.34](https://arxiv.org/html/2611.21791#A3.E34)\)\. For a promptxx, drawG≥2G\\geq 2independent responsesy\(g\)∼μ\(⋅∣x\)y^\{\(g\)\}\\sim\{\\mu\}\(\\cdot\\mid x\)\. For responseggand token positiontt, writeht\(g\)=\(x,a<t\(g\)\)h\_\{t\}^\{\(g\)\}=\(x,a\_\{<t\}^\{\(g\)\}\)and define the centered weighted score St\(g\)=a\(ρθ,t\(ht\(g\),at\(g\)\)\)ψθ,t\(ht\(g\),at\(g\)\)−𝔼b∼μ\(⋅∣ht\(g\)\)\[a\(ρθ,t\(ht\(g\),b\)\)ψθ,t\(ht\(g\),b\)\]\.S\_\{t\}^\{\(g\)\}=a\\\!\\left\(\\rho\_\{\\theta,t\}\(h\_\{t\}^\{\(g\)\},a\_\{t\}^\{\(g\)\}\)\\right\)\\psi\_\{\\theta,t\}\(h\_\{t\}^\{\(g\)\},a\_\{t\}^\{\(g\)\}\)\-\\mathbb\{E\}\_\{b\\sim\{\\mu\}\(\\cdot\\mid h\_\{t\}^\{\(g\)\}\)\}\\\!\\left\[a\\\!\\left\(\\rho\_\{\\theta,t\}\(h\_\{t\}^\{\(g\)\},b\)\\right\)\\psi\_\{\\theta,t\}\(h\_\{t\}^\{\(g\)\},b\)\\right\]\.The expectation averages over the current token at the sampled history\.ggindexes responses andttindexes tokens\. The grouped update is g^xgrp=1G−1∑g=1G\(R\(y\(g\)\)−R¯\)∑t=1HSt\(g\),R¯=1G∑g=1GR\(y\(g\)\)\.\\widehat\{g\}\_\{x\}^\{\\mathrm\{grp\}\}=\\frac\{1\}\{G\-1\}\\sum\_\{g=1\}^\{G\}\\bigl\(R\(y^\{\(g\)\}\)\-\\overline\{R\}\\bigr\)\\sum\_\{t=1\}^\{H\}S\_\{t\}^\{\(g\)\},\\qquad\\overline\{R\}=\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}R\(y^\{\(g\)\}\)\.\(C\.36\)The factor1/\(G−1\)1/\(G\-1\)removes the bias from using the same group to estimate the baseline and the update\. Since𝔼\[St∣x\]=0\\mathbb\{E\}\[S\_\{t\}\\mid x\]=0, the grouped estimator has the same conditional expectation as the original update\. Averaging independent groups with prompts drawn from𝒟\\mathcal\{D\}therefore preserves the objective in \([C\.3](https://arxiv.org/html/2611.21791#A3.E3)\)\. Both the group baseline and the sampled returns are detached during differentiation\. Appendix[C\.3\.2](https://arxiv.org/html/2611.21791#A3.SS3.SSS2)proves this identity and bounds the within\-prompt variance by4B2H2Ma/\(G−1\)4B^\{2\}H^\{2\}M\_\{a\}/\(G\-1\)under the score and weight\-moment assumptions of \([C\.35](https://arxiv.org/html/2611.21791#A3.E35)\)\. Random prompt sampling also contributes between\-prompt variance\. Forβ=0\\beta=0, Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) gives𝒞^k\\widehat\{\\mathcal\{C\}\}\_\{k\}, whose gradient is the average of Equation \([C\.36](https://arxiv.org/html/2611.21791#A3.E36)\) over the sampled prompts\. The dataset stays fixed during fitting; minibatches estimate the gradient of this empirical objective\. Regularized updates and grouped fitting\.Writeψθ,t\(h,u\)=∇θlogπθ\(u∣h\)\\psi\_\{\\theta,t\}\(h,u\)=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(u\\mid h\)for the score from Equation \([1](https://arxiv.org/html/2611.21791#S3.E1)\)\. For a sampled response with rewardR=R\(Y\)R=R\(Y\), use the main\-text weightsρtTIS=a\(ρt\)\\rho\_\{t\}^\{\\mathrm\{TIS\}\}=a\(\\rho\_\{t\}\)andρtKL=b\(ρt\)\\rho\_\{t\}^\{\\mathrm\{KL\}\}=b\(\\rho\_\{t\}\)\. The response update is Gk\(θ,Y\)=∑t\[\\displaystyle G\_\{k\}\(\\theta,Y\)=\\sum\_\{t\}\\Big\[R\(ρtTISψθ,t\(ht,at\)−𝔼u∼μ\(⋅∣ht\)\[ρtTIS\(ht,u\)ψθ,t\(ht,u\)\]\)\\displaystyle R\\bigl\(\\rho\_\{t\}^\{\\mathrm\{TIS\}\}\\psi\_\{\\theta,t\}\(h\_\{t\},a\_\{t\}\)\-\\mathbb\{E\}\_\{u\\sim\\mu\(\\cdot\\mid h\_\{t\}\)\}\[\\rho\_\{t\}^\{\\mathrm\{TIS\}\}\(h\_\{t\},u\)\\psi\_\{\\theta,t\}\(h\_\{t\},u\)\]\\bigr\)\(C\.37\)−βρtKLlogρtKLψθ,t\(ht,at\)\]\.\\displaystyle\-\\beta\\rho\_\{t\}^\{\\mathrm\{KL\}\}\\log\\rho\_\{t\}^\{\\mathrm\{KL\}\}\\psi\_\{\\theta,t\}\(h\_\{t\},a\_\{t\}\)\\Big\]\.The sampled rewardR\(Y\)R\(Y\)is held fixed inside the token expectation\. The reward term is score\-centered\. The penalty coefficient satisfies−1/e≤b\(ρ\)logb\(ρ\)≤cblogcb\-1/e\\leq b\(\\rho\)\\log b\(\\rho\)\\leq c\_\{b\}\\log c\_\{b\}\. Group centering\.For a promptxx, drawG≥2G\\geq 2independent responsesy\(g\)∼μ\(⋅∣x\)y^\{\(g\)\}\\sim\{\\mu\}\(\\cdot\\mid x\)and define the corrected centered rewards R¯=1G∑g=1GR\(y\(g\)\),R~\(g\)=GG−1\(R\(y\(g\)\)−R¯\)\.\\overline\{R\}=\\frac\{1\}\{G\}\\sum\_\{g=1\}^\{G\}R\(y^\{\(g\)\}\),\\qquad\\widetilde\{R\}^\{\(g\)\}=\\frac\{G\}\{G\-1\}\\bigl\(R\(y^\{\(g\)\}\)\-\\overline\{R\}\\bigr\)\.For responsegg, letht\(g\)h\_\{t\}^\{\(g\)\}be its history and abbreviateψt\(g\)=ψθ,t\(ht\(g\),at\(g\)\)\\psi\_\{t\}^\{\(g\)\}=\\psi\_\{\\theta,t\}\(h\_\{t\}^\{\(g\)\},a\_\{t\}^\{\(g\)\}\)andρt\(g\)=ρt\(ht\(g\),at\(g\)\)\\rho\_\{t\}^\{\(g\)\}=\\rho\_\{t\}\(h\_\{t\}^\{\(g\)\},a\_\{t\}^\{\(g\)\}\)\. The grouped update is g^k,xgrp=1G∑g,t\[\\displaystyle\\widehat\{g\}\_\{k,x\}^\{\\mathrm\{grp\}\}=\\frac\{1\}\{G\}\\sum\_\{g,t\}\\Big\[R~\(g\)\(a\(ρt\(g\)\)ψt\(g\)−𝔼u∼μ\(⋅∣ht\(g\)\)\[a\(ρt\(ht\(g\),u\)\)ψθ,t\(ht\(g\),u\)\]\)\\displaystyle\\widetilde\{R\}^\{\(g\)\}\\bigl\(a\(\\rho\_\{t\}^\{\(g\)\}\)\\psi\_\{t\}^\{\(g\)\}\-\\mathbb\{E\}\_\{u\\sim\\mu\(\\cdot\\mid h\_\{t\}^\{\(g\)\}\)\}\[a\(\\rho\_\{t\}\(h\_\{t\}^\{\(g\)\},u\)\)\\psi\_\{\\theta,t\}\(h\_\{t\}^\{\(g\)\},u\)\]\\bigr\)\(C\.38\)−βb\(ρt\(g\)\)logb\(ρt\(g\)\)ψt\(g\)\]\.\\displaystyle\-\\beta b\(\\rho\_\{t\}^\{\(g\)\}\)\\log b\(\\rho\_\{t\}^\{\(g\)\}\)\\psi\_\{t\}^\{\(g\)\}\\Big\]\. ###### Proposition C\.15\(The empirical objective and its stochastic gradient\)\. Letπθ\\pi\_\{\\theta\}be differentiable with full support, and hold behaviorμ=πk\\mu=\\pi\_\{k\}fixed\. With exact conditional centers, for the empirical objective in Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) and grouped update in Equation \([C\.38](https://arxiv.org/html/2611.21791#A3.E38)\), ∇θ𝒲^k\(πθ\)=1m∑j=1mg^k,xjgrp\.\\nabla\_\{\\theta\}\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\_\{\\theta\}\)=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\widehat\{g\}\_\{k,x\_\{j\}\}^\{\\mathrm\{grp\}\}\.For each fixedθ\\theta, independent prompts and conditionally independent responses give, whenever differentiation and expectation can be interchanged, 𝔼∇θ𝒲^k\(πθ\)=𝔼μGk\(θ,Y\)=∇θ𝒲k\(πθ\)\.\\mathbb\{E\}\\nabla\_\{\\theta\}\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\_\{\\theta\}\)=\\mathbb\{E\}\_\{\\mu\}G\_\{k\}\(\\theta,Y\)=\\nabla\_\{\\theta\}\\mathcal\{W\}\_\{k\}\(\\pi\_\{\\theta\}\)\.The empirical identity holds on the fixed dataset, including when that dataset is reused for fitting\. The expectation identity is for a fixed candidate, or for a candidate independent of the fresh data\. Replacing exact centers by conditionally unbiased estimates, sampled independently of the response actions and rewards given the visited history, preserves the population expectation identity\. The empirical identity above uses exact centers\. The centered rewardR~\(g\)\\widetilde\{R\}^\{\(g\)\}is held fixed inside the token expectation\. As in \([C\.36](https://arxiv.org/html/2611.21791#A3.E36)\), the factorG/\(G−1\)G/\(G\-1\)corrects the bias from estimating the baseline with the same group\. The penalty is averaged over responses with coefficientβ\\beta\. Averaging independent prompt groups gives an unbiased gradient estimator of𝒲k\\mathcal\{W\}\_\{k\}\. Equation \([C\.37](https://arxiv.org/html/2611.21791#A3.E37)\) also gives an unbiased single\-response gradient\. The grouped objective and convergence theorems useG≥2G\\geq 2\. When implementing the update through weighted log\-policy gradients, detach the reward baseline and both score coefficients\. The following sample objective gives a direct proof of these identities\. For the proofs, writeψθ,t\(h,u\)=∇θlogπθ\(u∣h\)\\psi\_\{\\theta,t\}\(h,u\)=\\nabla\_\{\\theta\}\\log\{\\pi\_\{\\theta\}\}\(u\\mid h\)\. UsingFaF\_\{a\}from Equation \([C\.2](https://arxiv.org/html/2611.21791#A3.E2)\) andKbK\_\{b\}from Equation \([C\.4](https://arxiv.org/html/2611.21791#A3.E4)\), define za,t\(h,u\)=Fa\(ρt\(h,u\)\)−𝔼v∼μ\(⋅∣h\)Fa\(ρt\(h,v\)\)\.z\_\{a,t\}\(h,u\)=F\_\{a\}\(\\rho\_\{t\}\(h,u\)\)\-\\mathbb\{E\}\_\{v\\sim\{\\mu\}\(\\cdot\\mid h\)\}F\_\{a\}\(\\rho\_\{t\}\(h,v\)\)\.An unbiased sample objective is ℋk\(π,Y\)=R\(Y\)∑tza,t\(ht,at\)−β∑tKb\(ρt\(ht,at\)\)\.\\mathcal\{H\}\_\{k\}\(\\pi,Y\)=R\(Y\)\\sum\_\{t\}z\_\{a,t\}\(h\_\{t\},a\_\{t\}\)\-\\beta\\sum\_\{t\}K\_\{b\}\(\\rho\_\{t\}\(h\_\{t\},a\_\{t\}\)\)\.\(C\.39\) ###### Proof of Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15)\. Because behavior is fixed,∇θρt=ρtψθ,t\\nabla\_\{\\theta\}\\rho\_\{t\}=\\rho\_\{t\}\\psi\_\{\\theta,t\}\. The identitiesρFa′\(ρ\)=a\(ρ\)\\rho F\_\{a\}^\{\\prime\}\(\\rho\)=a\(\\rho\)andρKb′\(ρ\)=b\(ρ\)logb\(ρ\)\\rho K\_\{b\}^\{\\prime\}\(\\rho\)=b\(\\rho\)\\log b\(\\rho\)give ∇θza,t=a\(ρt\)ψθ,t−𝔼μta\(ρt\)ψθ,t,∇θKb\(ρt\)=b\(ρt\)logb\(ρt\)ψθ,t\.\\nabla\_\{\\theta\}z\_\{a,t\}=a\(\\rho\_\{t\}\)\\psi\_\{\\theta,t\}\-\\mathbb\{E\}\_\{\{\\mu\_\{t\}\}\}a\(\\rho\_\{t\}\)\\psi\_\{\\theta,t\},\\qquad\\nabla\_\{\\theta\}K\_\{b\}\(\\rho\_\{t\}\)=b\(\\rho\_\{t\}\)\\log b\(\\rho\_\{t\}\)\\psi\_\{\\theta,t\}\.Differentiating \([C\.39](https://arxiv.org/html/2611.21791#A3.E39)\) proves the exact sample\-gradient identity∇θℋk\(πθ,Y\)=Gk\(θ,Y\)\\nabla\_\{\\theta\}\\mathcal\{H\}\_\{k\}\(\{\\pi\_\{\\theta\}\},Y\)=G\_\{k\}\(\\theta,Y\)\. Conditional advantage centering makes the reward potential unbiased for𝒞k\\mathcal\{C\}\_\{k\}, and the penalty is averaged under the same fixed behavior distribution\. This proves the population identity\. Same\-prompt group centering preserves the reward expectation by Appendix[C\.3\.2](https://arxiv.org/html/2611.21791#A3.SS3.SSS2)\. Response averaging preserves the penalty expectation and its gradient\. Specifically, forG≥2G\\geq 2, use the unbiased same\-prompt sample covariance ofRRwith∑tza,t\\sum\_\{t\}z\_\{a,t\}for the reward term and the response average for the penalty\. Averaging these quantities overmmprompts is precisely𝒲^k\\widehat\{\\mathcal\{W\}\}\_\{k\}in Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\)\. Differentiating this objective gives the average of Equation \([C\.38](https://arxiv.org/html/2611.21791#A3.E38)\) over prompts\. For independent conditionally unbiased center estimates, conditioning on the response and then averaging the auxiliary draws recovers the exact\-center update\. The tower property therefore preserves the population gradient identity\. ∎ Properties of the penalty\.Monotonicity andb\(1\)=1b\(1\)=1implyb\(ρ\)logb\(ρ\)≤0b\(\\rho\)\\log b\(\\rho\)\\leq 0forρ<1\\rho<1andb\(ρ\)logb\(ρ\)≥0b\(\\rho\)\\log b\(\\rho\)\\geq 0forρ\>1\\rho\>1\. Integrating from11therefore givesKb\(ρ\)≥0K\_\{b\}\(\\rho\)\\geq 0\. The bound onblogbb\\log bfollows from the minimum ofxlogxx\\log xatx=e−1x=e^\{\-1\}and its monotonicity forx≥1x\\geq 1\. Ifbbis differentiable at11, thenKb′\(1\)=0K\_\{b\}^\{\\prime\}\(1\)=0andKb′′\(1\)=b′\(1\)K\_\{b\}^\{\\prime\\prime\}\(1\)=b^\{\\prime\}\(1\)\. ThusKb\(ρ\)=12b′\(1\)\(ρ−1\)2\+o\(\(ρ−1\)2\)K\_\{b\}\(\\rho\)=\\tfrac\{1\}\{2\}b^\{\\prime\}\(1\)\(\\rho\-1\)^\{2\}\+o\(\(\\rho\-1\)^\{2\}\)\. Choosingb′\(1\)=1b^\{\\prime\}\(1\)=1matches KL’s local quadratic form\. Forb\(ρ\)=min\(ρ,cb\)b\(\\rho\)=\\min\(\\rho,c\_\{b\}\), integrate separately below and above the cap to obtain Kb\(ρ\)=\{ρlogρ−ρ\+1,ρ≤cb,cblogcb−cb\+1\+cblogcblog\(ρ/cb\),ρ\>cb\.K\_\{b\}\(\\rho\)=\\begin\{cases\}\\rho\\log\\rho\-\\rho\+1,&\\rho\\leq c\_\{b\},\\\\ c\_\{b\}\\log c\_\{b\}\-c\_\{b\}\+1\+c\_\{b\}\\log c\_\{b\}\\log\(\\rho/c\_\{b\}\),&\\rho\>c\_\{b\}\.\\end\{cases\}Above the cap, its score coefficient iscblogcbc\_\{b\}\\log c\_\{b\}\. The proof of Lemma[C\.22](https://arxiv.org/html/2611.21791#A3.Thmtheorem22)uses the interval on whichb\(ρ\)=ρb\(\\rho\)=\\rho\. #### C\.3\.2Variance and sample complexity for SC variants We begin with variance bounds for the SC and TIS score estimators\. These also describe the reward component of RSC; the regularized objective bounds below include the penalty\. For a fixed conditional centering policyτt\(⋅∣h\)\\tau\_\{t\}\(\\cdot\\mid h\)and a token weightχt\(h,a\)\\chi\_\{t\}\(h,a\), consider one behavior response’s contribution X\(θ\)=∑tχt\(ht,at\)Ut\[∇θlogπθ\(at∣ht\)−∑aτt\(a∣ht\)∇θlogπθ\(a∣ht\)\]\.X\(\\theta\)=\\sum\_\{t\}\\chi\_\{t\}\(h\_\{t\},a\_\{t\}\)U\_\{t\}\\left\[\\nabla\_\{\\theta\}\\log\{\\pi\_\{\\theta\}\}\(a\_\{t\}\\mid h\_\{t\}\)\-\\sum\_\{a\}\\tau\_\{t\}\(a\\mid h\_\{t\}\)\\nabla\_\{\\theta\}\\log\{\\pi\_\{\\theta\}\}\(a\\mid h\_\{t\}\)\\right\]\.Suppose the logits satisfy‖∇θfθ\(h,a\)‖≤B\\\|\\nabla\_\{\\theta\}f\_\{\\theta\}\(h,a\)\\\|\\leq B,\|Ut\|≤M\|U\_\{t\}\|\\leq M, and the token weights have bounded conditional second moments\. Fornnindependent responses and their averageg^\\widehat\{g\}, 𝔼∥g^−𝔼g^∥2≤4B2M2H2nsupt,h𝔼a∼μ\(⋅∣h\)χt\(h,a\)2\.\\mathbb\{E\}\\\|\\widehat\{g\}\-\\mathbb\{E\}\\widehat\{g\}\\\|^\{2\}\\leq\\frac\{4B^\{2\}M^\{2\}H^\{2\}\}\{n\}\\sup\_\{t,h\}\\mathbb\{E\}\_\{a\\sim\{\\mu\}\(\\cdot\\mid h\)\}\\chi\_\{t\}\(h,a\)^\{2\}\.\(C\.40\)Indeed the centered token score is∇fθ\(h,a\)−∑uτt\(u∣h\)∇fθ\(h,u\)\\nabla f\_\{\\theta\}\(h,a\)\-\\sum\_\{u\}\\tau\_\{t\}\(u\\mid h\)\\nabla f\_\{\\theta\}\(h,u\), whose norm is at most2B2B\. Apply‖∑tXt‖2≤H∑t‖Xt‖2\\\|\\sum\_\{t\}X\_\{t\}\\\|^\{2\}\\leq H\\sum\_\{t\}\\\|X\_\{t\}\\\|^\{2\}, then average over independent responses\. No independence between tokens of one response is assumed\. For SC,χt=1\\chi\_\{t\}=1,Ut=RU\_\{t\}=R, andM=1M=1\. For TIS,χt=min\(ρt,c\)\\chi\_\{t\}=\\min\(\\rho\_\{t\},c\)and𝔼μχt2≤c𝔼μρt=c\\mathbb\{E\}\_\{\\mu\}\\chi\_\{t\}^\{2\}\\leq c\\mathbb\{E\}\_\{\\mu\}\\rho\_\{t\}=c, giving the corresponding second\-moment constant\. Appendix[C\.3\.3](https://arxiv.org/html/2611.21791#A3.SS3.SSS3)treats the full RSC \(TIS\) objective, including its sample penalty\. For the centered weighted TIS score, use the moment calculation following Proposition[C\.14](https://arxiv.org/html/2611.21791#A3.Thmtheorem14), withMa=cM\_\{a\}=cand‖∇θlogπθ‖≤2B\\\|\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\\\|\\leq 2B\. This gives a bound of16B2H2c/n16B^\{2\}H^\{2\}c/nwhenBBbounds the logit gradient\. Same\-prompt group centering\.Fix the prompt and current iterate, and drawG≥2G\\geq 2independent responses\. WriteU¯t=G−1∑gUt\(g\)\\overline\{U\}\_\{t\}=G^\{\-1\}\\sum\_\{g\}U\_\{t\}^\{\(g\)\}for their mean token return andSt\(g\)S\_\{t\}^\{\(g\)\}for responsegg’s centered token score, as in Equation \([C\.36](https://arxiv.org/html/2611.21791#A3.E36)\)\. For that equationUt\(g\)=R\(y\(g\)\)U\_\{t\}^\{\(g\)\}=R\(y^\{\(g\)\}\)\. Suppose each token score satisfies𝔼\[St∣x\]=0\\mathbb\{E\}\[S\_\{t\}\\mid x\]=0\. For the estimator in \([C\.36](https://arxiv.org/html/2611.21791#A3.E36)\), independence between distinct responses gives 𝔼\[U¯tSt\(g\)∣x\]=1G𝔼\[UtSt∣x\]\.\\mathbb\{E\}\[\\overline\{U\}\_\{t\}S\_\{t\}^\{\(g\)\}\\mid x\]=\\frac\{1\}\{G\}\\mathbb\{E\}\[U\_\{t\}S\_\{t\}\\mid x\]\.Summing over responses and dividing byG−1G\-1proves 𝔼\[g^xgrp∣x\]=𝔼\[∑tUtSt\|x\]\.\\mathbb\{E\}\[\\widehat\{g\}\_\{x\}^\{\\mathrm\{grp\}\}\\mid x\]=\\mathbb\{E\}\\left\[\\sum\_\{t\}U\_\{t\}S\_\{t\}\\,\\middle\|\\,x\\right\]\.Thus group centering preserves the generalized SC reward gradient\. For the regularized objective in Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15), group\-center this reward term and average theb\(ρ\)logb\(ρ\)b\(\\rho\)\\log b\(\\rho\)penalty gradient directly\. This preserves the gradient of the stated empirical objective\. For a variance bound, assume\|Ut\|≤M\|U\_\{t\}\|\\leq Mand𝔼\[‖St‖2∣x\]≤σS2\\mathbb\{E\}\[\\\|S\_\{t\}\\\|^\{2\}\\mid x\]\\leq\\sigma\_\{S\}^\{2\}uniformly\. The grouped estimator is the average over all unordered response pairs of the symmetric kernel ℋ\(y,y′\)=12∑t\(Ut\(y\)−Ut\(y′\)\)\(St\(y\)−St\(y′\)\)\.\\mathcal\{H\}\(y,y^\{\\prime\}\)=\\frac\{1\}\{2\}\\sum\_\{t\}\(U\_\{t\}\(y\)\-U\_\{t\}\(y^\{\\prime\}\)\)\(S\_\{t\}\(y\)\-S\_\{t\}\(y^\{\\prime\}\)\)\.The two responses are independent, and their scores have mean zero\. Consequently,𝔼\[‖ℋ‖2∣x\]≤2H2M2σS2\\mathbb\{E\}\[\\\|\\mathcal\{H\}\\\|^\{2\}\\mid x\]\\leq 2H^\{2\}M^\{2\}\\sigma\_\{S\}^\{2\}\. To bound the variance of the pair average, average over random pairings of theGGresponses, leaving one response unused ifGGis odd\. Each pairing contains⌊G/2⌋\\lfloor G/2\\rfloorindependent pairs\. Its conditional expectation given the responses is the full pair average\. Conditional Jensen’s inequality therefore gives 𝔼\[‖g^xgrp−𝔼\[g^xgrp∣x\]‖2∣x\]≤4H2M2σS2G−1\.\\mathbb\{E\}\\bigl\[\\\|\\widehat\{g\}\_\{x\}^\{\\mathrm\{grp\}\}\-\\mathbb\{E\}\[\\widehat\{g\}\_\{x\}^\{\\mathrm\{grp\}\}\\mid x\]\\\|^\{2\}\\mid x\\bigr\]\\leq\\frac\{4H^\{2\}M^\{2\}\\sigma\_\{S\}^\{2\}\}\{G\-1\}\.\(C\.41\)For generalized SC,M=1M=1andσS2=B2Ma\\sigma\_\{S\}^\{2\}=B^\{2\}M\_\{a\}whenBBbounds the token score norm\. The separate penalty term is not part of this centered\-score bound\. Appendix[C\.3\.3](https://arxiv.org/html/2611.21791#A3.SS3.SSS3)bounds the full regularized objective\. Averagingmmindependent prompt groups divides the variance bymm\. The unconditional variance also includesVarx\(𝔼\[g^xgrp∣x\]\)/m\\operatorname\{Var\}\_\{x\}\(\\mathbb\{E\}\[\\widehat\{g\}\_\{x\}^\{\\mathrm\{grp\}\}\\mid x\]\)/m, where vector variance denotes expected squared distance from the mean\. This separates response noise from variation between prompts\. The group baseline removes a constant return exactly within a group, but its estimation noise can increase variance for other return and score distributions\. Statistical fitting from the radius margin\.Theorem[C\.29](https://arxiv.org/html/2611.21791#A3.Thmtheorem29)combines the radius margin with separate concentration bounds for prompts and responses\. Throughout the radius\-margin argument,μ=πk\{\\mu\}=\\pi\_\{k\},0<r≤1/\(8H\)0<r\\leq 1/\(8H\), and the TIS cap satisfiesc≥erc\\geq e^\{r\}\. For a fixed prompt define ax=∑t𝔼ht∼dtμ\(⋅∣x\)𝔼b∼μ\(⋅∣ht\)\|Atμ\(ht,b\)\|,Mx\(s\)=supπ∈Γk\(s\)𝒞k,x\(π\)\.a\_\{x\}=\\sum\_\{t\}\\mathbb\{E\}\_\{h\_\{t\}\\sim d\_\{t\}^\{\\mu\}\(\\cdot\\mid x\)\}\\mathbb\{E\}\_\{b\\sim\{\\mu\}\(\\cdot\\mid h\_\{t\}\)\}\|A\_\{t\}^\{\\mu\}\(h\_\{t\},b\)\|,\\qquad M\_\{x\}\(s\)=\\sup\_\{\\pi\\in\\Gamma\_\{k\}\(s\)\}\\mathcal\{C\}\_\{k,x\}\(\\pi\)\.WriteW=1W=1for SC andW=erW=e^\{r\}for TIS\. Underc≥erc\\geq e^\{r\}, the TIS potential isρ−1\\rho\-1throughout the feasible interval\. ###### Lemma C\.16\(Gain margin between two radii\)\. For SC or inactive\-cap TIS at0<r≤1/\(8H\)0<r\\leq 1/\(8H\),Mx\(r/2\)≥7rax/32M\_\{x\}\(r/2\)\\geq 7ra\_\{x\}/32,Mx\(r\)≥32Mx\(r/2\)M\_\{x\}\(r\)\\geq\\tfrac\{3\}\{2\}M\_\{x\}\(r/2\), andMx\(r\)−Mx\(r/2\)≥rax/16M\_\{x\}\(r\)\-M\_\{x\}\(r/2\)\\geq ra\_\{x\}/16at every prompt\. ###### Proof\. At one history the sign\-based policy from the proof of Lemma[C\.7](https://arxiv.org/html/2611.21791#A3.Thmtheorem7), at radiuss=r/2s=r/2, has linear advantage\(1−e−s\)𝔼μ\|A\|/2\(1\-e^\{\-s\}\)\\mathbb\{E\}\_\{\\mu\}\|A\|/2\. The scalar remainder is at mostB\(s\)s2𝔼μ\|A\|B\(s\)s^\{2\}\\mathbb\{E\}\_\{\\mu\}\|A\|, whereB\(s\)≤3/4B\(s\)\\leq 3/4\. Hence Mx\(r/2\)≥\(r/4−r2/4\)ax≥7r32ax\.M\_\{x\}\(r/2\)\\geq\(r/4\-r^\{2\}/4\)a\_\{x\}\\geq\\frac\{7r\}\{32\}a\_\{x\}\.The full class permits independent token choices, so the local bounds can be summed\. For SC, raise any feasible radius\-r/2r/2token ratioρ\\rhoto the power3/23/2and normalize\. Since1≤𝔼μρ3/2≤er/41\\leq\\mathbb\{E\}\_\{\\mu\}\\rho^\{3/2\}\\leq e^\{r/4\}, the new log ratios lie in\[−r,3r/4\]\[\-r,3r/4\]\. Conditional advantage centering cancels the normalization term and multiplies the objective by3/23/2\. For TIS use the normalized ratio1\+32\(ρ−1\)1\+\\tfrac\{3\}\{2\}\(\\rho\-1\)\. It lies in\[e−r,er\]\[e^\{\-r\},e^\{r\}\]forr≤1/8r\\leq 1/8, and also multiplies the objective by3/23/2\. Consequently, Mx\(r\)≥32Mx\(r/2\),Mx\(r\)−Mx\(r/2\)≥r16ax\.M\_\{x\}\(r\)\\geq\\frac\{3\}\{2\}M\_\{x\}\(r/2\),\\qquad M\_\{x\}\(r\)\-M\_\{x\}\(r/2\)\\geq\\frac\{r\}\{16\}a\_\{x\}\.\(C\.42\) ∎ ###### Lemma C\.17\(Conditional moments of objective differences\)\. For two radius\-rrcandidates, letZZbe the difference of their centered potentials\. The pair kernel12\(R−R′\)\(Z−Z′\)\\tfrac\{1\}\{2\}\(R\-R^\{\\prime\}\)\(Z\-Z^\{\\prime\}\)has conditional second moment at most36HW2r2ax36HW^\{2\}r^\{2\}a\_\{x\}\. The complete grouped average obeys the same bound\. ###### Proof\. For any two radius\-rrpolicies, letZZbe the difference of their centered potentials and writeZt=∑τ≤tξτZ\_\{t\}=\\sum\_\{\\tau\\leq t\}\\xi\_\{\\tau\}\. The increments have conditional mean zero, absolute value at most4Wr4Wr, and conditional second moment at most4W2r24W^\{2\}r^\{2\}\. Letsk\(x\)=𝔼μ\[R∣x\]s\_\{k\}\(x\)=\\mathbb\{E\}\_\{\\mu\}\[R\\mid x\]\. TelescopingZt2Z\_\{t\}^\{2\}, then conditioning on history and action, gives 𝔼μ\[RZ2∣x\]=∑t𝔼μ\[Qtμ\(2Zt−1ξt\+ξt2\)∣x\]≤16HW2r2sk\(x\)\+32HW2r2ax\.\\mathbb\{E\}\_\{\\mu\}\[RZ^\{2\}\\mid x\]=\\sum\_\{t\}\\mathbb\{E\}\_\{\\mu\}\[Q\_\{t\}^\{\\mu\}\(2Z\_\{t\-1\}\\xi\_\{t\}\+\\xi\_\{t\}^\{2\}\)\\mid x\]\\leq 16HW^\{2\}r^\{2\}s\_\{k\}\(x\)\+32HW^\{2\}r^\{2\}a\_\{x\}\.For the cross term, conditional centering replacesQtμQ\_\{t\}^\{\\mu\}byAtμA\_\{t\}^\{\\mu\}, and\|Zt−1\|≤4Wr\(t−1\)\|Z\_\{t\-1\}\|\\leq 4Wr\(t\-1\)\. ReplacingRRby1−R1\-Rgives the analogous bound withsk\(x\)s\_\{k\}\(x\)replaced by1−sk\(x\)1\-s\_\{k\}\(x\)\. Also𝔼μZ2≤4HW2r2\\mathbb\{E\}\_\{\\mu\}Z^\{2\}\\leq 4HW^\{2\}r^\{2\}\. For two conditionally independent responses defineD\(Y,Y′\)=12\(R\(Y\)−R\(Y′\)\)\(Z\(Y\)−Z\(Y′\)\)D\(Y,Y^\{\\prime\}\)=\\tfrac\{1\}\{2\}\(R\(Y\)\-R\(Y^\{\\prime\}\)\)\(Z\(Y\)\-Z\(Y^\{\\prime\}\)\)\. Since𝔼μZ=0\\mathbb\{E\}\_\{\\mu\}Z=0andRRis binary, 𝔼\[D2∣x\]=12\[\(1−sk\(x\)\)𝔼\(RZ2∣x\)\+sk\(x\)𝔼\(\(1−R\)Z2∣x\)\]\+\(𝔼\[RZ∣x\]\)2≤36HW2r2ax\.\\mathbb\{E\}\[D^\{2\}\\mid x\]=\\frac\{1\}\{2\}\\bigl\[\(1\-s\_\{k\}\(x\)\)\\mathbb\{E\}\(RZ^\{2\}\\mid x\)\+s\_\{k\}\(x\)\\mathbb\{E\}\(\(1\-R\)Z^\{2\}\\mid x\)\\bigr\]\+\(\\mathbb\{E\}\[RZ\\mid x\]\)^\{2\}\\leq 36HW^\{2\}r^\{2\}a\_\{x\}\.Here covariance Cauchy\-Schwarz gives\(𝔼RZ\)2≤4HW2r2sk\(x\)\(1−sk\(x\)\)\(\\mathbb\{E\}RZ\)^\{2\}\\leq 4HW^\{2\}r^\{2\}s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\), and the reward\-martingale identity impliessk\(x\)\(1−sk\(x\)\)≤axs\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\\leq a\_\{x\}\. The same second\-moment bound holds for the complete group average by Jensen’s inequality\. ∎ ###### Lemma C\.18\(A fitting margin controls response noise\)\. For each candidateπ\\pi, the full feasible family contains a comparison policyνπ\\nu\_\{\\pi\}whose conditional objective gaindπd\_\{\\pi\}dominates the positive shortfallΔπ\\Delta\_\{\\pi\}from the radius\-r/2r/2optimum\. Their pair objective difference has conditional second moment at most96dπ96d\_\{\\pi\}and absolute value less than one\. ###### Proof\. Choose a radius\-rrpolicyπr∗\\pi\_\{r\}^\{\*\}maximizing the objective at each prompt\. For a candidateπ∈Γk\(r\)\\pi\\in\\Gamma\_\{k\}\(r\), letνπ\\nu\_\{\\pi\}useπr∗\\pi\_\{r\}^\{\*\}on prompts where𝒞k,x\(π\)<Mx\(r/2\)\\mathcal\{C\}\_\{k,x\}\(\\pi\)<M\_\{x\}\(r/2\), and useπ\\pielsewhere\. The full token family containsνπ\\nu\_\{\\pi\}\. Write Δπ\(x\)=\[Mx\(r/2\)−𝒞k,x\(π\)\]\+,dπ\(x\)=𝒞k,x\(νπ\)−𝒞k,x\(π\)\.\\Delta\_\{\\pi\}\(x\)=\[M\_\{x\}\(r/2\)\-\\mathcal\{C\}\_\{k,x\}\(\\pi\)\]\_\{\+\},\\qquad d\_\{\\pi\}\(x\)=\\mathcal\{C\}\_\{k,x\}\(\\nu\_\{\\pi\}\)\-\\mathcal\{C\}\_\{k,x\}\(\\pi\)\.\(C\.43\)Letℋπ\(Y,Y′\)\\mathcal\{H\}\_\{\\pi\}\(Y,Y^\{\\prime\}\)be the pair objective difference betweenνπ\\nu\_\{\\pi\}andπ\\pi\. On a prompt with a shortfall, the radius margin givesdπ\(x\)≥rax/16d\_\{\\pi\}\(x\)\\geq ra\_\{x\}/16\. Elsewhere the pair difference is zero\. Lemma[C\.17](https://arxiv.org/html/2611.21791#A3.Thmtheorem17)therefore gives, at every prompt, dπ\(x\)≥Δπ\(x\)≥0,𝔼\[ℋπ2∣x\]≤576W2Hrdπ\(x\)≤96dπ\(x\)\.d\_\{\\pi\}\(x\)\\geq\\Delta\_\{\\pi\}\(x\)\\geq 0,\\qquad\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\pi\}^\{2\}\\mid x\]\\leq 576W^\{2\}Hr\\,d\_\{\\pi\}\(x\)\\leq 96d\_\{\\pi\}\(x\)\.HereHr≤1/8Hr\\leq 1/8andW2≤e1/4<4/3W^\{2\}\\leq e^\{1/4\}<4/3\. Also\|ℋπ\|≤4HWr<1\|\\mathcal\{H\}\_\{\\pi\}\|\\leq 4HWr<1, so the centered pair difference has absolute value less than two\. ∎ ###### Lemma C\.19\(Promptwise fitting shortfall controls reward\)\. Forr≤1/\(8H\)r\\leq 1/\(8H\), a feasible candidate satisfiesJx\(π\)−sk\(x\)≥\(r/32\)sk\(x\)\(1−sk\(x\)\)−Δπ\(x\)J\_\{x\}\(\\pi\)\-s\_\{k\}\(x\)\\geq\(r/32\)s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\-\\Delta\_\{\\pi\}\(x\)\. OverK≥\(64/r\)log\(6N/ϵ\)K\\geq\(64/r\)\\log\(6N/\\epsilon\)rounds, VN\(π0\)−J\(πK\)≤ϵ/2\+∑k<K𝔼xℒk\(x\),ℒk\(x\)=min\{1,κΔπk\+1\(x\)\},κ=4π2NK\.V\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilon/2\+\\sum\_\{k<K\}\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}\(x\),\\qquad\\mathcal\{L\}\_\{k\}\(x\)=\\min\\\{1,\\kappa\\Delta\_\{\\pi\_\{k\+1\}\}\(x\)\\\},\\quad\\kappa=4\\pi^\{2\}NK\.\(C\.44\) ###### Proof\. At a history putlh=𝔼π\(⋅∣h\)Atμl\_\{h\}=\\mathbb\{E\}\_\{\\pi\(\\cdot\\mid h\)\}A\_\{t\}^\{\\mu\}and letchc\_\{h\}be its SC or TIS objective contribution\. Then\|lh−ch\|≤B\(r\)r2𝔼μ\|Atμ\|\|l\_\{h\}\-c\_\{h\}\|\\leq B\(r\)r^\{2\}\\mathbb\{E\}\_\{\\mu\}\|A\_\{t\}^\{\\mu\}\|and\|lh\|≤\(er−1\)𝔼μ\|Atμ\|\|l\_\{h\}\|\\leq\(e^\{r\}\-1\)\\mathbb\{E\}\_\{\\mu\}\|A\_\{t\}^\{\\mu\}\|\. The learner\-to\-behavior history ratio differs from one by at moste\(H−1\)r−1e^\{\(H\-1\)r\}\-1\. The performance\-difference identity gives Jx\(π\)−sk\(x\)≥𝒞k,x\(π\)−\[B\(r\)r2\+\(e\(H−1\)r−1\)\(er−1\)\]ax\.J\_\{x\}\(\\pi\)\-s\_\{k\}\(x\)\\geq\\mathcal\{C\}\_\{k,x\}\(\\pi\)\-\\left\[B\(r\)r^\{2\}\+\(e^\{\(H\-1\)r\}\-1\)\(e^\{r\}\-1\)\\right\]a\_\{x\}\.For0≤u≤1/80\\leq u\\leq 1/8,eu−1≤8u/7e^\{u\}\-1\\leq 8u/7, andB\(r\)≤3/4B\(r\)\\leq 3/4\. The bracket is at most\(4/3\)Hr2≤r/6\(4/3\)Hr^\{2\}\\leq r/6\. Combining this withMx\(r/2\)≥7rax/32M\_\{x\}\(r/2\)\\geq 7ra\_\{x\}/32yields Jx\(π\)−sk\(x\)≥r32sk\(x\)\(1−sk\(x\)\)−Δπ\(x\)\.J\_\{x\}\(\\pi\)\-s\_\{k\}\(x\)\\geq\\frac\{r\}\{32\}s\_\{k\}\(x\)\(1\-s\_\{k\}\(x\)\)\-\\Delta\_\{\\pi\}\(x\)\.The ideal log\-odds incrementr/64r/64improves success probability by at most the positive term\. By the second deficit bound of Lemma[C\.3](https://arxiv.org/html/2611.21791#A3.Thmtheorem3), its squared one\-sided angular shortfall is therefore at mostπ2Δπ\(x\)/4\\pi^\{2\}\\Delta\_\{\\pi\}\(x\)/4\. WithK≥\(64/r\)log\(6N/ϵ\)K\\geq\(64/r\)\\log\(6N/\\epsilon\), the pointwise bound \([C\.18](https://arxiv.org/html/2611.21791#A3.E18)\) at accuracyϵ/2\\epsilon/2gives Equation \([C\.44](https://arxiv.org/html/2611.21791#A3.E44)\)\. Indeed, the second form of \([C\.18](https://arxiv.org/html/2611.21791#A3.E18)\) bounds the accumulated angular error by4π2NK∑kΔπk\+1\(x\)4\\pi^\{2\}NK\\sum\_\{k\}\\Delta\_\{\\pi\_\{k\+1\}\}\(x\), andmin\{1,∑kyk\}≤∑kmin\{1,yk\}\\min\\\{1,\\sum\_\{k\}y\_\{k\}\\\}\\leq\\sum\_\{k\}\\min\\\{1,y\_\{k\}\\\}foryk≥0y\_\{k\}\\geq 0\. ∎ The next lemma turns an empirical fitting tolerance into the capped prompt loss used by Lemma[C\.19](https://arxiv.org/html/2611.21791#A3.Thmtheorem19)\. ###### Lemma C\.20\(Uniform fitting certificate for unregularized SC and TIS\)\. Use the radius\-rrfamily and weights of Lemma[C\.18](https://arxiv.org/html/2611.21791#A3.Thmtheorem18)\. LetL=⌊G/2⌋L=\\lfloor G/2\\rfloor,κ=4π2NK\\kappa=4\\pi^\{2\}NK, andℓ=log\(4K𝒩\(u\)2/δ\)\\ell=\\log\(4K\\mathcal\{N\}\(u\)^\{2\}/\\delta\)for a radius\-uucover in Equation \([C\.12](https://arxiv.org/html/2611.21791#A3.E12)\)\. Defineℒπ\(x\)=min\{1,κΔπ\(x\)\}\\mathcal\{L\}\_\{\\pi\}\(x\)=\\min\\\{1,\\kappa\\Delta\_\{\\pi\}\(x\)\\\}, whereΔπ\\Delta\_\{\\pi\}is defined in Equation \([C\.43](https://arxiv.org/html/2611.21791#A3.E43)\)\. Conditional on the past, with probability at least1−δ/K1\-\\delta/K, every empirical fitπ^\\widehat\{\\pi\}with full\-family error at mosteke\_\{k\}satisfies 𝔼xℒπ^\(x\)≤4κek\+392κℓmL\+4ℓm\+120κHu\.\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{\\widehat\{\\pi\}\}\(x\)\\leq 4\\kappa e\_\{k\}\+\\frac\{392\\kappa\\ell\}\{mL\}\+\\frac\{4\\ell\}\{m\}\+120\\kappa Hu\.\(C\.45\) ###### Proof\. Fix one round and condition on its reference\. PutL=⌊G/2⌋L=\\lfloor G/2\\rfloor\. For a candidate and its comparison policy, write d¯π=1m∑jdπ\(xj\),d^π=𝒞^k\(νπ\)−𝒞^k\(π\)\.\\overline\{d\}\_\{\\pi\}=\\frac\{1\}\{m\}\\sum\_\{j\}d\_\{\\pi\}\(x\_\{j\}\),\\qquad\\widehat\{d\}\_\{\\pi\}=\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\nu\_\{\\pi\}\)\-\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\pi\)\.Conditional on the sampled prompts, randomly pair the responses in each group\. The actual group estimator is the conditional expectation of the paired estimator\. Exponential Jensen and one\-sided Bernstein for themLmLindependent pairs yield d¯π−d^π≤192d¯πℓmL\+4ℓ3mL,d¯π≤2d^π\+196ℓmL\.\\overline\{d\}\_\{\\pi\}\-\\widehat\{d\}\_\{\\pi\}\\leq\\sqrt\{\\frac\{192\\overline\{d\}\_\{\\pi\}\\ell\}\{mL\}\}\+\\frac\{4\\ell\}\{3mL\},\\qquad\\overline\{d\}\_\{\\pi\}\\leq 2\\widehat\{d\}\_\{\\pi\}\+\\frac\{196\\ell\}\{mL\}\.For the capped prompt lossℒπ\(x\)=min\{1,κΔπ\(x\)\}\\mathcal\{L\}\_\{\\pi\}\(x\)=\\min\\\{1,\\kappa\\Delta\_\{\\pi\}\(x\)\\\}, writePπ=𝔼xℒπP\_\{\\pi\}=\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{\\pi\}andP^π=m−1∑jℒπ\(xj\)\\widehat\{P\}\_\{\\pi\}=m^\{\-1\}\\sum\_\{j\}\\mathcal\{L\}\_\{\\pi\}\(x\_\{j\}\)\. SinceVarxℒπ≤Pπ\\operatorname\{Var\}\_\{x\}\\mathcal\{L\}\_\{\\pi\}\\leq P\_\{\\pi\}, prompt\-level Bernstein givesPπ≤2P^π\+4ℓ/mP\_\{\\pi\}\\leq 2\\widehat\{P\}\_\{\\pi\}\+4\\ell/m\. A radius\-uucover ofΓk\(r\)\\Gamma\_\{k\}\(r\)gives at most𝒩\(u\)2\\mathcal\{N\}\(u\)^\{2\}cells for the pairs\(νπ,π\)\(\\nu\_\{\\pi\},\\pi\)\. Choose one actual pair in each nonempty cell\. Each component lies within distance2u2uof its representative\. The group objective and conditional objective are2HW2HW\-Lipschitz\. The shortfall has the same Lipschitz bound\. With ℓ=log4K𝒩\(u\)2δ,\\ell=\\log\\frac\{4K\\mathcal\{N\}\(u\)^\{2\}\}\{\\delta\},a union bound and extension from the pair representatives give d¯π≤2d^π\+196ℓmL\+24HWu,Pπ≤2P^π\+4ℓm\+12κHWu\.\\overline\{d\}\_\{\\pi\}\\leq 2\\widehat\{d\}\_\{\\pi\}\+\\frac\{196\\ell\}\{mL\}\+24HWu,\\qquad P\_\{\\pi\}\\leq 2\\widehat\{P\}\_\{\\pi\}\+\\frac\{4\\ell\}\{m\}\+12\\kappa HWu\.BecauseΔπ≤dπ\\Delta\_\{\\pi\}\\leq d\_\{\\pi\}, we haveP^π≤κd¯π\\widehat\{P\}\_\{\\pi\}\\leq\\kappa\\overline\{d\}\_\{\\pi\}\. The empirical oracle andνπ^∈Γk\(r\)\\nu\_\{\\widehat\{\\pi\}\}\\in\\Gamma\_\{k\}\(r\)gived^π^≤ek\\widehat\{d\}\_\{\\widehat\{\\pi\}\}\\leq e\_\{k\}, whereeke\_\{k\}is its empirical fitting tolerance\. SinceW<2W<2, this proves Equation \([C\.45](https://arxiv.org/html/2611.21791#A3.E45)\)\. ∎ Covering\-number control of empirical fitting\.The following lemma treats a restricted model class, as an alternative to fitting against the full family in Theorem[C\.29](https://arxiv.org/html/2611.21791#A3.Thmtheorem29)\. ###### Lemma C\.21\(Additive fitting over a model class\)\. Use the radius\-rrfamily, withWa=1W\_\{a\}=1for SC andWa=min\{c,er\}W\_\{a\}=\\min\\\{c,e^\{r\}\\\}for TIS\. Let𝒫k⊆Γk\\mathcal\{P\}\_\{k\}\\subseteq\\Gamma\_\{k\}be the policies represented by the model\. In this lemma,𝒩k\(u\)\\mathcal\{N\}\_\{k\}\(u\)is the size of a radius\-uucover of𝒫k\\mathcal\{P\}\_\{k\}ind∞d\_\{\\infty\}with centers in𝒫k\\mathcal\{P\}\_\{k\}\. It is at most the size of a radius\-u/2u/2cover ofΓk\\Gamma\_\{k\}\. Define the representation gap and the empirical optimization gap of the fittedπ^∈𝒫k\\widehat\{\\pi\}\\in\\mathcal\{P\}\_\{k\}by bk=supπ∈Γk𝒞k\(π\)−supπ∈𝒫k𝒞k\(π\),ek=supπ∈𝒫k𝒞^k\(π\)−𝒞^k\(π^\)\.b\_\{k\}=\\sup\_\{\\pi\\in\\Gamma\_\{k\}\}\\mathcal\{C\}\_\{k\}\(\\pi\)\-\\sup\_\{\\pi\\in\\mathcal\{P\}\_\{k\}\}\\mathcal\{C\}\_\{k\}\(\\pi\),\\qquad e\_\{k\}=\\sup\_\{\\pi\\in\\mathcal\{P\}\_\{k\}\}\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\pi\)\-\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\widehat\{\\pi\}\)\.Then, with probability at least1−δk1\-\\delta\_\{k\}, the population gapsupΓk𝒞k−𝒞k\(π^\)\\sup\_\{\\Gamma\_\{k\}\}\\mathcal\{C\}\_\{k\}\-\\mathcal\{C\}\_\{k\}\(\\widehat\{\\pi\}\)is at mostbk\+ekb\_\{k\}\+e\_\{k\}plus2×2\\timesthe uniform error in \([C\.46](https://arxiv.org/html/2611.21791#A3.E46)\)\. ###### Proof\. WriteW=WaW=W\_\{a\}\. One prompt group’s contribution to the empirical objective is the complete average of the pair kernel ℋπ\(y,y′\)=12\(R\(y\)−R\(y′\)\)\(Zπ\(y\)−Zπ\(y′\)\)\.\\mathcal\{H\}\_\{\\pi\}\(y,y^\{\\prime\}\)=\\frac\{1\}\{2\}\(R\(y\)\-R\(y^\{\\prime\}\)\)\(Z\_\{\\pi\}\(y\)\-Z\_\{\\pi\}\(y^\{\\prime\}\)\)\.Conditional centering gives𝔼Zπ=0\\mathbb\{E\}Z\_\{\\pi\}=0, so its expectation is𝒞k\(π\)\\mathcal\{C\}\_\{k\}\(\\pi\)\. The potential isWW\-Lipschitz in the token log ratio\. Consequently the group contribution has absolute value at mostBpair=2HWrB\_\{\\mathrm\{pair\}\}=2HWr, and is2HW2HW\-Lipschitz ind∞d\_\{\\infty\}\. Under behavior, the centered token potentials form a martingale difference sequence\. The conditional range bound and orthogonality give𝔼μ\[Zπ2∣x\]≤HW2r2\\mathbb\{E\}\_\{\\mu\}\[Z\_\{\\pi\}^\{2\}\\mid x\]\\leq HW^\{2\}r^\{2\}\. Since rewards are binary, Varx\(𝔼\[ℋπ∣x\]\)≤HW2r24,𝔼xVar\(ℋπ∣x\)≤HW2r22\.\\operatorname\{Var\}\_\{x\}\(\\mathbb\{E\}\[\\mathcal\{H\}\_\{\\pi\}\\mid x\]\)\\leq\\frac\{HW^\{2\}r^\{2\}\}\{4\},\\qquad\\mathbb\{E\}\_\{x\}\\operatorname\{Var\}\(\\mathcal\{H\}\_\{\\pi\}\\mid x\)\\leq\\frac\{HW^\{2\}r^\{2\}\}\{2\}\.The first inequality follows from the reward covariance identity andVar\(R∣x\)≤1/4\\operatorname\{Var\}\(R\\mid x\)\\leq 1/4\. The second uses\|R−R′\|≤1\|R\-R^\{\\prime\}\|\\leq 1and independence ofZπ,Zπ′Z\_\{\\pi\},Z\_\{\\pi\}^\{\\prime\}at a fixed prompt\. Random pairings of theGGresponses therefore give group variance at most HW2r24\(1\+2⌊G/2⌋\)\.\\frac\{HW^\{2\}r^\{2\}\}\{4\}\\left\(1\+\\frac\{2\}\{\\lfloor G/2\\rfloor\}\\right\)\.Putℓ=log\(4𝒩k\(u\)/δk\)\\ell=\\log\(4\\mathcal\{N\}\_\{k\}\(u\)/\\delta\_\{k\}\)\. Bernstein’s inequality, a union bound over the cover, and the Lipschitz extension give a uniform error bound\. The extension term is4HWu4HWubecause both𝒞^k\\widehat\{\\mathcal\{C\}\}\_\{k\}and𝒞k\\mathcal\{C\}\_\{k\}are2HW2HW\-Lipschitz: supπ∈𝒫k\|𝒞^k\(π\)−𝒞k\(π\)\|≤4HWu\+WrH2m\(1\+2⌊G/2⌋\)ℓ\+8HWrℓ3m\.\\sup\_\{\\pi\\in\\mathcal\{P\}\_\{k\}\}\|\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\pi\)\-\\mathcal\{C\}\_\{k\}\(\\pi\)\|\\leq 4HWu\+Wr\\sqrt\{\\frac\{H\}\{2m\}\\left\(1\+\\frac\{2\}\{\\lfloor G/2\\rfloor\}\\right\)\\ell\}\+\\frac\{8HWr\\ell\}\{3m\}\.\(C\.46\)The full population gap is at mostbk\+ekb\_\{k\}\+e\_\{k\}plus2×2\\timesthe uniform error in \([C\.46](https://arxiv.org/html/2611.21791#A3.E46)\)\. Uniformity permits the fitted policy to depend on all observations\. Conditioning on earlier rounds and takingδk=δ/K\\delta\_\{k\}=\\delta/Kgives a simultaneous guarantee\. ∎ A sufficient prompt count\.For oracle toleranceζk\\zeta\_\{k\}, supposebk\+ek≤ζk/2b\_\{k\}\+e\_\{k\}\\leq\\zeta\_\{k\}/2\. Takingu=ζk/\(48HWa\)u=\\zeta\_\{k\}/\(48HW\_\{a\}\), a sufficient count is m≥⌈\[72HWa2r2ζk2\(1\+2⌊G/2⌋\)\+32HWarζk\]log4𝒩k\(ζk/\(48HWa\)\)δk⌉\.m\\geq\\left\\lceil\\left\[\\frac\{72HW\_\{a\}^\{2\}r^\{2\}\}\{\\zeta\_\{k\}^\{2\}\}\\left\(1\+\\frac\{2\}\{\\lfloor G/2\\rfloor\}\\right\)\+\\frac\{32HW\_\{a\}r\}\{\\zeta\_\{k\}\}\\right\]\\log\\frac\{4\\mathcal\{N\}\_\{k\}\(\\zeta\_\{k\}/\(48HW\_\{a\}\)\)\}\{\\delta\_\{k\}\}\\right\\rceil\.Each of the three error terms is then at mostζk/12\\zeta\_\{k\}/12, so2×2\\timesthe uniform error is at mostζk/2\\zeta\_\{k\}/2\. #### C\.3\.3Fitting TIS with KL regularization Separate concentration for prompts and responses\.Fix a round and useβ≥2\\beta\\geq 2,η=1/β\\eta=1/\\beta, andG≥2G\\geq 2\. Writeℓ\>0\\ell\>0for the tail parameter in the concentration bounds below\. The fitted objective is Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\) with cap\-two reward and penalty weights\. Its unbiasedness and gradient identity are given by Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15)\. Throughout this argument,Dk\(x\)D\_\{k\}\(x\)is the TIS objective gap in Equation \([C\.30](https://arxiv.org/html/2611.21791#A3.E30)\), andℒk\\mathcal\{L\}\_\{k\}andκ\\kappaare the capped loss and coefficient in Equation \([C\.33](https://arxiv.org/html/2611.21791#A3.E33)\)\. We first bound the response noise by the objective gap; accurate candidates then require less response sampling\. For a candidateπ\\pi, writeρt=π\(at∣ht\)/μ\(at∣ht\)\\rho\_\{t\}=\\pi\(a\_\{t\}\\mid h\_\{t\}\)/\\mu\(a\_\{t\}\\mid h\_\{t\}\)andρt∗=πk∗\(at∣ht\)/μ\(at∣ht\)\\rho\_\{t\}^\{\*\}=\\pi\_\{k\}^\{\*\}\(a\_\{t\}\\mid h\_\{t\}\)/\\mu\(a\_\{t\}\\mid h\_\{t\}\)\. LetD=𝔼xDk\(x\)D=\\mathbb\{E\}\_\{x\}D\_\{k\}\(x\)be the exact gap in \([C\.30](https://arxiv.org/html/2611.21791#A3.E30)\), and putV=∑t𝔼dtμ𝔼μt\(ρt∗−ρt\)2V=\\sum\_\{t\}\\mathbb\{E\}\_\{d\_\{t\}^\{\\mu\}\}\\mathbb\{E\}\_\{\{\\mu\_\{t\}\}\}\(\\rho\_\{t\}^\{\*\}\-\\rho\_\{t\}\)^\{2\}\. For one response defineZ=∑t\(ρt∗−ρt\)Z=\\sum\_\{t\}\(\\rho\_\{t\}^\{\*\}\-\\rho\_\{t\}\)andB=∑t\[k\(ρt∗\)−k\(ρt\)\]B=\\sum\_\{t\}\[k\(\\rho\_\{t\}^\{\*\}\)\-k\(\\rho\_\{t\}\)\]\. The sample objective excess isℋ=RZ−βB\\mathcal\{H\}=RZ\-\\beta B\. For two independent responses to the same prompt, define ℋ2\(Y,Y′\)=12\(R−R′\)\(Z−Z′\)−β2\(B\+B′\)\.\\mathcal\{H\}\_\{2\}\(Y,Y^\{\\prime\}\)=\\tfrac\{1\}\{2\}\(R\-R^\{\\prime\}\)\(Z\-Z^\{\\prime\}\)\-\\tfrac\{\\beta\}\{2\}\(B\+B^\{\\prime\}\)\.\(C\.47\) ###### Lemma C\.22\(Curvature and conditional response moments\)\. The objective gap satisfies D≥12βe−ηV\.D\\geq\\tfrac\{1\}\{2\}\\beta e^\{\-\\eta\}V\.\(C\.48\)For the symmetric pair kernel in \([C\.47](https://arxiv.org/html/2611.21791#A3.E47)\), conditional on a prompt,𝔼ℋ2=Dk\(x\)\\mathbb\{E\}\\mathcal\{H\}\_\{2\}=D\_\{k\}\(x\),𝔼ℋ22≤16HDk\(x\)/β\\mathbb\{E\}\\mathcal\{H\}\_\{2\}^\{2\}\\leq 16HD\_\{k\}\(x\)/\\beta, and\|ℋ2\|≤9H/β\|\\mathcal\{H\}\_\{2\}\|\\leq 9H/\\beta\. ###### Proof\. The scalar curvatureβk′′\(ρ\)=β/ρ\\beta k^\{\\prime\\prime\}\(\\rho\)=\\beta/\\rhoand feasibility ofπk∗\\pi\_\{k\}^\{\*\}imply Equation \([C\.48](https://arxiv.org/html/2611.21791#A3.E48)\)\. Normalization makes the increments ofZZmartingale differences, so𝔼Z2=V\\mathbb\{E\}Z^\{2\}=V\. Since\|k′\(ρ\)\|≤η\|k^\{\\prime\}\(\\rho\)\|\\leq\\eta,𝔼B2≤Hη2V\\mathbb\{E\}B^\{2\}\\leq H\\eta^\{2\}V\. Therefore 𝔼ℋ2≤2\(1\+H\)V≤4eη\(1\+H\)βD≤16HβD\.\\mathbb\{E\}\\mathcal\{H\}^\{2\}\\leq 2\(1\+H\)V\\leq\\frac\{4e^\{\\eta\}\(1\+H\)\}\{\\beta\}D\\leq\\frac\{16H\}\{\\beta\}D\.All these bounds also hold conditional on a prompt, withDk\(x\)D\_\{k\}\(x\)and the corresponding conditional squared\-ratio error\. ForG≥2G\\geq 2, the group excess is the average of the symmetric pair kernel \([C\.47](https://arxiv.org/html/2611.21791#A3.E47)\)\. Its conditional expectation isDk\(x\)D\_\{k\}\(x\)\. The squared reward term has expectation at most𝔼\[Z2∣x\]/2\\mathbb\{E\}\[Z^\{2\}\\mid x\]/2\. Jensen’s inequality bounds the squared penalty term byβ2𝔼\[B2∣x\]\\beta^\{2\}\\mathbb\{E\}\[B^\{2\}\\mid x\]\. WritingVxV\_\{x\}forVVconditional on the prompt, we obtain 𝔼\[ℋ22∣x\]≤\(1\+2H\)Vx≤2eη\(1\+2H\)βDk\(x\)≤16HβDk\(x\),\\mathbb\{E\}\[\\mathcal\{H\}\_\{2\}^\{2\}\\mid x\]\\leq\(1\+2H\)V\_\{x\}\\leq\\frac\{2e^\{\\eta\}\(1\+2H\)\}\{\\beta\}D\_\{k\}\(x\)\\leq\\frac\{16H\}\{\\beta\}D\_\{k\}\(x\),Jensen’s inequality gives the same bound for the complete pair averageUGU\_\{G\}\. The interval bounds give\|Z\|≤2eηH/β\|Z\|\\leq 2e^\{\\eta\}H/\\betaand0≤k\(ρ\)≤eηη20\\leq k\(\\rho\)\\leq e^\{\\eta\}\\eta^\{2\}\. In particular, the pair kernel satisfies\|ℋ2\|≤9H/β\|\\mathcal\{H\}\_\{2\}\|\\leq 9H/\\beta\. Its conditional mean isDk\(x\)D\_\{k\}\(x\)and its conditional second moment is at most16HDk\(x\)/β16HD\_\{k\}\(x\)/\\beta\. We use these moment bounds in Lemma[C\.23](https://arxiv.org/html/2611.21791#A3.Thmtheorem23)\. ∎ The moment bound gives a concentration inequality conditional on the sampled prompts\. ###### Lemma C\.23\(Response concentration within sampled prompts\)\. For each fixed candidate, conditional on the prompts, with probability at least1−e−ℓ1\-e^\{\-\\ell\},D¯≤2D^\+44Hℓ/\(βm⌊G/2⌋\)\\overline\{D\}\\leq 2\\widehat\{D\}\+44H\\ell/\(\\beta m\\lfloor G/2\\rfloor\)\. HereD¯=m−1∑jDk\(xj\)\\overline\{D\}=m^\{\-1\}\\sum\_\{j\}D\_\{k\}\(x\_\{j\}\), withDkD\_\{k\}from Equation \([C\.30](https://arxiv.org/html/2611.21791#A3.E30)\), andD^=𝒲^k\(πk∗\)−𝒲^k\(π\)\\widehat\{D\}=\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\_\{k\}^\{\*\}\)\-\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\)is the empirical gap for the objective in Equation \([C\.9](https://arxiv.org/html/2611.21791#A3.E9)\)\. ###### Proof\. WriteL=⌊G/2⌋L=\\lfloor G/2\\rfloor\. Conditional on themmprompts, randomly pair the responses in each group, leaving one unused whenGGis odd\. The complete pair average is the conditional expectation of this paired estimator\. Jensen’s inequality for its exponential moment therefore transfers the Bernstein bound formLmLindependent pairs to the actual grouped estimator\. LetD¯=m−1∑jDk\(xj\)\\overline\{D\}=m^\{\-1\}\\sum\_\{j\}D\_\{k\}\(x\_\{j\}\)and letD^\\widehat\{D\}be the empirical objective gap toπk∗\\pi\_\{k\}^\{\*\}\. The bounds on the pair kernel give, with tail parameterℓ\\ell, D¯−D^≤32HD¯ℓβmL\+6HℓβmL≤D¯/2\+22HℓβmL\.\\overline\{D\}\-\\widehat\{D\}\\leq\\sqrt\{\\frac\{32H\\overline\{D\}\\ell\}\{\\beta mL\}\}\+\\frac\{6H\\ell\}\{\\beta mL\}\\leq\\overline\{D\}/2\+\\frac\{22H\\ell\}\{\\beta mL\}\.ThusD¯≤2D^\+44Hℓ/\(βmL\)\\overline\{D\}\\leq 2\\widehat\{D\}\+44H\\ell/\(\\beta mL\)\. This step controls response noise conditional on the prompts, so its effective sample size ismLmL\. ∎ A separate concentration bound handles variation between prompts\. The cap in Equation \([C\.33](https://arxiv.org/html/2611.21791#A3.E33)\) bounds each prompt loss by one\. ###### Lemma C\.24\(Concentration of capped prompt losses\)\. For a fixed candidate andℒk\(x\)=min\{1,κDk\(x\)\}\\mathcal\{L\}\_\{k\}\(x\)=\\min\\\{1,\\kappa D\_\{k\}\(x\)\\\}, with probability at least1−e−ℓ1\-e^\{\-\\ell\},𝔼xℒk≤2m−1∑jℒk\(xj\)\+4ℓ/m\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}\\leq 2m^\{\-1\}\\sum\_\{j\}\\mathcal\{L\}\_\{k\}\(x\_\{j\}\)\+4\\ell/m\. ###### Proof\. Useℒk\(x,π\)=min\{1,κDk\(x,π\)\}\\mathcal\{L\}\_\{k\}\(x,\\pi\)=\\min\\\{1,\\kappa D\_\{k\}\(x,\\pi\)\\\}from \([C\.33](https://arxiv.org/html/2611.21791#A3.E33)\)\. For a fixed candidate, writeP=𝔼xℒkP=\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}andP^=m−1∑jℒk\(xj\)\\widehat\{P\}=m^\{\-1\}\\sum\_\{j\}\\mathcal\{L\}\_\{k\}\(x\_\{j\}\)\. Since0≤ℒk≤10\\leq\\mathcal\{L\}\_\{k\}\\leq 1, its variance is at mostPP\. One\-sided Bernstein gives P−P^≤2Pℓ/m\+ℓ/\(3m\),P≤2P^\+4ℓ/m\.P\-\\widehat\{P\}\\leq\\sqrt\{2P\\ell/m\}\+\\ell/\(3m\),\\qquad P\\leq 2\\widehat\{P\}\+4\\ell/m\.The cap bounds the cost of each prompt by its possible reward loss\. Its range is therefore independent ofHHandβ\\beta\. ∎ A covering argument makes both concentration bounds hold for the policy selected using the same dataset\. ###### Lemma C\.25\(Uniform fitting certificate for TIS with KL\)\. LetL=⌊G/2⌋L=\\lfloor G/2\\rfloor\. For a radius\-vvcover andℓ=log\(4K𝒩\(v\)/δ\)\\ell=\\log\(4K\\mathcal\{N\}\(v\)/\\delta\), outside an event of probability at mostδ/K\\delta/K, every fitted policy satisfying the empirical accuracy condition in Equation \([C\.10](https://arxiv.org/html/2611.21791#A3.E10)\) obeys 𝔼xℒk\(x,πk\+1\)≤4κek\+88κHℓβmL\+4ℓm\+54κHv\.\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}\(x,\\pi\_\{k\+1\}\)\\leq 4\\kappa e\_\{k\}\+\\frac\{88\\kappa H\\ell\}\{\\beta mL\}\+\\frac\{4\\ell\}\{m\}\+54\\kappa Hv\.\(C\.49\) ###### Proof\. For the log\-policy metric in Equation \([C\.12](https://arxiv.org/html/2611.21791#A3.E12)\), the sampled reward term is2Heη2He^\{\\eta\}\-Lipschitz, and the penalty isβHηeη\\beta H\\eta e^\{\\eta\}\-Lipschitz\. Their sum is at most6H6H\. The same bound holds for the conditional gapDk\(x,π\)D\_\{k\}\(x,\\pi\)and for the grouped empirical gapD^\\widehat\{D\}: its reward weights satisfy∑g\|Rg−R¯\|/\(G−1\)≤G/\(2\(G−1\)\)≤1\\sum\_\{g\}\|R\_\{g\}\-\\overline\{R\}\|/\(G\-1\)\\leq G/\(2\(G\-1\)\)\\leq 1, and its penalty is a response average\. The capped loss is6κH6\\kappa H\-Lipschitz\. Apply both concentration inequalities to a radius\-vvcover, with ℓ=log4K𝒩\(v\)δ\.\\ell=\\log\\frac\{4K\\mathcal\{N\}\(v\)\}\{\\delta\}\.Conditional on the past, a union bound and extension from the cover give, simultaneously for every candidate, D¯≤2D^\+44HℓβmL\+18Hv,P≤2P^\+4ℓm\+18κHv\.\\overline\{D\}\\leq 2\\widehat\{D\}\+\\frac\{44H\\ell\}\{\\beta mL\}\+18Hv,\\qquad P\\leq 2\\widehat\{P\}\+\\frac\{4\\ell\}\{m\}\+18\\kappa Hv\.SinceP^≤κD¯\\widehat\{P\}\\leq\\kappa\\overline\{D\}, the fitted policy satisfies Equation \([C\.49](https://arxiv.org/html/2611.21791#A3.E49)\)\. HereD^≤ek\\widehat\{D\}\\leq e\_\{k\}follows directly from the empirical oracle, since the target belongs to the full feasible family\. ∎ #### C\.3\.4Fitting unit\-weight SC with KL regularization The preceding TIS fitting analysis uses exact conditional centers\. For unit\-weight SC, we also allow sampled centers, so their randomness must be included in the empirical objective\. Use the independent auxiliary tokens and fixed\-dataset sampling scheme of Equation \([C\.13](https://arxiv.org/html/2611.21791#A3.E13)\)\. For the unit reward weight,Fa\(ρ\)=logρF\_\{a\}\(\\rho\)=\\log\\rho\. The following sample terms make that specialization explicit for the moment bounds below\. For an augmented response\(Y,U\)\(Y,U\), put Zπ\(Y,U\)\\displaystyle Z\_\{\\pi\}\(Y,U\)=∑t=1H\[logρtπ\(ht,at\)−1M∑i=1Mlogρtπ\(ht,Ut,i\)\],\\displaystyle=\\sum\_\{t=1\}^\{H\}\\left\[\\log\\rho\_\{t\}^\{\\pi\}\(h\_\{t\},a\_\{t\}\)\-\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\log\\rho\_\{t\}^\{\\pi\}\(h\_\{t\},U\_\{t,i\}\)\\right\],Bπ\(Y\)\\displaystyle B\_\{\\pi\}\(Y\)=∑t=1Hk\(ρtπ\(ht,at\)\)\.\\displaystyle=\\sum\_\{t=1\}^\{H\}k\\\!\\left\(\\rho\_\{t\}^\{\\pi\}\(h\_\{t\},a\_\{t\}\)\\right\)\.\(C\.50\)The empirical objective is 𝒲^k\(π\)=1m∑j=1m\[1G−1∑g=1G\(Rjg−R¯j\)Zπ\(Yjg,Ujg\)−2G∑g=1GBπ\(Yjg\)\]\.\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\)=\\frac\{1\}\{m\}\\sum\_\{j=1\}^\{m\}\\left\[\\frac\{1\}\{G\-1\}\\sum\_\{g=1\}^\{G\}\(R\_\{jg\}\-\\overline\{R\}\_\{j\}\)Z\_\{\\pi\}\(Y\_\{jg\},U\_\{jg\}\)\-\\frac\{2\}\{G\}\\sum\_\{g=1\}^\{G\}B\_\{\\pi\}\(Y\_\{jg\}\)\\right\]\.\(C\.51\)HereR¯j=G−1∑g=1GRjg\\overline\{R\}\_\{j\}=G^\{\-1\}\\sum\_\{g=1\}^\{G\}R\_\{jg\}\. The reward term is the same\-prompt sample covariance, including itsG/\(G−1\)G/\(G\-1\)correction\. Since𝔼\[Zπ∣x\]=0\\mathbb\{E\}\[Z\_\{\\pi\}\\mid x\]=0and𝔼\[RZπ∣x\]=Ck,x\(π\)\\mathbb\{E\}\[RZ\_\{\\pi\}\\mid x\]=C\_\{k,x\}\(\\pi\), this objective is unbiased for𝒲k\(π\)\\mathcal\{W\}\_\{k\}\(\\pi\)for each fixed candidate\. Its reward gradient uses the unweighted conditional score center, estimated byMMindependent draws\. The penalty potential isk\(ρ\)k\(\\rho\), with gradientρlogρ∇θlogπθ\\rho\\log\\rho\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\. No random advantage\-standard\-deviation normalization or variable\-length token weighting is included\. The fitting condition is Equation \([C\.10](https://arxiv.org/html/2611.21791#A3.E10)\), with the radius\-1/21/2family and the covering numbers defined before Theorem[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)\. Curvature and conditional response moments\.Use the unit\-weight objective gap in Equation \([C\.27](https://arxiv.org/html/2611.21791#A3.E27)\)\. Writeρt∗=πk∗/μ\\rho\_\{t\}^\{\*\}=\\pi\_\{k\}^\{\*\}/\{\\mu\}and define 𝒱k\(x,π\)=𝔼μ\[∑t\(ρt∗\(ht,at\)−ρtπ\(ht,at\)\)2\|x\]\.\\mathcal\{V\}\_\{k\}\(x,\\pi\)=\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[\\sum\_\{t\}\\bigl\(\\rho\_\{t\}^\{\*\}\(h\_\{t\},a\_\{t\}\)\-\\rho\_\{t\}^\{\\pi\}\(h\_\{t\},a\_\{t\}\)\\bigr\)^\{2\}\\,\\middle\|\\,x\\right\]\. ###### Lemma C\.26\(Quadratic control by the objective gap\)\. For the familyΓk\\Gamma\_\{k\}in Section[C\.2\.3](https://arxiv.org/html/2611.21791#A3.SS2.SSS3)and the gap in Equation \([C\.27](https://arxiv.org/html/2611.21791#A3.E27)\), everyπ∈Γk\\pi\\in\\Gamma\_\{k\}satisfiesDk\(x,π\)≥𝒱k\(x,π\)/4D\_\{k\}\(x,\\pi\)\\geq\\mathcal\{V\}\_\{k\}\(x,\\pi\)/4\. ###### Proof\. The negative local objective has scalar integrandϕA\(u\)=−Alogu\+2k\(u\)\\phi\_\{A\}\(u\)=\-A\\log u\+2k\(u\)\. For\|A\|≤1\|A\|\\leq 1andu∈\[e−1/2,e1/2\]u\\in\[e^\{\-1/2\},e^\{1/2\}\], ϕA′′\(u\)=A\+2uu2≥2u−1u2≥12\.\\phi\_\{A\}^\{\\prime\\prime\}\(u\)=\\frac\{A\+2u\}\{u^\{2\}\}\\geq\\frac\{2u\-1\}\{u^\{2\}\}\\geq\\frac\{1\}\{2\}\.The last inequality holds on\[2−2,2\+2\]\[2\-\\sqrt\{2\},2\+\\sqrt\{2\}\], which contains the feasible ratio interval\. At each history, feasible ratio vectors form a convex compact set with linear normalization\. Strong convexity and the first\-order optimality inequality, including at boundary optima, yield the stated gap bound after averaging histories\. ∎ For Lemma[C\.27](https://arxiv.org/html/2611.21791#A3.Thmtheorem27), fixk,x,πk,x,\\piand abbreviateD=Dk\(x,π\)D=D\_\{k\}\(x,\\pi\),𝒱=𝒱k\(x,π\)\\mathcal\{V\}=\\mathcal\{V\}\_\{k\}\(x,\\pi\),Z=Zπk∗−ZπZ=Z\_\{\\pi\_\{k\}^\{\*\}\}\-Z\_\{\\pi\}, andB=Bπk∗−BπB=B\_\{\\pi\_\{k\}^\{\*\}\}\-B\_\{\\pi\}, using the sample terms in Equation \([C\.50](https://arxiv.org/html/2611.21791#A3.E50)\)\. For two conditionally independent augmented responses define ℋ\(\(Y,U\),\(Y′,U′\)\)=12\(R−R′\)\(Z−Z′\)−\(B\+B′\)\.\\mathcal\{H\}\(\(Y,U\),\(Y^\{\\prime\},U^\{\\prime\}\)\)=\\tfrac\{1\}\{2\}\(R\-R^\{\\prime\}\)\(Z\-Z^\{\\prime\}\)\-\(B\+B^\{\\prime\}\)\.\(C\.52\)This kernel has conditional meanDD\. Its complete pair average is the grouped empirical excess objective in Equation \([C\.51](https://arxiv.org/html/2611.21791#A3.E51)\)\. ###### Lemma C\.27\(Conditional moments with finite Monte Carlo centers\)\. For everyπ∈Γk\\pi\\in\\Gamma\_\{k\}, withΓk\\Gamma\_\{k\}as in Section[C\.2\.3](https://arxiv.org/html/2611.21791#A3.SS2.SSS3), the kernel in Equation \([C\.52](https://arxiv.org/html/2611.21791#A3.E52)\) satisfies 𝔼\[ℋ2∣x\]≤CMHD,\|ℋ\|≤3H,CM=8\+4e\(1\+1/M\)\.\\mathbb\{E\}\[\\mathcal\{H\}^\{2\}\\mid x\]\\leq C\_\{M\}HD,\\qquad\|\\mathcal\{H\}\|\\leq 3H,\\qquad C\_\{M\}=8\+4e\(1\+1/M\)\. ###### Proof\. PutΔzt=logρt∗−logρtπ\\Delta z\_\{t\}=\\log\\rho\_\{t\}^\{\*\}\-\\log\\rho\_\{t\}^\{\\pi\}\. The increments Δzt\(ht,at\)−1M∑i=1MΔzt\(ht,Ut,i\)\\Delta z\_\{t\}\(h\_\{t\},a\_\{t\}\)\-\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\Delta z\_\{t\}\(h\_\{t\},U\_\{t,i\}\)are martingale differences on the augmented response filtration\. Their conditional variance is\(1\+1/M\)Varμt\(Δzt\)\(1\+1/M\)\\operatorname\{Var\}\_\{\{\{\\mu\_\{t\}\}\}\}\(\\Delta z\_\{t\}\)\. The logarithm ise1/2e^\{1/2\}\-Lipschitz on the feasible ratio interval, so orthogonality of these increments gives 𝔼\[Z∣x\]=0,𝔼\[Z2∣x\]≤\(1\+1/M\)e𝒱\.\\mathbb\{E\}\[Z\\mid x\]=0,\\qquad\\mathbb\{E\}\[Z^\{2\}\\mid x\]\\leq\(1\+1/M\)e\\,\\mathcal\{V\}\.Since\|k′\(u\)\|=\|logu\|≤1/2\|k^\{\\prime\}\(u\)\|=\|\\log u\|\\leq 1/2, Cauchy\-Schwarz gives𝔼\[B2∣x\]≤\(H/4\)𝒱\\mathbb\{E\}\[B^\{2\}\\mid x\]\\leq\(H/4\)\\mathcal\{V\}\. Using\(a−b\)2≤2a2\+2b2\(a\-b\)^\{2\}\\leq 2a^\{2\}\+2b^\{2\},\|R−R′\|≤1\|R\-R^\{\\prime\}\|\\leq 1, and independence of the two augmented responses, 𝔼\[ℋ2∣x\]\\displaystyle\\mathbb\{E\}\[\\mathcal\{H\}^\{2\}\\mid x\]≤𝔼\[Z2∣x\]\+8𝔼\[B2∣x\]\\displaystyle\\leq\\mathbb\{E\}\[Z^\{2\}\\mid x\]\+8\\mathbb\{E\}\[B^\{2\}\\mid x\]≤\[\(1\+1/M\)e\+2H\]𝒱≤CMHD,\\displaystyle\\leq\[\(1\+1/M\)e\+2H\]\\mathcal\{V\}\\leq C\_\{M\}HD,where the last step uses Lemma[C\.26](https://arxiv.org/html/2611.21791#A3.Thmtheorem26)andH≥1H\\geq 1\. Also\|Δzt\|≤1\|\\Delta z\_\{t\}\|\\leq 1gives\|Z\|≤2H\|Z\|\\leq 2H\. On the feasible interval0≤k\(u\)<1/40\\leq k\(u\)<1/4, so\|B\|≤H/4\|B\|\\leq H/4\. Equation \([C\.52](https://arxiv.org/html/2611.21791#A3.E52)\) then gives\|ℋ\|≤\(5/2\)H≤3H\|\\mathcal\{H\}\|\\leq\(5/2\)H\\leq 3H\. ∎ The moment bound vanishes with the objective gap\. Finite independent Monte Carlo centers retain this property because the same auxiliary draws are used for all candidate policies in the empirical objective\. When the candidate equals the comparator, their sampled centers agree on every dataset, so this source of comparison noise also vanishes\. From empirical fitting to population fitting\.Condition on the past before roundkk\. For sampled promptsxjx\_\{j\}, let D¯\(π\)=1m∑jDk\(xj,π\),D^\(π\)=𝒲^k\(πk∗\)−𝒲^k\(π\)\.\\overline\{D\}\(\\pi\)=\\frac\{1\}\{m\}\\sum\_\{j\}D\_\{k\}\(x\_\{j\},\\pi\),\\qquad\\widehat\{D\}\(\\pi\)=\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\_\{k\}^\{\*\}\)\-\\widehat\{\\mathcal\{W\}\}\_\{k\}\(\\pi\)\.Forκ\>0\\kappa\>0, define ℒk\(x,π\)=min\{1,κDk\(x,π\)\},P\(π\)=𝔼xℒk\(x,π\),P^\(π\)=1m∑jℒk\(xj,π\)\.\\mathcal\{L\}\_\{k\}\(x,\\pi\)=\\min\\\{1,\\kappa D\_\{k\}\(x,\\pi\)\\\},\\qquad P\(\\pi\)=\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}\(x,\\pi\),\\qquad\\widehat\{P\}\(\\pi\)=\\frac\{1\}\{m\}\\sum\_\{j\}\\mathcal\{L\}\_\{k\}\(x\_\{j\},\\pi\)\. ###### Lemma C\.28\(Uniform fitting certificate for unit\-weight SC with KL\)\. LetL=⌊G/2⌋L=\\lfloor G/2\\rfloor,BM=2\(CM\+4\)B\_\{M\}=2\(C\_\{M\}\+4\), andℓ=log\(4K𝒩\(ν\)/δ\)\\ell=\\log\(4K\\mathcal\{N\}\(\\nu\)/\\delta\)\. Except on an event of conditional probability at mostδ/K\\delta/K, every candidate satisfying Equation \([C\.10](https://arxiv.org/html/2611.21791#A3.E10)\) obeys P\(πk\+1\)≤4κek\+2κBMHℓmL\+4ℓm\+36κHν\.P\(\\pi\_\{k\+1\}\)\\leq 4\\kappa e\_\{k\}\+\\frac\{2\\kappa B\_\{M\}H\\ell\}\{mL\}\+\\frac\{4\\ell\}\{m\}\+36\\kappa H\\nu\.\(C\.53\) ###### Proof\. Step 1: response concentration\.First fix a candidate and condition on the sampled prompts\. Randomly pair theGGresponses in each group, leaving one out whenGGis odd\. For a fixed pairing there aremLmLconditionally independent kernels\. The complete pair average is the conditional expectation over these pairings, so Jensen’s inequality transfers the exponential\-moment bound to the actual grouped estimator\. By Lemma[C\.27](https://arxiv.org/html/2611.21791#A3.Thmtheorem27), the summed conditional variance is at mostmLCMHD¯mLC\_\{M\}H\\overline\{D\}, and each centered kernel has absolute bound6H6H\. Bernstein’s inequality therefore implies, with probability at least1−e−ℓ1\-e^\{\-\\ell\}, D¯−D^\\displaystyle\\overline\{D\}\-\\widehat\{D\}≤2CMHD¯ℓmL\+4HℓmL\\displaystyle\\leq\\sqrt\{\\frac\{2C\_\{M\}H\\overline\{D\}\\,\\ell\}\{mL\}\}\+\\frac\{4H\\ell\}\{mL\}≤12D¯\+\(CM\+4\)HℓmL,\\displaystyle\\leq\\tfrac\{1\}\{2\}\\overline\{D\}\+\\frac\{\(C\_\{M\}\+4\)H\\ell\}\{mL\},and hence D¯≤2D^\+BMHℓmL\.\\overline\{D\}\\leq 2\\widehat\{D\}\+\\frac\{B\_\{M\}H\\ell\}\{mL\}\.\(C\.54\)Step 2: prompt concentration\.For the independent prompt draws,0≤ℒk≤10\\leq\\mathcal\{L\}\_\{k\}\\leq 1andVarxℒk≤P\\operatorname\{Var\}\_\{x\}\\mathcal\{L\}\_\{k\}\\leq P\. The same Bernstein inversion gives P≤2P^\+4ℓmP\\leq 2\\widehat\{P\}\+\\frac\{4\\ell\}\{m\}\(C\.55\)with probability at least1−e−ℓ1\-e^\{\-\\ell\}\. Step 3: extend uniformly over fitted policies\.To make Equations \([C\.54](https://arxiv.org/html/2611.21791#A3.E54)\) and \([C\.55](https://arxiv.org/html/2611.21791#A3.E55)\) uniform, use the continuity of the empirical objective\. The centered sampled log\-potentialZπZ\_\{\\pi\}is2H2H\-Lipschitz ind∞d\_\{\\infty\}\. ForRg∈\[0,1\]R\_\{g\}\\in\[0,1\], 1G−1∑g\|Rg−R¯\|≤G2\(G−1\)≤1\.\\frac\{1\}\{G\-1\}\\sum\_\{g\}\|R\_\{g\}\-\\overline\{R\}\|\\leq\\frac\{G\}\{2\(G\-1\)\}\\leq 1\.The derivative of2k\(u\)2k\(u\)with respect tologu\\log uis2ulogu2u\\log u, whose absolute value is at moste1/2e^\{1/2\}on the feasible interval\. Thus the grouped empirical objective and its population expectation are at most\(2\+e1/2\)H<4H\(2\+e^\{1/2\}\)H<4H\-Lipschitz\. The conditional gapDk\(x,π\)D\_\{k\}\(x,\\pi\)has the same bound, andℒk\\mathcal\{L\}\_\{k\}is4κH4\\kappa H\-Lipschitz\. Apply Equations \([C\.54](https://arxiv.org/html/2611.21791#A3.E54)\) and \([C\.55](https://arxiv.org/html/2611.21791#A3.E55)\) to a feasible radius\-ν\\nucover\. The union bound has failure probability at most2𝒩\(ν\)e−ℓ≤δ/K2\\mathcal\{N\}\(\\nu\)e^\{\-\\ell\}\\leq\\delta/K\. Extending from each cover center to its candidate gives, simultaneously, D¯≤2D^\+BMHℓmL\+12Hν,P≤2P^\+4ℓm\+12κHν\.\\overline\{D\}\\leq 2\\widehat\{D\}\+\\frac\{B\_\{M\}H\\ell\}\{mL\}\+12H\\nu,\\qquad P\\leq 2\\widehat\{P\}\+\\frac\{4\\ell\}\{m\}\+12\\kappa H\\nu\.FinallyP^≤κD¯\\widehat\{P\}\\leq\\kappa\\overline\{D\}andD^\(πk\+1\)≤ek\\widehat\{D\}\(\\pi\_\{k\+1\}\)\\leq e\_\{k\}, becauseπk∗\\pi\_\{k\}^\{\*\}is a feasible empirical comparator\. Substitution proves Equation \([C\.53](https://arxiv.org/html/2611.21791#A3.E53)\)\. ∎ We now combine the outer\-loop progress lemmas with the uniform fitting certificates\. Fresh data at each round allow the certificates to be applied conditional on all previous rounds\. The resulting sufficient round counts areO\(Hlog\(N/ϵ\)\)O\(H\\log\(N/\\epsilon\)\)forβ=0\\beta=0andO\(log\(N/ϵ\)\)O\(\\log\(N/\\epsilon\)\)for the regularized settings in the main theorems\. #### C\.3\.5Combined bounds for unregularized SC and TIS The uniform fitting bound controls each round’s population loss\. Substituting this bound into the outer progress inequality gives the following reward and sample guarantee\. ###### Theorem C\.29\(Best\-of\-N improvement and sample budget\)\. Use SC, or SC\+TIS withc≥erc\\geq e^\{r\}, and set r=18H,K=⌈512Hlog6Nϵ⌉,N≥1,0<ϵ<1\.r=\\frac\{1\}\{8H\},\\qquad K=\\left\\lceil 512H\\log\\frac\{6N\}\{\\epsilon\}\\right\\rceil,\\quad N\\geq 1,\\quad 0<\\epsilon<1\.At each round setμ=πk\{\\mu\}=\\pi\_\{k\}and let Γk\(r\)=\{π:\|logπ\(b∣ht\)μ\(b∣ht\)\|≤rfor every reachableht,b,t\}\.\\Gamma\_\{k\}\(r\)=\\left\\\{\\pi:\\left\|\\log\\frac\{\\pi\(b\\mid h\_\{t\}\)\}\{\{\\mu\}\(b\\mid h\_\{t\}\)\}\\right\|\\leq r\\text\{ for every reachable \}h\_\{t\},b,t\\right\\\}\.These are full families of normalized token policies\. Let𝒩\(u\)\\mathcal\{N\}\(u\)be a common nonincreasing upper bound on the radius\-uucovering numbers ofΓk\(r\)\\Gamma\_\{k\}\(r\)ind∞d\_\{\\infty\}, with centers inΓk\(r\)\\Gamma\_\{k\}\(r\)\. Drawmmfresh prompts per round andG≥2G\\geq 2independent behavior responses per prompt\. Suppose the fitted policyπk\+1∈Γk\(r\)\\pi\_\{k\+1\}\\in\\Gamma\_\{k\}\(r\)satisfies 𝒞^k\(πk\+1\)≥supπ∈Γk\(r\)𝒞^k\(π\)−ϵ128π2NK2\.\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\pi\_\{k\+1\}\)\\geq\\sup\_\{\\pi\\in\\Gamma\_\{k\}\(r\)\}\\widehat\{\\mathcal\{C\}\}\_\{k\}\(\\pi\)\-\\frac\{\\epsilon\}\{128\\pi^\{2\}NK^\{2\}\}\.For0<δ<10<\\delta<1, if m≥⌈32K\+12544π2NK2/⌊G/2⌋ϵlog\[4Kδ𝒩\(ϵ3840π2NK2H\)2\]⌉,m\\geq\\left\\lceil\\frac\{32K\+12544\\pi^\{2\}NK^\{2\}/\\lfloor G/2\\rfloor\}\{\\epsilon\}\\log\\\!\\left\[\\frac\{4K\}\{\\delta\}\\,\\mathcal\{N\}\\\!\\left\(\\frac\{\\epsilon\}\{3840\\pi^\{2\}NK^\{2\}H\}\\right\)^\{2\}\\right\]\\right\\rceil,\(C\.56\)then afterKKrounds, usingKmKmprompt draws andKmGKmGresponses,VN\(π0\)−J\(πK\)≤ϵV\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\epsilonwith probability at least1−δ1\-\\delta\. ###### Proof of Theorem[C\.29](https://arxiv.org/html/2611.21791#A3.Thmtheorem29)\. Apply Lemma[C\.20](https://arxiv.org/html/2611.21791#A3.Thmtheorem20)withek=ϵ/\(128π2NK2\)e\_\{k\}=\\epsilon/\(128\\pi^\{2\}NK^\{2\}\)andκ=4π2NK\\kappa=4\\pi^\{2\}NK\. Chooseu=ϵ/\(3840π2NK2H\)u=\\epsilon/\(3840\\pi^\{2\}NK^\{2\}H\)\. Each of the four terms is at mostϵ/\(8K\)\\epsilon/\(8K\)under \([C\.56](https://arxiv.org/html/2611.21791#A3.E56)\)\. Thus each round contributes at mostϵ/\(2K\)\\epsilon/\(2K\)to \([C\.44](https://arxiv.org/html/2611.21791#A3.E44)\)\. The concentration events hold conditionally on earlier rounds and uniformly over fitted policies\. Fresh samples and a union bound give simultaneous success with probability at least1−δ1\-\\delta\. Equation \([C\.44](https://arxiv.org/html/2611.21791#A3.E44)\) then proves the reward guarantee\. Finally,K≤513Hlog\(6N/ϵ\)K\\leq 513H\\log\(6N/\\epsilon\)and⌊G/2⌋≥G/3\\lfloor G/2\\rfloor\\geq G/3give \([C\.57](https://arxiv.org/html/2611.21791#A3.E57)\)\. ∎ Unregularized budget substitution\.Choosingmmas the sufficient ceiling in Equation \([C\.56](https://arxiv.org/html/2611.21791#A3.E56)\) gives the total prompt budget Km=O\(CLOSE\\displaystyle Km=O\\\!\\Biggl\(H2log2\(6N/ϵ\)\+\(NH3/G\)log3\(6N/ϵ\)ϵ\\displaystyle\\frac\{H^\{2\}\\log^\{2\}\(6N/\\epsilon\)\+\(NH^\{3\}/G\)\\log^\{3\}\(6N/\\epsilon\)\}\{\\epsilon\}\(C\.57\)×log\[2052Hlog\(6N/ϵ\)δ𝒩\(ϵ3840⋅5132π2NH3log2\(6N/ϵ\)\)2\]\)\.\\displaystyle\\times\\log\\\!\\left\[\\frac\{2052H\\log\(6N/\\epsilon\)\}\{\\delta\}\\,\\mathcal\{N\}\\\!\\left\(\\frac\{\\epsilon\}\{3840\\cdot 513^\{2\}\\pi^\{2\}NH^\{3\}\\log^\{2\}\(6N/\\epsilon\)\}\\right\)^\{2\}\\right\]\\Biggr\)\. #### C\.3\.6Proof of Theorem[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1) For regularized TIS, the same combination uses the exact target gap and its reward\-transfer bound\. The proof below uses the fitting certificate to bound the error in each round\. ###### Proof of Theorem[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)\. Choose v=ϵ2160π2NK2H,ek≤ϵ160π2NK2\.v=\\frac\{\\epsilon\}\{2160\\pi^\{2\}NK^\{2\}H\},\\qquad e\_\{k\}\\leq\\frac\{\\epsilon\}\{160\\pi^\{2\}NK^\{2\}\}\.Withκ=10π2NK/β\\kappa=10\\pi^\{2\}NK/\\betaandβ=2\\beta=2, the first and fourth terms in \([C\.49](https://arxiv.org/html/2611.21791#A3.E49)\) are each at mostϵ/\(8K\)\\epsilon/\(8K\)\. The sample count \([C\.14](https://arxiv.org/html/2611.21791#A3.E14)\) bounds each of the other two terms byϵ/\(8K\)\\epsilon/\(8K\)\. Thus𝔼xℒk≤ϵ/\(2K\)\\mathbb\{E\}\_\{x\}\\mathcal\{L\}\_\{k\}\\leq\\epsilon/\(2K\)at every successful round\. Combining the rounds\.The concentration statements are uniform over fitted policies and hold conditionally on all earlier rounds\. Fresh data at each round and a union bound give simultaneous success with probability at least1−δ1\-\\delta\. Summing \([C\.33](https://arxiv.org/html/2611.21791#A3.E33)\) proves Theorem[C\.1](https://arxiv.org/html/2611.21791#A3.Thmtheorem1)\. Finally, choosemmas the sufficient ceiling in Equation \([C\.14](https://arxiv.org/html/2611.21791#A3.E14)\)\. ThenK≤5log\(6N/ϵ\)K\\leq 5\\log\(6N/\\epsilon\)and⌊G/2⌋≥G/3\\lfloor G/2\\rfloor\\geq G/3yield \([C\.58](https://arxiv.org/html/2611.21791#A3.E58)\)\. The first budget term estimates capped prompt losses\. The second estimates conditional objective gaps using the responses within each group\. ∎ Choosing the sufficient prompt count in Equation \([C\.14](https://arxiv.org/html/2611.21791#A3.E14)\) gives the total prompt budget Km=O\(CLOSE\\displaystyle Km=O\\\!\\Biggl\(log2\(6N/ϵ\)\+\(NH/G\)log3\(6N/ϵ\)ϵ\\displaystyle\\frac\{\\log^\{2\}\(6N/\\epsilon\)\+\(NH/G\)\\log^\{3\}\(6N/\\epsilon\)\}\{\\epsilon\}\(C\.58\)×log\[20log\(6N/ϵ\)δ𝒩\(ϵ54000π2NHlog2\(6N/ϵ\)\)\]\)\.\\displaystyle\\times\\log\\\!\\left\[\\frac\{20\\log\(6N/\\epsilon\)\}\{\\delta\}\\,\\mathcal\{N\}\\\!\\left\(\\frac\{\\epsilon\}\{54000\\pi^\{2\}NH\\log^\{2\}\(6N/\\epsilon\)\}\\right\)\\right\]\\Biggr\)\. #### C\.3\.7Proof of Theorem[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2) Reward transfer and the outer\-round guarantee\. ###### Proof of Theorem[C\.2](https://arxiv.org/html/2611.21791#A3.Thmtheorem2)\. Writesk\(x\)=Jx\(πk\)s\_\{k\}\(x\)=J\_\{x\}\(\\pi\_\{k\}\)andf\(v\)=67v/\(60\+7v\)f\(v\)=67v/\(60\+7v\)\. Lemma[C\.9](https://arxiv.org/html/2611.21791#A3.Thmtheorem9)andv\+760v\(1−v\)≥f\(v\)v\+\\tfrac\{7\}\{60\}v\(1\-v\)\\geq f\(v\)imply \[f\(sk\(x\)\)−sk\+1\(x\)\]\+≤Dk\(x,πk\+1\)\.\[f\(s\_\{k\}\(x\)\)\-s\_\{k\+1\}\(x\)\]\_\{\+\}\\leq D\_\{k\}\(x,\\pi\_\{k\+1\}\)\.Letαk=arcsinsk\\alpha\_\{k\}=\\arcsin\\sqrt\{s\_\{k\}\}andT\(θ\)=arctan\(eλ/2tanθ\)T\(\\theta\)=\\arctan\(e^\{\\lambda/2\}\\tan\\theta\), withλ=log\(67/60\)\\lambda=\\log\(67/60\)\. Thensin2T\(αk\)=f\(sk\)\\sin^\{2\}T\(\\alpha\_\{k\}\)=f\(s\_\{k\}\)\. The probability\-shortfall part of Lemma[C\.3](https://arxiv.org/html/2611.21791#A3.Thmtheorem3)gives ξk\(x\)2:=\[T\(αk\(x\)\)−αk\+1\(x\)\]\+2≤π24Dk\(x,πk\+1\)\.\\xi\_\{k\}\(x\)^\{2\}:=\[T\(\\alpha\_\{k\}\(x\)\)\-\\alpha\_\{k\+1\}\(x\)\]\_\{\+\}^\{2\}\\leq\\frac\{\\pi^\{2\}\}\{4\}D\_\{k\}\(x,\\pi\_\{k\+1\}\)\.Use the same lemma at accuracyϵ/2\\epsilon/2\. The choice ofKKin Equation \([C\.15](https://arxiv.org/html/2611.21791#A3.E15)\) andmin\{1,∑kyk\}≤∑kmin\{1,yk\}\\min\\\{1,\\sum\_\{k\}y\_\{k\}\\\}\\leq\\sum\_\{k\}\\min\\\{1,y\_\{k\}\\\}foryk≥0y\_\{k\}\\geq 0yield VN\(π0\)−J\(πK\)≤ϵ2\+∑k<K𝔼xmin\{1,4π2NKDk\(x,πk\+1\)\}\.V\_\{N\}\(\\pi\_\{0\}\)\-J\(\\pi\_\{K\}\)\\leq\\frac\{\\epsilon\}\{2\}\+\\sum\_\{k<K\}\\mathbb\{E\}\_\{x\}\\min\\\{1,4\\pi^\{2\}NKD\_\{k\}\(x,\\pi\_\{k\+1\}\)\\\}\.\(C\.59\)For the statedeke\_\{k\},ν\\nu, andmm, each of the four terms in Equation \([C\.53](https://arxiv.org/html/2611.21791#A3.E53)\) is at mostϵ/\(8K\)\\epsilon/\(8K\)\. Thus each expectation in Equation \([C\.59](https://arxiv.org/html/2611.21791#A3.E59)\) is at mostϵ/\(2K\)\\epsilon/\(2K\), except on an event of conditional probabilityδ/K\\delta/K\. The data are fresh at every round, so the uniform certificates apply conditionally on all previous rounds\. A union bound overKKrounds and Equation \([C\.59](https://arxiv.org/html/2611.21791#A3.E59)\) prove the claim\. ∎ Total sample budget\.For fixedMM, choosingmmas the ceiling in Equation \([C\.16](https://arxiv.org/html/2611.21791#A3.E16)\) gives the sufficient total counts Km=O\[K2\+NHK3/⌊G/2⌋ϵlog\(4K𝒩\(ν\)δ\)\],responses=GKm\.Km=O\\\!\\left\[\\frac\{K^\{2\}\+NHK^\{3\}/\\lfloor G/2\\rfloor\}\{\\epsilon\}\\log\\\!\\left\(\\frac\{4K\\mathcal\{N\}\(\\nu\)\}\{\\delta\}\\right\)\\right\],\\qquad\\text\{responses\}=GKm\.\(C\.60\)SinceK=O\(log\(N/ϵ\)\)K=O\(\\log\(N/\\epsilon\)\), the sufficient prompt and response factors are\(1\+NH/G\)/ϵ\(1\+NH/G\)/\\epsilonand\(G\+NH\)/ϵ\(G\+NH\)/\\epsilon, respectively, up to logarithms and the covering\-number term\. Finite Monte Carlo centers enter throughBMB\_\{M\}in Equation \([C\.15](https://arxiv.org/html/2611.21791#A3.E15)\)\. For finiteMM, they require at mostMHGKmMHGKmadditional token draws\. ### C\.4Surrogate integrands and derivatives used in the main text The convergence arguments above use population objectives and fitting gaps\. To relate these objectives to the token updates plotted in the main text, the following calculations give the integrands and derivatives in Table[5](https://arxiv.org/html/2611.21791#A2.T5)and Figure[2](https://arxiv.org/html/2611.21791#S4.F2)\. At a fixed history, writeρ=πθ\(a∣h\)/μ\(a∣h\)\\rho=\\pi\_\{\\theta\}\(a\\mid h\)/\\mu\(a\\mid h\)andA=Aμ\(h,a\)A=A^\{\\mu\}\(h,a\), and holdμ\\mu,AA, and the other learner logits fixed\. HereAAis the expected advantage underμ\\mudefined in Equation \([C\.1](https://arxiv.org/html/2611.21791#A3.E1)\);AiA\_\{i\}in the sampled update of Equation \([7](https://arxiv.org/html/2611.21791#S3.E7)\) is a group\-centered reward\. For the clipped reward weight used in Section[3\.1](https://arxiv.org/html/2611.21791#S3.SS1), a\(u\)=clip\(u,clow,chigh\),0≤clow≤1≤chigh,a\(u\)=\\operatorname\{clip\}\(u,c\_\{\\mathrm\{low\}\},c\_\{\\mathrm\{high\}\}\),\\qquad 0\\leq c\_\{\\mathrm\{low\}\}\\leq 1\\leq c\_\{\\mathrm\{high\}\},\(C\.61\)integratinga\(u\)/ua\(u\)/ugives Fa\(u\)=\{clow−1\+clowlog\(u/clow\),u<clow,u−1,clow≤u≤chigh,chigh−1\+chighlog\(u/chigh\),u\>chigh\.F\_\{a\}\(u\)=\\begin\{cases\}c\_\{\\mathrm\{low\}\}\-1\+c\_\{\\mathrm\{low\}\}\\log\(u/c\_\{\\mathrm\{low\}\}\),&u<c\_\{\\mathrm\{low\}\},\\\\ u\-1,&c\_\{\\mathrm\{low\}\}\\leq u\\leq c\_\{\\mathrm\{high\}\},\\\\ c\_\{\\mathrm\{high\}\}\-1\+c\_\{\\mathrm\{high\}\}\\log\(u/c\_\{\\mathrm\{high\}\}\),&u\>c\_\{\\mathrm\{high\}\}\.\\end\{cases\}\(C\.62\)The first branch is absent whenclow=0c\_\{\\mathrm\{low\}\}=0\. Thus the intervals\[1,1\]\[1,1\],\[0,c\]\[0,c\], and\[0,∞\)\[0,\\infty\)give unit\-weight SC, one\-sided TIS, and full IS, respectively\. In particular, SC usesFa\(u\)=loguF\_\{a\}\(u\)=\\log u, whereas full IS usesFa\(u\)=u−1F\_\{a\}\(u\)=u\-1\. With standard reverse KL, definek\(ρ\)=ρlogρ−ρ\+1k\(\\rho\)=\\rho\\log\\rho\-\\rho\+1\. The single\-action contribution to the main\-text objective is w=AFa\(ρ\)−βk\(ρ\)\.w=AF\_\{a\}\(\\rho\)\-\\beta k\(\\rho\)\.Averaging over actions underμ\\muand over behavior histories gives Equation \([5](https://arxiv.org/html/2611.21791#S3.E5)\), since𝔼μ\[ρ∣h\]=1\\mathbb\{E\}\_\{\\mu\}\[\\rho\\mid h\]=1and𝔼μ\[k\(ρ\)∣h\]=KL\(πθ∥μ\)\\mathbb\{E\}\_\{\\mu\}\[k\(\\rho\)\\mid h\]=\\operatorname\{KL\}\(\\pi\_\{\\theta\}\\\|\\mu\)\. The identityρFa′\(ρ\)=a\(ρ\)\\rho F\_\{a\}^\{\\prime\}\(\\rho\)=a\(\\rho\)gives ∂w∂ρ=Aa\(ρ\)ρ−βlogρ\.\\frac\{\\partial w\}\{\\partial\\rho\}=\\frac\{Aa\(\\rho\)\}\{\\rho\}\-\\beta\\log\\rho\.For the selected action’s softmax logitzaz\_\{a\},∂ρ/∂za=ρ\[1−πθ\(a∣h\)\]\\partial\\rho/\\partial z\_\{a\}=\\rho\[1\-\\pi\_\{\\theta\}\(a\\mid h\)\]\. Therefore ∂w∂za=\[Aa\(ρ\)−βρlogρ\]\[1−πθ\(a∣h\)\]\.\\frac\{\\partial w\}\{\\partial z\_\{a\}\}=\\bigl\[Aa\(\\rho\)\-\\beta\\rho\\log\\rho\\bigr\]\[1\-\\pi\_\{\\theta\}\(a\\mid h\)\]\.\(C\.63\)Settingβ=0\\beta=0gives the unregularized rows of Table[5](https://arxiv.org/html/2611.21791#A2.T5); choosinga=1a=1ora\(ρ\)=min\(ρ,c\)a\(\\rho\)=\\min\(\\rho,c\)gives the two RSC rows\. At the TIS cap, the derivatives from both sides agree\. Equation \([C\.63](https://arxiv.org/html/2611.21791#A3.E63)\) describes the selected action’s contribution\. The complete conditional objective averages all action contributions underμ\\mu\. Figure[2](https://arxiv.org/html/2611.21791#S4.F2)usesA=1A=1, TIS capc=2c=2,β=0\.05\\beta=0\.05for RSC, and uncapped reverse KL\. Its derivative coefficients are those in Equation \([C\.63](https://arxiv.org/html/2611.21791#A3.E63)\) after removing the common factor1−πθ\(a∣h\)1\-\\pi\_\{\\theta\}\(a\\mid h\)\. The curves use positive ratios from10−510^\{\-5\}to1010, with denser sampling near zero and explicit points at one and two\. At fixedμ\(a∣h\)\>0\\mu\(a\\mid h\)\>0, the ratio is bounded by1/μ\(a∣h\)1/\\mu\(a\\mid h\)and the displayed logit derivative vanishes asπθ\(a∣h\)→1\\pi\_\{\\theta\}\(a\\mid h\)\\to 1\. Across histories, large ratios can arise from small behavior probabilities, as discussed in Section[4](https://arxiv.org/html/2611.21791#S4)\. For an active penalty cap, the corresponding coefficient is instead−βb\(ρ\)logb\(ρ\)\-\\beta b\(\\rho\)\\log b\(\\rho\), derived in Proposition[C\.15](https://arxiv.org/html/2611.21791#A3.Thmtheorem15)\. ##### Population gradient identities\. With behaviorμ\\mu, its advantages, and clipping bounds fixed,∇θρ=ρψθ\\nabla\_\{\\theta\}\\rho=\\rho\\psi\_\{\\theta\}andρFa′\(ρ\)=a\(ρ\)\\rho F\_\{a\}^\{\\prime\}\(\\rho\)=a\(\\rho\)give ∇θ𝔼μ\[∑t=1HAtμFa\(ρt\)\]=𝔼μ\[∑t=1HAtμρtTISψθ\(ht,at\)\]\.\\nabla\_\{\\theta\}\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[\\sum\_\{t=1\}^\{H\}A\_\{t\}^\{\\mu\}F\_\{a\}\(\\rho\_\{t\}\)\\right\]=\\mathbb\{E\}\_\{\\mu\}\\\!\\left\[\\sum\_\{t=1\}^\{H\}A\_\{t\}^\{\\mu\}\\rho\_\{t\}^\{\\mathrm\{TIS\}\}\\psi\_\{\\theta\}\(h\_\{t\},a\_\{t\}\)\\right\]\.\(C\.64\)For standard reverse KL at a fixed history, the same differentiation gives ∇θKL\(πθ\(⋅∣h\)∥μ\(⋅∣h\)\)\\displaystyle\\nabla\_\{\\theta\}\\operatorname\{KL\}\(\{\\pi\_\{\\theta\}\}\(\\cdot\\mid h\)\\\|\{\\mu\}\(\\cdot\\mid h\)\)=𝔼μ\[ρ\(logρ\+1\)ψθ∣h\]\\displaystyle=\\mathbb\{E\}\_\{\\mu\}\[\\rho\(\\log\\rho\+1\)\\psi\_\{\\theta\}\\mid h\]\(C\.65\)=𝔼μ\[ρlogρψθ∣h\]\.\\displaystyle=\\mathbb\{E\}\_\{\\mu\}\[\\rho\\log\\rho\\,\\psi\_\{\\theta\}\\mid h\]\.Normalization gives𝔼μ\[ρψθ∣h\]=0\\mathbb\{E\}\_\{\\mu\}\[\\rho\\psi\_\{\\theta\}\\mid h\]=0, which removes the\+1\+1term\. Combining the reward and penalty identities gives Equation \([6](https://arxiv.org/html/2611.21791#S3.E6)\)\.
相似文章
@qingke_ai:最近,RL 社区两个非常热门的工作——BPO 和 Score Centering——看似是两条完全不同的路径……
分析表明,在理想条件下,RL 后训练中的 BPO 与 Score Centering 收敛到相同的全词表 token 梯度,其中 Score Centering 的作用是针对 off-policy 漂移的偏差校正,而非方差缩减。
@zyh2022:我认为这个系列文章是目前在博客或论文中能找到的、关于 LLM 异步 RL 分数中心化最直观、最直接的讲解……
推荐 HuggingFace 博客文章系列《Score Function Centering》,从直觉出发系统讲解 LLM 异步强化学习中的 trainer-generator 失配问题以及分数函数中心化(SFC)如何修复策略梯度中零期望性质,只需了解 GRPO 即可阅读。
关于KL正则化策略优化的研究
一篇探讨KL正则化策略优化的研究论文,可能与RLHF(基于人类反馈的强化学习)相关,发布于arXiv。
Self-Credit:通过组内重缩放实现长程智能体强化学习的信用分配
本文提出 Self-Credit,一种智能体强化学习方法,通过自生成的信用分配和组内重缩放来稠密化稀疏的 RLVR 奖励,在 BrowseComp-Plus、GAIA 和 xBench-DeepSearch 基准上使用 Qwen3 模型取得了显著提升。
通过近端KL正则化实现Off-Policy RLVR中的Token级重要性采样
这篇论文研究了RLVR中的off-policy数据问题,证明了在覆盖性与局部表达性条件下,完整未来KL信用可使精确近端优化器成为更新的唯一平稳策略,并提出ProximalKL方法,在Qwen3-1.7B和Qwen3-4B上相比GRPO在数学基准上最高提升2.58%的微平均准确率。