ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

# ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards
Source: [https://arxiv.org/html/2609.36058](https://arxiv.org/html/2609.36058)
Florent DrayeAffiliation:MPI\-ISBernhard SchölkopfAffiliation:MPI\-ISAffiliation:ELLIS Institute Tübingen

###### Abstract

Recent progress in reinforcement learning with verifiable rewards \(RLVR\) has highlighted the effectiveness of simple critic\-free policy\-gradient methods such as Group Relative Policy Optimization \(GRPO\)\. In contrast, actor\-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal\-difference error\. Motivated by this observation, we revisit trajectory\-level control variates through an advantage–value formulation, which we call Advantage\-Based Control Variates \(ABC\)\. This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation \(DAE\)\. Finally, we combine ABC with DAE into a single actor\-critic algorithm and evaluate it in an offline\-to\-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning\. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) has become a powerful approach for improving large language models \(LLMs\) beyond supervised fine\-tuning, particularly in domains such as mathematical reasoning, code generation, and formal theorem proving, where rewards can be computed automatically\([Lambert et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib8);[Shao et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib3);[Guo et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib4)\)\.

Actor\-critic methods\([Konda and Tsitsiklis, 1999](https://arxiv.org/html/2609.36058#bib.bib11);[Mnih et al\., 2016](https://arxiv.org/html/2609.36058#bib.bib1)\)such as Proximal Policy Optimization \(PPO\) estimate the advantage function using a learned value function\([Schulman et al\., 2015](https://arxiv.org/html/2609.36058#bib.bib2);[Schulman et al\., 2017](https://arxiv.org/html/2609.36058#bib.bib7)\)to reduce policy optimization variance, and remain a strong baseline for deep RL applications\([Rudin et al\., 2022](https://arxiv.org/html/2609.36058#bib.bib9);[Radosavovic et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib10)\)\. In LLM post\-training, however, maintaining a separate critic can be computationally expensive\. As such, modern RLVR methods often adopt Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib3)\), which removes the learned critic and estimates advantages by comparing groups of on\-policy rollouts\. GRPO has achieved strong results despite discarding the learned critic, raising the possibility that learning value functions in this case can be difficult and the bias from the approximation of the advantages can offset their variance\-reduction benefits\. However, GRPO’s group\-based estimator requires multiple fresh rollouts per prompt, provides trajectory\-level feedback, and no learning signal when rewards within a group are identical\([Yu et al\., 2026](https://arxiv.org/html/2609.36058#bib.bib12)\)\. These limitations motivate a method that allows fine\-grained token\-level feedback while ensuring that the bias stays controlled\.

Motivated by these observations, we studyAdvantage\-BasedControl variate \(ABC\), an advantage\-value formulation of trajectory\-level control variates\([Cheng et al\., 2020](https://arxiv.org/html/2609.36058#bib.bib38)\)that naturally combines learned advantage and value functions for variance reduction\. We show that its covariance is closely related to Direct Advantage Estimation \(DAE\)\([Pan et al\., 2022](https://arxiv.org/html/2609.36058#bib.bib5)\), a method that directly estimates the advantage and value functions, and develop a simple actor\-critic method that combines both ABC and DAE\. We evaluate our approach in an offline\-to\-online setting, where the critic is first trained on an offline dataset to approximate advantage and value functions for the target policy, and the actor and critic are subsequently updated using online sampled trajectories\. We show that, despite its simplicity, this combination achieves performance competitive with Dr\. GRPO\([Liu et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib15)\), an extension of the original GRPO\. To summarize, our contributions are:

1. 1\.We present ABC, an advantage\-value formulation of trajectory\-level control variates that incorporates learned advantage and value functions into policy\-gradient estimation while preserving unbiasedness\.
2. 2\.We theoretically characterize its variance and show that the true advantage and value functions minimize its covariance\. In deterministic environments, we bound the excess variance in terms of the return prediction error from the estimated advantage and value functions, motivating DAE as the critic objective\.
3. 3\.We evaluate the combination of ABC and DAE in an offline\-to\-online RLVR setup, and show that it achieves competitive performance with Dr\. GRPO using as little as 100 policy gradient updates\.

## 2Background

We consider a discounted Markov decision process \(MDP\) defined by\(𝒮,𝒜,r,p,γ\)\(\{\\mathcal\{S\}\},\{\\mathcal\{A\}\},r,p,\\gamma\), where𝒮\{\\mathcal\{S\}\}is the state space,𝒜\{\\mathcal\{A\}\}is the action space,r⁡\(s,a\)r\(s,a\)is the reward function,p⁡\(s′\|s,a\)p\(s^\{\\prime\}\|s,a\)is the transition probability, andγ∈\[0,1\)\\gamma\\in\[0,1\)is the discount factor\. We consider the discounted setting here for theoretical considerations, but the results carry over to other settings \(e\.g\., finite horizon\) as well\. The policy is denoted byπ⁡\(a\|s\)\\pi\(a\|s\), the value functions for a given policyπ\\piare denoted byVπ​\(s\)=𝔼π​\[∑t=0∞γt​r​\(st,at\)∣s0=s\]V^\{\\pi\}\(s\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}\{=\}s\\right\],Qπ\(s,a\)=𝔼π\[∑t=0∞γtr\(st,at\)∣s0=s,a0=a\]Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\mid s\_\{0\}\{=\}s,a\_\{0\}\{=\}a\\right\], andAπ​\(s,a\)=Qπ​\(s,a\)−Vπ​\(s\)A^\{\\pi\}\(s,a\)=Q^\{\\pi\}\(s,a\)\-V^\{\\pi\}\(s\)\. The objective of RL is to optimizeJ⁡\(π\)=𝔼⁡\[Vπ​\(s0\)\]J\(\\pi\)=\\mathbb\{E\}\[V^\{\\pi\}\(s\_\{0\}\)\]\.

##### RLVR

In RLVR, states correspond to the prompts and previously generated tokens, while the actions correspond to the generated tokens\. The rewards are typically rule\-based, for example indicating whether a mathematical answer is correct, and often without discounting \(i\.e\.,γ=1\\gamma=1\)\. The environment dynamics are typically deterministic conditional on the selected action, since appending a token uniquely determines the next state\. Finally, we note that RLVR is typically finite\-horizon and undiscounted \(γ=1\\gamma=1\), and we use the infinite\-horizon discounted formulation only for the theoretical presentation\.

##### Policy Optimization

One popular method for optimizing parametrized policies is by estimating the gradient of the expected return directly\. The REINFORCE algorithm does this by,

∇J​\(θ\)=𝔼πθ​\[G​∑t=0∞∇log⁡πθ​\(at\|st\)\],\\nabla J\(\\theta\)=\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\\left\[G\\sum\_\{t=0\}^\{\\infty\}\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\right\],\(1\)whereG=∑t=0∞γt​r​\(st,at\)G=\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\([Williams, 1992](https://arxiv.org/html/2609.36058#bib.bib13)\)\. Equivalently, by the policy gradient theorem, we have∇J\(θ\)=𝔼\(s,a\)∼dπθ\[G∇logπθ\(a\|s\)\]=𝔼\(s,a\)∼dπθ\[Aπ\(s,a\)∇logπθ\(a\|s\)\]\\nabla J\(\\theta\)=\\mathbb\{E\}\_\{\(s,a\)\\sim d^\{\\pi\_\{\\theta\}\}\}\\left\[G\\nabla\\log\\pi\_\{\\theta\}\(a\|s\)\\right\]=\\mathbb\{E\}\_\{\(s,a\)\\sim d^\{\\pi\_\{\\theta\}\}\}\\left\[A^\{\\pi\}\(s,a\)\\nabla\\log\\pi\_\{\\theta\}\(a\|s\)\\right\], wheredπθd^\{\\pi\_\{\\theta\}\}is the \(unnormalized\) occupancy measure induced byπθ\\pi\_\{\\theta\}\([Sutton et al\., 1999](https://arxiv.org/html/2609.36058#bib.bib14)\)\. We focus on the on\-policy case and omitdπθd^\{\\pi\_\{\\theta\}\}in the following unless otherwise stated\.

This approach forms the basis of modern policy optimization algorithms such as PPO or GRPO, where we optimize the clipping objective given by:

ℒ⁡\(θ\)=𝔼πθold​\[min⁡\(ρθ​\(s,a\)​A^​\(s,a\),clip⁡\(ρθ​\(s,a\),1−ϵ,1\+ϵ\)​A^​\(s,a\)\)\],\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\}\\left\[\\min\\left\(\\rho\_\{\\theta\}\(s,a\)\\hat\{A\}\(s,a\),\\operatorname\{clip\}\\left\(\\rho\_\{\\theta\}\(s,a\),1\-\\epsilon,1\+\\epsilon\\right\)\\hat\{A\}\(s,a\)\\right\)\\right\],\(2\)whereϵ\\epsilonis the clipping parameter,πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}is the data collecting policy,ρθ​\(s,a\)=πθ​\(a∣s\)πθold​\(a∣s\)\\rho\_\{\\theta\}\(s,a\)=\\frac\{\\pi\_\{\\theta\}\(a\\mid s\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\\mid s\)\}is the importance ratio andA^​\(s,a\)\\hat\{A\}\(s,a\)is an estimate of the advantage function\. The importance ratio permits multiple optimization steps on data collected byπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}, while clipping limits the changes to the policy\.

In practice, the advantage function is unknown, and has to be estimated from sampled trajectories\. PPO achieves this using a combination of TD\(λ\\lambda\) and TD\-error\([Schulman et al\., 2015](https://arxiv.org/html/2609.36058#bib.bib2)\), while GRPO achieves this by samplingnnrollouts from the same initial state \(prompt\) via

A^i=Gi−μ^,μ^=1n​∑i=1nGi,\\hat\{A\}\_\{i\}=G\_\{i\}\-\\hat\{\\mu\},\\qquad\\hat\{\\mu\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}G\_\{i\},\(3\)whereGiG\_\{i\}denote the return of rolloutii\. Here,A^i\\hat\{A\}\_\{i\}is a trajectory\-level advantage estimate: every state\-action pair along rolloutiiis assigned the same advantage\. Note that here, unlike the original GRPO formulation, we do not standardize group\-relative advantages, as standardization can introduce undesirable bias\([Liu et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib15)\)\.

##### Control Variate for Policy Gradient

The original REINFORCE algorithm is known to have high variance\. As such, it is common for practitioners to incorporate the value function into the policy gradient estimator via∇J\(θ\)=𝔼\[\(G−V^\(s\)\)∇logπθ\(a\|s\)\]\\nabla J\(\\theta\)=\\mathbb\{E\}\\left\[\(G\-\\hat\{V\}\(s\)\)\\nabla\\log\\pi\_\{\\theta\}\(a\|s\)\\right\], where the value functionV^≈Vπ\\hat\{V\}\\approx V^\{\\pi\}acts as a control variate to reduce the variance\([Greensmith et al\., 2004](https://arxiv.org/html/2609.36058#bib.bib16)\)\. This does not bias the gradient as𝔼\[V^\(s\)∇logπθ\(a\|s\)\]=0\\mathbb\{E\}\\left\[\\hat\{V\}\(s\)\\nabla\\log\\pi\_\{\\theta\}\(a\|s\)\\right\]=0for arbitraryV^\\hat\{V\}, and the choice ofV^\\hat\{V\}only affects the variance of the estimator\. While it is common in practice to chooseV^≈Vπ\\hat\{V\}\\approx V^\{\\pi\}as the baseline\([Mnih et al\., 2016](https://arxiv.org/html/2609.36058#bib.bib1)\), it should be noted that this choice is neither optimal\([Greensmith et al\., 2004](https://arxiv.org/html/2609.36058#bib.bib16)\)nor guaranteed to reduce variance even whenV^≡Vπ\\hat\{V\}\\equiv V^\{\\pi\}\(see Appendix[A](https://arxiv.org/html/2609.36058#A1)for a counterexample\)\.

It is possible to extend this idea beyond just state\-dependent control variates\. TrajCV\([Cheng et al\., 2020](https://arxiv.org/html/2609.36058#bib.bib38)\)considers trajectory\-level control variates, where instead of just using the value of the first state, TrajCV incorporates estimated state\-action valuesQ^​\(st,at\)\\hat\{Q\}\(s\_\{t\},a\_\{t\}\)along the trajectory by noticing that𝔼\[\(Q^\(st,at\)−𝔼\[Q^\(st,at\)\|st\]\)∇logπθ\(a0\|s0\)\]=0\\mathbb\{E\}\[\(\\hat\{Q\}\(s\_\{t\},a\_\{t\}\)\-\\mathbb\{E\}\[\\hat\{Q\}\(s\_\{t\},a\_\{t\}\)\|s\_\{t\}\]\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]=0\(t\>0t\>0\)\. This can further explain away part of the variance caused by stochastic actions from the trajectory\.

##### Direct Advantage Estimation \(DAE\)

DAE is a method that directly parametrizes and estimates the advantage function from sampled trajectories\([Pan et al\., 2022](https://arxiv.org/html/2609.36058#bib.bib5);[Pan and Schölkopf, 2024](https://arxiv.org/html/2609.36058#bib.bib6)\)\. For deterministic environments, DAE minimizes the following constrained least square objective:

ℒ⁡\(A^,V^\)=𝔼μ​\[\(∑t=0∞γt​\(r⁡\(st,at\)−A^​\(st,at\)\)−V^​\(s0\)\)2\],\\displaystyle\{\\mathcal\{L\}\}\(\\hat\{A\},\\hat\{V\}\)=\\mathbb\{E\}\_\{\\mu\}\\left\[\\left\(\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\(r\(s\_\{t\},a\_\{t\}\)\-\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\\right\)\-\\hat\{V\}\(s\_\{0\}\)\\right\)^\{2\}\\right\],\(4\)subject​to​∑a∈𝒜π⁡\(a\|s\)​A^​\(s,a\)=0∀s∈𝒮,\\displaystyle\\mathrm\{subject\\;to\}\\sum\_\{a\\in\{\\mathcal\{A\}\}\}\\pi\(a\|s\)\\hat\{A\}\(s,a\)=0\\quad\\forall s\\in\{\\mathcal\{S\}\},\(5\)whereμ\\muis the behavior policy\. Under mild assumptions on the coverages ofμ\\muandπ\\pi, minimizing this constrained objective gives\(Aπ,Vπ\)=argmin⁡ℒ⁡\(A^,V^\)\(A^\{\\pi\},V^\{\\pi\}\)=\\operatorname\{argmin\}\{\\mathcal\{L\}\}\(\\hat\{A\},\\hat\{V\}\)\. DAE is particularly attractive for RLVR as it can learn from off\-policy data by simply minimizing a least squares without relying on techniques such as importance sampling, which can cause high variance for long trajectories\. Finally, we note that we do not use bootstrapping and simply optimize the objective as is\.

## 3Advantage\-Based Control Variate for Policy Gradient

In this section, we present the ABC estimator, which extends value\-based control variate by considering advantages from future state\-actions, and show how it allows us to combine estimated advantage and value functions without biasing policy gradients\. We then study its variance structure and show that minimum variance is achieved when the estimates of the advantage and value functions are exact\. Finally, we show that the variance is closely related to the return prediction error from the advantage and value estimates, and draw a connection to DAE\.

We first provide some intuitions for the ABC estimator\. For state\-based control variates, using the value function can be seen as removing the variance of the weighting \(i\.e\.,Var⁡\(G−Vπ​\(s0\)\)≤Var⁡\(G\)\\mathrm\{Var\}\(G\-V^\{\\pi\}\(s\_\{0\}\)\)\\leq\\mathrm\{Var\}\(G\)\) that is explained by the initial state\. A sampled trajectory, however, contains more information than just the initial state \(e\.g\., intermediate actions\)\.[Pan and Schölkopf \(2024\)](https://arxiv.org/html/2609.36058#bib.bib6)showed that the return can be decomposed into a sum of advantages along the trajectory, and the advantages capture the variance caused by the intermediate actions\. ABC incorporates this insight into policy gradients to reduce their variance\.

### 3\.1Advantage\-Based Control Variate

Sampledtrajectoryτ\\tau\(s0,a0\)\(s\_\{0\},a\_\{0\}\)↓\\downarrow\(s1,a1\)\(s\_\{1\},a\_\{1\}\)↓\\downarrow⋮\\vdotsReturnG=∑t=0∞γt​rtG=\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}REINFORCE\(no control variate\)g^=G​∇θ​log⁡πθ​\(a0\|s0\)\\displaystyle\\hat\{g\}=G\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)Value baseline\(state\-level control variate\)g^V=\(G−V^​\(s0\)\)​∇θ​log⁡πθ​\(a0\|s0\)\\displaystyle\\hat\{g\}\_\{V\}=\\left\(G\-\\hat\{V\}\(s\_\{0\}\)\\right\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)ABC\(trajectory\-level control variate\)g^ABC=\(G−V^​\(s0\)−∑t=0∞γt​A^​\(st,at\)\)​∇θ​log⁡πθ​\(a0\|s0\)\\displaystyle\\hat\{g\}\_\{\\mathrm\{ABC\}\}=\\left\(G\-\\hat\{V\}\(s\_\{0\}\)\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\\right\)\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\+∑aA^\(s0,a\)∇θπθ\(a\|s0\)\\displaystyle\+\\sum\_\{a\}\\hat\{A\}\(s\_\{0\},a\)\\,\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(a\|s\_\{0\}\)Figure 1:Comparison of unbiased policy\-gradient estimators\. REINFORCE uses the sampled return directly, while a value baseline removes state\-dependent variation\. ABC additionally uses estimated advantages as a trajectory\-level control variate and adds an analytic correction term to preserve unbiasedness\.We begin with the centering property of the advantage function:

∑aπ⁡\(a\|s\)​Aπ​\(s,a\)=∑aπ⁡\(a\|s\)​Qπ​\(s,a\)−Vπ​\(s\)=0\.\\sum\_\{a\}\\pi\(a\|s\)A^\{\\pi\}\(s,a\)=\\sum\_\{a\}\\pi\(a\|s\)Q^\{\\pi\}\(s,a\)\-V^\{\\pi\}\(s\)=0\.\(6\)This centering property has an interesting implication on the policy gradient, namely:

###### Lemma 1\.

IfA^\\hat\{A\}satisfies the centering property with respect toπθ\\pi\_\{\\theta\}, then we have, fort′≠tt^\{\\prime\}\\neq t,

𝔼πθ\[A^\(st′,at′\)∇logπθ\(at\|st\)\]=0,\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\right\]=0,\(7\)where the expectation is with respect to the policyπθ\\pi\_\{\\theta\}\.

See Appendix[B](https://arxiv.org/html/2609.36058#A2)for a proof\. This shows that if an advantage estimate is centered, then it does not bias the gradient from different time\-steps\. Intuitively, this is because the advantages only capture the effects from the corresponding actions\. Now, we can combine this with an estimated value function to arrive at the following theorem\.

###### Theorem 2\(Advantage\-Based Control Variate for Policy Gradient\)\.

LetV^:𝒮→ℝ\\hat\{V\}:\{\\mathcal\{S\}\}\\rightarrow\\mathbb\{R\}, andA^:𝒮×𝒜→ℝ\\hat\{A\}:\{\\mathcal\{S\}\}\\times\{\\mathcal\{A\}\}\\rightarrow\\mathbb\{R\}satisfy the centering property\. Define the following gradient estimator:

g^ABC\(A^,V^\)=\(G−∑t=0∞γtA^\(st,at\)−V^\(s0\)\)∇logπθ\(a0\|s0\)\+∑aA^\(s0,a\)∇πθ\(a\|s0\),\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(\\hat\{A\},\\hat\{V\}\)=\\left\(G\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\-\\hat\{V\}\(s\_\{0\}\)\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\+\\sum\_\{a\}\\hat\{A\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\),\(8\)whereGGis the discounted return along\(s0,a0,s1,a1,…\)\(s\_\{0\},a\_\{0\},s\_\{1\},a\_\{1\},\\dots\)\. First of all,g^ABC\\hat\{g\}\_\{\\mathrm\{ABC\}\}is an unbiased estimator of the policy gradient\. Secondly, defineΔ​V=V^−Vπ\\Delta V=\\hat\{V\}\-V^\{\\pi\},Δ​A=A^−Aπ\\Delta A=\\hat\{A\}\-A^\{\\pi\},

D=−\(ΔV\(s0\)\+∑t=0∞γtΔA\(st,at\)\)∇logπθ\(a0\|s0\)\+∑aΔA\(s0,a\)∇πθ\(a\|s0\),D=\-\\left\(\\Delta V\(s\_\{0\}\)\+\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\Delta A\(s\_\{t\},a\_\{t\}\)\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\+\\sum\_\{a\}\\Delta A\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\),\(9\)and the covariance matrixΣ⁡\(A^,V^\)=Cov⁡\(g^ABC​\(A^,V^\)\)\\Sigma\(\\hat\{A\},\\hat\{V\}\)=\\mathrm\{Cov\}\(\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(\\hat\{A\},\\hat\{V\}\)\), we have

Σ⁡\(A^,V^\)=Σ⁡\(Aπ,Vπ\)\+𝔼⁡\[D​DT\]\.\\Sigma\(\\hat\{A\},\\hat\{V\}\)=\\Sigma\(A^\{\\pi\},V^\{\\pi\}\)\+\\mathbb\{E\}\\left\[DD^\{T\}\\right\]\.\(10\)Consequently,\(A^,V^\)=\(Aπ,Vπ\)\(\\hat\{A\},\\hat\{V\}\)=\(A^\{\\pi\},V^\{\\pi\}\)minimizes the covariance ofg^ABC\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(Loewner order\)\.

See Appendix[B](https://arxiv.org/html/2609.36058#A2)for a proof\. We note that the value\-baseline estimator is recovered as a special case of ABC by settingA^≡0\\hat\{A\}\\equiv 0\. Finally, Theorem[2](https://arxiv.org/html/2609.36058#Thmtheorem2)shows that using the exact functions\(Aπ,Vπ\)\(A^\{\\pi\},V^\{\\pi\}\)yields covariance no worse than any value\-baseline, including REINFORCE without a baseline\.

##### Relationship to Rao\-Blackwellization

Readers familiar with the skill\-luck decomposition\([Pan and Schölkopf, 2024](https://arxiv.org/html/2609.36058#bib.bib6)\)may notice that the weighting of the score functionG−∑t=0∞γt​Aπ​\(st,at\)−Vπ​\(s0\)G\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}A^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\-V^\{\\pi\}\(s\_\{0\}\)is precisely the cumulative effect from the environment stochasticity,∑t=0∞γt​Bπ​\(st,at,st\+1\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\), whereBπ​\(st,at,st\+1\)=rt\+γ​Vπ​\(st\+1\)−Qπ​\(st,at\)B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)=r\_\{t\}\+\\gamma V^\{\\pi\}\(s\_\{t\+1\}\)\-Q^\{\\pi\}\(s\_\{t\},a\_\{t\}\)captures how the stochastic transition from\(st,at\)\(s\_\{t\},a\_\{t\}\)tost\+1s\_\{t\+1\}affects the return\. As such, the estimator can be seen as separating environment noise \(first term\) from action effects \(second term, integrated over the action space\)\. Now, if we consider deterministic environments, whereBπ≡0B^\{\\pi\}\\equiv 0, such that the following equality holdsG=Vπ​\(s0\)\+∑t=0∞γt​Aπ​\(st,at\)G=V^\{\\pi\}\(s\_\{0\}\)\+\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}A^\{\\pi\}\(s\_\{t\},a\_\{t\}\), then the first term \(score function\) of the ABC estimator vanishes and the estimator reduces to

g^ABC\(Aπθ,Vπθ\)=∑aAπθ\(s0,a\)∇πθ\(a\|s0\)=𝔼\[Aπθ\(s0,a\)∇logπθ\(a\|s0\)∣s0\],\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(A^\{\\pi\_\{\\theta\}\},V^\{\\pi\_\{\\theta\}\}\)=\\sum\_\{a\}A^\{\\pi\_\{\\theta\}\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)=\\mathbb\{E\}\\left\[A^\{\\pi\_\{\\theta\}\}\(s\_\{0\},a\)\\nabla\\log\\pi\_\{\\theta\}\(a\|s\_\{0\}\)\\mid s\_\{0\}\\right\],which is precisely the policy gradient theorem conditioned on the state\([Sutton et al\., 1999](https://arxiv.org/html/2609.36058#bib.bib14)\)\. As such, the ABC estimator \(givenAπA^\{\\pi\}andVπV^\{\\pi\}\) can also be understood as Rao\-Blackwellizing the policy gradient estimatorAπθ\(s,a\)∇logππθ\(a\|s\)A^\{\\pi\_\{\\theta\}\}\(s,a\)\\nabla\\log\\pi\_\{\\pi\_\{\\theta\}\}\(a\|s\)\([Blackwell, 1947](https://arxiv.org/html/2609.36058#bib.bib17)\)\.

Finally, we note that ABC is algebraically equivalent to TrajCV\([Cheng et al\., 2020](https://arxiv.org/html/2609.36058#bib.bib38)\)under the reparameterizationQ^​\(s,a\)=A^​\(s,a\)\+V^​\(s\)\\hat\{Q\}\(s,a\)=\\hat\{A\}\(s,a\)\+\\hat\{V\}\(s\), where the centering constraint implies𝔼a∼π\(⋅∣s\)\[Q^\(s,a\)\]=V^\(s\)\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\\mid s\)\}\[\\hat\{Q\}\(s,a\)\]=\\hat\{V\}\(s\)\. However, our advantage\-value formulation makes the multi\-step return prediction from\(A^,V^\)\(\\hat\{A\},\\hat\{V\}\)explicit, motivating the connection to DAE developed next\.

### 3\.2Learning Control Variates

In practice, we often only have estimates of\(Aπ,Vπ\)\(A^\{\\pi\},V^\{\\pi\}\), and it is not immediately clear how much we suffer from the approximations\. Here, we show that this can be bounded by the error we make in predicting the returnGG\.

###### Corollary 3\.

For deterministic MDPs, if\(A^,V^\)\(\\hat\{A\},\\hat\{V\}\)satisfies\|G−∑t=0∞γt​A^​\(st,at\)−V^​\(s0\)\|<ϵ\|G\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\-\\hat\{V\}\(s\_\{0\}\)\|<\\epsilon, then

Σ⁡\(A^,V^\)−Σ⁡\(Aπ,Vπ\)≤ϵ2​F,\\Sigma\(\\hat\{A\},\\hat\{V\}\)\-\\Sigma\(A^\{\\pi\},V^\{\\pi\}\)\\leq\\epsilon^\{2\}F,whereF=𝔼\[∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]F=\\mathbb\{E\}\[\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]is the Fisher information matrix, and≤\\leqfollows the Loewner order\.

See Appendix[B](https://arxiv.org/html/2609.36058#A2)for a proof\. This suggests that, to reduce the variance of policy gradients, we should try to minimize the error\|G−∑t=0∞γt​A^​\(st,at\)−V^​\(s0\)\|\|G\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\-\\hat\{V\}\(s\_\{0\}\)\|\. Notably, the error appearing in this bound is similar to the quantity minimized by the DAE objective \(cf\. Equation[4](https://arxiv.org/html/2609.36058#S2.E4)\)\. This motivates our practical actor\-critic algorithm, which iteratively updates the actor with the ABC gradient estimate and the critic with DAE\. In the next section, we empirically verify the performance of this combination\.

## 4Experiments

In this section, we examine the effectiveness of combining the ABC estimator and DAE\([Pan et al\., 2022](https://arxiv.org/html/2609.36058#bib.bib5);[Pan and Schölkopf, 2024](https://arxiv.org/html/2609.36058#bib.bib6)\)in RLVR\. We consider an offline\-to\-online actor\-critic setting, where an offline dataset of rollouts is available\. This allows the critic LLM to learn useful representations and outputs before online training of the actor, which is particularly useful as value and advantage prediction differ substantially from next\-token prediction\. Figure[2](https://arxiv.org/html/2609.36058#S4.F2)summarizes the overall training pipeline\. Next, we detail the experimental design\.

Trajectory CollectionDeepMath promptsQwen3 rolloutsmath\-verifyOffline Critic TrainingDAE on offline trajectoriesLearnAϕA\_\{\\phi\}andVϕV\_\{\\phi\}Online Actor–Critic TrainingOn\-policy rolloutsABC actor updatesDAE critic updatesOffline PhaseOnline PhaseFigure 2:Overview of the offline\-to\-online training pipeline\. We first collect trajectories with verifiable rewards and use DAE to pretrain the critic\. During online training, trajectories sampled from the current actor are used both for ABC policy\-gradient updates and continued DAE critic training\.##### Model Class

We consider the Qwen3 family of models, which includes models of various sizes that share the same tokenizer and hence the same action space\. This allows us to train asymmetric actor\-critic pairs and control for the sizes of the critic independently\. For the following experiments, both actors and critics are fine\-tuned from base models using LoRA\([Hu et al\., 2021](https://arxiv.org/html/2609.36058#bib.bib19)\)\. We test our method on actors of sizes \{0\.6/1\.7/4\}B, while fixing the size of the critic to 8B\.

##### Math Dataset

We use the DeepMath\-103k dataset\([He et al\., 2026](https://arxiv.org/html/2609.36058#bib.bib18)\), a large dataset of challenging math question\-answer pairs spanning a wide range of topics\. This dataset is used for both offline dataset generation and for online training\. For evaluation, we consider the standard benchmark datasets MATH500\([Lightman et al\., 2023](https://arxiv.org/html/2609.36058#bib.bib31)\), AMC23\([He, 2024](https://arxiv.org/html/2609.36058#bib.bib30)\), AIME25\([Zhang and Math\-AI Team, 2025](https://arxiv.org/html/2609.36058#bib.bib32)\), and AIME26\([Zhang and Math\-AI Team, 2026](https://arxiv.org/html/2609.36058#bib.bib33)\)\. We usemath\-verifyfor answer extraction and verification111[https://github\.com/huggingface/math\-verify](https://github.com/huggingface/math-verify)\. Finally, we do not use additional templates for prompting, following the approach by[Liu et al\. \(2025\)](https://arxiv.org/html/2609.36058#bib.bib15)\.

##### Offline Critic Training

The offline dataset consists of trajectories generated by the Qwen3\-\{0\.6/1\.7/4/8/14\}B base models\. For each question in DeepMath\-103k dataset, we collect 16 rollouts from each model\. This results in a dataset of 103k \(questions\)×\\times16 \(rollouts\)×\\times5 \(base models\)≈\\approx8\.2 million trajectories\. We then use the DAE objective \(Equation[4](https://arxiv.org/html/2609.36058#S2.E4)\) to fine\-tune the critic model\. For fine\-tuning, we use LoRA\([Hu et al\., 2021](https://arxiv.org/html/2609.36058#bib.bib19)\)for intermediate layers, and add separate value and advantage heads on top of the base model to parametrize the advantage and value functions, similar in spirit to the dueling architecture\([Wang et al\., 2016](https://arxiv.org/html/2609.36058#bib.bib20)\)\.

##### Online Actor\-Critic Training

Algorithm[1](https://arxiv.org/html/2609.36058#alg1)summarizes the online actor\-critic training\. In contrast to many modern policy optimization methods, we strive to keep the algorithm simple and*do not*use clipping, importance sampling \(actor update uses fully on\-policy data\) or additional regularizations \(e\.g\., KL or entropy regularization\)\. At each iteration, the actor is updated with exactly one gradient step, avoiding the need for off\-policy corrections, while the critic is updated forKKsteps\. We train the actor for exactly 100 gradient steps \(8,192 trajectories per step,∼0\.8​M\{\\sim\}0\.8Mtrajectories in total\)\.

It should be noted that there is a lag between the critic and the actor: the critic learns from trajectories collected before the actor update, but the target policy is, in fact, the updated actor\. This is mitigated by the off\-policy nature of DAE, which can estimate the value and advantage functions of a target policy using data collected by a different behavior policy without additional corrections\.

Algorithm 1Online Actor\-Critic Training with ABC and DAE1:Problem dataset

𝒟\\mathcal\{D\}, Actor

πθ\\pi\_\{\\theta\}, Critic

\(Aϕ,Vϕ\)\(A\_\{\\phi\},V\_\{\\phi\}\), Batch size

NN, Num\. critic update

KK
2:whilenot convergeddo

3:Sample a batch of prompts

\{xi\}i=1,…,N∼𝒟\\\{x\_\{i\}\\\}\_\{i=1,\\dots,N\}\\sim\{\\mathcal\{D\}\}
4:Sample trajectories

\{τi\}i=1,…,N\\\{\\tau\_\{i\}\\\}\_\{i=1,\\dots,N\}for each prompt using

πθ\\pi\_\{\\theta\}
5:Compute

\(Aϕ,Vϕ\)\(A\_\{\\phi\},V\_\{\\phi\}\)for all states\(\-actions\) in

\{τi\}i=1,…,N\\\{\\tau\_\{i\}\\\}\_\{i=1,\\dots,N\}\(centered by

πθ\\pi\_\{\\theta\}\)

6:Compute the ABC policy\-gradient estimator \(

\|τ\|\|\\tau\|denotes the trajectory length\)

g^=1N​∑i=1N∑t=0\|τi\|−1g^ABC​\(Aϕ,Vϕ\)​\(st,at\)\\hat\{g\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\sum\_\{t=0\}^\{\|\\tau\_\{i\}\|\-1\}\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(A\_\{\\phi\},V\_\{\\phi\}\)\(s\_\{t\},a\_\{t\}\)
7:Update the actor parameters

θ\\thetawith gradient

g^\\hat\{g\}
8:for

k=1,…,Kk=1,\\ldots,Kdo

9:Sample a minibatch

ℬ∼\{τi\}i=1,…,N\\mathcal\{B\}\\sim\\\{\\tau\_\{i\}\\\}\_\{i=1,\\dots,N\}
10:Optimize the DAE critic loss

ℒDAE​\(ϕ,ℬ\)\\mathcal\{L\}\_\{\\mathrm\{DAE\}\}\(\\phi;\\mathcal\{B\}\)with respect to policy

πθ\\pi\_\{\\theta\}
11:endfor

12:endwhile

### 4\.1Results

##### Offline Advantage & Value Learning

We first consider the offline training phase\. To evaluate the quality of the estimated advantage and value functions, we split the offline dataset into training and validation sets \(10000 trajectories for validation\)\. We note that this phase is simply supervised learning with a mean squared error \(MSE\) objective \(Equation[4](https://arxiv.org/html/2609.36058#S2.E4)\)\. To further assess the quality of the learned advantage, we compare the MSE with or without the advantages\. Ideally, adding the advantages should allow the model to better approximate the return and reduce the MSE\. Figure[4](https://arxiv.org/html/2609.36058#S4.F4)shows the offline training loss curves\. Including the advantage term substantially reduces held\-out MSE, indicating that the learned advantages capture the part of the return that is not explained by the value function alone\. Interestingly, we see the gap between the MSE \(with or without the advantages\) decreases with actor size\. This is likely due to larger actors having lower entropy policies \(note that the gap reflects how much the variance can be explained by the actions, and the more deterministic the policy is, the less the variance can be explained by the actions\), and we see a similar effect over the course of online training \(see Figure[11](https://arxiv.org/html/2609.36058#A3.F11)and Appendix[C](https://arxiv.org/html/2609.36058#A3)for additional discussion\)\.

Figure 3:Offline critic training loss\.Figure 4:Variance ratio between estimators\. REINFORCE=1=1\.

##### Variance Reduction

We now compare the variance of the policy gradients between different methods\. We estimate the first gradient step variance by computing per\-sample policy gradient over 4096 sample trajectories with respect to each method \(REINFORCE, Value baseline, and ABC\), and consider the sum of the variances across parameters \(i\.e\., trace of the estimated covariance\)\. Figure[4](https://arxiv.org/html/2609.36058#S4.F4)shows the ratio of the sum of variances across estimators\. Across model sizes, we see that using value baselines already reduces the variance by approximately30%30\\%\. We also see that, aside from the 1\.7B model, adding advantages \(ABC\) further reduces the variance compared to the value baseline\. For the 1\.7B model, we see a few outlier states with extreme critic prediction error, which likely explains why the variance is not reduced \(see Appendix[C](https://arxiv.org/html/2609.36058#A3)for additional discussion\)\.

Figure 5:ABC training curves\.
##### Policy Optimization Performance

To demonstrate the effectiveness of our method, we compare with Dr\. GRPO\([Liu et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib15)\), an extension of the original GRPO\([Shao et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib3)\)that fixes the bias from advantage standardization and length normalization\. The Dr\. GRPO baseline is trained with 1 epoch over the training set, corresponding to approximately 6000 gradient steps, or 1\.6M generated trajectories \(see Appendix[C](https://arxiv.org/html/2609.36058#A3)for hyperparameters\)\. Table[1](https://arxiv.org/html/2609.36058#S4.T1)shows that our method \(ABC\) achieves performance competitive with the Dr\. GRPO baseline on the evaluation datasets\. Figure[5](https://arxiv.org/html/2609.36058#S4.F5)shows the learning curves of ABC, and we find that ABC achieves similar training performance to Dr\. GRPO using only about 50 gradients \(each actor update corresponds to exactly one policy gradient\)\. In addition, we observe similar increases in completion length under ABC despite using an unbiased policy\-gradient estimator, suggesting that this phenomenon is not solely due to biases specific to GRPO\([Liu et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib15)\)\. Finally, we find that the variance reduction is crucial, as REINFORCE is not able to achieve similar performance as Dr\. GRPO \(see Figure[9](https://arxiv.org/html/2609.36058#A3.F9)\)\.

Table 1:Final\-model evaluation scores in percent, reported as \(mean/pass\)@32\. Mean is the average reward over 32 sampled completions per problem; pass is the percentage of problems with at least one correct completion\.Figure 6:Comparison with biased gradients\.
##### Comparison with Biased Estimator

As DAE learns an approximation of the advantage function, one may wonder if we can use this directly in the policy gradient viaA^\(s,a\)∇logπθ\(a\|s\)\\hat\{A\}\(s,a\)\\nabla\\log\\pi\_\{\\theta\}\(a\|s\)\. Unlike the unbiased ABC estimator, the performance of this approach depends heavily on the critic, as the actor update depends on the environment reward only through the learned critic\. Figure[6](https://arxiv.org/html/2609.36058#S4.F6)compares the training score between ABC and this biased approach\. We see that at initialization, the biased approach actually outperforms ABC, which suggests that the offline training is, in fact, effective in learning the advantage function\. However, as learning proceeds, the biased approach slows down while ABC steadily improves\. This suggests that critic bias can limit performance even when the initial advantage estimates are informative\.

## 5Related Work

##### RLVR

RLVR has recently emerged as an effective approach for improving the reasoning capabilities of LLMs, particularly for domains where rewards can be verified automatically\. A prominent family of methods builds on Group Relative Policy Optimization \(GRPO\)\([Shao et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib3)\), which removes the learned critic used in PPO and instead estimates advantages by comparing multiple rollouts generated from the same prompt\. GRPO has been used extensively in large\-scale reasoning models such as DeepSeek\-R1\([Guo et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib4)\), and has motivated a number of subsequent methods aimed at improving its stability and efficiency\([Yu et al\., 2026](https://arxiv.org/html/2609.36058#bib.bib12);[Liu et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib15);[Piché et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib22);[Yue et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib23);[Li et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib24)\)\. In contrast to critic\-free approaches, our method revisits the use of learned advantages, while avoiding the need to estimate them solely from online MC rollouts\.

##### Advantage Estimation and Variance Reduction

Reducing the variance of policy\-gradient estimators has been studied extensively, beginning with state\-dependent baselines and actor\-critic methods\([Williams, 1992](https://arxiv.org/html/2609.36058#bib.bib13);[Sutton et al\., 1999](https://arxiv.org/html/2609.36058#bib.bib14)\), and later through methods such as Generalized Advantage Estimation \(GAE\)\([Schulman et al\., 2015](https://arxiv.org/html/2609.36058#bib.bib2)\)\. More general control\-variate constructions have also been proposed\. For example, Q\-Prop uses an off\-policy critic as a control variate for policy gradients\([Gu et al\., 2016](https://arxiv.org/html/2609.36058#bib.bib25)\), while[Liu et al\. \(2017\)](https://arxiv.org/html/2609.36058#bib.bib26)propose action\-dependent control variates based on Stein’s identity\. Trajectory\-level information has also been used for variance reduction; for example,[Mesnard et al\. \(2020\)](https://arxiv.org/html/2609.36058#bib.bib27)introduce future\-conditional value functions that exploit future information while preserving unbiasedness\. TrajCV\([Cheng et al\., 2020](https://arxiv.org/html/2609.36058#bib.bib38)\)similarly studied trajectory\-level control variates using estimatedQQ\-values, and ABC can be viewed as an advantage\-value reparameterization, that naturally connects to DAE\. Direct Advantage Estimation \(DAE\) has been shown to improve policy optimization\([Pan et al\., 2022](https://arxiv.org/html/2609.36058#bib.bib5);[Pan and Schölkopf, 2024](https://arxiv.org/html/2609.36058#bib.bib6);[Pan and Schölkopf, 2026a](https://arxiv.org/html/2609.36058#bib.bib28)\)and reduce the variance of policy evaluation\([Pan and Schölkopf, 2026b](https://arxiv.org/html/2609.36058#bib.bib29)\)\. In contrast to prior works on DAE, we study how estimates of the advantage function can be used as control variates to reduce policy\-gradient variance while preserving an unbiased gradient estimator, which is particularly relevant in light of the strong empirical performance of critic\-free RLVR methods\.

##### On\-policy Distillation

Recent on\-policy distillation \(OPD\) methods also exploit dense teacher feedback on tokens visited by the current policy\([Agarwal et al\., 2024](https://arxiv.org/html/2609.36058#bib.bib37)\)\. In full\-vocabulary OPD, objectives such as reverse KL take the form∑aw\(s,a\)∇πθ\(a\|s\)\\sum\_\{a\}w\(s,a\)\\nabla\\pi\_\{\\theta\}\(a\|s\), which closely resembles the analytic advantage term in ABC,∑aA^​\(s,a\)​∇θπθ​\(a\|s\)\\sum\_\{a\}\\hat\{A\}\(s,a\)\\nabla\_\{\\theta\}\\pi\_\{\\theta\}\(a\|s\)\. The two methods differ mainly in the goal of the objectives: OPD uses teacher log\-probabilities to move the student toward the teacher policy, whereas ABC uses estimated advantages as a control variate and combines the analytic term with the residual term to preserve an unbiased gradient of the original RL objective\.

## 6Discussion

Recent RLVR methods have shown that critic\-free approaches can perform strongly despite potentially higher variance, raising the possibility that bias introduced through approximate advantage estimation from the critic can be more harmful than the variance they are intended to reduce\. The ABC estimator provides a natural way to integrate estimated advantages and values from DAE into the policy gradient estimator without introducing bias\. Furthermore, our theoretical analysis demonstrates that the variance of the ABC estimator is closely related to the objective function of DAE, which motivates a natural combination of both algorithms\.

Empirically, our offline\-to\-online RL results suggest that this combination can make critic\-based optimization competitive with modern critic\-free RLVR methods\. This may be especially attractive when large amounts of rollout data already exist, since critic learning can partially be moved to the offline phase rather than starting from scratch during online RL\. Furthermore, critic training can in principle reuse off\-policy data and proceed independently of rollout collection\. Combined with the fact that ABC does not rely on group\-based advantage estimates, this decoupling may make the approach more amenable to asynchronous RL settings\([Piché et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib22);[Noukhovitch et al\., 2025](https://arxiv.org/html/2609.36058#bib.bib21)\)\.

##### Limitations

First, our empirical evaluation focuses on mathematical problems, and it remains to be seen how well ABC transfers to other RL domains and to settings with stochastic environment dynamics\. Second, the quality of the control variate depends on the learned critic\. Although approximation errors do not bias the ABC estimator, poor estimates may provide little or even negative variance reduction\. Third, our current experiments focus primarily on optimization performance rather than end\-to\-end computational efficiency\. Training and maintaining a critic incurs additional memory and computation costs, and a complete systems\-level comparison with critic\-free methods is an important direction for future work\. Finally, understanding how the distribution of the offline data affects the critic is another important direction for future study\.

#### Acknowledgments

HRP would like to thank Zeju Qiu, Le Chen, Luis Bauer, Simon Guist, Jan Schneider, and Max Mordig for \(virtually\) supporting this project\.

## References

- Agarwalet al\.\(2024\)R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. Ramos Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 21246–21263\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px3.p1.1)\.
- Blackwell \(1947\)D\. BlackwellConditional Expectation and Unbiased Sequential Estimation\.The Annals of Mathematical Statistics18\(1\),pp\. 105 – 110\.External Links:[Document](https://dx.doi.org/10.1214/aoms/1177730497),[Link](https://doi.org/10.1214/aoms/1177730497)Cited by:[§3\.1](https://arxiv.org/html/2609.36058#S3.SS1.SSS0.Px1.p1.2)\.
- Chenget al\.\(2020\)C\. Cheng, X\. Yan, and B\. BootsTrajectory\-wise control variates for variance reduction in policy gradient methods\.InConference on Robot Learning,pp\. 1379–1394\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p3.1),[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px3.p2.1),[§3\.1](https://arxiv.org/html/2609.36058#S3.SS1.SSS0.Px1.p2.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Daoet al\.\(2022\)T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. RéFlashattention: fast and memory\-efficient exact attention with io\-awareness\.Advances in neural information processing systems35,pp\. 16344–16359\.Cited by:[§C\.4](https://arxiv.org/html/2609.36058#A3.SS4.SSS0.Px1.p1.3)\.
- Greensmithet al\.\(2004\)E\. Greensmith, P\. L\. Bartlett, and J\. BaxterVariance reduction techniques for gradient estimates in reinforcement learning\.Journal of Machine Learning Research5\(Nov\),pp\. 1471–1530\.Cited by:[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px3.p1.1)\.
- Guet al\.\(2016\)S\. Gu, T\. Lillicrap, Z\. Ghahramani, R\. E\. Turner, and S\. LevineQ\-prop: sample\-efficient policy gradient with an off\-policy critic\.arXiv preprint arXiv:1611\.02247\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.DeepSeek\-r1 incentivizes reasoning in llms through reinforcement learning\.Nature645\(8081\),pp\. 633–638\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p1.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1)\.
- Heet al\.\(2026\)Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang,et al\.Deepmath\-103k: a large\-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning\.InInternational Conference on Learning Representations,Vol\.2026,pp\. 138306–138322\.Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px2.p1.1)\.
- He \(2024\)Z\. HeAMC23 dataset\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/zwhe99/amc23)Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px2.p1.1)\.
- Huet al\.\(2021\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLora: low\-rank adaptation of large language models\.arXiv preprint arXiv:2106\.09685\.Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px1.p1.1),[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px3.p1.1)\.
- Konda and Tsitsiklis \(1999\)V\. Konda and J\. TsitsiklisActor\-critic algorithms\.Advances in neural information processing systems12\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1)\.
- Kwonet al\.\(2023\)W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. StoicaEfficient memory management for large language model serving with pagedattention\.InProceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles,Cited by:[§C\.1](https://arxiv.org/html/2609.36058#A3.SS1.p1.1)\.
- Lambertet al\.\(2024\)N\. Lambert, J\. Morrison, V\. Pyatkin, S\. Huang, H\. Ivison, F\. Brahman, L\. J\. V\. Miranda, A\. Liu, N\. Dziri, S\. Lyu,et al\.Tulu 3: pushing frontiers in open language model post\-training\.arXiv preprint arXiv:2411\.15124\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p1.1)\.
- Liet al\.\(2025\)S\. Li, Z\. Zhou, W\. Lam, C\. Yang, and C\. LuRepo: replay\-enhanced policy optimization\.arXiv preprint arXiv:2506\.09340\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2017\)H\. Liu, Y\. Feng, Y\. Mao, D\. Zhou, J\. Peng, and Q\. LiuAction\-dependent control variates for policy optimization via stein’s identity\.arXiv preprint arXiv:1710\.11198\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§C\.3](https://arxiv.org/html/2609.36058#A3.SS3.SSS0.Px1.p1.1),[Table 2](https://arxiv.org/html/2609.36058#A3.T2.3.3.2.1),[§1](https://arxiv.org/html/2609.36058#S1.p3.1),[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px2.p3.2),[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.36058#S4.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1)\.
- Mesnardet al\.\(2020\)T\. Mesnard, T\. Weber, F\. Viola, S\. Thakoor, A\. Saade, A\. Harutyunyan, W\. Dabney, T\. Stepleton, N\. Heess, A\. Guez,et al\.Counterfactual credit assignment in model\-free reinforcement learning\.arXiv preprint arXiv:2011\.09464\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Mnihet al\.\(2016\)V\. Mnih, A\. P\. Badia, M\. Mirza, A\. Graves, T\. Lillicrap, T\. Harley, D\. Silver, and K\. KavukcuogluAsynchronous methods for deep reinforcement learning\.InInternational conference on machine learning,pp\. 1928–1937\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1),[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px3.p1.1)\.
- Noukhovitchet al\.\(2025\)M\. Noukhovitch, S\. Huang, S\. Xhonneux, A\. Hosseini, R\. Agarwal, and A\. CourvilleAsynchronous rlhf: faster and more efficient off\-policy rl for language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 4003–4029\.Cited by:[§6](https://arxiv.org/html/2609.36058#S6.p2.1)\.
- Panet al\.\(2022\)H\. Pan, N\. Gürtler, A\. Neitz, and B\. SchölkopfDirect advantage estimation\.Advances in Neural Information Processing Systems35,pp\. 11869–11880\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p3.1),[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2609.36058#S4.p1.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Pan and Schölkopf \(2024\)H\. Pan and B\. SchölkopfSkill or luck? return decomposition via advantage functions\.InThe Twelfth International Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2609.36058#A2.p5.2.1),[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px4.p1.1),[§3\.1](https://arxiv.org/html/2609.36058#S3.SS1.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2609.36058#S3.p2.1),[§4](https://arxiv.org/html/2609.36058#S4.p1.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Pan and Schölkopf \(2026a\)H\. Pan and B\. SchölkopfDirect advantage estimation for scalable and sample\-efficient deep reinforcement learning\.arXiv preprint arXiv:2606\.20411\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Pan and Schölkopf \(2026b\)H\. Pan and B\. SchölkopfOn the variance of temporal difference learning and its reduction using control variates\.arXiv preprint arXiv:2606\.20357\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Pichéet al\.\(2025\)A\. Piché, E\. Kamalloo, R\. Pardinas, X\. Chen, and D\. BahdanauPipelinerl: faster on\-policy reinforcement learning for long sequence generation\.arXiv preprint arXiv:2509\.19128\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2609.36058#S6.p2.1)\.
- Radosavovicet al\.\(2024\)I\. Radosavovic, T\. Xiao, B\. Zhang, T\. Darrell, J\. Malik, and K\. SreenathReal\-world humanoid locomotion with reinforcement learning\.Science Robotics9\(89\),pp\. eadi9579\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1)\.
- Rudinet al\.\(2022\)N\. Rudin, D\. Hoeller, P\. Reist, and M\. HutterLearning to walk in minutes using massively parallel deep reinforcement learning\.InConference on robot learning,pp\. 91–100\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1)\.
- Schulmanet al\.\(2015\)J\. Schulman, P\. Moritz, S\. Levine, M\. Jordan, and P\. AbbeelHigh\-dimensional continuous control using generalized advantage estimation\.arXiv preprint arXiv:1506\.02438\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1),[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px2.p3.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p1.1),[§1](https://arxiv.org/html/2609.36058#S1.p2.1),[§4\.1](https://arxiv.org/html/2609.36058#S4.SS1.SSS0.Px3.p1.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. MansourPolicy gradient methods for reinforcement learning with function approximation\.Advances in neural information processing systems12\.Cited by:[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px2.p1.2),[§3\.1](https://arxiv.org/html/2609.36058#S3.SS1.SSS0.Px1.p1.2),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- von Werraet al\.\(2020\)TRL: Transformers Reinforcement LearningExternal Links:[Link](https://github.com/huggingface/trl)Cited by:[§C\.3](https://arxiv.org/html/2609.36058#A3.SS3.SSS0.Px1.p1.1)\.
- Wanget al\.\(2016\)Z\. Wang, T\. Schaul, M\. Hessel, H\. Hasselt, M\. Lanctot, and N\. FreitasDueling network architectures for deep reinforcement learning\.InInternational conference on machine learning,pp\. 1995–2003\.Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px3.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine learning8\(3\),pp\. 229–256\.Cited by:[§2](https://arxiv.org/html/2609.36058#S2.SS0.SSS0.Px2.p1.2),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2026\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.Advances in Neural Information Processing Systems38,pp\. 113222–113244\.Cited by:[§1](https://arxiv.org/html/2609.36058#S1.p2.1),[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, Y\. Yuan, Q\. Yu, X\. Zuo, R\. Zhu, W\. Xu, J\. Chen, C\. Wang, T\. Fan, Z\. Du,et al\.Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks\.arXiv preprint arXiv:2504\.05118\.Cited by:[§5](https://arxiv.org/html/2609.36058#S5.SS0.SSS0.Px1.p1.1)\.
- Zhang and Math\-AI Team \(2025\)Y\. Zhang and Math\-AI TeamAmerican invitational mathematics examination \(AIME\) 2025\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime25)Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px2.p1.1)\.
- Zhang and Math\-AI Team \(2026\)Y\. Zhang and Math\-AI TeamAmerican invitational mathematics examination \(AIME\) 2026\.Hugging Face\.External Links:[Link](https://huggingface.co/datasets/math-ai/aime26)Cited by:[§4](https://arxiv.org/html/2609.36058#S4.SS0.SSS0.Px2.p1.1)\.

## Appendix ABaseline Counterexample

Here, we give a concrete example showing that usingVπV^\{\\pi\}as the baseline function may increase the variance of policy gradients\.

Consider a two\-armed bandit problem with action space \(𝒜=\{0,1\}\\mathcal\{A\}=\\\{0,1\\\}\) and rewardsr⁡\(0\)=0r\(0\)=0,r⁡\(1\)=1r\(1\)=1\. We parameterize the policy as

πθ​\(a\)=\{11\+eθ,a=0,eθ1\+eθ,a=1\.\\pi\_\{\\theta\}\(a\)=\\begin\{cases\}\\frac\{1\}\{1\+e^\{\\theta\}\},\\qquad a=0,\\\\ \\frac\{e^\{\\theta\}\}\{1\+e^\{\\theta\}\},\\qquad a=1\.\\end\{cases\}\(11\)
The score function is then

∇θ​log​πθ​\(a\)=a−πθ​\(1\),\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\)=a\-\\pi\_\{\\theta\}\(1\),\(12\)and the exact value function is

Vπθ=πθ​\(1\)\.V^\{\\pi\_\{\\theta\}\}=\\pi\_\{\\theta\}\(1\)\.\(13\)Denoteπθ​\(1\)\\pi\_\{\\theta\}\(1\)bypp, then the REINFORCE estimator is equal to

g=\(a−p\)​r​\(a\),g=\(a\-p\)r\(a\),\(14\)and

𝔼⁡\[g\]=p⁡\(1−p\),Var⁡\(g\)=p​\(1−p\)3\.\\mathbb\{E\}\[g\]=p\(1\-p\),\\quad\\mathrm\{Var\}\(g\)=p\(1\-p\)^\{3\}\.\(15\)Now, the baseline estimator with exact value function is equal to

gb=\(a−p\)​\(r⁡\(a\)−p\)=\(a−p\)2,g\_\{b\}=\(a\-p\)\\left\(r\(a\)\-p\\right\)=\(a\-p\)^\{2\},\(16\)and

Var⁡\(gb\)=p⁡\(1−p\)​\(1−2​p\)2\.\\mathrm\{Var\}\(g\_\{b\}\)=p\(1\-p\)\(1\-2p\)^\{2\}\.\(17\)As such, when\(1−2​p\)2\>\(1−p\)2\(1\-2p\)^\{2\}\>\(1\-p\)^\{2\}\(e\.g\.,p\>23p\>\\frac\{2\}\{3\}\), we haveVar⁡\(gb\)\>Var⁡\(g\)\\mathrm\{Var\}\(g\_\{b\}\)\>\\mathrm\{Var\}\(g\)\.

## Appendix BProofs

See[1](https://arxiv.org/html/2609.36058#Thmtheorem1)

###### Proof\.

Ift′\>tt^\{\\prime\}\>t, then

𝔼\[A^\(st′,at′\)∇logπθ\(at\|st\)\]\\displaystyle\\mathbb\{E\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\right\]=𝔼\[𝔼\[A^\(st′,at′\)∇logπθ\(at\|st\)∣st,at\]\]\\displaystyle=\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\mid s\_\{t\},a\_\{t\}\\right\]\\right\]=𝔼\[𝔼\[A^\(st′,at′\)∣st,at\]∇logπθ\(at\|st\)\]=0,\\displaystyle=\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\mid s\_\{t\},a\_\{t\}\\right\]\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\right\]=0,where the last equation follows from the centering property\. Ift′<tt^\{\\prime\}<t, then

𝔼\[A^\(st′,at′\)∇logπθ\(at\|st\)\]\\displaystyle\\mathbb\{E\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\right\]=𝔼\[𝔼\[A^\(st′,at′\)∇logπθ\(at\|st\)∣st′,at′\]\]\\displaystyle=\\mathbb\{E\}\\left\[\\mathbb\{E\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\mid s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\]\\right\]=𝔼\[A^\(st′,at′\)𝔼\[∇logπθ\(at\|st\)∣st′,at′\]\]=0,\\displaystyle=\\mathbb\{E\}\\left\[\\hat\{A\}\\left\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\)\\mathbb\{E\}\\left\[\\nabla\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\)\\mid s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\\right\]\\right\]=0,∎

See[2](https://arxiv.org/html/2609.36058#Thmtheorem2)

###### Proof\.

We begin by proving that the estimator is unbiased\.

𝔼​\[g^ABC​\(A^,V^\)\]=\\displaystyle\\mathbb\{E\}\[\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(\\hat\{A\},\\hat\{V\}\)\]=𝔼\[G∇logπθ\(a0\|s0\)\]−∑t=0∞γt𝔼\[A^\(st,at\)∇logπθ\(a0\|s0\)\]\\displaystyle\\mathbb\{E\}\[G\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\mathbb\{E\}\[\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]\(18\)−𝔼\[V^\(s0\)∇logπθ\(a0\|s0\)\]\+𝔼\[∑aA^\(s0,a\)∇πθ\(a\|s0\)\]\\displaystyle\-\\mathbb\{E\}\[\\hat\{V\}\(s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]\+\\mathbb\{E\}\[\\sum\_\{a\}\\hat\{A\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)\]\(19\)By Lemma[1](https://arxiv.org/html/2609.36058#Thmtheorem1),𝔼\[A^\(st,at\)∇logπθ\(a0\|s0\)\]=0\\mathbb\{E\}\[\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]=0fort\>0t\>0\. Next, use the log derivative trick, we have

𝔼\[∑aA^\(s0,a\)∇πθ\(a\|s0\)\]=𝔼\[A^\(s0,a0\)∇logπθ\(a0\|s0\)\]\\mathbb\{E\}\[\\sum\_\{a\}\\hat\{A\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)\]=\\mathbb\{E\}\[\\hat\{A\}\(s\_\{0\},a\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]Consequently,

𝔼​\[g^ABC​\(A^,V^\)\]=\\displaystyle\\mathbb\{E\}\[\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(\\hat\{A\},\\hat\{V\}\)\]=𝔼\[G∇logπθ\(a0\|s0\)\]\.\\displaystyle\\mathbb\{E\}\[G\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\]\.\(20\)This concludes the first part of the proof\. Next, we consider the covariance of the estimator\. To simplify the notation, we first denoteg^ABCπ=g^ABC​\(Aπ,Vπ\)\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\}=\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(A^\{\\pi\},V^\{\\pi\}\)\. It follows that

g^ABC​\(A^,V^\)=g^ABCπ\+D\.\\hat\{g\}\_\{\\mathrm\{ABC\}\}\(\\hat\{A\},\\hat\{V\}\)=\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\}\+D\.\(21\)The covariance follows

Σ⁡\(A^,V^\)=Σ⁡\(Aπ,Vπ\)\+𝔼⁡\[D​DT\]\+Cov⁡\(g^ABCπ,D\)\+Cov⁡\(D,g^ABCπ\),\\Sigma\(\\hat\{A\},\\hat\{V\}\)=\\Sigma\(A^\{\\pi\},V^\{\\pi\}\)\+\\mathbb\{E\}\[DD^\{T\}\]\+\\mathrm\{Cov\}\(\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\},D\)\+\\mathrm\{Cov\}\(D,\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\}\),\(22\)whereCov⁡\(D\)=𝔼⁡\[D​DT\]\\mathrm\{Cov\}\(D\)=\\mathbb\{E\}\[DD^\{T\}\]as𝔼⁡\[D\]=0\\mathbb\{E\}\[D\]=0\(unbiasedness\)\. Consequently, we only need to show that

Cov⁡\(g^ABCπ,D\)\+Cov⁡\(D,g^ABCπ\)=0\\mathrm\{Cov\}\(\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\},D\)\+\\mathrm\{Cov\}\(D,\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\}\)=0\(23\)
We first decompose the return of a trajectory into:

G=Vπ​\(s0\)\+∑t=0∞γt​\(Aπ​\(st,at\)\+Bπ​\(st,at,st\+1\)\),G=V^\{\\pi\}\(s\_\{0\}\)\+\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\left\(A^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\+B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\right\),\(24\)whereBπ​\(st,at,st\+1\)=rt\+γ​Vπ​\(st\+1\)−Qπ​\(st,at\)B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)=r\_\{t\}\+\\gamma V^\{\\pi\}\(s\_\{t\+1\}\)\-Q^\{\\pi\}\(s\_\{t\},a\_\{t\}\)\[[22](https://arxiv.org/html/2609.36058#bib.bib6)\]\. Note thatBπB^\{\\pi\}satisfies𝔼\[Bπ\(st,at,st\+1\)\|st,at\]=0\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\|s\_\{t\},a\_\{t\}\]=0\. We can now rewrite the estimator:

g^ABCπ=\(∑t=0∞γtBπ\(st,at,st\+1\)\)∇logπθ\(a0\|s0\)\+∑aAπ\(s0,a\)∇πθ\(a\|s0\)\.\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\}=\\left\(\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\right\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\+\\sum\_\{a\}A^\{\\pi\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)\.\(25\)Note that the second term∑aAπ\(s0,a\)∇πθ\(a\|s0\)\\sum\_\{a\}A^\{\\pi\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)can be viewed as a function ofs0s\_\{0\}\. As such,𝔼\[\(∑aAπ\(s0,a\)∇πθ\(a\|s0\)\)DT\]=0\\mathbb\{E\}\[\(\\sum\_\{a\}A^\{\\pi\}\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)\)D^\{T\}\]=0\.

For theBπB^\{\\pi\}terms, we show that

𝔼\[Bπ\(st,at,st\+1\)ΔA\(st′,at′\)∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]=0\.\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]=0\.\(26\)
Ift′≤tt^\{\\prime\}\\leq t,

𝔼\[Bπ\(st,at,st\+1\)ΔA\(st′,at′\)∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]\\displaystyle\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]=\\displaystyle=𝔼\[𝔼\[Bπ\(st,at,st\+1\)ΔA\(st′,at′\)∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\|s0,a0,…,st,at\]\]\\displaystyle\\mathbb\{E\}\[\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\|s\_\{0\},a\_\{0\},\\dots,s\_\{t\},a\_\{t\}\]\]=\\displaystyle=𝔼\[𝔼\[Bπ\(st,at,st\+1\)\|st,at\]ΔA\(st′,at′\)∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]\\displaystyle\\mathbb\{E\}\[\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\|s\_\{t\},a\_\{t\}\]\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]=\\displaystyle=0\\displaystyle 0Ift′\>tt^\{\\prime\}\>t,

𝔼\[Bπ\(st,at,st\+1\)ΔA\(st′,at′\)∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]\\displaystyle\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]=\\displaystyle=𝔼\[𝔼\[Bπ\(st,at,st\+1\)ΔA\(st′,at′\)∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\|s0,a0,…,st′\]\]\\displaystyle\\mathbb\{E\}\[\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\|s\_\{0\},a\_\{0\},\\dots,s\_\{t^\{\\prime\}\}\]\]=\\displaystyle=𝔼\[Bπ\(st,at,st\+1\)𝔼\[ΔA\(st′,at′\)\|st′\]∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]\\displaystyle\\mathbb\{E\}\[B^\{\\pi\}\(s\_\{t\},a\_\{t\},s\_\{t\+1\}\)\\mathbb\{E\}\[\\Delta A\(s\_\{t^\{\\prime\}\},a\_\{t^\{\\prime\}\}\)\|s\_\{t^\{\\prime\}\}\]\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]=\\displaystyle=0\\displaystyle 0The terms involvingΔV\(s0\)∇logπθ\(a0\|s0\)\\Delta V\(s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)andΔA\(s0,a\)∇πθ\(a\|s0\)\\Delta A\(s\_\{0\},a\)\\nabla\\pi\_\{\\theta\}\(a\|s\_\{0\}\)similarly vanish by the same argument\. Consequently,Cov⁡\(g^ABCπ,D\)=0\\mathrm\{Cov\}\(\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\},D\)=0, and

Cov⁡\(g^ABCπ,D\)\+Cov⁡\(D,g^ABCπ\)=0\.\\operatorname\{Cov\}\(\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\},D\)\+\\operatorname\{Cov\}\(D,\\hat\{g\}\_\{\\mathrm\{ABC\}\}^\{\\pi\}\)=0\.\(27\)∎

##### Remark

The covariance optimality result applies to individual gradient contributions\. In practice \(e\.g\., Algorithm[1](https://arxiv.org/html/2609.36058#alg1)\), we sum the gradients along the trajectories\. This introduces additional covariances between the gradients across time\-steps, so optimality of the individual gradients does not, in general, imply the optimality of their sum\. Unbiasedness, however, is preserved\.

See[3](https://arxiv.org/html/2609.36058#Thmtheorem3)

###### Proof\.

From Theorem[2](https://arxiv.org/html/2609.36058#Thmtheorem2), we have

Σ⁡\(A^,V^\)=Σ⁡\(Aπ,Vπ\)\+𝔼⁡\[D​DT\]\.\\Sigma\(\\hat\{A\},\\hat\{V\}\)=\\Sigma\(A^\{\\pi\},V^\{\\pi\}\)\+\\mathbb\{E\}\\left\[DD^\{T\}\\right\]\.\(28\)We first denoteδ=G−∑t=0∞γt​A^​\(st,at\)−V^​\(s0\)\\delta=G\-\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\-\\hat\{V\}\(s\_\{0\}\)\.DDcan then be rewritten into

D=δ∇logπθ\(a0\|s0\)−𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]\.\\displaystyle D=\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\-\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]\.\(29\)We have

D​DT\\displaystyle DD^\{T\}=δ2∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\\displaystyle=\\delta^\{2\}\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\(30\)−δ∇logπθ\(a0\|s0\)𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]T\\displaystyle\-\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]^\{T\}\(31\)−𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]δ∇logπθ\(a0\|s0\)T\\displaystyle\-\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\(32\)\+𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]T\.\\displaystyle\+\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]^\{T\}\.\(33\)By law of total expectation, we have

𝔼⁡\[D​DT\]\\displaystyle\\mathbb\{E\}\[DD^\{T\}\]=𝔼\[δ2∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]\\displaystyle=\\mathbb\{E\}\[\\delta^\{2\}\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]\(34\)−𝔼\[𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]T\]\.\\displaystyle\-\\mathbb\{E\}\[\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]^\{T\}\]\.\(35\)Since𝔼\[𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]𝔼\[δ∇logπθ\(a0\|s0\)\|s0\]T\]\\mathbb\{E\}\[\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]\\mathbb\{E\}\[\\delta\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\|s\_\{0\}\]^\{T\}\]is positive semi\-definite, we have

𝔼\[DDT\]≤𝔼\[δ2∇logπθ\(a0\|s0\)∇logπθ\(a0\|s0\)T\]≤ϵ2F\\displaystyle\\mathbb\{E\}\[DD^\{T\}\]\\leq\\mathbb\{E\}\[\\delta^\{2\}\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)\\nabla\\log\\pi\_\{\\theta\}\(a\_\{0\}\|s\_\{0\}\)^\{T\}\]\\leq\\epsilon^\{2\}F\(36\)∎

## Appendix CExperiment Details

### C\.1Offline Dataset

We generate the offline dataset using models of different sizes to have a wider coverage of trajectories\. The dataset is generated once and used for all subsequent experiments\. We use vLLM\[[12](https://arxiv.org/html/2609.36058#bib.bib36)\]for efficient trajectory generation\.

### C\.2Additional Results

##### GRPO

Figure[7](https://arxiv.org/html/2609.36058#A3.F7)shows the Dr\. GRPO training curves\. Despite the length bias correction of Dr\. GRPO, we still see steady increase of completion lengths over the course of training\. This is, in fact, consistent with our method, where we also see increases in completion length despite being unbiased\.

Figure 7:Dr\. GRPO training curves\.Figure 8:Residual \(ww\) squared distribution\.
##### Gradient Variance

Figure[4](https://arxiv.org/html/2609.36058#S4.F4)showed that the 1\.7B model does not benefit from the advantages\. Recall that the policy gradient is weighted by either\(G−V^​\(s0\)\)\(G\-\\hat\{V\}\(s\_\{0\}\)\)\(value baseline\) or\(G−V^​\(s0\)−∑t=0∞A^​\(st,at\)\)\(G\-\\hat\{V\}\(s\_\{0\}\)\-\\sum\_\{t=0\}^\{\\infty\}\\hat\{A\}\(s\_\{t\},a\_\{t\}\)\)\(ABC\)\. We denote this weighting byww\. Corollary[3](https://arxiv.org/html/2609.36058#Thmtheorem3)suggests that variance is closely related to this residual error\. Figure[8](https://arxiv.org/html/2609.36058#A3.F8)shows that, in general, ABC’swwdistributions tend to be closer to 0\. However, for the 1\.7B model, we see an outlier withw2\>5w^\{2\}\>5, which may explain why the ABC gradient variance is higher\.

Figure 9:REINFORCE training curves\.
##### REINFORCE

Figure[9](https://arxiv.org/html/2609.36058#A3.F9)summarizes the learning curves for REINFORCE\.

Figure 10:Critic loss during online training\.Figure 11:Policy entropy during online training\.

##### Entropy and Critic Loss

In Section[4](https://arxiv.org/html/2609.36058#S4), we observed the pattern that larger actors tend to have a smaller gap between loss with and without advantages\. Figure[11](https://arxiv.org/html/2609.36058#A3.F11)shows the critic loss with respect to different actor sizes\. For the 4B model, we see that the gap almost vanishes towards the end of training\. This can likely be explained by the gradual decrease of the entropy\. Recall that the advantage function explains away part of the variance of the return caused by the actions\. As such, if actions are near deterministic, then the advantages will become less significant\.

### C\.3Hyperparameters

##### Dr\. GRPO baseline

We use the implementation by[von Werra et al\. \[32\]](https://arxiv.org/html/2609.36058#bib.bib35), which includes various corrections that are not present in the original implementation \(e\.g\., training\-inference mismatch\)\. Table[2](https://arxiv.org/html/2609.36058#A3.T2)summarizes the hyperparameters used for the GRPO baselines in Section[4](https://arxiv.org/html/2609.36058#S4)\. The hyperparameters largely follow the ones used by[Liu et al\. \[17\]](https://arxiv.org/html/2609.36058#bib.bib15), except that we increase the completion length to 4096, increase the number of generations per prompt from 8 to 16, and use LoRA for fine\-tuning\. We have also tried tuning the learning rates but found that larger learning rates often lead to performance collapsing\.

##### Offline DAE Critic Training

Table[3](https://arxiv.org/html/2609.36058#A3.T3)summarizes the hyperparameters used for the offline training\. We found that using a much smaller learning rate for the advantage head significantly stabilizes learning\. In our early experiments, we tried using the same learning rate across all parameters and found that the loss became unstable\. This is likely due to the fact that the DAE objective involves a sum over the advantages across time\-steps, which makes it more sensitive to the output advantages\. In addition, we initialize the advantage and value heads to zero, which also stabilizes early training\.

##### Online Actor\-Critic Training

Table[4](https://arxiv.org/html/2609.36058#A3.T4)summarizes the hyperparameters for the online RL training\. In general, the actor configuration follows the ones used in the Dr\. GRPO baseline \(e\.g\., LoRA configuration, Adamβ\\beta\)\. The critic hyperparameters largely follow the ones used in the offline training phase \(cf\. Table[3](https://arxiv.org/html/2609.36058#A3.T3)\), except we lowered the learning rate\. For critic training, we only pass through the online\-collected trajectories once\. As such, the number of gradient steps per iteration \(KKin Algorithm[1](https://arxiv.org/html/2609.36058#alg1)\) is8192256=32\\frac\{8192\}\{256\}=32\.

Table 2:Hyperparameters for the GRPO baselines\.Table 3:Hyperparameters for the offline critic training\.Table 4:Hyperparameters for the online actor\-critic training\.

### C\.4Implementation Details

##### Reducing memory cost

Training with DAE requires enforcing the centering constraint, which is done in practice by parametrizing the function in the following way:

A^θ​\(s,a\)=fθ​\(s,a\)−∑aπ⁡\(a\|s\)​fθ​\(s,a\)\.\\hat\{A\}\_\{\\theta\}\(s,a\)=f\_\{\\theta\}\(s,a\)\-\\sum\_\{a\}\\pi\(a\|s\)f\_\{\\theta\}\(s,a\)\.This ensures that the centering constraint is always satisfied\. However, naively implementing this can be expensive as it requires storing two tensors \(π⁡\(a\|s\)\\pi\(a\|s\),fθ​\(s,a\)f\_\{\\theta\}\(s,a\)\) of shape\(T,V\)\(T,V\), whereTTis the sequence length andVVis the vocabulary size \(typically in the order of∼\\sim100000\)\. Note that the output layer is linear with respect to the output of the base model \(i\.e\.,fθ​\(s,a\)=hθ​\(s\)​Wf\_\{\\theta\}\(s,a\)=h\_\{\\theta\}\(s\)W, whereWWis the output linear layer for the advantage head with dimension\(C,V\)\(C,V\)\)\. SinceCC\(∼1000\{\\sim\}1000\) is usually much smaller thanVV, we can reorder the operation by:

∑a′π⁡\(a′\|s\)​fθ​\(s,a′\)\\displaystyle\\sum\_\{a^\{\\prime\}\}\\pi\(a^\{\\prime\}\|s\)f\_\{\\theta\}\(s,a^\{\\prime\}\)=∑a′∑cπ⁡\(a′\|s\)​hθ​\(s\)c​Wc​a′\\displaystyle=\\sum\_\{a^\{\\prime\}\}\\sum\_\{c\}\\pi\(a^\{\\prime\}\|s\)h\_\{\\theta\}\(s\)\_\{c\}W\_\{ca^\{\\prime\}\}=∑c\(∑a′Wc​a′​π​\(a′\|s\)\)​hθ​\(s\)c\.\\displaystyle=\\sum\_\{c\}\\left\(\\sum\_\{a^\{\\prime\}\}W\_\{ca^\{\\prime\}\}\\pi\(a^\{\\prime\}\|s\)\\right\)h\_\{\\theta\}\(s\)\_\{c\}\.This avoids constructing the full tensorfθ​\(s,a′\)f\_\{\\theta\}\(s,a^\{\\prime\}\), and reduces its memory cost from\(T,V\)\(T,V\)to\(T,C\)\(T,C\)\. For future work, it might be valuable to try FlashAttention style fusion of the softmax\-value product to further reduce memory footprint\[[4](https://arxiv.org/html/2609.36058#bib.bib34)\]\.

##### Centering in the ABC estimator

Recall that the ABC estimator requires computing the following term:

∑aA^\(s,a\)∇πθ\(a\|s\)\.\\sum\_\{a\}\\hat\{A\}\(s,a\)\\nabla\\pi\_\{\\theta\}\(a\|s\)\.Note that in practice, we center the advantages via:

A^​\(s,a\)=f⁡\(s,a\)−∑a′π⁡\(a′\|s\)​f​\(s,a′\),\\hat\{A\}\(s,a\)=f\(s,a\)\-\\sum\_\{a^\{\\prime\}\}\\pi\(a^\{\\prime\}\|s\)f\(s,a^\{\\prime\}\),whereffis the underlying approximator \(linear advantage head output\)\. This means thatA^​\(s,a\)\\hat\{A\}\(s,a\)andf⁡\(s,a\)f\(s,a\)differ only by a function ofss\(i\.e\.,∑aπ⁡\(a\|s\)​f​\(s,a\)\\sum\_\{a\}\\pi\(a\|s\)f\(s,a\)\)\. As we only care about the gradient for the policy in this case, we can ignore the centering because:

∑aA^\(s,a\)∇πθ\(a\|s\)\\displaystyle\\sum\_\{a\}\\hat\{A\}\(s,a\)\\nabla\\pi\_\{\\theta\}\(a\|s\)=∑af\(s,a\)∇πθ\(a\|s\)−∑a\(∑a′π\(a′\|s\)f\(s,a′\)\)∇πθ\(a\|s\)\\displaystyle=\\sum\_\{a\}f\(s,a\)\\nabla\\pi\_\{\\theta\}\(a\|s\)\-\\sum\_\{a\}\\left\(\\sum\_\{a^\{\\prime\}\}\\pi\(a^\{\\prime\}\|s\)f\(s,a^\{\\prime\}\)\\right\)\\nabla\\pi\_\{\\theta\}\(a\|s\)\(37\)=∑af\(s,a\)∇πθ\(a\|s\)−\(∑a′π\(a′\|s\)f\(s,a′\)\)\(∑a∇πθ\(a\|s\)\)\\displaystyle=\\sum\_\{a\}f\(s,a\)\\nabla\\pi\_\{\\theta\}\(a\|s\)\-\\left\(\\sum\_\{a^\{\\prime\}\}\\pi\(a^\{\\prime\}\|s\)f\(s,a^\{\\prime\}\)\\right\)\\left\(\\sum\_\{a\}\\nabla\\pi\_\{\\theta\}\(a\|s\)\\right\)\(38\)=∑af\(s,a\)∇πθ\(a\|s\)\\displaystyle=\\sum\_\{a\}f\(s,a\)\\nabla\\pi\_\{\\theta\}\(a\|s\)\(39\)