Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards

# Gradients Know What Outcomes Don’t: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
Source: [https://arxiv.org/html/2609.03342](https://arxiv.org/html/2609.03342)
Leqi ZhengAffiliation:Tsinghua UniversityAffiliation:Equal contribution\.\* Correspondence:chaokun@tsinghua\.edu\.cnJinbo SuAffiliation:Renmin University of ChinaAffiliation:Equal contribution\.\* Correspondence:chaokun@tsinghua\.edu\.cnFang NiuAffiliation:Tsinghua UniversityAffiliation:Equal contribution\.\* Correspondence:chaokun@tsinghua\.edu\.cnWeiping WangAffiliation:Institute of Information Engineering, CASJiajun ZhangAffiliation:USTCShannan YanAffiliation:Tsinghua UniversityJie WuAffiliation:The Australian National UniversityZhaolu KangAffiliation:Peking UniversityRong FuAffiliation:University of MacauHang ZhangAffiliation:Tsinghua University

###### Abstract

Reinforcement learning from verifiable rewards \(RLVR\) drives chain\-of\-thought reasoning in large language models, yet its binary outcome reward cannot distinguish among correct trajectories\. Existing dense reward alternatives, from surface heuristics to process reward models, either ignore the expert solutions already present in training corpora or require expensive offline annotation\. We proposeGradient\-Aligned Reward \(GAR\), which operates in the policy’s own gradient space: truncated backpropagation through the output projection layer extracts a compact gradient vector for each rollout, and cosine similarity with an expert\-anchor gradient yields a dense, reasoning\-aware reward with less than 9% wall\-clock overhead\. We prove that this cosine admits a multiplicative decomposition into prediction\-error and activation\-pattern factors, providing a concrete characterization of what the alignment signal measures\. On Qwen3\-4B and Qwen3\-8B, GAR consistently improves over GRPO and other baselines on competition\-level math benchmarks and transfers to GPQA Diamond and MMLU\-Pro without domain\-specific data\. Code and data are available at[https://github\.com/LQgdwind/GAR](https://github.com/LQgdwind/GAR)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.03342v1/fig1_gradient_space.png)Figure 1:Gradient\-space visualization of GAR\. The expert anchor𝐯a\\mathbf\{v\}\_\{a\}\(star\) defines the reference direction in the parameter update manifold\. Correct rollouts \(circles\) are scored by cosine similarity with the anchor: high\-alignment rollouts \(dark\) receive a large GAR bonus, while low\-alignment rollouts \(light\) receive a smaller bonus despite also being correct\. Incorrect rollouts \(crosses\) are gated out and receive zero reward regardless of their gradient direction\.Reinforcement learning from verifiable rewards \(RLVR\) has emerged as a compelling paradigm for eliciting chain\-of\-thought reasoning in large language models without supervised fine\-tuning[Guo et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib1);[Shao et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib2);[Yu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib12);[Zheng et al\. \(2025b\)](https://arxiv.org/html/2609.03342#bib.bib42);[Zheng et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib43);[Zheng et al\. \(2025a\)](https://arxiv.org/html/2609.03342#bib.bib44)\. Under this setting, a base model is optimized solely with a binary outcome reward that verifies final\-answer correctness, and DeepSeek\-R1\-Zero[Guo et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib1)demonstrated that structured reasoning[Wei et al\. \(2022\)](https://arxiv.org/html/2609.03342#bib.bib24);[Kojima et al\. \(2022\)](https://arxiv.org/html/2609.03342#bib.bib25)can emerge spontaneously from such sparse supervision\. Despite this success, the exclusive reliance on binary rewards introduces a fundamental*credit assignment*problem: once multiple rollouts produce correct answers, the reward signal becomes flat over the correct subset, and the resulting policy gradient carries no information to preferentially reinforce higher\-quality reasoning trajectories\.

This pathology motivates a search for denser, process\-level supervision, yet existing remedies each suffer from significant limitations: \(1\)Expert solutions remain unused\.Outcome\-only RLVR assigns identical reward to every correct response, collapsing the group\-relative advantage among the correct subset to zero[Shao et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib2)\. Widely used math corpora such as NuminaMath\-CoT[Li et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib7)already ship expert chain\-of\-thought solutions alongside each problem, yet no existing reward mechanism exploits them to provide a process\-level training signal\. \(2\)Surface\-level heuristic shaping\.Rule\-based reward shaping methods[Wen et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib6);[Aggarwal and Welleck \(2025\)](https://arxiv.org/html/2609.03342#bib.bib13)introduce length or format penalties as proxies for reasoning quality, but such heuristics operate on surface attributes and cannot evaluate whether the underlying chain of thought genuinely engages the reasoning structures required by the task\. \(3\)Expensive and offline process supervision\.Process reward models \(PRMs\) provide step\-level feedback but require large\-scale expert annotations[Lightman et al\. \(2023\)](https://arxiv.org/html/2609.03342#bib.bib3);[Uesato et al\. \(2022\)](https://arxiv.org/html/2609.03342#bib.bib32); even automated alternatives[Wang et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib4);[Cui et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib15);[Setlur et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib31)are trained offline on fixed distributions that diverge from the evolving policy\. A detailed comparison is provided in Table[1](https://arxiv.org/html/2609.03342#S1.T1)\.

Table 1:Comparison of reward paradigms\. GAR is the only method that leverages expert CoT in gradient space\.These limitations point to a natural question:can we turn the expert solutions that training corpora already provide into a gradient\-space signal that differentiates among correct rollouts?

We find that we can, and introduceGradient\-Aligned Reward \(GAR\), a lightweight online process reward mechanism for RLVR\. The central insight is that if two correct trajectories implement substantively similar reasoning, their gradient directions through the output projection layer must remain close, irrespective of surface\-level textual variation\. GAR operationalizes this hypothesis by performing truncated backpropagation through only the LM head and scoring each rollout by its cosine similarity with an expert\-anchor gradient derived from existing chain\-of\-thought solutions in the training corpus \(Figure[1](https://arxiv.org/html/2609.03342#S1.F1)\)\. This design addresses all three limitations: gradient cosine with expert anchors turns readily available CoT solutions into a dense reward; the gradient signal captures reasoning\-level structure beyond surface attributes; and the online, truncated computation eliminates dependence on step\-level annotations or external reward models\.

In summary, our contributions are four\-fold:

\(1\)Gradient\-Space Process Rewards from Expert Anchors\.We introduce a reward mechanism that converts readily available expert chain\-of\-thought solutions into gradient\-space reference vectors, enabling dense, per\-rollout process supervision without additional annotation\. We ground the signal via the empirical neural tangent kernel and potential\-based reward shaping theory\.

\(2\)Multiplicative Decomposition of the Alignment Signal\.We prove that the gradient cosine decomposes multiplicatively into a prediction\-error factor and an activation\-pattern factor \(Theorem[2](https://arxiv.org/html/2609.03342#Thmtheorem2)\), providing a concrete characterization of the two complementary axes along which GAR distinguishes correct trajectories\.

\(3\)Lightweight Online Computation\.Truncated backpropagation through the output projection layer reduces alignment cost toO⁡\(V×d\)O\(V\\times d\), adding less than 9% wall\-clock overhead to standard GRPO training\.

\(4\)Empirical Validation\.On Qwen3\-4B and 8B base models, GAR yields consistent pass@kkgains on four competition\-level math benchmarks and transfers to GPQA Diamond and MMLU\-Pro without domain\-specific training data\.

## 2Method

GAR is motivated by a fundamental limitation of outcome\-only reinforcement learning\. Under a binary verifier rewardr⁡\(x,y\)=𝟏​\[Verify​\(x,y\)\]r\(x,y\)=\\mathbf\{1\}\[\\text\{Verify\}\(x,y\)\], all correct responses within a rollout group receive the same reward, so GRPO’s group\-relative advantage collapses to an identical value across allKcK\_\{c\}correct trajectories and cannot preferentially reinforce higher\-quality reasoning\. As training progresses andKc→KK\_\{c\}\\to K, this further induces instabilities in the policy update\. GAR addresses both pathologies by introducing intra\-group variance among correct responses through gradient\-space alignment with expert reasoning traces\. An end\-to\-end overview is provided in Figure[2](https://arxiv.org/html/2609.03342#S2.F2)\.

![Refer to caption](https://arxiv.org/html/2609.03342v1/figures/backbone.png)Figure 2:GAR pipeline within one GRPO step\.\(A\)The policy generatesKKrollouts; a verifier gates correct responses \(all receivingr=1r\{=\}1under standard GRPO\)\.\(B\)For each correct rollout, a teacher\-forcing forward pass produces hidden states𝐇~\\widetilde\{\\mathbf\{H\}\}, which are detached at the output projection boundary and backpropagated throughWoW\_\{o\}only to obtain gradient\-activation vector𝐯i\\mathbf\{v\}\_\{i\}\. The expert CoT undergoes the same procedure \(cached\) to yield𝐯a\\mathbf\{v\}\_\{a\}; their cosine similarity gives the alignment bonus\.\(C\)Rollouts whose gradient directions cluster near𝐯a\\mathbf\{v\}\_\{a\}receive higher rewards, breaking the flat\-reward pathology\.### 2\.1The Gradient Alignment Hypothesis

Letℒ\(θ;x,y\)=−1\|y\|∑t=1\|y\|logπθ\(yt∣x,y<t\)\\mathcal\{L\}\(\\theta;x,y\)=\-\\frac\{1\}\{\|y\|\}\\sum\_\{t=1\}^\{\|y\|\}\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)denote the teacher\-forcing negative log\-likelihood of a responseyyunder the current policy\. Consider a promptxxtogether with two correct responses: a candidateyygenerated by the policy and an expert anchoraa\. Define the \(full\-parameter\) gradient operator

𝐠⁡\(x,y\)=∇θℒ​\(θ,x,y\)∈ℝ\|θ\|\.\\mathbf\{g\}\(x,y\)=\\nabla\_\{\\theta\}\\mathcal\{L\}\(\\theta;x,y\)\\in\\mathbb\{R\}^\{\|\\theta\|\}\.\(1\)The gradient alignment hypothesis posits that whenever the candidate response implements the same underlying reasoning process as the anchor, the two gradients are close in direction, i\.e\.,cos⁡\(𝐠⁡\(x,y\),𝐠⁡\(x,a\)\)≈1\\cos\(\\mathbf\{g\}\(x,y\),\\mathbf\{g\}\(x,a\)\)\\approx 1, whereas trajectories that arrive at the correct answer through qualitatively different computational pathways yield gradients that are only weakly aligned\. The intuition is that a policy gradient step on either trajectory nudges the same set of reasoning\-relevant computational circuits, whereas semantically incompatible trajectories modify non\-overlapping subnetworks\.

Computing the full gradient𝐠⁡\(x,y\)∈ℝ\|θ\|\\mathbf\{g\}\(x,y\)\\in\\mathbb\{R\}^\{\|\\theta\|\}for every rollout is prohibitive in practice\. Our key algorithmic contribution is to replace the full gradient with a truncated surrogate obtained by freezing the Transformer body and backpropagating only through the output projection layer, resulting in a vector of dimensionddrather than\|θ\|\|\\theta\|\. We show in Section[3](https://arxiv.org/html/2609.03342#S3)that the cosine similarity of these truncated surrogates is bi\-Lipschitz equivalent to the normalized output\-layer NTK similarity, preserving relative ordering among trajectories under mild regularity conditions\.

### 2\.2Truncated Gradient Signal

For a prompt\-response pair\(x,y\)\(x,y\), we perform a no\-grad forward pass through the full Transformer and intercept the hidden state𝐇∈ℝn×d\\mathbf\{H\}\\in\\mathbb\{R\}^\{n\\times d\}at the input of the output projection via a forward pre\-hook, detach it to block gradient flow into the Transformer body, and compute a truncated cross\-entropy loss through only the LM head:

ℒ=1\|𝒯y\|​∑t∈𝒯yℓ⁡\(Wo​𝐡~t\+𝐛o,yt\+1\),\\mathcal\{L\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\ell\\\!\\left\(W\_\{o\}\\widetilde\{\\mathbf\{h\}\}\_\{t\}\+\\mathbf\{b\}\_\{o\},\\;y\_\{t\+1\}\\right\),\(2\)where𝒯y\\mathcal\{T\}\_\{y\}indexes the response span \(or, when delimited, the explicit thinking span\),𝐡~t\\widetilde\{\\mathbf\{h\}\}\_\{t\}is the detached hidden state at positiontt,yt\+1y\_\{t\+1\}is the next\-token target, andWo,𝐛oW\_\{o\},\\mathbf\{b\}\_\{o\}are the output projection parameters\. The gradient𝐆t=∂ℒ/∂𝐡~t\\mathbf\{G\}\_\{t\}=\\partial\\mathcal\{L\}/\\partial\\widetilde\{\\mathbf\{h\}\}\_\{t\}is then combined element\-wise with the activation to form the gradient\-activation signal

𝐒t=𝐆t⊙𝐡~t,∀t∈𝒯y,\\mathbf\{S\}\_\{t\}=\\mathbf\{G\}\_\{t\}\\odot\\widetilde\{\\mathbf\{h\}\}\_\{t\},\\quad\\forall\\,t\\in\\mathcal\{T\}\_\{y\},\(3\)The element\-wise product emphasizes dimensions where loss sensitivity and activation magnitude jointly concentrate, making the signal more robust to surface\-level wording variation than the raw gradient alone\. Proposition[1](https://arxiv.org/html/2609.03342#Thmproposition1)shows that𝐒t\\mathbf\{S\}\_\{t\}admits a first\-order interpretation as the linearized per\-dimension contribution to the log\-likelihood\. The per\-token signals are averaged across response positions and L2\-normalized,

𝐯=𝐬¯‖𝐬¯‖2,𝐬¯=1\|𝒯y\|​∑t∈𝒯y𝐒t,\\mathbf\{v\}=\\frac\{\\bar\{\\mathbf\{s\}\}\}\{\\\|\\bar\{\\mathbf\{s\}\}\\\|\_\{2\}\},\\quad\\bar\{\\mathbf\{s\}\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\mathbf\{S\}\_\{t\},\(4\)which removes magnitude drift across training steps so that cosine comparisons remain commensurate across trajectories of different lengths\.

### 2\.3Reward Formulation

#### Verifier Gate\.

GAR retains the outcome verifier as a hard gate: a responseyiy\_\{i\}with binary outcome rewardrraw​\(x,yi\)=0r\_\{\\text\{raw\}\}\(x,y\_\{i\}\)=0receives no alignment bonus and triggers no gradient computation, which both prevents the policy from inflating its reward through spurious directions in gradient space and confines the additional cost of GAR to verifier\-passing rollouts\.

#### Anchor and Cosine Score\.

For each promptxxwe designate an expert anchora⁡\(x\)a\(x\): the chain\-of\-thought solution shipped with the training dataset\. Crucially, widely used math corpora such as NuminaMath\-CoT[Li et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib7)already provide such solutions as standard metadata, yet conventional GRPO pipelines ignore them entirely\. GAR repurposes these annotations as gradient\-space references, converting an underutilized resource into dense process supervision at no additional annotation cost\. The anchor is mapped to its gradient vector𝐯a\\mathbf\{v\}\_\{a\}via the same truncated procedure, with the anchor vector cached per prompt across the rollout batch to avoid redundant computation\. For a verifier\-passing responseyiy\_\{i\}with gradient vector𝐯i\\mathbf\{v\}\_\{i\}, the alignment score is the cosine similarity with the anchor, which reduces to the inner product because both vectors are L2\-normalized:

b⁡\(yi\)=𝐯i⊤​𝐯a\.b\(y\_\{i\}\)=\\mathbf\{v\}\_\{i\}^\{\\top\}\\mathbf\{v\}\_\{a\}\.\(5\)

#### Final Reward\.

The final GAR reward applies the gated, group\-centered, non\-negatively clipped bonus on top of the base reward:

rGAR​\(x,yi\)=\\displaystyle r\_\{\\text\{GAR\}\}\(x,y\_\{i\}\)=\(6\)\{rbase\+β⋅max⁡\(0,b^​\(yi\)\)\+p⁡\(yi\),rraw\>0,p⁡\(yi\),otherwise,\\displaystyle\\begin\{cases\}r\_\{\\text\{base\}\}\+\\beta\\cdot\\max\(0,\\hat\{b\}\(y\_\{i\}\)\)\+p\(y\_\{i\}\),&r\_\{\\text\{raw\}\}\>0,\\\\ p\(y\_\{i\}\),&\\text\{otherwise\},\\end\{cases\}whereb^​\(yi\)=b⁡\(yi\)−1\|𝒫⁡\(x\)\|​∑yj∈𝒫⁡\(x\)b⁡\(yj\)\\hat\{b\}\(y\_\{i\}\)=b\(y\_\{i\}\)\-\\frac\{1\}\{\|\\mathcal\{P\}\(x\)\|\}\\sum\_\{y\_\{j\}\\in\\mathcal\{P\}\(x\)\}b\(y\_\{j\}\)is the bonus centered over the correct subset𝒫⁡\(x\)\\mathcal\{P\}\(x\)of the rollout group,rbaser\_\{\\text\{base\}\}\(default1\.01\.0\) is the base reward for correctness,β≥0\\beta\\geq 0\(default0\.50\.5\) scales the alignment bonus, andp⁡\(yi\)≤0p\(y\_\{i\}\)\\leq 0is a small format penalty on responses that violate the prescribed think/answer structure\. Themax⁡\(0,⋅\)\\max\(0,\\cdot\)clip keeps the GAR reward weakly above the outcome\-only baseline on every correct trajectory and is essential for the safe reward\-shaping guarantee of Theorem[3](https://arxiv.org/html/2609.03342#Thmtheorem3)\.

## 3Theoretical Analysis

All proofs are deferred to the appendix\.

### 3\.1NTK Interpretation and Multiplicative Decomposition

Let𝐠o​\(x,y\)=∇Woℒ​\(Wo,x,y\)∈ℝ\|𝒱\|×d\\mathbf\{g\}\_\{o\}\(x,y\)=\\nabla\_\{W\_\{o\}\}\\mathcal\{L\}\(W\_\{o\};x,y\)\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}denote the output\-layer gradient\. Straightforward differentiation yields𝐠o​\(x,y\)=1\|𝒯y\|​∑t\(𝐩t−𝐞yt\+1\)⊗𝐡t\\mathbf\{g\}\_\{o\}\(x,y\)=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\}\(\\mathbf\{p\}\_\{t\}\-\\mathbf\{e\}\_\{y\_\{t\+1\}\}\)\\otimes\\mathbf\{h\}\_\{t\}\.

###### Proposition 1\(Gradient\-activation product as linearized log\-likelihood\)\.

Let𝐇y∈ℝn×d\\mathbf\{H\}\_\{y\}\\in\\mathbb\{R\}^\{n\\times d\}denote the matrix of final\-layer hidden states at the input of the output projection, obtained from a no\-gradient forward pass\. The per\-token signal𝐮t=𝐆t⊙𝐡t\\mathbf\{u\}\_\{t\}=\\mathbf\{G\}\_\{t\}\\odot\\mathbf\{h\}\_\{t\}of Eq\. \([3](https://arxiv.org/html/2609.03342#S2.E3)\) satisfiesdd​η​ℒ​\(\(1\+η\)​𝐇y\)\|η=0=1\|𝒯y\|​∑t𝟏⊤​𝐮t\\frac\{d\}\{d\\eta\}\\mathcal\{L\}\\big\(\(1\+\\eta\)\\mathbf\{H\}\_\{y\}\\big\)\\big\|\_\{\\eta=0\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\}\\mathbf\{1\}^\{\\top\}\\mathbf\{u\}\_\{t\}, so the aggregated signal𝐬¯\\bar\{\\mathbf\{s\}\}captures the linearized contribution of each hidden dimension to the log\-likelihood\.

###### Theorem 1\(NTK\-induced functional alignment\)\.

LetΘo​\(y,a\)=⟨vec⁡\(𝐠o​\(x,y\)\),vec⁡\(𝐠o​\(x,a\)\)⟩\\Theta\_\{o\}\(y,a\)=\\langle\\mathrm\{vec\}\(\\mathbf\{g\}\_\{o\}\(x,y\)\),\\mathrm\{vec\}\(\\mathbf\{g\}\_\{o\}\(x,a\)\)\\rangledenote the output\-layer empirical NTK, and let𝐯y,𝐯a\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}denote the L2\-normalized gradient\-activation vectors of Eq\. \([4](https://arxiv.org/html/2609.03342#S2.E4)\)\. Under bounded error signals \(‖𝛅¯y‖2≤Bδ\\\|\\bar\{\\boldsymbol\{\\delta\}\}\_\{y\}\\\|\_\{2\}\\leq B\_\{\\delta\}\) and hidden states \(‖𝐡¯y‖2≤Bh\\\|\\bar\{\\mathbf\{h\}\}\_\{y\}\\\|\_\{2\}\\leq B\_\{h\}\), there exist constantsc1,c2\>0c\_\{1\},c\_\{2\}\>0such that

c1cos\(𝐯y,𝐯a\)\\displaystyle c\_\{1\}\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)≤Θo​\(y,a\)‖𝐠o​\(x,y\)‖F​‖𝐠o​\(x,a\)‖F\\displaystyle\\leq\\frac\{\\Theta\_\{o\}\(y,a\)\}\{\\\|\\mathbf\{g\}\_\{o\}\(x,y\)\\\|\_\{F\}\\\|\\mathbf\{g\}\_\{o\}\(x,a\)\\\|\_\{F\}\}\(7\)≤c2cos\(𝐯y,𝐯a\)\+𝒪\(κy\+κa\),\\displaystyle\\leq c\_\{2\}\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)\+\\mathcal\{O\}\(\\kappa\_\{y\}\+\\kappa\_\{a\}\),whereκy,κa\\kappa\_\{y\},\\kappa\_\{a\}measure the per\-token error dispersion around the trajectory means\.

Theorem[1](https://arxiv.org/html/2609.03342#Thmtheorem1)shows that rewarding high\-cosine trajectories is equivalent, up to bounded distortion, to rewarding high output\-layer NTK similarity with the expert anchor\.

###### Theorem 2\(Multiplicative decomposition of gradient alignment\)\.

Under the same assumptions, the NTK inner product decomposes as

⟨𝐠o​\(x,y\),𝐠o​\(x,a\)⟩\\displaystyle\\langle\\mathbf\{g\}\_\{o\}\(x,y\),\\;\\mathbf\{g\}\_\{o\}\(x,a\)\\rangle\(8\)=\(𝜹¯y⊤​𝜹¯a\)⏟gradient\-direction​\(𝐡¯y⊤​𝐡¯a\)⏟activation\-pattern\\displaystyle=\\underbrace\{\\big\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\\big\)\}\_\{\\text\{gradient\-direction\}\}\\underbrace\{\\big\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\\big\)\}\_\{\\text\{activation\-pattern\}\}\+𝒪⁡\(Bh2​κy​κa\+Bδ​\(κy\+κa\)​Bh\)\.\\displaystyle\+\\mathcal\{O\}\\\!\\big\(B\_\{h\}^\{2\}\\kappa\_\{y\}\\kappa\_\{a\}\+B\_\{\\delta\}\(\\kappa\_\{y\}\+\\kappa\_\{a\}\)B\_\{h\}\\big\)\.

The multiplicative structure requires agreement in*both*prediction\-error profile and activation pattern; a trajectory reaching the correct answer through a different computational pathway receives a low cosine score\. Empirically, across six training\-set problems where correct rollouts use identifiably different solution methods, same\-method rollouts receive3\.5×3\.5\\timeshigher cosine scores than alternative correct methods \(meanb¯same=0\.45\\bar\{b\}\_\{\\text\{same\}\}=0\.45vs\.b¯diff=0\.13\\bar\{b\}\_\{\\text\{diff\}\}=0\.13; Appendix[M](https://arxiv.org/html/2609.03342#A13)\), confirming that the signal discriminates based on the underlying derivation strategy rather than surface\-level correlates such as length or formatting\.

###### Corollary 1\(Orthogonality under subspace separation\)\.

If theε\\varepsilon\-effective supports𝒮εy\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}and𝒮εa\\mathcal\{S\}\_\{\\varepsilon\}^\{a\}are disjoint, then\|cos⁡\(𝐯y,𝐯a\)\|≤2​ε​d\+𝒪⁡\(κy\+κa\)\|\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)\|\\leq 2\\varepsilon d\+\\mathcal\{O\}\(\\kappa\_\{y\}\+\\kappa\_\{a\}\)\.

### 3\.2Safe Reward Shaping

###### Definition 1\(Outcome\-verified optimal policy\)\.

A policyπ⋆\\pi^\{\\star\}is outcome\-verified optimal if it maximizes𝒥0​\(π\)=𝔼x,y∼π​1​\[Verify⁡\(x,y\)\]\\mathcal\{J\}\_\{0\}\(\\pi\)=\\mathbb\{E\}\_\{x,y\\sim\\pi\}\\,\\mathbf\{1\}\[\\mathrm\{Verify\}\(x,y\)\]\.

###### Theorem 3\(Safe reward shaping\)\.

The GAR reward of Eq\. \([6](https://arxiv.org/html/2609.03342#S2.E6)\) satisfies two properties\. \(i\)Non\-degradation: for every correct responseyywithrraw​\(x,y\)\>0r\_\{\\text\{raw\}\}\(x,y\)\>0,rGAR​\(x,y\)−p⁡\(y\)≥rbase\>0r\_\{\\text\{GAR\}\}\(x,y\)\-p\(y\)\\geq r\_\{\\text\{base\}\}\>0, so GAR never reduces the reward of a correct response below the outcome\-only baseline\. \(ii\)Strict incentive separation: for every incorrect responsey′y^\{\\prime\}withrraw​\(x,y′\)=0r\_\{\\text\{raw\}\}\(x,y^\{\\prime\}\)=0,rGAR​\(x,y′\)=p⁡\(y′\)≤0<rbase≤rGAR​\(x,y\)−p⁡\(y\)r\_\{\\text\{GAR\}\}\(x,y^\{\\prime\}\)=p\(y^\{\\prime\}\)\\leq 0<r\_\{\\text\{base\}\}\\leq r\_\{\\text\{GAR\}\}\(x,y\)\-p\(y\), so the policy gradient consistently assigns higher advantage to correct responses than to incorrect ones\.

Direction \(i\) follows from the verifier gate andmax⁡\(0,b^\)\\max\(0,\\hat\{b\}\)clip ensuring every correct response contributes at leastrbaser\_\{\\text\{base\}\}; direction \(ii\) follows from the verifier gate assigningrGAR​\(x,y′\)=p⁡\(y′\)≤0r\_\{\\text\{GAR\}\}\(x,y^\{\\prime\}\)=p\(y^\{\\prime\}\)\\leq 0to incorrect responses, while every correct response receives at leastrbase\>0r\_\{\\text\{base\}\}\>0on top of the format penalty\.

### 3\.3Variance and Unbiasedness

###### Proposition 2\(Variance amplification\)\.

Under the flat outcome reward,Vari:rraw=1\[Aioutcome\]=0\\mathrm\{Var\}\_\{i:r\_\{\\text\{raw\}\}=1\}\[A\_\{i\}^\{\\text\{outcome\}\}\]=0\. Under GAR,Vari:rraw=1\[AiGAR\]\>0\\mathrm\{Var\}\_\{i:r\_\{\\text\{raw\}\}=1\}\[A\_\{i\}^\{\\text\{GAR\}\}\]\>0whenever the alignment bonuses have non\-zero variance within the correct subset\.

###### Proposition 3\(Unbiasedness of prompt\-group normalization\)\.

The group\-normalized bonus satisfies𝔼y:rraw=1\[b^\(y\)\]=0\\mathbb\{E\}\_\{y:r\_\{\\text\{raw\}\}=1\}\[\\hat\{b\}\(y\)\]=0for every promptxx\.

### 3\.4Monotonic Alignment Improvement

###### Proposition 4\(Monotonic alignment improvement\)\.

LetC¯t=𝔼x,y∼πt,rraw=1​\[cos⁡\(𝐯y,𝐯a\)\]\\bar\{C\}\_\{t\}=\\mathbb\{E\}\_\{x,y\\sim\\pi\_\{t\},r\_\{\\text\{raw\}\}=1\}\[\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)\]denote the expected cosine alignment at iterationtt\. Under GAR\-shaped GRPO updates with sufficiently small KL penalty,C¯t\+1≥C¯t\\bar\{C\}\_\{t\+1\}\\geq\\bar\{C\}\_\{t\}\.

This follows because GAR assigns positive advantages exclusively to above\-mean\-cosine trajectories, soCov⁡\[Ai,ci\]≥0\\mathrm\{Cov\}\[A\_\{i\},c\_\{i\}\]\\geq 0\.

## 4Experimental Setup

Table 2:Pass@kkaccuracy \(%, abbreviated as P@kkin headers\) on competition\-level math benchmarks, averaged over 10 independent runs \(standard deviations ingray; all GAR improvements over the corresponding base optimizer are statistically significant withp<0\.05p<0\.05by pairedtt\-test across runs\)\. All methods train from Qwen3 base checkpoints without SFT warmup using identical training data and compute budget\. Best results per model inbold\.### 4\.1Training

GAR is implemented within the SLIME / Megatron training stack as an online reward hook invoked before GRPO advantage computation \(Appendix[C](https://arxiv.org/html/2609.03342#A3)\)\. We train Qwen3\-4B\-Base and Qwen3\-8B\-Base[Yang et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib8)from their base checkpoints without SFT warmup using full\-parameter reinforcement learning, for400400steps with batch size128128andK=16K\{=\}16rollouts per prompt on∼\{\\sim\}10k NuminaMath\-CoT[Li et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib7)problems; each problem’s chain\-of\-thought solution serves as the expert anchor\. Each optimization step therefore contains2,0482\{,\}048sampled trajectories before verifier gating, which provides multiple candidate solutions per prompt for estimating within\-group reward differences\. Training directly from the base checkpoints prevents the observed gains from being attributed to an SFT warmup and isolates the contribution of the reward signal\. The default GAR hyperparameters areβ=0\.5\\beta\{=\}0\.5, max GAR spanL=768L\{=\}768tokens, activation thresholdτa=0\.05\\tau\_\{a\}\{=\}0\.05, and anchor filterpf=0\.7p\_\{f\}\{=\}0\.7\. The expert\-anchor vector is cached within each rollout group, and the alignment bonus is evaluated only for verifier\-passing responses, limiting additional computation to the trajectories for which process\-level differentiation is meaningful\.

### 4\.2Baselines

We compare againstGRPO[Shao et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib2);[Yu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib12),REINFORCE\+\+[Hu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib27),MASPO[Fu et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib37),Grad2Reward[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib35), andG2RL[Liang et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib41)\. We reportGAR\-GRPOandGAR\-REINFORCE\+\+to isolate GAR’s contribution from the optimizer\. GRPO and REINFORCE\+\+ provide two distinct policy optimization backbones, while MASPO, Grad2Reward, and G2RL test whether GAR remains competitive with recent methods designed to improve signal reliability or exploit gradient information\. All methods share the same base checkpoint, training data, compute budget, and rollout infrastructure; only the reward function differs\. Consequently, each comparison between an optimizer and its GAR variant measures the effect of reward shaping without conflating it with additional data, supervised initialization, or a larger rollout budget\.

### 4\.3Evaluation

We report pass@kk\(k∈\{1,4,16\}k\{\\in\}\\\{1,4,16\\\}\) on four held\-out math benchmarks \(IMO\-AnswerBench, HMMT ’25/​’26, AIME ’26\) and two general reasoning benchmarks \(GPQA Diamond, MMLU\-Pro\) to assess cross\-domain transfer\. Correctness is determined by exact answer match after normalization\. Pass@1 measures whether training concentrates probability mass on a correct solution in a single attempt, whereas pass@16 measures whether the policy preserves broader solution coverage across repeated samples\. We average every configuration over 10 independently trained runs and use paired significance tests under matched sampling conditions; complete decoding, statistical, and contamination\-control protocols are provided in Appendix[H](https://arxiv.org/html/2609.03342#A8)\.

## 5Results

### 5\.1Main Results

Table[2](https://arxiv.org/html/2609.03342#S4.T2)presents the main results across four competition\-level benchmarks\. GAR improves over the corresponding base optimizer on every benchmark–model combination, with relative gains of up to 52\.4% at pass@1 \(HMMT 2025, 4B\)\. The improvements are most pronounced at lowerkk, indicating that GAR steers the policy toward higher\-probability correct solutions rather than merely expanding coverage\. For GAR\-GRPO at the 4B scale, pass@1 increases from 2\.42 to 3\.18 on HMMT 2026 and from 3\.83 to 5\.00 on HMMT 2025, corresponding to relative gains of 31\.4% and 30\.5%, respectively\. The same comparisons at the 8B scale increase from 4\.24 to 5\.15 and from 4\.50 to 6\.00, showing that the benefit persists as model capacity grows\. Improvements also remain positive at pass@16 across every dataset and optimizer, which indicates that concentrating probability on strong solutions does not reduce the overall coverage of correct reasoning paths\. Layering GAR on REINFORCE\+\+[Hu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib27)yields comparable gains to the GRPO variant, demonstrating that the reward signal is complementary to optimizer\-side design choices\. The largest relative gain occurs for GAR\-REINFORCE\+\+ on HMMT 2025 at the 4B scale, where pass@1 rises from 3\.17 to 4\.83\. Among gradient\-based competitors, Grad2Reward[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib35)and G2RL[Liang et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib41)underperform GAR across all benchmarks, while MASPO[Fu et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib37)provides only modest improvement over GRPO\. Together, these results indicate that the principal advantage arises from aligning the reward with expert reasoning in gradient space rather than from the choice of policy optimizer alone\.

## 6Empirical Analysis

### 6\.1General Reasoning Transfer

Table 3:General reasoning transfer \(Qwen3\-4B\-Base, 10 runs\)\. All methods are trained exclusively on mathematical data and evaluated zero\-shot\. GPQA reports pass@kkand maj@16; MMLU\-Pro reports micro\-averaged pass@1\.To assess whether gradient\-aligned rewards transfer beyond the training domain, Table[3](https://arxiv.org/html/2609.03342#S6.T3)evaluates all methods, trained exclusively on mathematical data, on two general reasoning benchmarks\. GAR\-GRPO outperforms all baselines on both GPQA Diamond and MMLU\-Pro, indicating that the gradient alignment signal captures domain\-general reasoning structure rather than math\-specific heuristics\. Relative to GRPO, the 4B model gains 2\.57 points on GPQA pass@1, 1\.79 points on pass@4, 1\.94 points on majority voting, and 2\.41 points on MMLU\-Pro\. GAR\-GRPO also exceeds the strongest alternative baseline in every reported column, so the transfer improvement is not explained solely by a weak GRPO reference\. Results on Qwen3\-8B\-Base \(Appendix[L](https://arxiv.org/html/2609.03342#A12)\) confirm consistent improvements at the larger scale\. At 8B, the absolute gains over GRPO remain between 2\.02 and 2\.29 points across all four metrics, indicating stable transfer across model scales\.

### 6\.2Ablation Study

Figure 3:Ablation on Qwen3\-8B\-Base \(AIME 2026\), removing one GAR component at a time from \(a\) GAR\-GRPO and \(b\) GAR\-REINFORCE\+\+\. Activation weighting is essential for consistent gains across both optimizers\.Figure[3](https://arxiv.org/html/2609.03342#S6.F3)isolates the contribution of each GAR component on Qwen3\-8B across two optimizers\. Full GAR consistently achieves the best performance regardless of optimizer choice\. Dropping the outcome reward entirely \(GAR\-only\) degrades performance well below the baseline, confirming that the verifier gate is essential\. This result shows that gradient alignment is effective as a process\-level refinement of correctness but is not a substitute for the binary outcome signal\. Interestingly, using raw gradients without activation weighting yields inconsistent results: it helps with GRPO but slightly hurts with REINFORCE\+\+, suggesting that the unweighted gradient signal is noisy and optimizer\-sensitive\. The activation\-weighted formulation of Eq\. \([3](https://arxiv.org/html/2609.03342#S2.E3)\) resolves this instability, delivering reliable gains in both settings\. The agreement across both optimizer backbones further indicates that activation weighting is a structural component of the reward rather than an optimizer\-specific tuning effect\.

### 6\.3Reward Distribution

Figure 4:Cosine alignment distribution among correct rollouts at training steps 0, 100, 200, 400 \(Qwen3\-8B\)\. Red dashed lines mark means\.Figure[4](https://arxiv.org/html/2609.03342#S6.F4)shows how the cosine alignment distribution evolves during training\. The meanb⁡\(yi\)b\(y\_\{i\}\)increases from 0\.10 to 0\.42 while variance narrows, confirming progressive alignment with expert anchors\. Because the distribution contains only verifier\-passing rollouts, the fourfold increase cannot be attributed merely to a higher fraction of correct answers\. Instead, it reflects a redistribution within the correct subset toward trajectories whose gradient directions more closely match the expert solution\. The simultaneous reduction in variance suggests that this behavior becomes systematic across rollouts rather than being driven by a small number of highly aligned outliers\.

### 6\.4Computational Overhead

Figure 5:Per\-step wall\-clock time distribution \(Qwen3\-8B, 4×\\timesA100,K=16K\{=\}16, 100 steps\)\. GAR adds only 8\.3% overhead vialm\_head\-only backpropagation\.Figure[5](https://arxiv.org/html/2609.03342#S6.F5)compares per\-step training cost across gradient\-based methods\. GAR’s truncated backpropagation through only thelm\_headlayer adds just 8\.3% overhead over outcome\-only GRPO\. G2RL’s pairwise gradient diversity computation requires multi\-layer backpropagation \(\+37\.1%\), while Grad2Reward[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib35)performs a full backward pass through the Judge for each rubric \(\+43\.7%\)\. Thus, GAR incurs less than one quarter of G2RL’s additional cost and less than one fifth of Grad2Reward’s additional cost under the reported setup\. This efficiency is consistent with the design of GAR: gradients are truncated at the output projection layer, incorrect responses are removed by the verifier gate before gradient extraction, and each expert anchor is cached across the rollout group\. The measured overhead also falls within the analytical range derived in Appendix[J](https://arxiv.org/html/2609.03342#A10), connecting the implementation\-level cost model with observed wall\-clock behavior\.

### 6\.5Additional Analysis

We provide further empirical analysis in the appendix: hyperparameter sensitivity \(Appendix[K](https://arxiv.org/html/2609.03342#A11)\), analysis of alignment and solution diversity \(Appendix[M](https://arxiv.org/html/2609.03342#A13)\), and Qwen3\-8B general reasoning transfer results \(Appendix[L](https://arxiv.org/html/2609.03342#A12)\)\.

## 7Related Work

#### RL for LLM reasoning and reward design\.

Critic\-free policy gradient methods[Shao et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib2);[Yu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib12);[Fu et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib37);[Hu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib27);[Liu et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib33);[Melo et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib34)and direct RL from base models[Guo et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib1);[Zeng et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib14);[Wen et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib6);[Aggarwal and Welleck \(2025\)](https://arxiv.org/html/2609.03342#bib.bib13);[Yue et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib40)have shown that chain\-of\-thought reasoning[Wei et al\. \(2022\)](https://arxiv.org/html/2609.03342#bib.bib24)emerges from outcome rewards alone\. Process reward models[Lightman et al\. \(2023\)](https://arxiv.org/html/2609.03342#bib.bib3);[Uesato et al\. \(2022\)](https://arxiv.org/html/2609.03342#bib.bib32);[Wang et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib4);[Luo et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib5);[Cui et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib15);[Khalifa et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib16);[Zhao et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib17);[Wang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib38);[Cobbe et al\. \(2021\)](https://arxiv.org/html/2609.03342#bib.bib26);[Snell et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib30)offer denser supervision but require step\-level annotations or a separately trained verifier\. Classical potential\-based shaping[Ng et al\. \(1999\)](https://arxiv.org/html/2609.03342#bib.bib9);[Harutyunyan et al\. \(2015\)](https://arxiv.org/html/2609.03342#bib.bib10);[Devlin and Kudenko \(2012\)](https://arxiv.org/html/2609.03342#bib.bib11)establishes when shaped rewards preserve the optimal policy yet leaves the potential function unspecified\.

#### Gradient signals in LLM training\.

Gradient information has been used for data attribution[Koh and Liang \(2017\)](https://arxiv.org/html/2609.03342#bib.bib19);[Pruthi et al\. \(2020\)](https://arxiv.org/html/2609.03342#bib.bib28);[Park et al\. \(2023\)](https://arxiv.org/html/2609.03342#bib.bib20), curriculum learning[Mindermann et al\. \(2022\)](https://arxiv.org/html/2609.03342#bib.bib22);[Fifty et al\. \(2021\)](https://arxiv.org/html/2609.03342#bib.bib21), and representation engineering[Zou et al\. \(2023\)](https://arxiv.org/html/2609.03342#bib.bib23), with NTK theory[Jacot et al\. \(2018\)](https://arxiv.org/html/2609.03342#bib.bib18);[Mohamadi et al\. \(2023\)](https://arxiv.org/html/2609.03342#bib.bib29);[Tomihari \(2026\)](https://arxiv.org/html/2609.03342#bib.bib39)providing formal grounding\. Grad2Reward[Zhang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib35), G2RL[Liang et al\. \(2025\)](https://arxiv.org/html/2609.03342#bib.bib41), and GradAlign[Yang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib36)recently apply gradient signals to LLM reward design, exploration diversity, and data selection, respectively\. GAR differs from these by anchoring the gradient signal to expert chain\-of\-thought solutions, combining the information richness of reference\-based methods with the structural depth of gradient\-space operation, while requiring no external judge or offline annotations\. Unlike recent distillation approaches that study information transfer and on\-policy optimization granularity[Fang et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib45);[Li et al\. \(2026\)](https://arxiv.org/html/2609.03342#bib.bib46), GAR does not optimize toward a fixed reference distribution: it preserves the RL exploration loop and uses expert CoTs only as a gradient\-space reference signal\.

## 8Conclusion

We have presented Gradient\-Aligned Reward \(GAR\), which converts readily available expert chain\-of\-thought solutions into gradient\-space reference vectors and scores rollouts by cosine similarity with these anchors, providing dense process supervision within standard RLVR training at less than 9% wall\-clock overhead\. Experiments on four competition\-level math benchmarks with Qwen3\-4B and 8B base models show consistent pass@kkgains over GRPO, REINFORCE\+\+, and gradient\-based competitors, with positive transfer to GPQA Diamond and MMLU\-Pro\.

## Acknowledgments

This work is supported in part by the National Natural Science Foundation of China \(No\. 62372264 and No\. 92467203 \)\. Chaokun Wang is the corresponding author\.

## Limitations

While GAR demonstrates consistent improvements across two model scales and multiple benchmarks, it has not yet been deployed in an industrial production environment\.

## Ethics Statement

GAR operates exclusively on publicly available mathematical reasoning benchmarks with open licenses, and does not involve human subjects, private data, or dual\-use applications\.

## References

- Aggarwal and Welleck \(2025\)P\. Aggarwal and S\. WelleckL1: controlling how long a reasoning model thinks with reinforcement learning\.arXiv preprint arXiv:2503\.04697\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Cobbeet al\.\(2021\)K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano,et al\.Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Cuiet al\.\(2025\)G\. Cui, L\. Yuan, Z\. Wang, H\. Wang, Y\. Zhang, J\. Chen, W\. Li, B\. He, Y\. Fan, T\. Yu,et al\.Process reinforcement through implicit rewards\.arXiv preprint arXiv:2502\.01456\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Devlin and Kudenko \(2012\)S\. M\. Devlin and D\. KudenkoDynamic potential\-based reward shaping\.In11th International Conference on Autonomous Agents and Multiagent Systems \(AAMAS 2012\),pp\. 433–440\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Fanget al\.\(2026\)H\. Fang, T\. Zhang, T\. Zhuang, J\. Kong, K\. Gao, B\. Chen, L\. Zheng, S\. Xia, and K\. XuTowards distillation\-resistant large language models: an information\-theoretic perspective\.External Links:2602\.03396,[Link](https://arxiv.org/abs/2602.03396)Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Fiftyet al\.\(2021\)C\. Fifty, E\. Amid, Z\. Zhao, T\. Yu, R\. Anil, and C\. FinnEfficiently identifying task groupings for multi\-task learning\.Advances in Neural Information Processing Systems34,pp\. 27503–27516\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Fuet al\.\(2026\)X\. Fu, J\. Lin, Y\. Fang, B\. Zheng, C\. Hu, Z\. Shao, C\. Qin, L\. Pan, K\. Zeng, and X\. CaiMaspo: unifying gradient utilization, probability mass, and signal reliability for robust and sample\-efficient llm reasoning\.arXiv preprint arXiv:2602\.17550\.Cited by:[§4\.2](https://arxiv.org/html/2609.03342#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.03342#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Guoet al\.\(2025\)D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Harutyunyanet al\.\(2015\)A\. Harutyunyan, S\. Devlin, P\. Vrancx, and A\. NowéExpressing arbitrary reward functions as potential\-based advice\.InProceedings of the AAAI conference on artificial intelligence,Vol\.29\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Huet al\.\(2025\)J\. Hu, J\. K\. Liu, H\. Xu, and W\. ShenReinforce\+\+: stabilizing critic\-free policy optimization with global advantage normalization\.arXiv preprint arXiv:2501\.03262\.Cited by:[§4\.2](https://arxiv.org/html/2609.03342#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.03342#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Jacotet al\.\(2018\)A\. Jacot, F\. Gabriel, and C\. HonglerNeural tangent kernel: convergence and generalization in neural networks\.Advances in neural information processing systems31\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Khalifaet al\.\(2025\)M\. Khalifa, R\. Agarwal, L\. Logeswaran, J\. Kim, H\. Peng, M\. Lee, H\. Lee, and L\. WangProcess reward models that think\.arXiv preprint arXiv:2504\.16828\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Koh and Liang \(2017\)P\. W\. Koh and P\. LiangUnderstanding black\-box predictions via influence functions\.InInternational conference on machine learning,pp\. 1885–1894\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Kojimaet al\.\(2022\)T\. Kojima, S\. S\. Gu, M\. Reid, Y\. Matsuo, and Y\. IwasawaLarge language models are zero\-shot reasoners\.Advances in neural information processing systems35,pp\. 22199–22213\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1)\.
- Liet al\.\(2024\)J\. Li, E\. Beeching, L\. Tunstall, B\. Lipkin, R\. Soletskyi, S\. Huang, K\. Rasul, L\. Yu, A\. Q\. Jiang, Z\. Shen,et al\.Numinamath: the largest public dataset in ai4maths with 860k pairs of competition math problems and solutions\.Hugging Face repository13\(9\),pp\. 9\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§2\.3](https://arxiv.org/html/2609.03342#S2.SS3.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2609.03342#S4.SS1.p1.1)\.
- Liet al\.\(2026\)Y\. Li, L\. Zheng, Y\. Yu, W\. Zhou, X\. Zhong, X\. Hu, J\. Jin, H\. Yuan, and T\. FengFilter, then reweight: rethinking optimization granularity in on\-policy distillation\.External Links:2606\.02684,[Link](https://arxiv.org/abs/2606.02684)Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Lianget al\.\(2025\)Z\. Liang, S\. Lu, W\. Yu, K\. Panaganti, Y\. Zhou, H\. Mi, and D\. YuCan llms guide their own exploration? gradient\-guided reinforcement learning for llm reasoning\.arXiv preprint arXiv:2512\.15687\.Cited by:[§4\.2](https://arxiv.org/html/2609.03342#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.03342#S5.SS1.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Lightmanet al\.\(2023\)H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. CobbeLet’s verify step by step\.InThe twelfth international conference on learning representations,Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, C\. Chen, W\. Li, P\. Qi, T\. Pang, C\. Du, W\. S\. Lee, and M\. LinUnderstanding r1\-zero\-like training: a critical perspective\.arXiv preprint arXiv:2503\.20783\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Luoet al\.\(2024\)L\. Luo, Y\. Liu, R\. Liu, S\. Phatale, M\. Guo, H\. Lara, Y\. Li, L\. Shu, Y\. Zhu, L\. Meng,et al\.Improve mathematical reasoning in language models by automated process supervision\.arXiv preprint arXiv:2406\.06592\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Meloet al\.\(2025\)L\. C\. Melo, A\. Abate, and Y\. GalStabilizing policy gradients for sample\-efficient reinforcement learning in llm reasoning\.arXiv preprint arXiv:2510\.00819\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Mindermannet al\.\(2022\)S\. Mindermann, J\. M\. Brauner, M\. T\. Razzak, M\. Sharma, A\. Kirsch, W\. Xu, B\. Höltgen, A\. N\. Gomez, A\. Morisot, S\. Farquhar,et al\.Prioritized training on points that are learnable, worth learning, and not yet learnt\.InInternational Conference on Machine Learning,pp\. 15630–15649\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Mohamadiet al\.\(2023\)M\. A\. Mohamadi, W\. Bae, and D\. J\. SutherlandA fast, well\-founded approximation to the empirical neural tangent kernel\.InInternational conference on machine learning,pp\. 25061–25081\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Nget al\.\(1999\)A\. Y\. Ng, D\. Harada, and S\. RussellPolicy invariance under reward transformations: theory and application to reward shaping\.InIcml,Vol\.99,pp\. 278–287\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Parket al\.\(2023\)S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. MadryTrak: attributing model behavior at scale\.arXiv preprint arXiv:2303\.14186\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Pruthiet al\.\(2020\)G\. Pruthi, F\. Liu, S\. Kale, and M\. SundararajanEstimating training data influence by tracing gradient descent\.Advances in Neural Information Processing Systems33,pp\. 19920–19930\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Setluret al\.\(2024\)A\. Setlur, C\. Nagpal, A\. Fisch, X\. Geng, J\. Eisenstein, R\. Agarwal, A\. Agarwal, J\. Berant, and A\. KumarRewarding progress: scaling automated process verifiers for llm reasoning\.arXiv preprint arXiv:2410\.08146\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[Appendix E](https://arxiv.org/html/2609.03342#A5.p1.2),[§1](https://arxiv.org/html/2609.03342#S1.p1.1),[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§4\.2](https://arxiv.org/html/2609.03342#S4.SS2.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Snellet al\.\(2024\)C\. Snell, J\. Lee, K\. Xu, and A\. KumarScaling llm test\-time compute optimally can be more effective than scaling model parameters\.arXiv preprint arXiv:2408\.03314\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Tomihari \(2026\)A\. TomihariLearning dynamics in rl post\-training for language models\.arXiv preprint arXiv:2601\.04670\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Uesatoet al\.\(2022\)J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. HigginsSolving math word problems with process\-and outcome\-based feedback\.arXiv preprint arXiv:2211\.14275\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2026\)J\. Wang, L\. Zhu, T\. Weng, S\. Wu, H\. Tan, J\. Chen, C\. Tao, H\. Bai, L\. Hou, L\. Shang,et al\.GRPO\-vps: enhancing group relative policy optimization with verifiable process supervision for effective reasoning\.arXiv preprint arXiv:2604\.20659\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)P\. Wang, L\. Li, Z\. Shao, R\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. SuiMath\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9426–9439\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Weiet al\.\(2022\)J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Wenet al\.\(2025\)L\. Wen, Y\. Cai, F\. Xiao, X\. He, Q\. An, Z\. Duan, Y\. Du, J\. Liu, T\. Tanglifu, X\. Lv,et al\.Light\-r1: curriculum sft, dpo and rl for long cot from scratch and beyond\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 6: Industry Track\),pp\. 318–327\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p2.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§4\.1](https://arxiv.org/html/2609.03342#S4.SS1.p1.1)\.
- Yanget al\.\(2026\)N\. Yang, W\. Du, W\. Sun, S\. Welleck, and Y\. YangGradAlign: gradient\-aligned data selection for llm reinforcement learning\.arXiv preprint arXiv:2602\.21492\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Yuet al\.\(2025\)Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.Dapo: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1),[§4\.2](https://arxiv.org/html/2609.03342#S4.SS2.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Yueet al\.\(2025\)Y\. Yue, Y\. Yuan, Q\. Yu, X\. Zuo, R\. Zhu, W\. Xu, J\. Chen, C\. Wang, T\. Fan, Z\. Du,et al\.Vapo: efficient and reliable reinforcement learning for advanced reasoning tasks\.arXiv preprint arXiv:2504\.05118\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Zenget al\.\(2025\)W\. Zeng, Y\. Huang, Q\. Liu, W\. Liu, K\. He, Z\. Ma, and J\. HeSimplerl\-zoo: investigating and taming zero reinforcement learning for open base models in the wild\.arXiv preprint arXiv:2503\.18892\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2026\)Z\. Zhang, A\. Lu, Y\. Zeng, Z\. Shan, J\. Guo, L\. Li, Y\. Li, and K\. RenGrad2Reward: from sparse judgment to dense rewards for improving open\-ended llm reasoning\.arXiv preprint arXiv:2602\.01791\.Cited by:[§4\.2](https://arxiv.org/html/2609.03342#S4.SS2.p1.1),[§5\.1](https://arxiv.org/html/2609.03342#S5.SS1.p1.1),[§6\.4](https://arxiv.org/html/2609.03342#S6.SS4.p1.1),[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.
- Zhaoet al\.\(2026\)J\. Zhao, R\. Liu, K\. Zhang, Z\. Zhou, J\. Gao, D\. Li, J\. Lyu, Z\. Qian, B\. Qi, X\. Li,et al\.Genprm: scaling test\-time compute of process reward models via generative reasoning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 34932–34940\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px1.p1.1)\.
- Zhenget al\.\(2025a\)L\. Zheng, C\. Wang, C\. Chen, J\. Zhang, C\. Wu, Z\. Song, S\. Yan, Z\. Liu, and H\. LiLAGCL4Rec: when LLMs activate interactions potential in graph contrastive learning for recommendation\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 1163–1184\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.61/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.61),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1)\.
- Zhenget al\.\(2025b\)L\. Zheng, C\. Wang, Z\. Song, C\. Wu, S\. Yan, J\. Zhang, and Z\. LiuNegative feedback really matters: signed dual\-channel graph contrastive learning framework for recommendation\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38, Main Conference,pp\. 107595–107624\.External Links:[Document](https://dx.doi.org/10.52202/085713-3589),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/9a92ea37efa0d290bd7015558166c056-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1)\.
- Zhenget al\.\(2026\)L\. Zheng, J\. Zhang, C\. Chen, C\. Wang, H\. Li, Y\. Li, Y\. Mao, S\. Yan, Z\. Song, Z\. Feng, Z\. Kang, Z\. Chen, H\. Zhang, Q\. Liu, L\. Wang, and Z\. LiuWhat should i cite? a rag benchmark for academic citation prediction\.InProceedings of the ACM Web Conference 2026,WWW ’26,New York, NY, USA,pp\. 1852–1863\.External Links:ISBN 9798400723070,[Document](https://dx.doi.org/10.1145/3774904.3792075),[Link](https://doi.org/10.1145/3774904.3792075)Cited by:[§1](https://arxiv.org/html/2609.03342#S1.p1.1)\.
- Zouet al\.\(2023\)A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika, A\. Dombrowski,et al\.Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§7](https://arxiv.org/html/2609.03342#S7.SS0.SSS0.Px2.p1.1)\.

## Appendix ANotation

We denote byπθ\\pi\_\{\\theta\}an autoregressive language model policy parameterized byθ\\theta, where for any promptxxand responsey=\(y1,…,yT\)y=\(y\_\{1\},\\dots,y\_\{T\}\)of lengthTTthe policy factorizes asπθ​\(y∣x\)=∏t=1Tπθ​\(yt∣x,y<t\)\\pi\_\{\\theta\}\(y\\mid x\)=\\prod\_\{t=1\}^\{T\}\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)\. The model consists of an embedding matrix, a stack ofLLTransformer layers, and an output projection layer with weightWo∈ℝ\|𝒱\|×dW\_\{o\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}and optional bias𝐛o∈ℝ\|𝒱\|\\mathbf\{b\}\_\{o\}\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\}, where𝒱\\mathcal\{V\}is the vocabulary andddthe hidden dimension\. For a prompt\-response pair\(x,y\)\(x,y\), we write𝐡t∈ℝd\\mathbf\{h\}\_\{t\}\\in\\mathbb\{R\}^\{d\}for the hidden state at positionttimmediately before the output projection, and𝐇=\(𝐡1,…,𝐡n\)⊤∈ℝn×d\\mathbf\{H\}=\(\\mathbf\{h\}\_\{1\},\\dots,\\mathbf\{h\}\_\{n\}\)^\{\\top\}\\in\\mathbb\{R\}^\{n\\times d\}for the stacked hidden states of the full sequence of lengthn=\|x\|\+\|y\|n=\|x\|\+\|y\|\. The per\-token log\-likelihood of a response under the policy islog⁡πθ​\(yt∣x,y<t\)=log⁡softmax​\(Wo​𝐡t\+𝐛o\)yt\\log\\pi\_\{\\theta\}\(y\_\{t\}\\mid x,y\_\{<t\}\)=\\log\\mathrm\{softmax\}\(W\_\{o\}\\mathbf\{h\}\_\{t\}\+\\mathbf\{b\}\_\{o\}\)\_\{y\_\{t\}\}\. For brevity, we denote by𝒯y⊆\{1,…,n\}\\mathcal\{T\}\_\{y\}\\subseteq\\\{1,\\dots,n\\\}the set of token positions that correspond to the response span ofyy\(or, when available, the explicit thinking span delimited by the special tags⟨think⟩\\langle\\text\{think\}\\rangleand⟨/think⟩\\langle/\\text\{think\}\\rangle\)\.

## Appendix BAlgorithm

Algorithm[1](https://arxiv.org/html/2609.03342#algorithm1)presents the complete GAR online reward computation within a single GRPO step\. For each prompt, the procedure first gates on outcome verification, skipping gradient computation for incorrect or malformed responses \(lines 4–6\)\. For each verified\-correct rollout, it captures the hidden state via a forward pre\-hook on the output projection layer and computes the truncated gradient\-activation vector through Eq\. \([3](https://arxiv.org/html/2609.03342#S2.E3)\)–\([4](https://arxiv.org/html/2609.03342#S2.E4)\) \(lines 7–8\)\. Expert\-anchor gradient vectors are computed on demand and cached per prompt to avoid redundant computation across theKKrollouts \(lines 9–11\)\. Each correct rollout is then scored by cosine similarity with the expert anchor via Eq\. \([5](https://arxiv.org/html/2609.03342#S2.E5)\) \(line 12\)\. Finally, the bonuses are group\-centered over the correct subset and combined with the base reward through the non\-negatively clipped formulation of Eq\. \([6](https://arxiv.org/html/2609.03342#S2.E6)\) \(lines 15–17\)\.

Algorithm 1Gradient\-Aligned Reward \(GAR\) online computation within a single GRPO step\.Input:Model

πθ\\pi\_\{\\theta\}, rollout batch

\{\(xj,\{yj,1,…,yj,K\}\)\}\\\{\(x\_\{j\},\\\{y\_\{j,1\},\\dots,y\_\{j,K\}\\\}\)\\\}, outcome verifier, anchor

a⁡\(xj\)a\(x\_\{j\}\)per prompt

Output:GAR rewards

\{rGAR​\(xj,yj,i\)\}\\\{r\_\{\\text\{GAR\}\}\(x\_\{j\},y\_\{j,i\}\)\\\}
Initialize anchor gradient cache

𝒞←\{\}\\mathcal\{C\}\\leftarrow\\\{\\\};

foreach*promptxjx\_\{j\}in batch*do

foreach*responseyj,iy\_\{j,i\},i=1,…,Ki=1,\\dots,K*do

if*Verify​\(yj,i\)=False\\textup\{Verify\}\(y\_\{j,i\}\)=\\textup\{False\}orFormatValid​\(yj,i\)=False\\textup\{FormatValid\}\(y\_\{j,i\}\)=\\textup\{False\}*then

rGAR​\(xj,yj,i\)←p⁡\(yj,i\)r\_\{\\text\{GAR\}\}\(x\_\{j\},y\_\{j,i\}\)\\leftarrow p\(y\_\{j,i\}\);

continue;

Capture hidden state

𝐇\\mathbf\{H\}via forward pre\-hook on output layer;

Compute

𝐯i←GradAct​\(𝐇,Wo,𝒯yj,i\)\\mathbf\{v\}\_\{i\}\\leftarrow\\text\{GradAct\}\(\\mathbf\{H\},W\_\{o\},\\mathcal\{T\}\_\{y\_\{j,i\}\}\);//Eq\. \([3](https://arxiv.org/html/2609.03342#S2.E3)\)\-\-\([4](https://arxiv.org/html/2609.03342#S2.E4)\)

if*a⁡\(xj\)∉𝒞a\(x\_\{j\}\)\\notin\\mathcal\{C\}*then

𝒞⁡\[a⁡\(xj\)\]←GradAct​\(𝐇a,Wo,𝒯a\)\\mathcal\{C\}\[a\(x\_\{j\}\)\]\\leftarrow\\text\{GradAct\}\(\\mathbf\{H\}\_\{a\},W\_\{o\},\\mathcal\{T\}\_\{a\}\);

b⁡\(yj,i\)←𝐯i⊤​𝒞​\[a⁡\(xj\)\]b\(y\_\{j,i\}\)\\leftarrow\\mathbf\{v\}\_\{i\}^\{\\top\}\\mathcal\{C\}\[a\(x\_\{j\}\)\];//Eq\. \([5](https://arxiv.org/html/2609.03342#S2.E5)\)

Compute

b¯j←1\|𝒫⁡\(xj\)\|​∑y∈𝒫⁡\(xj\)b⁡\(y\)\\bar\{b\}\_\{j\}\\leftarrow\\frac\{1\}\{\|\\mathcal\{P\}\(x\_\{j\}\)\|\}\\sum\_\{y\\in\\mathcal\{P\}\(x\_\{j\}\)\}b\(y\);

foreach*correctyj,i∈𝒫⁡\(xj\)y\_\{j,i\}\\in\\mathcal\{P\}\(x\_\{j\}\)*do

rGAR​\(xj,yj,i\)←rbase\+β⋅max⁡\(0,b⁡\(yj,i\)−b¯j\)\+p⁡\(yj,i\)r\_\{\\text\{GAR\}\}\(x\_\{j\},y\_\{j,i\}\)\\leftarrow r\_\{\\text\{base\}\}\+\\beta\\cdot\\max\(0,b\(y\_\{j,i\}\)\-\\bar\{b\}\_\{j\}\)\+p\(y\_\{j,i\}\);//Eq\. \([6](https://arxiv.org/html/2609.03342#S2.E6)\)

return

\{rGAR\}\\\{r\_\{\\text\{GAR\}\}\\\};

## Appendix CImplementation Details

This appendix provides the low\-level engineering details for deploying GAR within the SLIME / Megatron training stack\.

### C\.1Forward Pre\-Hook and Hidden State Capture

The forward pre\-hook is registered once at actor initialization time and captures the hidden state tensor passed as input to the output projection\. In tensor\-parallel configurations with sequence parallelism enabled, the hook fires on each TP rank and captures only the local shard\. We subsequently all\-gather the shards across the tensor\-parallel group to reconstruct the full hidden state of shape\(n,d\)\(n,d\), wherennis the padded sequence length\. The reconstruction is necessary because the gradient\-activation signal is defined over the full response span, whereas sequence parallelism typically partitions the sequence axis\. To avoid re\-gathering the hidden state for every rollout, we batch the forward pass across rollouts that share the same prompt and call the hook only once per batch\.

### C\.2Truncated Backpropagation and Loss Computation

The cross\-entropy loss of Eq\. \([2](https://arxiv.org/html/2609.03342#S2.E2)\) is computed through the same Megatron output\-layer wrapper used for training, but with per\-token reduction so that losses can be summed over the response span alone\. We then invoke a non\-graph\-creating, non\-graph\-retaining backward call to obtain∂ℒ/∂𝐇~\\partial\\mathcal\{L\}/\\partial\\widetilde\{\\mathbf\{H\}\}, after which the computational graph is immediately released\. Although the output\-layer weightWoW\_\{o\}is large \(\|𝒱\|×d\|\\mathcal\{V\}\|\\times d\), the backward pass does not accumulate gradients inWoW\_\{o\}itself because𝐇~\\widetilde\{\\mathbf\{H\}\}is the only leaf carrying a gradient requirement, andWoW\_\{o\}is accessed as a frozen constant through the standard column\-parallel linear call\.

### C\.3Anchor Caching and Deduplication

Anchor gradients are cached in a dictionary keyed by the tuple\(tokenized\_prompt,anchor\_text,max\_resp\_tokens\)\(\\text\{tokenized\\\_prompt\},\\text\{anchor\\\_text\},\\text\{max\\\_resp\\\_tokens\}\)for the duration of a rollout batch\. This design ensures that within a single GRPO step, an expert anchor that is shared across allKKrollouts contributes only a single gradient computation rather thanKK\.

### C\.4Anchor Text Extraction

When an anchor is provided as a full decoded response, we extract the supervised span by first attempting to locate the content between⟨think⟩\\langle\\text\{think\}\\rangleand⟨/think⟩\\langle/\\text\{think\}\\rangletags; if these tags are absent, we fall back to stripping trailing special tokens while preserving a small set of semantically meaningful closing tags\. This mirrors the extraction logic applied to candidate responses, so that the candidate and anchor gradient vectors are computed over comparable surface forms and are not contaminated by formatting\-specific tokens such as⟨bos⟩\\langle\\text\{bos\}\\rangleor⟨eos⟩\\langle\\text\{eos\}\\rangle\.

### C\.5Failure Handling and Graceful Degradation

Several edge cases are handled defensively\. If an anchor text is absent or empty, the candidate falls back to the outcome\-only reward plus penalties\. If the gradient norm is below10−810^\{\-8\}, the corresponding vector is set to zero and the bonus reduces to zero, equivalent to the unshaped baseline\. If the verifier reports format invalidity, the response is excluded from gradient computation regardless of whether the numerical answer is correct, because numerical correctness without the mandated think/answer structure is considered suspect and is therefore not rewarded\.

## Appendix DReward Formulation Details

This appendix expands on the design choices in the GAR reward formulation of Section[2\.3](https://arxiv.org/html/2609.03342#S2.SS3)\.

### D\.1Verifier Gate

The verifier gate serves two complementary purposes\. First, it ensures that GAR never rewards an incorrect answer regardless of how closely its gradient aligns with any expert anchor, eliminating the possibility that the policy exploits spurious directions in gradient space to inflate the reward without solving the underlying problem\. Second, it confines the additional cost of GAR to the verifier\-passing fraction of rollouts, which is particularly valuable in the initial low\-accuracy phase of training when the majority of rollouts fail verification and would otherwise trigger unnecessary truncated\-backward computation\.

### D\.2Anchor Source Specification

In our SLIME\-based implementation, the loader inspects each per\-prompt metadata record for the expert anchor \(an expert\-written solution or verified chain\-of\-thought trace\), accepting a configurable, ordered list of field names so that datasets following different naming conventions are supported without additional configuration\. In our experiments with NuminaMath\-CoT, each problem provides exactly one chain\-of\-thought solution, which serves as the sole anchor; the underlying cache key structure is described in Appendix[C\.3](https://arxiv.org/html/2609.03342#A3.SS3)\.

### D\.3Anchor\-Cache Cost Reduction

For each promptxxwithKKrollouts that share the same anchor, a naive implementation would re\-compute the anchor gradient vector for every rollout, incurring a cost of𝒪⁡\(K\)\\mathcal\{O\}\(K\)truncated forward\-backward pairs per prompt\. Caching the anchor vector at the start of the batch and reusing it across allKKrollouts reduces this cost to𝒪⁡\(1\)\\mathcal\{O\}\(1\), aKK\-fold reduction that is the single largest source of anchor\-pathway savings in practice and corresponds to the anchor\-cache amortization term in the overhead analysis of Appendix[J](https://arxiv.org/html/2609.03342#A10)\.

### D\.4Group Centering of the Bonus

Following the group\-relative philosophy of GRPO, the bonusb⁡\(yi\)b\(y\_\{i\}\)of Eq\. \([5](https://arxiv.org/html/2609.03342#S2.E5)\) is centered over the correct subset𝒫⁡\(x\)=\{yj:rraw​\(x,yj\)\>0\}\\mathcal\{P\}\(x\)=\\\{y\_\{j\}:r\_\{\\text\{raw\}\}\(x,y\_\{j\}\)\>0\\\}of the rollout group before being passed to the final reward of Eq\. \([6](https://arxiv.org/html/2609.03342#S2.E6)\):

b^​\(yi\)=b⁡\(yi\)−1\|𝒫⁡\(x\)\|​∑yj∈𝒫⁡\(x\)b⁡\(yj\)\.\\hat\{b\}\(y\_\{i\}\)=b\(y\_\{i\}\)\-\\frac\{1\}\{\|\\mathcal\{P\}\(x\)\|\}\\sum\_\{y\_\{j\}\\in\\mathcal\{P\}\(x\)\}b\(y\_\{j\}\)\.\(9\)The centering removes the confound that easy prompts systematically attract higher cosine scores than hard ones and makes the bonus measure how much betteryiy\_\{i\}aligns with the expert reference than the other correct rollouts of the same prompt\. The prompt\-group mean is computed only over𝒫⁡\(x\)\\mathcal\{P\}\(x\)rather than over the full rollout group of sizeKK: incorrect responses are gated out atrraw=0r\_\{\\text\{raw\}\}=0and produce no cosine score, so including them in the centering with a default bonus of zero would bias the empirical mean downward and inflate the centered bonusb^\\hat\{b\}of every correct rollout, defeating the purpose of group normalization\. Restricting the mean to𝒫⁡\(x\)\\mathcal\{P\}\(x\)keeps the centering unbiased over the set of trajectories that actually receive the alignment bonus, which is the precise condition under which Proposition[3](https://arxiv.org/html/2609.03342#Thmproposition3)holds\.

### D\.5Role of the Non\-Negative Clip and Format Penalty

Themax⁡\(0,⋅\)\\max\(0,\\cdot\)clip in Eq\. \([6](https://arxiv.org/html/2609.03342#S2.E6)\) keeps the GAR reward weakly above the outcome\-only baseline on every correct trajectory\. Concretely, a correct response with below\-mean alignment receives onlyrbase\+p⁡\(yi\)r\_\{\\text\{base\}\}\+p\(y\_\{i\}\), identical to what it would receive under outcome\-only RLVR augmented with the same format penalty, while a correct response with above\-mean alignment receives a strictly larger reward\. This non\-decreasing\-in\-alignment structure prevents the shaped reward from ever penalizing a correct response relative to outcome\-only training and is essential for the safe reward\-shaping guarantee of Theorem[3](https://arxiv.org/html/2609.03342#Thmtheorem3)\. The format penaltyp⁡\(yi\)≤0p\(y\_\{i\}\)\\leq 0applies to responses that violate the prescribed output structure \(typically a missing or malformed think/answer delimiter\) and is applied symmetrically to both correct and incorrect responses, so that the policy is encouraged to produce well\-formed reasoning traces alongside correct final answers\.

## Appendix EIntegration with GRPO

GAR builds on Group Relative Policy Optimization \(GRPO\)[Shao et al\. \(2024\)](https://arxiv.org/html/2609.03342#bib.bib2), which samplesKKresponses\{y1,…,yK\}\\\{y\_\{1\},\\dots,y\_\{K\}\\\}per promptxxand computes a group\-relative advantage

Ai=r⁡\(x,yi\)−μxσx\+ϵ,A\_\{i\}=\\frac\{r\(x,y\_\{i\}\)\-\\mu\_\{x\}\}\{\\sigma\_\{x\}\+\\epsilon\},\(10\)whereμx\\mu\_\{x\}andσx\\sigma\_\{x\}are the intra\-group mean and standard deviation of the rewards\. The policy is updated via a clipped surrogate objective with KL regularization:

ℒGRPO\\displaystyle\\mathcal\{L\}\_\{\\text\{GRPO\}\}=𝔼x,yi\[min\(ρiAi,clip\(ρi,1−ϵc,1\+ϵc\)Ai\)\\displaystyle=\\mathbb\{E\}\_\{x,y\_\{i\}\}\\Big\[\\min\\\!\\big\(\\rho\_\{i\}A\_\{i\},\\;\\text\{clip\}\(\\rho\_\{i\},1\{\-\}\\epsilon\_\{c\},1\{\+\}\\epsilon\_\{c\}\)A\_\{i\}\\big\)\(11\)−βKLDKL\(πθ∥πref\)\],\\displaystyle\-\\beta\_\{\\text\{KL\}\}D\_\{\\text\{KL\}\}\(\\pi\_\{\\theta\}\\\|\\pi\_\{\\text\{ref\}\}\)\\Big\],whereρi=πθ​\(yi∣x\)/πold​\(yi∣x\)\\rho\_\{i\}=\\pi\_\{\\theta\}\(y\_\{i\}\\mid x\)/\\pi\_\{\\text\{old\}\}\(y\_\{i\}\\mid x\)\. Under the outcome\-only reward, all correct responses share reward11, collapsing the intra\-correct advantage to

Aioutcome=1−Kc/K\(Kc/K\)​\(1−Kc/K\)\+ϵ,A\_\{i\}^\{\\text\{outcome\}\}=\\frac\{1\-K\_\{c\}/K\}\{\\sqrt\{\(K\_\{c\}/K\)\(1\-K\_\{c\}/K\)\}\+\\epsilon\},\(12\)which is identical across allKcK\_\{c\}correct trajectories\.

GAR is implemented as an online reward hook within the GRPO training pipeline and is activated by pointing SLIME’s custom\-reward entry point to the GAR module\. The integration proceeds in the following stages\. First, the rollout engine \(SGLang\-based\) generatesKKresponses for each prompt and evaluates them with the outcome verifier, producing the binary rewardrrawr\_\{\\text\{raw\}\}\. Second, the rollout data, including token sequences, response lengths, raw rewards, and the metadata carrying the anchor texts, is forwarded from the rollout workers to the actor through the standard SLIME sample\-metadata channel\. Third, before advantage computation, the actor invokes the GAR reward hook, which performs the truncated gradient computation described above for each verified\-correct response and its anchors, computes the prompt\-group normalized bonus, and overwrites the raw reward with the GAR rewardrGARr\_\{\\text\{GAR\}\}\. Fourth, the standard GRPO advantage computation and policy update proceed using the enriched reward signal through Eq\. \([10](https://arxiv.org/html/2609.03342#A5.E10)\)–\([11](https://arxiv.org/html/2609.03342#A5.E11)\)\.

Because GAR performs gradient computation only through the output projection layer and only for responses that pass the verifier gate, its computational cost is a small fraction of the full model forward\-backward pass\. The gradient vectors have dimensionalitydd\(the hidden size of the model\), and all operations are local to the actor process, requiring no additional inter\-node communication beyond the standard tensor\-parallel all\-gather used to reconstruct the full sequence of hidden states\. We make the following parallelism assumptions in the current implementation: the model uses the packed token\-sequence \(thd\) attention format, the context\-parallel size equals one, GAR executes only on the last pipeline stage, and only a single Megatron model chunk is supported\. Each of these assumptions is a convenience rather than a fundamental constraint and could be relaxed with additional engineering\.

## Appendix FProofs

We first state the output\-layer gradient derivation referenced in Section[3\.1](https://arxiv.org/html/2609.03342#S3.SS1)\. Letℒ⁡\(Wo,x,y\)\\mathcal\{L\}\(W\_\{o\};x,y\)denote the truncated cross\-entropy loss of Eq\. \([2](https://arxiv.org/html/2609.03342#S2.E2)\) viewed as a function ofWoW\_\{o\}with hidden states held fixed\. The per\-example output\-layer gradient is

𝐠o​\(x,y\)=∇Woℒ​\(Wo,x,y\)∈ℝ\|𝒱\|×d\.\\mathbf\{g\}\_\{o\}\(x,y\)=\\nabla\_\{W\_\{o\}\}\\mathcal\{L\}\(W\_\{o\};x,y\)\\in\\mathbb\{R\}^\{\|\\mathcal\{V\}\|\\times d\}\.\(13\)Straightforward differentiation yields the rank\-one expansion

𝐠o​\(x,y\)\\displaystyle\\mathbf\{g\}\_\{o\}\(x,y\)=1\|𝒯y\|​∑t∈𝒯y\(𝐩t−𝐞yt\+1\)⊗𝐡t,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\big\(\\mathbf\{p\}\_\{t\}\-\\mathbf\{e\}\_\{y\_\{t\+1\}\}\\big\)\\otimes\\mathbf\{h\}\_\{t\},\(14\)𝐩t\\displaystyle\\mathbf\{p\}\_\{t\}=softmax⁡\(Wo​𝐡t\+𝐛o\)\.\\displaystyle=\\mathrm\{softmax\}\(W\_\{o\}\\mathbf\{h\}\_\{t\}\+\\mathbf\{b\}\_\{o\}\)\.
### F\.1Proof of Proposition[1](https://arxiv.org/html/2609.03342#Thmproposition1)

By the chain rule applied to the scaling𝐡t↦\(1\+η\)​𝐡t\\mathbf\{h\}\_\{t\}\\mapsto\(1\+\\eta\)\\mathbf\{h\}\_\{t\},

dd​η​ℒ​\(\(1\+η\)​𝐇y\)\|η=0\\displaystyle\\frac\{d\}\{d\\eta\}\\mathcal\{L\}\\big\(\(1\+\\eta\)\\mathbf\{H\}\_\{y\}\\big\)\\Big\|\_\{\\eta=0\}=1\|𝒯y\|​∑t∈𝒯y𝐆t⊤​𝐡t\\displaystyle=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\mathbf\{G\}\_\{t\}^\{\\top\}\\mathbf\{h\}\_\{t\}\(15\)=1\|𝒯y\|​∑t∈𝒯y𝟏⊤​\(𝐆t⊙𝐡t\),\\displaystyle=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\mathbf\{1\}^\{\\top\}\(\\mathbf\{G\}\_\{t\}\\odot\\mathbf\{h\}\_\{t\}\),where the second equality follows from𝐆t⊤​𝐡t=∑jGt,j​ht,j=𝟏⊤​\(𝐆t⊙𝐡t\)\\mathbf\{G\}\_\{t\}^\{\\top\}\\mathbf\{h\}\_\{t\}=\\sum\_\{j\}G\_\{t,j\}h\_\{t,j\}=\\mathbf\{1\}^\{\\top\}\(\\mathbf\{G\}\_\{t\}\\odot\\mathbf\{h\}\_\{t\}\)\. This is precisely the claim of Proposition[1](https://arxiv.org/html/2609.03342#Thmproposition1)\. ∎

### F\.2Proof of Theorem[1](https://arxiv.org/html/2609.03342#Thmtheorem1)

We formalize the connection between the gradient\-activation cosine similarity and NTK similarity through a three\-step argument\.

#### Step 1: Rank\-one expansion of the NTK inner product\.

By straightforward differentiation, the output\-layer gradient admits the rank\-one decomposition𝐠o​\(x,y\)=1\|𝒯y\|​∑t∈𝒯y𝜹ty⊗𝐡ty\\mathbf\{g\}\_\{o\}\(x,y\)=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\boldsymbol\{\\delta\}\_\{t\}^\{y\}\\otimes\\mathbf\{h\}\_\{t\}^\{y\}, where𝜹ty=𝐩t−𝐞yt\+1∈ℝV\\boldsymbol\{\\delta\}\_\{t\}^\{y\}=\\mathbf\{p\}\_\{t\}\-\\mathbf\{e\}\_\{y\_\{t\+1\}\}\\in\\mathbb\{R\}^\{V\}and𝐡ty∈ℝd\\mathbf\{h\}\_\{t\}^\{y\}\\in\\mathbb\{R\}^\{d\}\. The Frobenius inner product therefore expands as

Θo​\(y,a\)\\displaystyle\\Theta\_\{o\}\(y,a\)=⟨𝐠o​\(x,y\),𝐠o​\(x,a\)⟩F\\displaystyle=\\langle\\mathbf\{g\}\_\{o\}\(x,y\),\\mathbf\{g\}\_\{o\}\(x,a\)\\rangle\_\{F\}\(16\)=1\|𝒯y\|​\|𝒯a\|​∑t∈𝒯y∑s∈𝒯a\(𝜹ty⊤​𝜹sa\)​\(𝐡ty⊤​𝐡sa\)\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\|\\mathcal\{T\}\_\{a\}\|\}\\sum\_\{t\\in\\mathcal\{T\}\_\{y\}\}\\sum\_\{s\\in\\mathcal\{T\}\_\{a\}\}\\big\(\\boldsymbol\{\\delta\}\_\{t\}^\{y\\top\}\\boldsymbol\{\\delta\}\_\{s\}^\{a\}\\big\)\\big\(\\mathbf\{h\}\_\{t\}^\{y\\top\}\\mathbf\{h\}\_\{s\}^\{a\}\\big\)\.

#### Step 2: Mean\-residual splitting with bounded dispersion\.

Write𝜹ty=𝜹¯y\+𝜹~ty\\boldsymbol\{\\delta\}\_\{t\}^\{y\}=\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\+\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}and𝐡ty=𝐡¯y\+𝐡~ty\\mathbf\{h\}\_\{t\}^\{y\}=\\bar\{\\mathbf\{h\}\}^\{y\}\+\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}, where𝜹¯y=1\|𝒯y\|​∑t𝜹ty\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\}\\boldsymbol\{\\delta\}\_\{t\}^\{y\}and𝜹~ty\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}is the mean\-zero residual satisfying‖𝜹~ty‖2≤κy\\\|\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}\\\|\_\{2\}\\leq\\kappa\_\{y\}by assumption, and analogously for𝐡¯y,𝐡~ty\\bar\{\\mathbf\{h\}\}^\{y\},\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}\. Substituting into Eq\. \([16](https://arxiv.org/html/2609.03342#A6.E16)\) and expanding the product\(𝜹¯y\+𝜹~ty\)⊤​\(𝜹¯a\+𝜹~sa\)⋅\(𝐡¯y\+𝐡~ty\)⊤​\(𝐡¯a\+𝐡~sa\)\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\+\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}\)^\{\\top\}\(\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\+\\tilde\{\\boldsymbol\{\\delta\}\}\_\{s\}^\{a\}\)\\cdot\(\\bar\{\\mathbf\{h\}\}^\{y\}\+\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}\)^\{\\top\}\(\\bar\{\\mathbf\{h\}\}^\{a\}\+\\tilde\{\\mathbf\{h\}\}\_\{s\}^\{a\}\)yields the leading term\(𝜹¯y⊤​𝜹¯a\)​\(𝐡¯y⊤​𝐡¯a\)\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\)\. The cross terms involving exactly one residual factor vanish upon averaging over the trajectory index of the residual, because1\|𝒯y\|​∑t𝜹~ty=𝟎\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\}\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}=\\mathbf\{0\}and similarly for𝐡~ty\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}\. The remaining terms involve products of two or more residuals and are bounded by Cauchy–Schwarz:

\|1\|𝒯y\|​\|𝒯a\|​∑t,s𝜹~ty⊤​𝜹~sa⋅𝐡ty⊤​𝐡sa\|\\displaystyle\\bigg\|\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\|\\mathcal\{T\}\_\{a\}\|\}\\sum\_\{t,s\}\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\\top\}\\tilde\{\\boldsymbol\{\\delta\}\}\_\{s\}^\{a\}\\cdot\\mathbf\{h\}\_\{t\}^\{y\\top\}\\mathbf\{h\}\_\{s\}^\{a\}\\bigg\|\(17\)≤κy​κa​Bh2\+Bδ​\(κy​Bh\+κa​Bh\),\\displaystyle\\leq\\kappa\_\{y\}\\kappa\_\{a\}B\_\{h\}^\{2\}\+B\_\{\\delta\}\(\\kappa\_\{y\}B\_\{h\}\+\\kappa\_\{a\}B\_\{h\}\),where we used‖𝜹¯y‖≤Bδ\\\|\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\\\|\\leq B\_\{\\delta\}and‖𝐡¯y‖≤Bh\\\|\\bar\{\\mathbf\{h\}\}^\{y\}\\\|\\leq B\_\{h\}\. This establishes thatΘo​\(y,a\)=\(𝜹¯y⊤​𝜹¯a\)​\(𝐡¯y⊤​𝐡¯a\)\+R\\Theta\_\{o\}\(y,a\)=\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\)\+R, where\|R\|≤κy​κa​Bh2\+Bδ​\(κy\+κa\)​Bh\|R\|\\leq\\kappa\_\{y\}\\kappa\_\{a\}B\_\{h\}^\{2\}\+B\_\{\\delta\}\(\\kappa\_\{y\}\+\\kappa\_\{a\}\)B\_\{h\}\.

#### Step 3: Relating gradient\-activation cosine to NTK cosine viaWoW\_\{o\}conditioning\.

The gradient\-activation vector for trajectoryyyis𝐬y=1\|𝒯y\|​∑t𝐆ty⊙𝐡ty\\mathbf\{s\}^\{y\}=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\}\\mathbf\{G\}\_\{t\}^\{y\}\\odot\\mathbf\{h\}\_\{t\}^\{y\}, where𝐆ty=Wo⊤​𝜹ty∈ℝd\\mathbf\{G\}\_\{t\}^\{y\}=W\_\{o\}^\{\\top\}\\boldsymbol\{\\delta\}\_\{t\}^\{y\}\\in\\mathbb\{R\}^\{d\}\. To leading order,𝐬y≈\(Wo⊤​𝜹¯y\)⊙𝐡¯y\\mathbf\{s\}^\{y\}\\approx\(W\_\{o\}^\{\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\)\\odot\\bar\{\\mathbf\{h\}\}^\{y\}, and the normalized vector is𝐯y=𝐬y/‖𝐬y‖2\\mathbf\{v\}\_\{y\}=\\mathbf\{s\}^\{y\}/\\\|\\mathbf\{s\}^\{y\}\\\|\_\{2\}\. LetSSdenote the subspace spanned by\{𝜹¯y,𝜹¯a\}\\\{\\bar\{\\boldsymbol\{\\delta\}\}^\{y\},\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\\\}and letσmin,σmax\\sigma\_\{\\min\},\\sigma\_\{\\max\}denote the extreme singular values ofWoW\_\{o\}restricted toSS\. Define the condition numberκW=σmax/σmin\\kappa\_\{W\}=\\sigma\_\{\\max\}/\\sigma\_\{\\min\}\. The cosinecos⁡\(𝐯y,𝐯a\)=⟨𝐬y,𝐬a⟩/\(‖𝐬y‖​‖𝐬a‖\)\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)=\\langle\\mathbf\{s\}^\{y\},\\mathbf\{s\}^\{a\}\\rangle/\(\\\|\\mathbf\{s\}^\{y\}\\\|\\\|\\mathbf\{s\}^\{a\}\\\|\)can be related to the normalized leading term ofΘo\\Theta\_\{o\}via the substitutionWo⊤​𝜹¯y=σy​𝐰yW\_\{o\}^\{\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}=\\sigma\_\{y\}\\mathbf\{w\}\_\{y\}whereσmin≤σy≤σmax\\sigma\_\{\\min\}\\leq\\sigma\_\{y\}\\leq\\sigma\_\{\\max\}and‖𝐰y‖=‖𝜹¯y‖\\\|\\mathbf\{w\}\_\{y\}\\\|=\\\|\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\\\|\. After algebraic manipulation, we obtain

1κW2cos\(𝐯y,𝐯a\)\\displaystyle\\frac\{1\}\{\\kappa\_\{W\}^\{2\}\}\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)≤Θo​\(y,a\)‖𝐠o​\(x,y\)‖F​‖𝐠o​\(x,a\)‖F\\displaystyle\\leq\\frac\{\\Theta\_\{o\}\(y,a\)\}\{\\\|\\mathbf\{g\}\_\{o\}\(x,y\)\\\|\_\{F\}\\\|\\mathbf\{g\}\_\{o\}\(x,a\)\\\|\_\{F\}\}\(18\)≤κW2cos\(𝐯y,𝐯a\)\+𝒪\(κy\+κa\),\\displaystyle\\leq\\kappa\_\{W\}^\{2\}\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)\+\\mathcal\{O\}\(\\kappa\_\{y\}\{\+\}\\kappa\_\{a\}\),which yields the claimed bound withc1=1/κW2c\_\{1\}=1/\\kappa\_\{W\}^\{2\}andc2=κW2c\_\{2\}=\\kappa\_\{W\}^\{2\}\. WhenWoW\_\{o\}is approximately isotropic \(i\.e\.,κW≈1\\kappa\_\{W\}\\approx 1\), the bound collapses to a tight bi\-Lipschitz equivalence between the gradient\-activation cosine and NTK cosine\. ∎

### F\.3Proof of Theorem[3](https://arxiv.org/html/2609.03342#Thmtheorem3)

We argue each direction in turn\. \(i\) For any correct responseyywithrraw​\(x,y\)=1r\_\{\\text\{raw\}\}\(x,y\)=1, we haverGAR​\(x,y\)−p⁡\(y\)=rbase\+β​max⁡\(0,b^​\(y\)\)≥rbaser\_\{\\text\{GAR\}\}\(x,y\)\-p\(y\)=r\_\{\\text\{base\}\}\+\\beta\\max\(0,\\hat\{b\}\(y\)\)\\geq r\_\{\\text\{base\}\}, where the inequality follows from non\-negativity of the clipped bonus\. Thus GAR never reduces the reward contribution of a correct response belowrbaser\_\{\\text\{base\}\}\. \(ii\) For any incorrect responsey′y^\{\\prime\}withrraw​\(x,y′\)=0r\_\{\\text\{raw\}\}\(x,y^\{\\prime\}\)=0, the verifier gate yieldsrGAR​\(x,y′\)=p⁡\(y′\)≤0r\_\{\\text\{GAR\}\}\(x,y^\{\\prime\}\)=p\(y^\{\\prime\}\)\\leq 0\. Sincerbase\>0r\_\{\\text\{base\}\}\>0, we haverGAR​\(x,y′\)=p⁡\(y′\)≤0<rbase≤rGAR​\(x,y\)−p⁡\(y\)r\_\{\\text\{GAR\}\}\(x,y^\{\\prime\}\)=p\(y^\{\\prime\}\)\\leq 0<r\_\{\\text\{base\}\}\\leq r\_\{\\text\{GAR\}\}\(x,y\)\-p\(y\)for any correctyy\. The strict separation between the reward contributions of correct and incorrect responses ensures that the policy gradient consistently reinforces producing correct answers; GAR’s contribution is to additionally differentiate*among*correct trajectories by their reasoning quality\. ∎

### F\.4Proof of Proposition[2](https://arxiv.org/html/2609.03342#Thmproposition2)

Under the outcome\-only reward, every correct response has reward11, so the empirical variance restricted to correct responses is identically zero\. Under the GAR reward, correct responses have rewardrbase\+β​max⁡\(0,b^​\(yi\)\)\+p⁡\(yi\)r\_\{\\text\{base\}\}\+\\beta\\max\(0,\\hat\{b\}\(y\_\{i\}\)\)\+p\(y\_\{i\}\)\. Assuming the penaltypphas zero intra\-correct variance \(a reasonable approximation when format constraints are satisfied by all correct responses\), the intra\-correct variance of the GAR reward equalsβ2⋅Vari:rraw=1\[max\(0,b^\(yi\)\)\]\\beta^\{2\}\\cdot\\mathrm\{Var\}\_\{i:r\_\{\\text\{raw\}\}=1\}\[\\max\(0,\\hat\{b\}\(y\_\{i\}\)\)\]\. By the clipping operation this variance is at leastmin⁡\(1,β2​σb2\)\>0\\min\(1,\\beta^\{2\}\\sigma\_\{b\}^\{2\}\)\>0wheneverσb2\>0\\sigma\_\{b\}^\{2\}\>0\. Scaling by the denominatorσx\+ϵ\\sigma\_\{x\}\+\\epsilonof Eq\. \([10](https://arxiv.org/html/2609.03342#A5.E10)\) \(which is bounded in expectation\) yields the claim\. ∎

### F\.5Proof of Proposition[3](https://arxiv.org/html/2609.03342#Thmproposition3)

By definitionb^​\(y\)=b​\(y\)−b¯​\(x\)\\hat\{b\}\(y\)=b\(y\)\-\\bar\{b\}\(x\)whereb¯​\(x\)\\bar\{b\}\(x\)is the empirical mean ofbbover the correct subset of the rollout group\. For every promptxx,

𝔼y:rraw​\(y\)=1\[b^\(y\)\]\\displaystyle\\mathbb\{E\}\_\{y:r\_\{\\text\{raw\}\}\(y\)=1\}\[\\hat\{b\}\(y\)\]=𝔼​\[b​\(y\)\]−b¯​\(x\)\\displaystyle=\\mathbb\{E\}\[b\(y\)\]\-\\bar\{b\}\(x\)\(19\)=b¯​\(x\)−b¯​\(x\)=0,\\displaystyle=\\bar\{b\}\(x\)\-\\bar\{b\}\(x\)=0,where the inner expectation is over the uniform distribution over correct rollouts\. ∎

### F\.6Proof of Proposition[5](https://arxiv.org/html/2609.03342#Thmproposition5)

The forward through the output layer costs𝒪⁡\(Ny​d​V\)\\mathcal\{O\}\(N\_\{y\}dV\)for the matrix multiply and𝒪⁡\(Ny​V\)\\mathcal\{O\}\(N\_\{y\}V\)for the softmax\. The backward through the output layer similarly costs𝒪⁡\(Ny​d​V\)\\mathcal\{O\}\(N\_\{y\}dV\)for the gradient with respect to the hidden states, becauseWoW\_\{o\}is applied as a frozen constant and no weight gradients are accumulated\. The gradient\-activation multiplication and normalization cost𝒪⁡\(Ny​d\)\\mathcal\{O\}\(N\_\{y\}d\), which is dominated by the output\-layer cost\. With anchor caching, each unique anchor incurs these costs once per rollout batch, reducing the amortized anchor cost by a factor ofKK\. ∎

### F\.7Proof of Theorem[2](https://arxiv.org/html/2609.03342#Thmtheorem2)

We prove the multiplicative decomposition of the NTK inner product stated in Theorem[2](https://arxiv.org/html/2609.03342#Thmtheorem2)\.

Starting from the rank\-one expansion established in Eq\. \([16](https://arxiv.org/html/2609.03342#A6.E16)\) of the proof of Theorem[1](https://arxiv.org/html/2609.03342#Thmtheorem1),

Θo​\(y,a\)=1\|𝒯y\|​\|𝒯a\|​∑t,s\(𝜹ty⊤​𝜹sa\)​\(𝐡ty⊤​𝐡sa\)\.\\Theta\_\{o\}\(y,a\)=\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\|\\mathcal\{T\}\_\{a\}\|\}\\sum\_\{t,s\}\(\\boldsymbol\{\\delta\}\_\{t\}^\{y\\top\}\\boldsymbol\{\\delta\}\_\{s\}^\{a\}\)\(\\mathbf\{h\}\_\{t\}^\{y\\top\}\\mathbf\{h\}\_\{s\}^\{a\}\)\.\(20\)Write𝜹ty=𝜹¯y\+𝜹~ty\\boldsymbol\{\\delta\}\_\{t\}^\{y\}=\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\+\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}and𝐡ty=𝐡¯y\+𝐡~ty\\mathbf\{h\}\_\{t\}^\{y\}=\\bar\{\\mathbf\{h\}\}^\{y\}\+\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}\(similarly for anchoraa\), where the tildes denote mean\-zero residuals with‖𝜹~ty‖≤κy\\\|\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}\\\|\\leq\\kappa\_\{y\}and‖𝐡~ty‖≤Bh\\\|\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}\\\|\\leq B\_\{h\}\. Expanding the product\(𝜹¯y\+𝜹~ty\)⊤​\(𝜹¯a\+𝜹~sa\)⋅\(𝐡¯y\+𝐡~ty\)⊤​\(𝐡¯a\+𝐡~sa\)\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\}\+\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}\)^\{\\top\}\(\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\+\\tilde\{\\boldsymbol\{\\delta\}\}\_\{s\}^\{a\}\)\\cdot\(\\bar\{\\mathbf\{h\}\}^\{y\}\+\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}\)^\{\\top\}\(\\bar\{\\mathbf\{h\}\}^\{a\}\+\\tilde\{\\mathbf\{h\}\}\_\{s\}^\{a\}\)yields sixteen terms, which we classify by the number of residual factors\.

#### Zero\-residual term \(leading order\)\.

The single term with no residuals is\(𝜹¯y⊤​𝜹¯a\)​\(𝐡¯y⊤​𝐡¯a\)\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\), which survives the double average unchanged\.

#### One\-residual terms\.

There are four such terms\. Consider\(𝜹~ty⊤​𝜹¯a\)​\(𝐡¯y⊤​𝐡¯a\)\(\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\); averaging overttgives\(1\|𝒯y\|​∑t𝜹~ty\)⊤​𝜹¯a⋅\(𝐡¯y⊤​𝐡¯a\)=0\(\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\}\\sum\_\{t\}\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}\)^\{\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\\cdot\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\)=0because∑t𝜹~ty=𝟎\\sum\_\{t\}\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}=\\mathbf\{0\}\. The same argument eliminates the other three one\-residual terms\.

#### Two\-residual terms\.

There are six such terms\. The dominant ones are: \(a\)1\|𝒯y\|​\|𝒯a\|​∑t,s\(𝜹~ty⊤​𝜹~sa\)​\(𝐡¯y⊤​𝐡¯a\)\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\|\\mathcal\{T\}\_\{a\}\|\}\\sum\_\{t,s\}\(\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\\top\}\\tilde\{\\boldsymbol\{\\delta\}\}\_\{s\}^\{a\}\)\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\), bounded in absolute value byκy​κa​Bh2\\kappa\_\{y\}\\kappa\_\{a\}B\_\{h\}^\{2\}; \(b\)1\|𝒯y\|​\|𝒯a\|​∑t,s\(𝜹¯y⊤​𝜹¯a\)​\(𝐡~ty⊤​𝐡~sa\)\\frac\{1\}\{\|\\mathcal\{T\}\_\{y\}\|\|\\mathcal\{T\}\_\{a\}\|\}\\sum\_\{t,s\}\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\\top\}\\tilde\{\\mathbf\{h\}\}\_\{s\}^\{a\}\), bounded byBδ2​Bh2B\_\{\\delta\}^\{2\}B\_\{h\}^\{2\}; and cross terms of the form\(𝜹~ty⊤​𝜹¯a\)​\(𝐡~ty⊤​𝐡¯a\)\(\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\), which do not vanish upon averaging overttbecause𝜹~ty\\tilde\{\\boldsymbol\{\\delta\}\}\_\{t\}^\{y\}and𝐡~ty\\tilde\{\\mathbf\{h\}\}\_\{t\}^\{y\}are correlated at the same position, but are bounded byκy​Bδ​Bh2\\kappa\_\{y\}B\_\{\\delta\}B\_\{h\}^\{2\}via Cauchy–Schwarz\.

#### Three\- and four\-residual terms\.

These are bounded by products of three or four dispersion terms and are absorbed into the𝒪\\mathcal\{O\}notation\.

Collecting all bounds, we obtain

Θo​\(y,a\)\\displaystyle\\Theta\_\{o\}\(y,a\)=\(𝜹¯y⊤​𝜹¯a\)​\(𝐡¯y⊤​𝐡¯a\)\\displaystyle=\(\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\)\(\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\)\(21\)\+𝒪⁡\(Bh2​κy​κa\+Bδ​\(κy​Bh\+κa​Bh\)\),\\displaystyle\+\\mathcal\{O\}\(B\_\{h\}^\{2\}\\kappa\_\{y\}\\kappa\_\{a\}\+B\_\{\\delta\}\(\\kappa\_\{y\}B\_\{h\}\+\\kappa\_\{a\}B\_\{h\}\)\),which is precisely Eq\. \([8](https://arxiv.org/html/2609.03342#S3.E8)\)\. ∎

### F\.8Proof of Corollary[1](https://arxiv.org/html/2609.03342#Thmcorollary1)

#### Setup\.

Recall the definition: theε\\varepsilon\-effective support of the gradient\-activation signal𝐬¯z\\bar\{\\mathbf\{s\}\}^\{z\}for trajectoryzzis𝒮εz=\{j∈\[d\]:\|s¯jz\|\>ε​‖𝐬¯z‖∞\}\\mathcal\{S\}\_\{\\varepsilon\}^\{z\}=\\\{j\\in\[d\]:\|\\bar\{s\}\_\{j\}^\{z\}\|\>\\varepsilon\\\|\\bar\{\\mathbf\{s\}\}^\{z\}\\\|\_\{\\infty\}\\\}\. We assume𝒮εy∩𝒮εa=∅\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}\\cap\\mathcal\{S\}\_\{\\varepsilon\}^\{a\}=\\varnothing\.

#### Decomposition of the inner product\.

Consider the un\-normalized inner product⟨𝐬¯y,𝐬¯a⟩=∑j=1ds¯jy​s¯ja\\langle\\bar\{\\mathbf\{s\}\}^\{y\},\\bar\{\\mathbf\{s\}\}^\{a\}\\rangle=\\sum\_\{j=1\}^\{d\}\\bar\{s\}\_\{j\}^\{y\}\\bar\{s\}\_\{j\}^\{a\}\. We split this sum into three parts: \(i\)j∈𝒮εy∖𝒮εaj\\in\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}\\setminus\\mathcal\{S\}\_\{\\varepsilon\}^\{a\}, where\|s¯ja\|≤ε​‖𝐬¯a‖∞\|\\bar\{s\}\_\{j\}^\{a\}\|\\leq\\varepsilon\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{\\infty\}; \(ii\)j∈𝒮εa∖𝒮εyj\\in\\mathcal\{S\}\_\{\\varepsilon\}^\{a\}\\setminus\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}, where\|s¯jy\|≤ε​‖𝐬¯y‖∞\|\\bar\{s\}\_\{j\}^\{y\}\|\\leq\\varepsilon\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{\\infty\}; and \(iii\)j∉𝒮εy∪𝒮εaj\\notin\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}\\cup\\mathcal\{S\}\_\{\\varepsilon\}^\{a\}, where both signals are at mostε\\varepsilontimes their respective maxima\.

For part \(i\), using\|𝒮εy\|≤d\|\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}\|\\leq dand\|s¯jy\|≤‖𝐬¯y‖∞\|\\bar\{s\}\_\{j\}^\{y\}\|\\leq\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{\\infty\}:

\|∑j∈𝒮εy∖𝒮εas¯jy​s¯ja\|\\displaystyle\\bigg\|\\sum\_\{j\\in\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}\\setminus\\mathcal\{S\}\_\{\\varepsilon\}^\{a\}\}\\bar\{s\}\_\{j\}^\{y\}\\bar\{s\}\_\{j\}^\{a\}\\bigg\|≤\|𝒮εy\|⋅‖𝐬¯y‖∞⋅ε​‖𝐬¯a‖∞\\displaystyle\\leq\|\\mathcal\{S\}\_\{\\varepsilon\}^\{y\}\|\\cdot\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{\\infty\}\\cdot\\varepsilon\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{\\infty\}\(22\)≤d​ε​‖𝐬¯y‖∞​‖𝐬¯a‖∞\.\\displaystyle\\leq d\\varepsilon\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{\\infty\}\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{\\infty\}\.Part \(ii\) yields the same bound by symmetry\. For part \(iii\), both factors are bounded byε\\varepsilontimes their respective maxima, giving a bound ofd​ε2​‖𝐬¯y‖∞​‖𝐬¯a‖∞d\\varepsilon^\{2\}\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{\\infty\}\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{\\infty\}\.

#### Normalization\.

Since‖𝐬¯z‖2≥‖𝐬¯z‖∞\\\|\\bar\{\\mathbf\{s\}\}^\{z\}\\\|\_\{2\}\\geq\\\|\\bar\{\\mathbf\{s\}\}^\{z\}\\\|\_\{\\infty\}\(theℓ2\\ell\_\{2\}norm is at least theℓ∞\\ell\_\{\\infty\}norm\), we have

\|cos⁡\(𝐯y,𝐯a\)\|\\displaystyle\|\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)\|=\|⟨𝐬¯y,𝐬¯a⟩\|‖𝐬¯y‖2​‖𝐬¯a‖2\\displaystyle=\\frac\{\|\\langle\\bar\{\\mathbf\{s\}\}^\{y\},\\bar\{\\mathbf\{s\}\}^\{a\}\\rangle\|\}\{\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{2\}\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{2\}\}\(23\)≤\(2​d​ε\+d​ε2\)​‖𝐬¯y‖∞​‖𝐬¯a‖∞‖𝐬¯y‖2​‖𝐬¯a‖2\\displaystyle\\leq\\frac\{\(2d\\varepsilon\{\+\}d\\varepsilon^\{2\}\)\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{\\infty\}\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{\\infty\}\}\{\\\|\\bar\{\\mathbf\{s\}\}^\{y\}\\\|\_\{2\}\\\|\\bar\{\\mathbf\{s\}\}^\{a\}\\\|\_\{2\}\}≤2​d​ε\+d​ε2\.\\displaystyle\\leq 2d\\varepsilon\+d\\varepsilon^\{2\}\.For smallε\\varepsilon, theε2\\varepsilon^\{2\}term is negligible, and incorporating the dispersion correction from Theorem[1](https://arxiv.org/html/2609.03342#Thmtheorem1)adds an𝒪⁡\(κy\+κa\)\\mathcal\{O\}\(\\kappa\_\{y\}\+\\kappa\_\{a\}\)term, yielding the bound\|cos⁡\(𝐯y,𝐯a\)\|≤2​ε​d\+𝒪⁡\(κy\+κa\)\|\\cos\(\\mathbf\{v\}\_\{y\},\\mathbf\{v\}\_\{a\}\)\|\\leq 2\\varepsilon d\+\\mathcal\{O\}\(\\kappa\_\{y\}\+\\kappa\_\{a\}\)as claimed\. ∎

### F\.9Proof of Proposition[4](https://arxiv.org/html/2609.03342#Thmproposition4)

We prove that a single GRPO step with GAR\-shaped advantages does not decrease the expected cosine alignment of correct rollouts\.

#### Setup\.

Letπt\\pi\_\{t\}denote the current policy and letci=cos⁡\(𝐯yi,𝐯a\)c\_\{i\}=\\cos\(\\mathbf\{v\}\_\{y\_\{i\}\},\\mathbf\{v\}\_\{a\}\)denote the cosine alignment of rolloutyiy\_\{i\}\. For a promptxxwithKKrollouts, the GAR reward for a correct responseyiy\_\{i\}isri=rbase\+β​max⁡\(0,ci−c¯\)r\_\{i\}=r\_\{\\text\{base\}\}\+\\beta\\max\(0,c\_\{i\}\-\\bar\{c\}\), wherec¯=1Kc∑j:rraw​\(yj\)=1cj\\bar\{c\}=\\frac\{1\}\{K\_\{c\}\}\\sum\_\{j:r\_\{\\text\{raw\}\}\(y\_\{j\}\)=1\}c\_\{j\}is the prompt\-group mean cosine andKcK\_\{c\}is the number of correct responses\. The GRPO advantage isAi=\(ri−r¯\)/\(σr\+ϵ\)A\_\{i\}=\(r\_\{i\}\-\\bar\{r\}\)/\(\\sigma\_\{r\}\+\\epsilon\)\.

#### One\-step improvement\.

After a policy gradient step with learning rateα\\alpha, the log\-probability of each response changes by approximatelyΔ​log⁡π⁡\(yi∣x\)≈α​Ai\\Delta\\log\\pi\(y\_\{i\}\\mid x\)\\approx\\alpha A\_\{i\}\. The expected cosine at the next iteration, restricted to correct responses, is

C¯t\+1\\displaystyle\\bar\{C\}\_\{t\+1\}≈∑i:rraw=1πt\(yi∣x\)eα​Ai⋅ci∑i:rraw=1πt\(yi∣x\)eα​Ai\\displaystyle\\approx\\frac\{\\sum\_\{i:r\_\{\\text\{raw\}\}=1\}\\pi\_\{t\}\(y\_\{i\}\\mid x\)e^\{\\alpha A\_\{i\}\}\\cdot c\_\{i\}\}\{\\sum\_\{i:r\_\{\\text\{raw\}\}=1\}\\pi\_\{t\}\(y\_\{i\}\\mid x\)e^\{\\alpha A\_\{i\}\}\}\(24\)≈C¯t\+α⋅Covπt\[Ai,ci∣rraw\(yi\)=1\],\\displaystyle\\approx\\bar\{C\}\_\{t\}\+\\alpha\\cdot\\mathrm\{Cov\}\_\{\\pi\_\{t\}\}\\big\[A\_\{i\},\\,c\_\{i\}\\mid r\_\{\\text\{raw\}\}\(y\_\{i\}\)=1\\big\],where the approximation useseα​Ai≈1\+α​Aie^\{\\alpha A\_\{i\}\}\\approx 1\+\\alpha A\_\{i\}for smallα\\alpha\.

#### Sign of the covariance\.

The GAR advantage among correct responses is a monotonically non\-decreasing function ofcic\_\{i\}: responses withci\>c¯c\_\{i\}\>\\bar\{c\}receive positive bonus and hence above\-mean advantage, while responses withci≤c¯c\_\{i\}\\leq\\bar\{c\}receive zero bonus \(due to themax⁡\(0,⋅\)\\max\(0,\\cdot\)clipping\) and hence below\-mean advantage\. Formally,Ai=f⁡\(ci\)A\_\{i\}=f\(c\_\{i\}\)whereffis non\-decreasing, which impliesCov⁡\[f⁡\(ci\),ci\]≥0\\mathrm\{Cov\}\[f\(c\_\{i\}\),c\_\{i\}\]\\geq 0by the covariance inequality for comonotone random variables\. The covariance is strictly positive wheneverVar⁡\[ci∣rraw=1\]\>0\\mathrm\{Var\}\[c\_\{i\}\\mid r\_\{\\text\{raw\}\}=1\]\>0, i\.e\., whenever the correct rollouts are not all equally aligned\.

#### KL penalty\.

The KL divergence penalty−λDKL\(πt\+1∥πref\)\-\\lambda D\_\{\\text\{KL\}\}\(\\pi\_\{t\+1\}\\\|\\pi\_\{\\text\{ref\}\}\)reduces the effective step size but does not change the sign of the improvement, providedλ\\lambdasatisfies the condition stated in the proposition\. Combining these observations,C¯t\+1≥C¯t\\bar\{C\}\_\{t\+1\}\\geq\\bar\{C\}\_\{t\}as claimed\. ∎

## Appendix GExtended Discussion of Theoretical Results

### G\.1NTK Interpretation: Functional Meaning of Gradient Alignment

Theorem[1](https://arxiv.org/html/2609.03342#Thmtheorem1)states that gradient cosine similarity in the gradient\-activation space is, up to a low\-order dispersion term, equivalent to the normalized output\-layer NTK similarity of the two responses, and therefore measures how similarly a gradient step induced byyywould influence the model’s predictions onaa, and vice versa\. In the idealized case where both trajectories fit their targets with comparable token\-level error distributions,κy,κa→0\\kappa\_\{y\},\\kappa\_\{a\}\\to 0and the bound collapses to a bi\-Lipschitz equivalence\. This provides a functional interpretation of GAR: rewarding high\-cosine trajectories amounts to rewarding responses whose parameter\-space influence on the expert anchor is large, i\.e\., trajectories that, if used for a gradient update, would improve the model’s prediction of the expert solution\.

### G\.2Multiplicative Decomposition: What Gradient Alignment Measures

The decomposition in Theorem[2](https://arxiv.org/html/2609.03342#Thmtheorem2)reveals that two trajectories can achieve high gradient alignment only if they agree in*both*their prediction\-error profile \(the gradient\-direction factor𝜹¯y⊤​𝜹¯a\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}\) and their internal representation usage \(the activation\-pattern factor𝐡¯y⊤​𝐡¯a\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\)\. A trajectory that reaches the correct answer through a qualitatively different computational pathway, for example one that activates a substantially different subset of hidden dimensions or produces a different distribution of token\-level prediction residuals, diverges from the expert reference in at least one of these two factors and therefore receives a low cosine score even when its surface\-level output appears plausible\.

Corollary[1](https://arxiv.org/html/2609.03342#Thmcorollary1)further formalizes this mechanism: if two trajectories concentrate their gradient\-activation signals on disjoint subsets of hidden dimensions, the resulting cosine similarity is near zero regardless of whether both produce the correct final answer\. In practice, expert solutions tend to activate a broad set of features spanning intermediate derivation steps, whereas alternative correct trajectories that arrive at the answer through different reasoning pathways may concentrate on narrower or qualitatively different feature subsets, which makes subspace separation a structurally meaningful property of the alignment signal\.

### G\.3Safe Reward Shaping: Detailed Proof Sketch

Theorem[3](https://arxiv.org/html/2609.03342#Thmtheorem3)establishes that GAR is a safe reward\-shaping mechanism: it*re\-weights preferences among correct responses*to favor expert\-aligned reasoning without degrading the reward signal for correctness\. The proof hinges on two observations\. First, the verifier gate ensuresrGAR​\(x,y\)=p⁡\(y\)≤0r\_\{\\text\{GAR\}\}\(x,y\)=p\(y\)\\leq 0on incorrect responses, maintaining a strict separation between the rewards assigned to correct and incorrect trajectories\. Second, themax⁡\(0,b^\)\\max\(0,\\hat\{b\}\)clipping ensures that the alignment bonus is non\-negative, so every correct response receives at leastrbaser\_\{\\text\{base\}\}and the shaped reward never falls below the outcome\-only baseline\. Together, these properties guarantee that the policy gradient under GAR consistently reinforces correctness while using the alignment bonus to differentiate among correct trajectories\.

### G\.4Variance Amplification and Unbiasedness

Proposition[2](https://arxiv.org/html/2609.03342#Thmproposition2)quantifies the pathology of the flat reward identified in Section[2](https://arxiv.org/html/2609.03342#S2): under the outcome\-only reward, the intra\-correct variance of the advantage is identically zero, so the policy gradient cannot distinguish among correct trajectories\. GAR strictly increases this variance, which, under standard assumptions on the log\-likelihood ratio, translates into a non\-trivial gradient signal in the direction of higher\-alignment trajectories\. Crucially, the prompt\-group normalization keeps the global mean reward invariant, so the extra variance does not come at the cost of bias\.

Propositions[2](https://arxiv.org/html/2609.03342#Thmproposition2)and[3](https://arxiv.org/html/2609.03342#Thmproposition3), taken together with Theorem[3](https://arxiv.org/html/2609.03342#Thmtheorem3), justify our design choice to always center the bonus before applyingβ\\beta: centering is a free variance\-increasing and bias\-removing transformation, and the downstreammax⁡\(0,⋅\)\\max\(0,\\cdot\)clip then ensures non\-negativity of the final reward contribution, preserving the strict separation between correct and incorrect responses\.

### G\.5Monotonic Improvement: Interpretation

The monotonic improvement result of Proposition[4](https://arxiv.org/html/2609.03342#Thmproposition4)follows from the observation that GAR assigns positive advantages exclusively to correct trajectories whose cosine similarity exceeds the prompt\-group mean, and zero or negative advantages to the remainder\. The resulting policy gradient therefore increases the log\-probability of high\-alignment trajectories relative to low\-alignment ones, and the expected cosine in the next iteration is the current expected cosine plus a non\-negative covariance term that vanishes only when all correct trajectories are equally aligned\. This provides a convergence\-like guarantee: GAR training cannot decrease the average alignment of the policy’s correct rollouts with the expert reference, as measured by gradient cosine\.

## Appendix HEvaluation Protocol Details

We evaluate using sampling\-based decoding with1616independent responses per problem\. Pass@kk\(k∈\{1,4,16\}k\\in\\\{1,4,16\\\}\) is estimated as the fraction of problems for which at least one ofkkrandomly sampled responses \(without replacement from the 16 samples\) is correct; we report the expectation over all\(16k\)\\binom\{16\}\{k\}subsets\. Maj@kktakes the plurality answer amongkksampled responses\.

Statistical methodology\.Each configuration is independently trained with a different random seed, yielding 10 runs per method\. We report the mean and standard deviation of each metric across these 10 runs\. Statistical significance is assessed by a two\-sided pairedtt\-test: for each runr∈\{1,…,10\}r\\in\\\{1,\\dots,10\\\}, both the baseline \(e\.g\., GRPO\) and the GAR variant \(e\.g\., GAR\-GRPO\) are evaluated under identical sampling conditions, and the test statistic is computed over the 10 paired differences in per\-run pass@kkscores\. Each configuration is trained for400400optimization steps with a global batch size of128128prompts \(∼8\{\\sim\}8epochs\)\. All runs use tensor parallelism of size44, context\-parallel size of11, and rollout group sizeK=16K\{=\}16\.

Mathematical benchmarks\.IMO\-AnswerBench \(400 problems\), HMMT Feb 2025 \(30\), HMMT Feb 2026 \(33\), and AIME 2026 \(30\) are strictly held out from training data and anchor corpora\. Correctness is determined by exact match of the final numerical answer after normalization\.

General reasoning benchmarks\.GPQA Diamond \(198 graduate\-level science questions\) and MMLU\-Pro \(12k broad\-domain reasoning questions\) are evaluated zero\-shot to assess cross\-domain transfer from mathematical training\. Both benchmarks use multiple\-choice format; correctness is determined by exact match of the predicted answer choice after normalization\.

Contamination control\.All mathematical evaluation benchmarks post\-date the Qwen3 base model’s pretraining cutoff \(HMMT ’25/​’26 and AIME ’26 were released after the model’s training data was frozen\), and we further verify that none of the evaluation problems appear verbatim in our training corpus by running exact\-match deduplication against the NuminaMath\-CoT training set\. IMO\-AnswerBench and GPQA Diamond were released before the pretraining cutoff; however, the base model’s low zero\-shot accuracy on these benchmarks \(below 4% pass@1 on IMO\-AnswerBench and below 29% on GPQA Diamond, see Tables[2](https://arxiv.org/html/2609.03342#S4.T2)and[3](https://arxiv.org/html/2609.03342#S6.T3)\) is consistent with minimal memorization\.

## Appendix IHyperparameter Details

In all experiments, we set the base rewardrbase=1\.0r\_\{\\text\{base\}\}=1\.0, the alignment bonus coefficientβ=0\.5\\beta=0\.5, and the maximum response length for GAR gradient computation to768768tokens\. The format penalty is set to−0\.6\-0\.6, and the rollout group size isK=16K=16for all runs\.

## Appendix JComputational Overhead Analysis

This appendix expands on the theoretical cost analysis summarized in Section[6\.4](https://arxiv.org/html/2609.03342#S6.SS4)\.

###### Proposition 5\(GAR overhead\)\.

The additional per\-response cost of GAR is𝒪⁡\(Ny​d​V\)\+𝒪⁡\(Ny​d\)\\mathcal\{O\}\(N\_\{y\}dV\)\+\\mathcal\{O\}\(N\_\{y\}d\), corresponding to one output\-layer forward and one truncated backward\. With anchor caching, the amortized per\-prompt anchor cost reduces from𝒪⁡\(K​Ny​d​V\)\\mathcal\{O\}\(KN\_\{y\}dV\)to𝒪⁡\(Ny​d​V\)\\mathcal\{O\}\(N\_\{y\}dV\)\.

For Qwen3\-8B \(d=4096d\{=\}4096,V=152064V\{=\}152064,L=36L\{=\}36\), the ratioV/\(L​d\)≈1\.03V/\(Ld\)\\approx 1\.03, so each truncated backward pass is comparable in cost to a single full forward, yielding a practical overhead of55–12%12\\%after amortization\.

#### Per\-operation cost\.

GAR adds a single truncated backward pass through the output projection layer for each verified\-correct response\. For a Qwen3\-8B model with hidden dimensiond=4096d=4096, vocabulary sizeV=152064V=152064, number of Transformer layersL=36L=36, and response lengthNy≈1024N\_\{y\}\\approx 1024, the ratio of the GAR per\-response cost to a full\-model forward pass is approximatelyV/\(L​d\)=152064/\(36⋅4096\)≈1\.03V/\(Ld\)=152064/\(36\\cdot 4096\)\\approx 1\.03, so each truncated backward pass is comparable in cost to a single full forward\. The overhead is dominated by the output\-layer matrix multiply of shapeV×d=152064×4096V\\times d=152064\\times 4096, which is intrinsic to the LM head and cannot be avoided as long as the gradient is defined with respect to the next\-token logits\.

#### Verifier\-gate amortization\.

Because GAR computes gradients only for responses that pass the outcome verifier, the per\-rollout cost is further reduced by the verifier rejection rate\. In the early training phase, typically4040–70%70\\%of rollouts fail verification and therefore incur no gradient computation at all, which makes the effective overhead substantially smaller than the per\-response figure above suggests\. As training progresses and the verifier pass rate increases, this amortization weakens, but by then the policy is also producing shorter and more focused responses, partially offsetting the increase\.

#### Anchor\-cache amortization\.

For each prompt group ofKKrollouts that share the same anchor, a naive implementation would invoke the anchor gradient computation once per rollout, incurring a cost of𝒪⁡\(K​Ny​d​V\)\\mathcal\{O\}\(KN\_\{y\}dV\)\. Anchor caching computes the anchor gradient vector once at the start of the batch and reuses it across allKKrollouts of the same prompt, reducing the anchor component to𝒪⁡\(Ny​d​V\)\\mathcal\{O\}\(N\_\{y\}dV\)and yielding aKK\-fold reduction\. For the default rollout group sizeK=16K=16used in our experiments, this amortization is the single largest source of practical savings in the anchor pathway\.

#### Aggregate prediction\.

Combining the per\-response cost, the verifier\-gate amortization, and the anchor\-cache amortization, the predicted end\-to\-end overhead falls within the range of approximately5%5\\%to12%12\\%\. The measured wall\-clock overhead reported in Section[6\.4](https://arxiv.org/html/2609.03342#S6.SS4)agrees with this analytical envelope\.

## Appendix KHyperparameter Sensitivity

Figure 6:Sensitivity of GAR\-GRPO \(Qwen3\-8B\-Base, AIME 2026\) to four hyperparameters, measured as percentage decrease from each metric’s peak\. Pass@1 peaks at the defaults; pass@16 favors slightly more aggressive settings and degrades more gracefully\.Figure[6](https://arxiv.org/html/2609.03342#A11.F6)sweeps four GAR hyperparameters on AIME 2026\. A consistent pattern emerges: pass@1 peaks at the defaults \(β=0\.5\\beta\{=\}0\.5,L=768L\{=\}768,τa=0\.05\\tau\_\{a\}\{=\}0\.05,pf=0\.7p\_\{f\}\{=\}0\.7\), while pass@16 favors slightly more aggressive settings \(higherβ\\beta, longer span, lower activation threshold, and less token filtering\), because coverage\-oriented metrics benefit from retaining broader gradient signal\. All four curves exhibit smooth, inverted\-U shapes with pass@16 degrading more gracefully than pass@1, indicating that GAR is robust across a wide hyperparameter range\.

## Appendix LGeneral Reasoning Transfer: 8B Results

Table 4:General reasoning transfer \(Qwen3\-8B\-Base, 10 runs\)\. All methods are trained exclusively on mathematical data and evaluated zero\-shot\.Table[4](https://arxiv.org/html/2609.03342#A12.T4)presents the 8B counterpart of the general reasoning transfer evaluation\. The pattern observed at the 4B scale carries over: GAR\-GRPO achieves the highest scores across all four metrics, with the largest margin on MMLU\-Pro \(\+2\.28 absolute over GRPO\), confirming that gradient\-aligned rewards capture domain\-general reasoning structure at both model scales\.

## Appendix MAlignment and Solution Diversity

To verify that gradient alignment captures genuine reasoning\-process similarity rather than superficial features such as response length or answer format, we analyze a representative set of training\-set problems where multiple well\-known solution methods exist\. For each problem, we collect correct rollouts from GAR training, classify them by solution method, and report the mean cosine alignmentb⁡\(yi\)b\(y\_\{i\}\)within each method category\.

Table 5:Mean cosine alignmentb⁡\(yi\)b\(y\_\{i\}\)grouped by solution method across six training\-set problems\. “Same” denotes rollouts following the same method as the expert anchor; “Different” denotes rollouts using an alternative correct method\. The consistent3\.5×3\.5\\timesgap confirms that gradient alignment captures reasoning\-process similarity\.Table[5](https://arxiv.org/html/2609.03342#A13.T5)confirms that same\-method rollouts receive substantially higher cosine scores \(meanb¯same=0\.45\\bar\{b\}\_\{\\text\{same\}\}=0\.45\) than different\-method rollouts \(meanb¯diff=0\.13\\bar\{b\}\_\{\\text\{diff\}\}=0\.13\), a3\.5×3\.5\\timesgap\. This validates that the gradient alignment signal is structurally meaningful: it discriminates based on the underlying computational pathway rather than surface\-level correlates\.

#### Why surface\-level confounds are unlikely to explain the gap\.

The multiplicative decomposition of Theorem[2](https://arxiv.org/html/2609.03342#Thmtheorem2)provides a structural argument: the cosine factorizes into a prediction\-error factor𝜹¯y⊤​𝜹¯a\\bar\{\\boldsymbol\{\\delta\}\}^\{y\\top\}\\bar\{\\boldsymbol\{\\delta\}\}^\{a\}and an activation\-pattern factor𝐡¯y⊤​𝐡¯a\\bar\{\\mathbf\{h\}\}^\{y\\top\}\\bar\{\\mathbf\{h\}\}^\{a\}\. Surface features such as response length, answer format, or template phrasing affect at most the activation\-pattern factor, but two trajectories that follow genuinely different derivation paths will diverge in their prediction\-error profiles, since the sequence of intermediate tokens \(and hence the per\-token prediction residuals\) differs substantially\. Concretely, a complex\-number rotation solution and a distance\-formula solution share neither the set of mathematical operations nor the intermediate numerical quantities, producing divergent𝜹¯\\bar\{\\boldsymbol\{\\delta\}\}vectors regardless of stylistic overlap\. This is further corroborated by Corollary[1](https://arxiv.org/html/2609.03342#Thmcorollary1): when two trajectories concentrate their gradient\-activation signals on disjoint hidden\-dimension subsets, the resulting cosine is near zero, a condition that holds for genuinely different reasoning pathways even when both are correct and similarly formatted\.

#### Effect on alternative correct strategies\.

Themax⁡\(0,b^\)\\max\(0,\\hat\{b\}\)clip in the final reward \(Eq\.[6](https://arxiv.org/html/2609.03342#S2.E6)\) guarantees that every correct rollout receives a raw reward of at leastrbaser\_\{\\text\{base\}\}, strictly above the zero reward assigned to incorrect responses\. Under GRPO’s group\-normalized advantage, this raw\-reward floor does not prevent the relative advantage of low\-alignment correct rollouts from being lower than that of high\-alignment ones; this relative reweighting within the correct set is precisely the intended mechanism by which GAR steers toward expert\-aligned reasoning\. Crucially, the verifier gate ensures that correct responses always receive higher raw reward than incorrect ones, so the policy gradient consistently reinforces producing correct answers; GAR’s contribution is to additionally differentiate*among*correct trajectories by their reasoning quality\.

The expert CoTs from NuminaMath\-CoT predominantly employ concise, theorem\-driven approaches \(e\.g\., complex\-number rotation over coordinate brute\-force, modular arithmetic over exhaustive enumeration\), and the consistent accuracy gains across all benchmarks \(Section[5\.1](https://arxiv.org/html/2609.03342#S5.SS1)\) confirm that steering the policy toward such strategies provides effective reasoning\-quality supervision\.