LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models

# LambdaPO: A Lambda Style Policy Optimization for Reasoning Language Models
Source: [https://arxiv.org/html/2605.19416](https://arxiv.org/html/2605.19416)
Zhe Yuan1, Yipeng Zhou2, Jinghan Li3, Xinyuan Chen4, Bowen Deng5, Zhiqian Chen4, Liang Zhao6 1Pinterest, San Francisco, CA 94107, USA 2Facebook, Menlo Park, CA 94025, USA 3University of Michigan \- Ann Arbor, Ann Arbor, MI 48109, USA 4Mississippi State University, Mississippi State, MS 39762, USA 5Carnegie Mellon University, Pittsburgh, PA 15213\. USA 6Emory University, Atlanta, GA 30322, USA

\(March 11, 2026\)

###### Abstract

Group Relative Policy Optimization\(GRPO\) has become a cornerstone of modern reinforcement learning alignment, prized for its efficacy in foregoing an explicit value\-critic by leveraging reward normalization across sampled trajectory cohorts\. However, the method’s reliance on a monolithic statistical baseline, such as the group mean, collapses the relational topology of the trajectory space into a single scalar, thereby erasing the fine\-grained preference information essential for navigating complex, rank\-sensitive reward landscapes\. To address this issue, we introduce a novel framework, Lambda Policy Optimization \(LambdaPO\), that addresses this information\-theoretic bottleneck by re\-conceptualizing advantage estimation from a scalar value to a decomposed, pairwise preference structure\. Specifically, the advantage for any given trajectory is formulated as the integrated sum of reward differentials against all peers in its cohort, where each pairwise comparison is dynamically attenuated by the policy’s own probabilistic confidence in the established preference\. To further mitigate the sparsity of binary outcome supervision, we augment the objective with a semantic density reward, derived from the precision\-recall alignment between generated reasoning traces and ground\-truth solutions\. As a result, our method can mine more fine\-grained optimization signals from a group of rollouts, guiding the LLM to a better optima\. Experimental results across challenging math reasoning and question\-answering tasks demonstrates that LambdaPO improves performance compared to the baseline methods\.

## 1Introduction

The evolution of Large Language Models \(LLMs\) has a paradigm shift in user expectations, transitioning from mere linguistic fluency to rigorous reasoning capabilities in domains such as mathematics and complex question answering\. Consequently, Reinforcement Learning \(RL\) has ascended as the basic technology for incentivizing and amplifying these cognitive behaviors\. While Proximal Policy Optimization \(PPO\)\[[14](https://arxiv.org/html/2605.19416#bib.bib1)\]established itself as thede factostandard in early human preference alignment, its computational dependence on auxiliary value networks presents scalability challenges\. Addressing this, recent frameworks such as Direct Preference Optimization \(DPO\)\[[11](https://arxiv.org/html/2605.19416#bib.bib18)\]have pioneered reference\-based offline alignment, while DeepSeekMath and DeepSeek\-R1 have introduced Group Relative Policy Optimization \(GRPO\)\[[15](https://arxiv.org/html/2605.19416#bib.bib2)\], a critic\-free refinement of PPO that significantly enhances training efficiency while maintaining robust optimization stability\. Furthermore, recent advances like SimPO\[[9](https://arxiv.org/html/2605.19416#bib.bib19)\]and ORPO\[[6](https://arxiv.org/html/2605.19416#bib.bib20)\]have further streamlined the alignment pipeline by eliminating the need for reference models or explicit reward modeling, setting a precedent for the efficiency LambdaPO aims to achieve\.

However, GRPO suffers from a fundamental information\-theoretic bottleneck: it relies on a monolithic statistical baseline \(the group mean\) for advantage estimation\. This scalar reduction fundamentally collapses the rich relational topology of the trajectory space, consequently erasing the fine\-grained, pairwise preference signals that are essential for navigating complex, rank\-sensitive reward landscapes\. By treating the entire cohort as an undifferentiated aggregate, existing methods incur suboptimal credit assignment and elevated gradient variance\.

To resolve this limitation, we draw upon the ”virtual gradient” principles established in Learning\-to\-Rank \(LtR\), specifically the LambdaRank framework\[[2](https://arxiv.org/html/2605.19416#bib.bib4)\], to propose Lambda Policy Optimization \(LambdaPO\)\. We re\-conceptualize the advantage function not as a simple deviation from a cohort mean, but as a pairwise decomposed gradient\. In this novel framework, the advantage for a specific trajectory is formulated as the integrated sum of its reward differentials against all cohort peers, dynamically attenuated by the policy’s own probabilistic confidence\. Furthermore, we address the challenge of reward sparsity in rigorous reasoning tasks—where binary outcome supervision fails to credit correct intermediate logic—by introducing a Semantic Density Reward\. This mechanism augments the optimization landscape with a dense, content\-aware signal derived from the token\-level precision and recall alignment between the generated chain\-of\-thought and the ground\-truth solution\. This approach effectively combines RL with the classical listwise ranking objectives, enabling the model to self\-anneal its learning rate based on the discordance between its internal belief and the external reward structure\.

We validate LambdaPO on complex mathematical reasoning and challenging question\-answering benchmarks\. Our results demonstrate that by recovering the pairwise information lost by standard group\-relative methods, LambdaPO helps the lightweighted LLMs to gain superior reasoning performance\. Our contributions include:

- •Methodological Innovation:We introduce LambdaPO, a framework that leverages pairwise advantage decomposition to materialize the gradients of a latent preference manifold, resolving the relational bottleneck of scalar baselines\.
- •Dense Signal Augmentation:We propose a Semantic Density Reward mechanism that mitigates the sparsity of outcome\-based supervision by explicitly incentivizing the recall of ground\-truth reasoning tokens and the precision of the generated rationale\.
- •Empirical Performance:We demonstrate through extensive experiments on mathematical reasoning and challenging question\-answering benchmarks that LambdaPO significantly outperforms GRPO, establishing a new state\-of\-the\-art for critic\-free alignment\.

## 2Related work

Research in LLM reasoning:

Many works have been devoted in enhancing the LLMs’ reasoning capabilities\. They can be divided into three directions: \(1\) explicit reasoning traces, \(2\) tool augmentation, and \(3\) reinforcement\-learning based optimization\.

Chain\-of\-Thought\(CoT\)\[[18](https://arxiv.org/html/2605.19416#bib.bib5)\]first proved that providing step\-by\-step rationales enables models to generalize on arithmetic, logic, and commonsense tasks beyond what is achievable with direct answer prompting\. Small models can acquire strong reasoning skills when explicitly trained on high\-quality chain\-of\-thought \(CoT\) rationales generated by a larger teacher model\. This has led to self\-improvement paradigms such as STaR\[[23](https://arxiv.org/html/2605.19416#bib.bib21)\], which leverages self\-generated rationales to bootstrap reasoning capabilities\. To further refine these traces, methods like Math\-Shepherd\[[17](https://arxiv.org/html/2605.19416#bib.bib22)\]utilize automated process\-level supervision to assign credit to individual reasoning steps, addressing the credit assignment problem that LambdaPO seeks to solve via its pairwise decomposition\.

Another area of work focuses on integrating LLMs with external tools to improve numerical accuracy and procedural reasoning\. ReAct\[[21](https://arxiv.org/html/2605.19416#bib.bib6)\]introduced a unified framework for interleaving natural\-language reasoning with tool actions \(e\.g\., search, calculator, code execution\), enabling LLMs to iteratively think and reflect\. Toolformer\[[13](https://arxiv.org/html/2605.19416#bib.bib7)\]further shows that LLMs can self\-supervise tool\-usage demonstrations, allowing models to autonomously decide when and how to invoke tools\. Together, these works demonstrate that augmenting LLMs with external tools or environments leads to more reliable and interpretable reasoning\.

Recent reasoning\-focused models adopt reinforcement learning to optimize long\-form reasoning traces\. Building on earlier RLHF pipelines\[[10](https://arxiv.org/html/2605.19416#bib.bib3)\], several works refine reward design for deliberate reasoning, including grouped\-rollout methods such as GRPO\[[15](https://arxiv.org/html/2605.19416#bib.bib2)\]\. DeepSeek\-R1\[[5](https://arxiv.org/html/2605.19416#bib.bib12)\]represents a recent milestone: it trains LLMs to perform extended internal reasoning through a combination of supervised step\-by\-step rationales and reinforcement learning over long reasoning trajectories\. These methods show that reinforcement learning—when applied to intermediate reasoning instead of just final answers—can yield significant gains in mathematical, logical, and scientific problem solving\.

From PPO to GRPO:Reinforcement learning has become a central approach for improving long chain\-of\-thought reasoning in large language models\. Early systems relied on Proximal Policy Optimization \(PPO\)\[[14](https://arxiv.org/html/2605.19416#bib.bib1)\], but directly applying PPO to language generation introduces sparse\-reward credit assignment issues and unstable updates\. Recent work therefore introduced GRPO, a PPO\-style policy optimization method tailored for LLMs, incorporating token\-level credit assignment and entropy preservation to enhance stability in long\-form reasoning tasks, and establishing the baseline for systems such as DeepSeek R1\-Zero\.

From GRPO to DAPO:While recent reasoning LLMs like OpenAI’s o1 and DeepSeek’s R1 have demonstrated impressive performance via test\-time scaling and reinforcement learning \(RL\), the full set of algorithmic details behind their training pipelines remains largely undisclosed\. The evolution of these methods sits within a broader theoretical effort to refine policy gradients, like KTO\[[4](https://arxiv.org/html/2605.19416#bib.bib23)\], which frames alignment through the lens of prospect theory\. DAPO \(Decoupled Clip and Dynamic sAmpling Policy Optimization\)\[[22](https://arxiv.org/html/2605.19416#bib.bib14)\]explicitly highlights this reproducibility gap: when they ran a naïve GRPO baseline on a Qwen2\.5\-32B model\[[20](https://arxiv.org/html/2605.19416#bib.bib15)\], they only achieved 30 points on the AIME benchmark — significantly below DeepSeek\-R1’s 47 points\. This performance gap motivated the design of DAPO’s core techniques and the authors open\-sourced not just the algorithm but also their training code and data to enable reproducibility\.

## 3Preliminary

### 3\.1Language Models as Stochastic Polices

We frame the task of text generation as a sequential decision\-making problem\. A LLM is treated as a stochastic policy, denoted byπθ\\pi\_\{\\theta\}, parameterized by weightsθ\\theta\. For each promptqq, the policy generates a sequence of tokenso=\(y1,y2,…,yr\)o=\(y\_\{1\},y\_\{2\},\.\.\.,y\_\{r\}\)\. Then the probability of the entire sequence is the product of conditional token probabilities:

πθ​\(o\|q\)=∏t=1Tπθ​\(yt\|q,y<t\)\\pi\_\{\\theta\}\(o\|q\)=\\prod\_\{t=1\}^\{T\}\\pi\_\{\\theta\}\(y\_\{t\}\|q,y\_\{<t\}\)\(1\)The goal is to optimizeθ\\thetasuch that sequences sampled fromπθ\\pi\_\{\\theta\}are highly valued by a given reawrd functionR​\(o\)R\(o\)\.

### 3\.2Group Relative Policy Optimization\(GRPO\)

Group Relative Policy Optimization\[[15](https://arxiv.org/html/2605.19416#bib.bib2)\]is an off\-policy, critic\-free version of the PPO\[[14](https://arxiv.org/html/2605.19416#bib.bib1)\]algorithm\. It improves sample efficiency and stabilizes training by utilizing an off\-policy framework and clipping the importance sampling ratio\.

For a given promptqq, the old policyπθo​l​d\\pi\_\{\\theta\_\{old\}\}generates a group ofGGresponseso1,…,oG\{o\_\{1\},\.\.\.,o\_\{G\}\}\. Each responseoio\_\{i\}receives a sequence\-level rewardR​\(oi\)R\(o\_\{i\}\)and it’s used to compute the advantage for each response\.

Ai^=R​\(oi\)−μRσR\+ϵ\\hat\{A\_\{i\}\}=\\frac\{R\(o\_\{i\}\)\-\\mu\_\{R\}\}\{\\sigma\_\{R\}\+\\epsilon\}\(2\)whereμR\\mu\_\{R\}andσR\\sigma\_\{R\}are the mean and standard deviation of rewards inOqO\_\{q\}\. Since the policyπθ\\pi\_\{\\theta\}is autoregressive, the optimization objective must be formulated at the token level\. The advantageAi^\\hat\{A\_\{i\}\}is a sequence\-level quantity, but it is distributed as a constant credit to every token\-level generation step within that sequence\. The full GRPO objective is thus given by:

𝒥GRPO​\(θ\)\\displaystyle\\mathcal\{J\}\_\{\\text\{GRPO\}\}\(\\theta\)\(3\)=\\displaystyle=𝔼q∼P​\(Q\),\{oi\}∼πθold​\[1G​∑i=1G1\|oi\|​∑t=1\|oi\|Lclip​\(oi,t,θ\)\]\\displaystyle\\mathbb\{E\}\_\{q\\sim P\(Q\),\\\{o\_\{i\}\\\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\}\\left\[\\frac\{1\}\{G\}\\sum\_\{i=1\}^\{G\}\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{t=1\}^\{\|o\_\{i\}\|\}L\_\{\\text\{clip\}\}\(o\_\{i,t\},\\theta\)\\right\]
whereLclip​\(oi,t,θ\)L\_\{\\text\{clip\}\}\(o\_\{i,t\},\\theta\)is the token\-level clipped surrogate objective:

ℒclip​\(𝐨i,t,θ\)=min\(w\(𝐨i,t\|θ\)A^i,clip\(w\(𝐨i,t\|θ\),1−ϵ,1\+ϵ\)A^i\)\\begin\{split\}\\mathcal\{L\}\_\{\\text\{clip\}\}\(\\mathbf\{o\}\_\{i,t\},\\theta\)&=\\min\\left\(w\(\\mathbf\{o\}\_\{i,t\}\|\\theta\)\\hat\{A\}\_\{i\},\\right\.\\\\ &\\quad\\left\.\\text\{clip\}\(w\(\\mathbf\{o\}\_\{i,t\}\|\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\_\{i\}\\right\)\\end\{split\}\(4\)The per\-token importance ratio isw​\(𝐨i,t\|θ\)=πθ​\(yt\|q,y<t\)πθold​\(yt\|q,y<t\)w\(\\mathbf\{o\}\_\{i,t\}\|\\theta\)=\\frac\{\\pi\_\{\\theta\}\(y\_\{t\}\|q,y\_\{<t\}\)\}\{\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(y\_\{t\}\|q,y\_\{<t\}\)\}, andε\\varepsilonis the clipping hyperparameter\. Often, a KL\-divergence penalty,−βDKL\[πθ\|\|πref\]\-\\beta D\_\{\\text\{KL\}\}\[\\pi\_\{\\theta\}\|\|\\pi\_\{\\text\{ref\}\}\], is added to regularize the policy and prevent it from deviating too far from a trusted reference policyπref\\pi\_\{\\text\{ref\}\}\(e\.g\., the initial SFT model\)\.

## 4Methodology

In this section, we present Lambda Policy Optimization \(LambdaPO\)\. We begin by dissecting the information\-theoretic limitations of scalar baseline methods like GRPO\. We then formally derive our Pairwise Decomposed Advantage \(PDA\) estimator, provide a theoretical intuition regarding its gradient dynamics, and conclude with the complete algorithmic framework\.

![Refer to caption](https://arxiv.org/html/2605.19416v1/x1.png)Figure 1:The architectural evolution from GRPO to LambdaPO\.Both frameworks generate a cohort of outputso1,…,oG\{o\_\{1\},\.\.\.,o\_\{G\}\}from a queryqq\. GRPO \(Top\) derives advantages via Z\-score normalization using a standard reward model\. In contrast, LambdaPO \(Bottom\) enhances it by \(1\) incorporating semantic density signals into the reward, and \(2\) replacing the scalar baseline with a fully\-connected pairwise comparison mechanism, enabling the policy to learn from the relative strengths of all generated trajectories\.### 4\.1Revisiting the Relational Bottleneck in GRPO

Group Relative Policy Optimization \(GRPO\) departs from conventional actor–critic methods by eliminating an explicit, learned value function and instead normalizing rewards within a locally sampled cohort of trajectories\. Concretely, given a queryqqand its associated output set𝒪q=\{o1,…,oK\}\\mathcal\{O\}\_\{q\}=\\\{o\_\{1\},\\dots,o\_\{K\}\\\}, the advantage assigned to a trajectoryoio\_\{i\}is computed via a centered statistic,

A^i∝R​\(oi\)−μ𝒪,\\hat\{A\}\_\{i\}\\;\\propto\\;R\(o\_\{i\}\)\-\\mu\_\{\\mathcal\{O\}\},\(5\)whereμ𝒪\\mu\_\{\\mathcal\{O\}\}denotes the empirical mean reward over𝒪q\\mathcal\{O\}\_\{q\}\(optionally rescaled by the standard deviationσ𝒪\\sigma\_\{\\mathcal\{O\}\}\)\. This design choice yields substantial computational savings and simplifies optimization, as discussed in Section 3\.

Despite these practical benefits, this form of aggregation introduces a fundamental limitation that we refer to as the*relational bottleneck*\. By reducing the cohort𝒪q\\mathcal\{O\}\_\{q\}to a small set of scalar moments—typically\(μ𝒪,σ𝒪\)\(\\mu\_\{\\mathcal\{O\}\},\\sigma\_\{\\mathcal\{O\}\}\)—GRPO discards the richer relational structure encoded in the joint distribution of peer trajectories\. In particular, the resulting baseline is permutation invariant: it is insensitive to which specific peers outperform or underperformoio\_\{i\}, and only reflects howoio\_\{i\}compares to an undifferentiated average\.

This invariance has important consequences for the learning dynamics\. From an optimization perspective, the mean\-centered baseline acts as a form of low\-pass filter on the policy gradient\. High\-frequency relational information—such as fine\-grained rank orderings or asymmetric performance gaps among candidates—is suppressed\. For example, knowing that trajectoryoio\_\{i\}is slightly better thanojo\_\{j\}but substantially worse thanoko\_\{k\}defines a directional preference structure that could guide targeted parameter updates\. In contrast, a comparison solely againstμ𝒪\\mu\_\{\\mathcal\{O\}\}collapses these distinctions into a single scalar signal, obscuring which improvements are most salient\.

The impact of this bottleneck is especially pronounced in rank\-sensitive domains, including multi\-step reasoning, program synthesis, and other structured generation tasks\. In such settings, progress often hinges on subtle qualitative differences between high\-performing trajectories rather than coarse separation from the mean\. By failing to preserve pairwise or higher\-order comparisons within𝒪q\\mathcal\{O\}\_\{q\}, GRPO provides an attenuated and sometimes ambiguous credit\-assignment signal\. As a result, even when multiple strong candidates are present, the algorithm may struggle to consistently reinforce the most informative distinctions among them\.

Taken together, these observations suggest that while GRPO’s statistical baseline is efficient, its coarse relational encoding can limit representational and optimization fidelity\. Addressing this relational bottleneck—by reintroducing richer peerwise or distribution\-aware comparisons—appears crucial for improving credit assignment and achieving finer\-grained discrimination among competing trajectories, without forfeiting the scalability advantages that motivate GRPO in the first place\.

### 4\.2Lambda Policy Optimization

To mitigate the relational bottleneck inherent in scalar\-baseline policy optimization, we introduce*Lambda Policy Optimization*\(LambdaPO\)\. The central idea of LambdaPO lies in how the advantage function is conceptualized and computed\. Instead of defining advantage as a deviation from a single global baseline \(e\.g\., a mean or value estimate\), LambdaPO models advantage as an aggregate of*pairwise preferences*among trajectories, thereby embedding relational structure directly into the optimization signal\.

Concretely, we define the*Pairwise Decomposed Advantage*\(PDA\), denoted byAλA\_\{\\lambda\}, for a trajectoryoio\_\{i\}sampled from a group ofGGcandidate trajectories\. The PDA is given by

Aλ​\(oi\)=1G−1​∑j=1j≠iGΔ​Ri​j​σ​\(Δ​si​jτ\),A\_\{\\lambda\}\(o\_\{i\}\)=\\frac\{1\}\{G\-1\}\\sum\_\{\\begin\{subarray\}\{c\}j=1\\\\ j\\neq i\\end\{subarray\}\}^\{G\}\\Delta R\_\{ij\}\\;\\sigma\\\!\\left\(\\frac\{\\Delta s\_\{ij\}\}\{\\tau\}\\right\),\(6\)where each term in the summation captures the relative dominance ofoio\_\{i\}over a peer trajectoryojo\_\{j\}\.

Here,Δ​Ri​j=R​\(oi\)−R​\(oj\)\\Delta R\_\{ij\}=R\(o\_\{i\}\)\-R\(o\_\{j\}\)denotes the extrinsic utility difference between two trajectories\. This reward signal subsumes both binary task correctness and dense reasoning\-alignment feedback \(see[Section4\.3](https://arxiv.org/html/2605.19416#S4.SS3)\), allowing LambdaPO to incorporate fine\-grained supervision beyond sparse success signals\. The quantity

Δ​si​j=log⁡πθ​\(oi∣q\)−log⁡πθ​\(oj∣q\)\\Delta s\_\{ij\}=\\log\\pi\_\{\\theta\}\(o\_\{i\}\\mid q\)\-\\log\\pi\_\{\\theta\}\(o\_\{j\}\\mid q\)\(7\)represents the intrinsic log\-probability gap induced by the current policyπθ\\pi\_\{\\theta\}, reflecting the model’s relative confidence in generatingoio\_\{i\}versusojo\_\{j\}given the same queryqq\. The logistic sigmoid functionσ​\(⋅\)\\sigma\(\\cdot\)maps this confidence gap to a soft weighting factor, while the temperature parameterτ∈ℝ\+\\tau\\in\\mathbb\{R\}^\{\+\}controls the sharpness of the induced preference surface\.

Intuitively, this construction replaces a static, sample\-independent baseline with a*dynamic, peer\-conditioned reference*\. A trajectory receives positive advantage not merely by exceeding an average reward, but by consistently outperforming alternative trajectories, weighted by how decisively the policy itself distinguishes between them\. As a result, the learning signal becomes inherently comparative and context\-aware\.

Equivalently, the PDA can be expressed in expectation form as

Aπθ​\(oi\):=𝔼oj∼πθ​\[Δ​R​\(oi,oj\)⋅σ​\(1τ​Δ​log⁡πθ​\(oi,oj\)\)\],A\_\{\\pi\_\{\\theta\}\}\(o\_\{i\}\):=\\mathbb\{E\}\_\{o\_\{j\}\\sim\\pi\_\{\\theta\}\}\\left\[\\Delta R\(o\_\{i\},o\_\{j\}\)\\cdot\\sigma\\\!\\left\(\\frac\{1\}\{\\tau\}\\Delta\\log\\pi\_\{\\theta\}\(o\_\{i\},o\_\{j\}\)\\right\)\\right\],\(8\)which highlights that LambdaPO estimates advantage by integrating reward differentials over a latent pairwise preference manifold induced by the policy\.

Algorithm 1Lambda Policy Optimization \(LambdaPO\)1:Input:Dataset

𝒟\\mathcal\{D\}, Policy

πθ\\pi\_\{\\theta\}, Reference

πref\\pi\_\{\\text\{ref\}\}, Group size

GG, Temperature

τ\\tau, KL coef

β\\beta\.

2:Initialize

θold←θ\\theta\_\{\\text\{old\}\}\\leftarrow\\theta
3:repeat

4:Sample prompt batch

B∼𝒟B\\sim\\mathcal\{D\}
5:foreach

q∈Bq\\in Bdo

6:*// Rollout Phase*

7:Generate cohort

𝒪q=\{o1,…,oG\}∼πθold\(⋅\|q\)\\mathcal\{O\}\_\{q\}=\\\{o\_\{1\},\\dots,o\_\{G\}\\\}\\sim\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(\\cdot\|q\)
8:Compute rewards

ℛ=\{R​\(o1\),…,R​\(oG\)\}\\mathcal\{R\}=\\\{R\(o\_\{1\}\),\\dots,R\(o\_\{G\}\)\\\}
9:Compute sequence log\-probs

si=log⁡πθold​\(oi\|q\)s\_\{i\}=\\log\\pi\_\{\\theta\_\{\\text\{old\}\}\}\(o\_\{i\}\|q\)
10:*// Advantage Estimation Phase*

11:for

i=1i=1to

GGdo

12:

Ai←0A\_\{i\}\\leftarrow 0
13:for

j=1j=1to

GG,

j≠ij\\neq ido

14:

Δ​Ri​j←R​\(oi\)−R​\(oj\)\\Delta R\_\{ij\}\\leftarrow R\(o\_\{i\}\)\-R\(o\_\{j\}\)
15:

Δ​si​j←si−sj\\Delta s\_\{ij\}\\leftarrow s\_\{i\}\-s\_\{j\}
16:

Ai←Ai\+Δ​Ri​j⋅σ​\(Δ​si​j/τ\)A\_\{i\}\\leftarrow A\_\{i\}\+\\Delta R\_\{ij\}\\cdot\\sigma\(\\Delta s\_\{ij\}/\\tau\)
17:endfor

18:

Aλ​\(oi\)←Ai/GA\_\{\\lambda\}\(o\_\{i\}\)\\leftarrow A\_\{i\}/G
19:endfor

20:Normalize

AλA\_\{\\lambda\}over cohort \(optional\)

21:endfor

22:*// Optimization Phase*

23:Update

θ\\thetaby maximizing the surrogate objective:

24:

ℒ​\(θ\)=𝔼𝒪q​\[1\|oi\|​∑tℒclip​\(𝐨i,t,θ\)−β​𝔻KL\]\\mathcal\{L\}\(\\theta\)=\\mathbb\{E\}\_\{\\mathcal\{O\}\_\{q\}\}\\left\[\\frac\{1\}\{\|o\_\{i\}\|\}\\sum\_\{t\}\\mathcal\{L\}\_\{\\text\{clip\}\}\(\\mathbf\{o\}\_\{i,t\},\\theta\)\-\\beta\\mathbb\{D\}\_\{\\text\{KL\}\}\\right\], where

ℒclip​\(𝐨i,t,θ\)\\mathcal\{L\}\_\{\\text\{clip\}\}\(\\mathbf\{o\}\_\{i,t\},\\theta\)is given by Eq\.[4](https://arxiv.org/html/2605.19416#S3.E4)

25:

θold←θ\\theta\_\{\\text\{old\}\}\\leftarrow\\theta
26:untilconvergence

### 4\.3Dense Reward Augmentation via Semantic Overlap

Standard reinforcement learning for reasoning tasks typically relies on a binary outcome reward,R∈\{0,1\}R\\in\\\{0,1\\\}which creates a sparse supervision signal\. The policy struggles to distinguish between a good attempt that failed at the last step and a complete hallucination\.

To mitigate this, we augment the extrinsic utility with a Semantic Density Reward,Rs​e​mR\_\{sem\}, which acts as a dense proxy for reasoning quality\. Drawing inspiration from verification\-based RL\[[16](https://arxiv.org/html/2605.19416#bib.bib24)\], we leverage standard n\-gram overlap metrics to quantify the alignment between the generated reasoning traceoio\_\{i\}and the ground\-truth solutiony∗y^\{\*\}\. Specifically, we utilize the ROUGE\-L \(Longest Common Subsequence\)\[[8](https://arxiv.org/html/2605.19416#bib.bib13)\]metric, which is particularly well\-suited for reasoning tasks as it captures both the lexical overlap and the sequential structure of the logical deduction\.

We define the semantic reward as the ROUGE\-L F1\-score:

Rsem​\(oi\)=ROUGE\-LF1​\(oi,y∗\)R\_\{\\text\{sem\}\}\(o\_\{i\}\)=\\text\{ROUGE\-L\}\_\{\\text\{F1\}\}\(o\_\{i\},y^\{\*\}\)\(9\)

### 4\.4Theoretical Explanation: Gradient Dynamics

The efficacy of LambdaPO stems from the unique kinematics of the resulting policy gradient\. By substitutingAλ​\(oi\)A\_\{\\lambda\}\(o\_\{i\}\)into the policy gradient theorem, the update for trajectoryoio\_\{i\}becomes proportional to:

∇θJ∝∑j≠i\(R​\(oi\)−R​\(oj\)\)⏟Direction & Magnitude⋅σ​\(si−sjτ\)⏟Confidence Weight⋅∇θlog⁡πθ​\(oi\)\\nabla\_\{\\theta\}J\\propto\\sum\_\{j\\neq i\}\\underbrace\{\(R\(o\_\{i\}\)\-R\(o\_\{j\}\)\)\}\_\{\\text\{Direction \\& Magnitude\}\}\\cdot\\underbrace\{\\sigma\\left\(\\frac\{s\_\{i\}\-s\_\{j\}\}\{\\tau\}\\right\)\}\_\{\\text\{Confidence Weight\}\}\\cdot\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(o\_\{i\}\)\(10\)This structure reveals two critical properties:

1\.Directional Coherence:Unlike GRPO, where the sign of the advantage depends on the variance\-sensitive group mean, LambdaPO’s gradient sign is strictly determined by the pairwise reward differentialΔ​Ri​j\\Delta R\_\{ij\}\. This ensures that the optimization landscape is homeomorphic to the true reward landscape\.

2\.Self\-Annealing via Confidence:The sigmoid term acts as a differentiable gate\.

- •Case I\(Model Confusion\):When the policy cannot distinguish betweenoio\_\{i\}andojo\_\{j\}\(i\.e\.,si≈sjs\_\{i\}\\approx s\_\{j\}\),σ≈0\.5\\sigma\\approx 0\.5, allowing the full magnitude of the reward difference to drive learning\.
- •Case II\(Model Misalignment:For example, when the model confidently ranks the worse trajectory higher \(sj≫sis\_\{j\}\\gg s\_\{i\}despiteRi\>RjR\_\{i\}\>R\_\{j\}\), the sigmoid term approaches 0, but the gradient with respect to the log\-odds \(derived from the implicit objective\) creates a strong correction signal \(analogous to the LambdaRank mechanism\)\.
- •Case III\(Convergence\):As the policy aligns with the reward \(si≫sjs\_\{i\}\\gg s\_\{j\}forRi\>RjR\_\{i\}\>R\_\{j\}\), the gradient contribution naturally decays, preventing over\-optimization on established patterns and focusing capacity on unresolved distinctions\.

The temperatureτ\\taufurther modulates this dynamic, serving as a spectral regularization term that prevents gradient saturation in the early stages of training\.

### 4\.5The LambdaPO Algorithm

The complete training procedure integrates the Pairwise Decomposed Advantage into the PPO clipping mechanism\. The algorithm preserves the critic\-free efficiency of GRPO while maximizing the information extraction from each sampled cohort\.

## 5Experiments

### 5\.1Experimental Setup

#### 5\.1\.1Datasets

We focus our evaluation on mathematical reasoning, a domain characterized by strict logical dependencies and binary reward signals \(correct/incorrect\)\.

- •RL Training Stage: We construct our training corpus using OpenR1\-Math\-220k and GSM8K\[[3](https://arxiv.org/html/2605.19416#bib.bib8)\]\. This combination provides a diverse spectrum of problem difficulties, ranging from grade\-school arithmetic to competition\-level mathematics\. For each training prompt, rewards are computed based on accuracy and format\.
- •Evaluation Benchmarks: We report performance on three progressively challenging benchmarks: AIME24\[[24](https://arxiv.org/html/2605.19416#bib.bib9)\], MATH\-500\[[7](https://arxiv.org/html/2605.19416#bib.bib11)\], and GPQA\-Diamond\[[12](https://arxiv.org/html/2605.19416#bib.bib10)\]\.

#### 5\.1\.2Configurations

- •Baseline: We primarily compare LambdaPO against Group Relative Policy Optimization \(GRPO\)\. Since LambdaPO is architected as a direct, plug\-and\-play replacement for the advantage estimation module of GRPO, this comparison strictly isolates the contribution of the pairwise decomposition mechanism\.
- •Models: We evaluate our method across different architectures and scales to assess generalization\. Specifically, we utilize the Qwen3 family\[[19](https://arxiv.org/html/2605.19416#bib.bib16)\]\(1\.7B and 4B parameters\) and Phi\-4\-mini\[[1](https://arxiv.org/html/2605.19416#bib.bib17)\]\(4B\)\.
- •Sampling Config: We employ a group cohort size ofG=8G=8
- •Reward Function: We utilize a composite reward functionR​\(o\)=Ra​c​c\+λ​Rf​m​t\+β​Rs​e​mR\(o\)=R\_\{acc\}\+\\lambda R\_\{fmt\}\+\\beta R\_\{sem\}\. Here,Ra​c​cR\_\{acc\}is a binary correctness reward,Rf​m​tR\_\{fmt\}is a format compliance reward that penalizes structural deviations andRs​e​mR\_\{sem\}is the semantic density reward\.
- •LambdaPO Settings: Unless otherwise stated, we set the temperature parameter toτ=1\.0\\tau=1\.0based on our ablation findings\.

Table 1:Model Performance Comparison on Benchmarks

### 5\.2Experimental results

Table[1](https://arxiv.org/html/2605.19416#S5.T1)presents the comparative performance of LambdaPO versus the GRPO baseline across two distinct model architectures \(Qwen3 and Phi\-4\-mini\)\.

Performance Superiority\.LambdaPO consistently outperforms GRPO across all evaluated tasks\. On the Qwen3\-4B model, our method achieves an average accuracy of 76\.49%, a substantial improvement of \+1\.45% over the baseline\. This suggests that pairwise comparisons are particularly effective for hard reasoning tasks where distinguishing between high\-quality reasoning paths is critical\.

Architecture Generalization\.The gains are not specific to a single architecture\. On Phi\-4\-mini, LambdaPO achieves an average improvement of \+1\.86% \(55\.35% \-\>\>56\.38%\)\. The robustness of these improvements across different model families validates the universality of the ”Relational Bottleneck” hypothesis and the effectiveness of our solution\.

### 5\.3Ablation Studies and Analysis

Parameter Sensitivity\(τ\\tau\)\. The temperature parameterτ\\tauin Eq\. \(4\) controls the sensitivity of the sigmoid weighting function, effectively determining the ”softness” of the pairwise ranking\. We conducted a sweep ofτ∈1,1\.5,2,5,7\\tau\\in\{1,1\.5,2,5,7\}using the Qwen3\-1\.7B model\.

- •Low Temperature\(τ=1\\tau=1\):The gradient signal becomes too deterministic, leading to optimization instability\. The performance \(58\.80%\) is better than the baseline but suboptimal\.
- •Optimal Temperature\(τ=1\.5\\tau=1\.5\):We observe a performance peak atτ=1\.5\\tau=1\.5with an average score of 59\.60%, significantly outperforming the base GRPO \(57\.72%\)\. This validates our hypothesis that a ”soft” ranking signal, which accounts for model uncertainty, provides the highest quality gradient\.
- •High Temperature\(τ≥5\\tau\\geq 5\):Asτ\\tauincreases, the sigmoid saturates towards linearity \(or flatness\), washing out the pairwise distinctions\. Atτ=7\\tau=7, performance degrades to 57\.68%, effectively reverting to the performance of mean\-aggregation\.

Impact of Semantic Density Reward\.To disentangle the gains attributed to the pairwise optimization algorithm from those provided by the dense supervision signal, we conduct an ablation study on the reward composition\. We observe that the Semantic Density Reward contributes a distinct performance gain on the AIME benchmark\.

![Refer to caption](https://arxiv.org/html/2605.19416v1/x2.png)Figure 2:Ablation study on Semantic Density Reward\.Training Dynamics and Stability\.To investigate the stability of the alignment process, we visualize the evolution of the accuracy reward over 1,800 optimization steps in[Figure3](https://arxiv.org/html/2605.19416#A1.F3)\. The training trajectories reveal a critical divergence in robustness\. While both methods exhibit comparable learning gains in the initial phase \(steps 0–400\), the GRPO baseline \(blue line\) suffers from severe instability in the latter stages, eventually leading to a catastrophic policy collapse after step 1,200 where the reward degrades to near\-zero\. This degradation is indicative of the model losing the capacity to generate valid terminal answers, likely due to high\-variance gradient updates pushing the policy into degenerate regions of the solution space\. In contrast, LambdaPO \(orange line\) demonstrates superior resilience\. By anchoring optimization on pairwise preference differentials rather than a volatile scalar baseline, LambdaPO maintains a consistently higher reward floor and effectively prevents the policy degradation observed in the baseline\. This confirms that the self\-annealing mechanism of the pairwise objective acts as a crucial regularizer, preserving generation quality even deep into the training stage\.

Table 2:Qualitative comparison of reasoning trajectories\.We illustrate the generation process for a geometry problem\. The GRPO baseline \(top\) suffers from reasoning loops and verbose repetition, consuming around 8k tokens without reaching a conclusion\. In contrast, LambdaPO \(bottom\) produces a structured result that is 50% more efficient \(around 4k tokens\) and successfully converges to the correct solution\.Response\(∼\\sim8k tokens\) from GRPO:<<think\>\>Okay, let’s try to solve this problem step by step\. First, I need to visualize the square ABCD and the circle O tangent to sides AB and AD\. The circle intersects BC at points M and P, and we’re given BM=8 cm and MC=17 cm\. The goal is to find the area of the circle\.First, let me sketch the square ABCD\. Let me label the square with AB, BC, CD, DA as sides\. Let’s assume it’s a standard square with AB as the bottom side, BC as the right side, CD as the top, and DA as the left side\. So, points A, B, C, D are in order\.…Hmm\. Let me try to think about the given values\. BM = 8 cm, MC = 17 cm\. So BC is 8 \+ 17 = 25 cm\. Therefore, the side length of the square is 25 cm\.Wait\! If BC is 25 cm, then the side length of the square is 25 cm\. So a = 25 cm\.So now, with a =Response\(∼\\sim4k tokens\) from LambdaPO \(Ours\):<<think\>\>Okay, let’s try to solve this problem step by step\. First, I need to visualize the square ABCD\. Let me sketch it mentally\. Let’s say the square is labeled clockwise as A, B, C, D\. So, AB is one side, BC is the next, and so on\. The circle O is tangent to sides AB and AD at points L and K, respectively\. Also, the circle intersects BC at points M and P\. Given BM = 8 cm and MC = 17 cm, so BC is 8 \+ 17 = 25 cm\.…Another way to think: The circle is tangent to AB and AD, so it’s a circle in the corner of the square\. The center is at \(r, r\)\. Then, the equation is correct\. Then, the intersection with BC is as calculated\. So all steps are correct\.Therefore, the area is169​π169\\pi\. So the answer is169​π169\\pi\.<</think\>\><<answer\>\>The area of the circle is169​π\\boxed\{169\\pi\}square centimeters\.<</answer\>\>
### 5\.4Case Studies

We further analyze specific failure cases of GRPO corrected by LambdaPO\. In multi\-step math problem reasoning, a common failure mode observed in GRPO is that the model fails to resolve the mathematical query within the strict token budget\. In contrast, LambdaPO demonstrates improved generation efficiency, successfully converging to valid solution within the allocated constraints\. This suggests that the pairwise advantage formulation provides a better gradient signal that implicitly regularizes the inefficient budget\-exhausted trajectories\. The concrete cases are presented in Table[2](https://arxiv.org/html/2605.19416#S5.T2)\.

## 6Conclusion

In this work, we presentedLambda Policy Optimization \(LambdaPO\), a principled framework that resolves the relational bottleneck inherent in group\-relative alignment algorithms\. By re\-conceptualizing advantage estimation from a scalar deviation to a Pairwise Decomposed Advantage, we demonstrated that the rich, high\-dimensional preference topology within sampled cohorts can be effectively harvested to drive policy improvement\.Furthermore, we augmented the training objective with a Semantic Density Reward, derived from precision\-recall alignment with ground\-truth reasoning, to mitigate the challenge of sparse outcome supervision in complex reasoning tasks\. Our method synthesizes the gradient kinematics of learning\-to\-rank with the stability of trust\-region optimization, yielding a learning signal that is both directionally coherent and dynamically self\-annealing\. Empirical evaluations across the Qwen3 and Phi\-4 families confirm that this approach significantly outperforms scalar\-baseline methods on rigorous mathematical reasoning benchmarks, particularly in regimes requiring complex, multi\-step logic\.

## Impact Statement

This work proposesLambda Policy Optimization \(LambdaPO\), a reinforcement learning framework for improving the reasoning capabilities of large language models via pairwise, ranking\-aware policy optimization\. The primary goal of this research is to advance the methodological foundations of machine learning, particularly in the areas of reinforcement learning, learning\-to\-rank, and large\-scale model alignment\.

Potential Positive Impacts\.By enabling more fine\-grained credit assignment over groups of generated trajectories, LambdaPO has the potential to improve the reliability, efficiency, and stability of reasoning\-oriented language models\. Such improvements may benefit downstream applications that rely on structured reasoning, including mathematics education, scientific problem solving, and complex decision support systems\. Additionally, the proposed framework enhances sample efficiency and reduces reliance on auxiliary value models, which may lower the computational and engineering barriers for training aligned language models, thereby promoting broader accessibility to advanced ML techniques in both academic and industrial settings\.

Ethical Considerations and Risks\.As with most advances in large language model training, the techniques introduced in this paper could be misused to amplify the persuasive or deceptive capabilities of automated systems if deployed without appropriate safeguards\. More effective reasoning and alignment methods may increase the plausibility of generated outputs, which underscores the importance of responsible deployment, transparency, and downstream monitoring\. Importantly, LambdaPO does not introduce new data sources, does not alter the supervision paradigm beyond reward shaping, and does not inherently increase model autonomy or agency\. The method operates entirely within standard reinforcement learning pipelines and inherits the same ethical considerations as existing RL\-based alignment approaches\.

Societal Implications\.We do not foresee immediate negative societal consequences that are unique to LambdaPO compared to prior work on reinforcement learning for language models\. The societal impact of this research is therefore expected to be incremental and aligned with the broader trajectory of progress in machine learning\. As with all such work, long\-term outcomes will depend primarily on how the resulting models are deployed, governed, and integrated into real\-world systems\.

Overall, this paper presents a technical contribution intended to advance the state of the art in machine learning optimization methods\. We believe that its ethical profile is comparable to existing research in reinforcement learning for large language models, and that any broader impacts—positive or negative—are best addressed through responsible use, evaluation, and governance at the system\-deployment level\.

## References

- \[1\]M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.\(2024\)Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[2nd item](https://arxiv.org/html/2605.19416#S5.I2.i2.p1.1)\.
- \[2\]C\. Burges, R\. Ragno, and Q\. Le\(2006\)Learning to rank with nonsmooth cost functions\.InAdvances in Neural Information Processing Systems,B\. Schölkopf, J\. Platt, and T\. Hoffman \(Eds\.\),Vol\.19,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2006/file/af44c4c56f385c43f2529f9b1b018f6a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2605.19416#S1.p3.1)\.
- \[3\]K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman\(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[1st item](https://arxiv.org/html/2605.19416#S5.I1.i1.p1.1)\.
- \[4\]K\. Ethayarajh, W\. Xu, N\. Muennighoff, D\. Jurafsky, and D\. Kiela\(2024\)KTO: model alignment as prospect theoretic optimization\.External Links:2402\.01306,[Link](https://arxiv.org/abs/2402.01306)Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p7.1)\.
- \[5\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p5.1)\.
- \[6\]J\. Hong, N\. Lee, and J\. Thorne\(2024\)ORPO: monolithic preference optimization without reference model\.External Links:2403\.07691,[Link](https://arxiv.org/abs/2403.07691)Cited by:[§1](https://arxiv.org/html/2605.19416#S1.p1.1)\.
- \[7\]H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe\(2023\)Let’s verify step by step\.arXiv preprint arXiv:2305\.20050\.Cited by:[2nd item](https://arxiv.org/html/2605.19416#S5.I1.i2.p1.1)\.
- \[8\]C\. Lin\(2004\-07\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§4\.3](https://arxiv.org/html/2605.19416#S4.SS3.p2.3)\.
- \[9\]Y\. Meng, M\. Xia, and D\. Chen\(2024\)SimPO: simple preference optimization with a reference\-free reward\.External Links:2405\.14734,[Link](https://arxiv.org/abs/2405.14734)Cited by:[§1](https://arxiv.org/html/2605.19416#S1.p1.1)\.
- \[10\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.\(2022\)Training language models to follow instructions with human feedback\.Advances in neural information processing systems35,pp\. 27730–27744\.Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p5.1)\.
- \[11\]R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn\(2024\)Direct preference optimization: your language model is secretly a reward model\.External Links:2305\.18290,[Link](https://arxiv.org/abs/2305.18290)Cited by:[§1](https://arxiv.org/html/2605.19416#S1.p1.1)\.
- \[12\]D\. Rein, B\. L\. Hou, A\. C\. Stickland, J\. Petty, R\. Y\. Pang, J\. Dirani, J\. Michael, and S\. R\. Bowman\(2023\)GPQA: a graduate\-level google\-proof q&a benchmark\.External Links:2311\.12022,[Link](https://arxiv.org/abs/2311.12022)Cited by:[2nd item](https://arxiv.org/html/2605.19416#S5.I1.i2.p1.1)\.
- \[13\]T\. Schick, J\. Dwivedi\-Yu, R\. Dessì, R\. Raileanu, M\. Lomeli, E\. Hambro, L\. Zettlemoyer, N\. Cancedda, and T\. Scialom\(2023\)Toolformer: language models can teach themselves to use tools\.Advances in Neural Information Processing Systems36,pp\. 68539–68551\.Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p4.1)\.
- \[14\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§1](https://arxiv.org/html/2605.19416#S1.p1.1),[§2](https://arxiv.org/html/2605.19416#S2.p6.1),[§3\.2](https://arxiv.org/html/2605.19416#S3.SS2.p1.1)\.
- \[15\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2605.19416#S1.p1.1),[§2](https://arxiv.org/html/2605.19416#S2.p5.1),[§3\.2](https://arxiv.org/html/2605.19416#S3.SS2.p1.1)\.
- \[16\]J\. Uesato, N\. Kushman, R\. Kumar, F\. Song, N\. Siegel, L\. Wang, A\. Creswell, G\. Irving, and I\. Higgins\(2022\)Solving math word problems with process\- and outcome\-based feedback\.External Links:2211\.14275,[Link](https://arxiv.org/abs/2211.14275)Cited by:[§4\.3](https://arxiv.org/html/2605.19416#S4.SS3.p2.3)\.
- \[17\]P\. Wang, L\. Li, Z\. Shao, R\. X\. Xu, D\. Dai, Y\. Li, D\. Chen, Y\. Wu, and Z\. Sui\(2024\)Math\-shepherd: verify and reinforce llms step\-by\-step without human annotations\.External Links:2312\.08935,[Link](https://arxiv.org/abs/2312.08935)Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p3.1)\.
- \[18\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. Chi, Q\. V\. Le, D\. Zhou,et al\.\(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in neural information processing systems35,pp\. 24824–24837\.Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p3.1)\.
- \[19\]A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[2nd item](https://arxiv.org/html/2605.19416#S5.I2.i2.p1.1)\.
- \[20\]A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu\(2024\)Qwen2\.5 technical report\.arXiv preprint arXiv:2412\.15115\.Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p7.1)\.
- \[21\]S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao\(2022\)React: synergizing reasoning and acting in language models\.InThe eleventh international conference on learning representations,Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p4.1)\.
- \[22\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu,et al\.\(2025\)Dapo: an open\-source llm reinforcement learning system at scale\.arXiv preprint arXiv:2503\.14476\.Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p7.1)\.
- \[23\]E\. Zelikman, Y\. Wu, J\. Mu, and N\. D\. Goodman\(2022\)STaR: bootstrapping reasoning with reasoning\.External Links:2203\.14465,[Link](https://arxiv.org/abs/2203.14465)Cited by:[§2](https://arxiv.org/html/2605.19416#S2.p3.1)\.
- \[24\]Y\. Zhang and T\. Math\-AI\(2024\)American invitational mathematics examination \(aime\) 2024\.Cited by:[2nd item](https://arxiv.org/html/2605.19416#S5.I1.i2.p1.1)\.

## Appendix ASupplementary materials\.

##### Logging files

##### Visualization of training dynamics

The training dynamics of Qwen3\-1\.7B, Qwen3\-4B, and Phi4\-mini in Figure[3](https://arxiv.org/html/2605.19416#A1.F3)\.

![Refer to caption](https://arxiv.org/html/2605.19416v1/x3.png)

![Refer to caption](https://arxiv.org/html/2605.19416v1/x4.png)

![Refer to caption](https://arxiv.org/html/2605.19416v1/x5.png)

![Refer to caption](https://arxiv.org/html/2605.19416v1/x6.png)

![Refer to caption](https://arxiv.org/html/2605.19416v1/x7.png)

![Refer to caption](https://arxiv.org/html/2605.19416v1/x8.png)

Figure 3:Comparison on Qwen3\-1\.7B, Qwen3\-4B, and Phi4\-mini\.