Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasoners

# Process Advantage Signal Shaping: A Paradigm-Agnostic Middleware for Process-Supervised RL in LLM Reasoners
Source: [https://arxiv.org/html/2606.29296](https://arxiv.org/html/2606.29296)
Chao Wang Tsinghua University &Hongtao Tian11footnotemark:1 WeChat, Tencent &Tao Yang WeChat, Tencent Contributed equally\.Work done while interning at Tencent\.Corresponding Author\.[luckytyang@tencent\.com](https://arxiv.org/html/2606.29296v1/[email protected]),[ding\.wenbo@sz\.tsinghua\.edu\.cn](https://arxiv.org/html/2606.29296v1/[email protected])Yunsheng Shi WeChat, Tencent &Ting Yao WeChat, Tencent &Wenbo Ding33footnotemark:3 Tsinghua University

###### Abstract

Group Relative Policy Optimization \(GRPO\) is a default recipe for process\-supervised reinforcement learning of LLM reasoners, and dense process supervision—via learned process reward models \(PRMs\) or on\-policy\-distillation KL signals—is a common way to densify its otherwise weak outcome reward\. Layering such a step\-level signal on top of GRPO’s group\-standardized advantage, however, exposes three structural pathologies:*channel contamination*between the pooled process, outcome, and format streams at group standardization;*resolution mismatch*between the granularity of the process signal and the granularity of the logical decisions being credited; and a*cumulative trap*by which GRPO’s return\-to\-go sum surfaces either length inflation or truncated exploration depending on the sign regime of the signal\. We proposePASS\(*Process Advantage Signal Shaping*\), a compact middleware that sits between any scalar step\-level process signal and GRPO’s clipped surrogate and addresses the three pathologies in turn:*Advantage Fusion*standardizes the three streams independently within each group,*Chunk\-by\-Value*derives value\-homogeneous chunks from the signal itself and broadcasts credit within each chunk, and*Divide\-Length*converts the cumulative objective into an average\-value\-density score\. We validate PASS across two domains and two process\-signal paradigms—a learned PRM on mathematical reasoning and an on\-policy\-distillation KL signal \(with a generalized variant\) on multi\-hop question answering—and under two group\-standardization operators\. In every regime PASS delivers a consistent pass@1 gain over the corresponding GRPO baseline\.

## 1Introduction

Group Relative Policy Optimization \(GRPO\)\(Shaoet al\.,[2024](https://arxiv.org/html/2606.29296#bib.bib1); Guoet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib2)\)has become a default recipe for process\-supervised reinforcement learning \(RL\) of large language model \(LLM\) reasoners\. When an outcome reward alone leaves long reasoning trajectories only weakly supervised, a natural remedy is to add a dense*process signal*that scores the trajectory step by step: a learned process reward model \(PRM\)\(Zhanget al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib7); Lightmanet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib10)\)in mathematical reasoning, or a token\-level KL\-style divergence against a stronger teacher, as in on\-policy distillation\(Agarwalet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib11)\), in more open\-ended reasoning\. In practice, however, attaching such a dense signal to GRPO’s clipped surrogate is rarely a drop\-in operation, and the difficulties it surfaces are not properties of the signal itself but artifacts of how GRPO forms its advantage\.

These difficulties align naturally with the three stages of GRPO’s advantage pipeline: aggregation, broadcast, and trajectory summation\. At the aggregation stage, pooling the process, outcome, and format streams into a single within\-group standardization lets the high\-variance outcome reward absorb the fine\-grained fluctuations ofsprcs\_\{\\text\{prc\}\}; recent work has linked such naïve channel mixing to sharp accuracy drops and has traced the instability back to the group\-relative statistic itself\(Yeet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib21); Liuet al\.,[2026](https://arxiv.org/html/2606.29296#bib.bib9)\)\. At the broadcast stage, the spatial granularity ofsprcs\_\{\\text\{prc\}\}rarely matches the granularity of the logical decisions the policy actually makes: a PRM is too coarse, a token\-level reverse KL is too fine, so the per\-token advantage is either blunt or noisy\. Recent surveys of credit\-assignment granularity indeed single out the absence of a signal\-driven mechanism for matching the resolution of credit to the resolution of reasoning as an open problem\(Zhang,[2026](https://arxiv.org/html/2606.29296#bib.bib26)\)\. At the trajectory\-summation stage, GRPO’s return\-to\-go objective unconditionally rewards any step with a non\-negative marginal contribution, which projects onto two different failure modes depending on the sign regime ofsprcs\_\{\\text\{prc\}\}: length inflation and reward hacking\(Liuet al\.,[2026](https://arxiv.org/html/2606.29296#bib.bib9); Sullivan,[2025](https://arxiv.org/html/2606.29296#bib.bib6); Liet al\.,[2026b](https://arxiv.org/html/2606.29296#bib.bib19)\)under a positively biased signal, and premature truncation under the negatively biased token\-level reverse KL of on\-policy distillation\. A mechanism that addresses only one of these three stages, or only one of these two sign regimes, will leave the rest intact\.

We proposePASS\(*Process Advantage Signal Shaping*\), a compact middleware that sits between any scalar step\-level process signalsprcs\_\{\\text\{prc\}\}and GRPO’s clipped surrogate\. PASS is deliberately not a new RL algorithm: it does not modify the clipped surrogate, the KL regularizer, or the rollout loop; it only reshapes how a generic scalar signal is turned into the per\-token advantage that GRPO consumes\. Its three rules line up, in the same order, with the three stages at which the difficulties above arise:*Advantage Fusion*\(AF\) at aggregation,*Chunk\-by\-Value*\(CV\) at broadcast, and*Divide\-Length*\(DL\) at trajectory summation\. The three rules are stated in terms of a generic scalar step\-level score and the same middleware is applied in both of our domains without algorithmic changes\. Section[4](https://arxiv.org/html/2606.29296#S4)presents the precise definition of each rule and a first\-order analysis of its effect on the GRPO gradient\.

#### Empirical scope\.

We validate PASS in two domains that exercise very different process\-signal paradigms\. In mathematical reasoning,sprcs\_\{\\text\{prc\}\}is a learned PRM score and we evaluate on a standard panel of competition and school\-level benchmarks\. In multi\-hop question answering,sprcs\_\{\\text\{prc\}\}is a token\-level reverse\-KL against a stronger teacher, realized both as vanilla on\-policy distillation \(OPD\) and as a generalized variant \(G\-OPD\) that anchors the divergence at an additional base policy; we evaluate on three multi\-hop benchmarks that vary in hop count and distractor difficulty\. To further isolate the contribution of PASS itself from the choice of numerical preprocessing, we additionally run every multi\-hop configuration under two group\-standardization operators in common use, masked\-mean/std \(Masked\-Norm\) and absolute\-maximum scaling \(Abs\-Max Scaling\)\. Across all of these axes, PASS is a positive and consistent contribution over its corresponding GRPO baseline\.

#### Contributions\.

Our work makes the following contributions\.

- •We identify three structural difficulties that arise when dense process supervision is layered on cumulative\-advantage GRPO—aggregation\-time channel contamination, broadcast\-time resolution mismatch, and trajectory\-time cumulative bias—and introducePASS, a thin advantage\-shaping middleware whose three rules \(AF, CV, DL\) address the three stages one by one\.
- •We prove a length\-collapse theorem for cumulative\-advantage GRPO \(Theorem[1](https://arxiv.org/html/2606.29296#Thmtheorem1)\) and show analytically that the DL rule converts the objective into an average\-value\-density form whose gradient is a two\-sided marginal\-value filter, symmetric in the sign of the marginal contribution\.
- •We validate PASS across two process\-signal paradigms, two domains, and two group\-standardization operators, observing consistent pass@1 gains over the corresponding GRPO baseline in every regime\.

## 2Related Work

#### Process\-reward models and mixed\-reward GRPO\.

Dense PRMs were popularized by step\-level verifiers\(Lightmanet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib10)\)and refined with self\-supervised process\-data pipelines\(Zhanget al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib7)\)\. Layering such signals on GRPO\-style group\-normalized RL\(Shaoet al\.,[2024](https://arxiv.org/html/2606.29296#bib.bib1); Guoet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib2)\), however, repeatedly misbehaves:Sullivan \([2025](https://arxiv.org/html/2606.29296#bib.bib6)\)shows that GRPO itself already acts as an implicit and frequency\-biased process reward estimator, andYeet al\.\([2025](https://arxiv.org/html/2606.29296#bib.bib21)\)reports that naïve mixing of outcome and process rewards can degrade accuracy by double\-digit margins, tracing the instability back to incompatible channels sharing one group\-standardization statistic\. Proposed remedies split into three routes—frequency reweighting within the group\(Sullivan,[2025](https://arxiv.org/html/2606.29296#bib.bib6)\), consistency filtering of rollouts\(Yeet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib21)\), and reward\-decoupled normalization\(Liuet al\.,[2026](https://arxiv.org/html/2606.29296#bib.bib9)\)—all of which motivate PASS’s decoupling at the per\-token advantage level rather than at the scalar reward\.

#### Credit\-assignment granularity\.

A parallel line targets the mismatch between the granularity of an RL signal and that of the logical transitions it should credit\.Zhang \([2026](https://arxiv.org/html/2606.29296#bib.bib26)\)organizes existing approaches into a granularity spectrum and identifies the absence of a signal\-driven mechanism for matching credit resolution to reasoning resolution as a central open problem\. Representative attacks on different points of this spectrum carry different costs: FIPO\(Maet al\.,[2026](https://arxiv.org/html/2606.29296#bib.bib20)\)reweights tokens by a discounted future\-KL term, which scales poorly on long trajectories; counterfactual span weighting\(Khandogaet al\.,[2026](https://arxiv.org/html/2606.29296#bib.bib18)\)masks candidate spans but assumes correctness concentrates on a small, identifiable set; attribution\-based approaches such as ACPO\(Yinet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib25)\)rely on upstream semantic segmenters that degrade on unstructured reasoning\. PASS addresses the similar spectrum without either auxiliary networks or correctness\-localization assumptions\.

#### Length inflation and the cumulative trap\.

Analyses of GRPO’s return\-to\-go structure consistently document a structural bias toward longer trajectories\(Razinet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib3); Liet al\.,[2026b](https://arxiv.org/html/2606.29296#bib.bib19)\)\. Proposed mitigations include multiplicative length rescaling \(GR3\(Liet al\.,[2026b](https://arxiv.org/html/2606.29296#bib.bib19)\)\), which requires careful penalty\-strength tuning, and correctness\-decoupled baselines \(DRPO\(Liet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib22)\)\), which presuppose an oracle that reliably labels rollouts as correct or incorrect\. A recent survey\(Liuet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib24)\)further notes that explicit length penalties cannot cleanly separate necessary long derivations from redundant verbosity\. Divide\-Length sidesteps both issues by rewriting the cumulative objective as an average\-value\-density score, expressing the admission criterion for additional tokens entirely in terms of the signal’s marginal contribution\.

#### On\-policy distillation dynamics\.

On\-policy distillation\(Agarwalet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib11)\)offers an alternative, token\-level KL\-style step signal\.Liet al\.\([2026a](https://arxiv.org/html/2606.29296#bib.bib16)\)shows that the reliability of this signal is not uniform across depth: as the student drifts out\-of\-distribution, elevated teacher entropy emerges at the suffix and propagates backward over training, eventually bottlenecking long\-horizon reasoning\. This substantiates, from a different angle than the GRPO\-native literature above, the need for a shaping mechanism that does not treat every token as a first\-class credit\-assignment unit\. Our HotpotQA experiments \(§[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)\) then show that PASS transfers without algorithmic change from PRM\-style signals to OPD and G\-OPD\(Yanget al\.,[2026](https://arxiv.org/html/2606.29296#bib.bib17)\)KL\-style signals\.

## 3Preliminaries

We briefly formalize Group Relative Policy Optimization \(GRPO\) with process supervision, following the notation ofShaoet al\.\([2024](https://arxiv.org/html/2606.29296#bib.bib1)\); Guoet al\.\([2025](https://arxiv.org/html/2606.29296#bib.bib2)\)\. Letqqdenote an input query and letπθ\\pi\_\{\\theta\}be the policy under optimization\. GRPO samples a group ofGGcandidate outputs\{o1,o2,…,oG\}\\\{o\_\{1\},o\_\{2\},\\dots,o\_\{G\}\\\}fromπθ\(⋅∣q\)\\pi\_\{\\theta\}\(\\cdot\\mid q\)\. Each outputoio\_\{i\}is further segmented intoKiK\_\{i\}reasoning chunks by a tokenizer\-level rule𝒟\\mathcal\{D\}\(e\.g\., step boundaries produced by a reasoning template, or boundaries induced by a score trace\)\. We letindex​\(j\)\\mathrm\{index\}\(j\)denote the ending token index of thejj\-th chunk ofoio\_\{i\}, and writeLiL\_\{i\}for the total length ofoio\_\{i\}\.

#### Process signal\.

We adopt a generic notationsprc​\(i,j\)∈ℝs\_\{\\text\{prc\}\}\(i,j\)\\in\\mathbb\{R\}for a scalar*process signal*evaluated at stepjjof outputoio\_\{i\}\. We intentionally leave the origin ofsprcs\_\{\\text\{prc\}\}unspecified at this stage: it may be a Process Reward Model \(PRM\) score\(Lightmanet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib10)\), a token\-level \(reverse\) KL divergence against a teacher policy, or any other step\-level scalar that ranks local quality\. Section[4](https://arxiv.org/html/2606.29296#S4)instantiatessprcs\_\{\\text\{prc\}\}for both the PRM and the on\-policy distillation\(Agarwalet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib11)\)regimes used in our experiments\.

#### Group normalization\.

Following standard GRPO with process supervision, the raw step\-level signals are pooled across theGGoutputs of the group,

𝐒=\{\{sprc​\(1,1\),…,sprc​\(1,K1\)\},…,\{sprc​\(G,1\),…,sprc​\(G,KG\)\}\},\\mathbf\{S\}=\\left\\\{\\\{s\_\{\\text\{prc\}\}\(1,1\),\\dots,s\_\{\\text\{prc\}\}\(1,K\_\{1\}\)\\\},\\,\\dots,\\,\\\{s\_\{\\text\{prc\}\}\(G,1\),\\dots,s\_\{\\text\{prc\}\}\(G,K\_\{G\}\)\\\}\\right\\\},\(1\)and standardized to produce the per\-step score

s~iindex​\(j\)=𝒩​\(sprc​\(i,j\);𝐒\),\\tilde\{s\}\_\{i\}^\{\\mathrm\{index\}\(j\)\}\\;=\\;\\mathcal\{N\}\\\!\\left\(s\_\{\\text\{prc\}\}\(i,j\);\\,\\mathbf\{S\}\\right\),\(2\)where𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\)denotes a group\-level standardization operator \(its concrete choice is deferred to §[5](https://arxiv.org/html/2606.29296#S5)\)\.

#### Cumulative advantage\.

For any token at positionttwithinoio\_\{i\}, the baseline GRPO with process supervision computes a*cumulative*advantage by summing all remaining normalized step\-level scores,

A^i,t=∑index​\(j\)≥ts~iindex​\(j\)\.\\hat\{A\}\_\{i,t\}\\;=\\;\\sum\_\{\\mathrm\{index\}\(j\)\\,\\geq\\,t\}\\tilde\{s\}\_\{i\}^\{\\mathrm\{index\}\(j\)\}\.\(3\)The policy is then optimized by the GRPO clipped surrogate objective withA^i,t\\hat\{A\}\_\{i,t\}plugged in as the per\-token advantage\. This cumulative\-sum formulation, together with the waysprcs\_\{\\text\{prc\}\}is combined with an outcome rewardroutr\_\{\\text\{out\}\}and a format rewardrfmtr\_\{\\text\{fmt\}\}, will motivate the PASS middleware described next\.

## 4Method: The PASS Middleware

PASS \(*Process Advantage Signal Shaping*\) is a middleware that transforms any scalar step\-level process signalsprcs\_\{\\text\{prc\}\}into a per\-token advantageA~i,t\\tilde\{A\}\_\{i,t\}to be consumed by GRPO’s clipped surrogate, as illustrated in Figure[1](https://arxiv.org/html/2606.29296#S4.F1), without otherwise modifying GRPO’s update rule, KL regularizer, or rollout loop\. It consists of three operators, each a direct response to one of the pathologies identified in §[1](https://arxiv.org/html/2606.29296#S1): \(i\)*Advantage Fusion*\(AF\) answers*channel contamination*by standardizing the process, outcome, and format streams independently within each group before they are recombined; \(ii\)*Chunk\-by\-Value*\(CV\) answers*resolution mismatch*by deriving value\-homogeneous chunks directly fromsprcs\_\{\\text\{prc\}\}and broadcasting credit within each chunk; \(iii\)*Divide\-Length*\(DL\) answers the*cumulative trap*by converting the trajectory objective from a return\-to\-go sum into an average\-value\-density score, whose gradient is a two\-sided marginal\-value filter\. We present each operator in turn; for the remainder of this sectionsprcs\_\{\\text\{prc\}\}retains its generic meaning from §[3](https://arxiv.org/html/2606.29296#S3), and the symbolA~\\tilde\{A\}always denotes the PASS\-reshaped advantage\.

AFFuse multi\-domain signalCVDensify process signalDLFilter marginalvalue \(RTG/\(Ki−j\+1\)k\{\}/\(K\_\{i\}\{\-\}j\{\+\}1\)^\{k\}\)PASS MiddlewareClipped Surrogate\(GRPO Objective\)s0s\_\{0\}s1s\_\{1\}⋯\\cdotssns\_\{n\}Baseline Aggregation

Figure 1:The process\-supervised RL computation graph\. Baseline GRPO pools all reward signals directly into a single surrogate\. PASS intercepts the multi\-domain process signalss0,s1,…,sns\_\{0\},s\_\{1\},\\ldots,s\_\{n\}, where eachsis\_\{i\}represents the process signal from a different domain, via a three\-stage middleware: Advantage Fusion \(AF\), Chunk\-by\-Value \(CV\), and Divide\-Length \(DL\)\. DL is implemented as a chunk\-count residual divisor\(Ki−j\+1\)k\(K\_\{i\}\-j\+1\)^\{k\}applied on top of the per\-chunk return\-to\-goGijG\_\{i\}^\{j\}, reshaping the per\-token advantage without modifying the underlying RL algorithm\.### 4\.1Advantage Fusion

A reasoning trajectoryoio\_\{i\}exposes three supervisory streams: the step\-level process signalsprc​\(i,j\)s\_\{\\text\{prc\}\}\(i,j\), a sequence\-level outcome rewardrout,ir\_\{\\text\{out\},i\}, and a sequence\-level format rewardrfmt,ir\_\{\\text\{fmt\},i\}\. The baseline GRPO recipe pools them into a single global set and standardizes that union within the group\. Because the outcome reward is near\-binary and therefore high\-variance, the union statistic is dominated by outcome variance, and the clamped group\-normalized advantage becomes largely insensitive to the step\-level fluctuations ofsprcs\_\{\\text\{prc\}\}\(channel contamination, Pathology 1\)\. AF instead normalizes the three streams*independently within the group*\. We denote the resulting per\-step normalized scores bys~prc\(t\)\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\},r~out\(t\)\\tilde\{r\}\_\{\\text\{out\}\}^\{\(t\)\}, andr~fmt\(t\)\\tilde\{r\}\_\{\\text\{fmt\}\}^\{\(t\)\}, and attach static mixing weightswprc,wout,wfmtw\_\{\\text\{prc\}\},w\_\{\\text\{out\}\},w\_\{\\text\{fmt\}\}with total magnitudeW=wprc\+wout\+wfmtW=w\_\{\\text\{prc\}\}\+w\_\{\\text\{out\}\}\+w\_\{\\text\{fmt\}\}\. A format indicator𝕀fmt,i∈\{0,1\}\\mathbb\{I\}\_\{\\text\{fmt\},i\}\\in\\\{0,1\\\}marks whetheroio\_\{i\}complies with the required output template\. The fused advantage is then the piecewise function

A~\(t\)=\{wprc​s~prc\(t\)\+wout​r~out\(t\)\+wfmt​r~fmt\(t\),𝕀fmt,i=1,W⋅r~fmt\(t\),𝕀fmt,i=0,\\tilde\{A\}^\{\(t\)\}=\\begin\{cases\}w\_\{\\text\{prc\}\}\\,\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\}\+w\_\{\\text\{out\}\}\\,\\tilde\{r\}\_\{\\text\{out\}\}^\{\(t\)\}\+w\_\{\\text\{fmt\}\}\\,\\tilde\{r\}\_\{\\text\{fmt\}\}^\{\(t\)\},&\\mathbb\{I\}\_\{\\text\{fmt\},i\}=1,\\\\\[2\.0pt\] W\\cdot\\tilde\{r\}\_\{\\text\{fmt\}\}^\{\(t\)\},&\\mathbb\{I\}\_\{\\text\{fmt\},i\}=0,\\end\{cases\}\(4\)which acts as a mild curriculum: when the format is violated, both the process and outcome channels are gated off and the gradient is devoted entirely to recovering the format, with the rescaling byWWkeeping the gradient norm comparable across the two branches\. This before\-standardization decoupling is conceptually complementary to post\-hoc reward\-calibration approaches that regress out spurious correlates such as length bias after the reward has already been aggregated\(Singhalet al\.,[2024](https://arxiv.org/html/2606.29296#bib.bib27)\): AF intervenes at the group aggregation statistic itself rather than at the scalar reward, and therefore does not require estimating a calibration function or assuming a specific functional form for the contaminating correlate\. AF is the PASS response to Pathology 1 \(channel contamination\); its pseudocode is given in Algorithm[1](https://arxiv.org/html/2606.29296#alg1), and an empirical comparison against the more common baseline of linearly combining the three raw rewards before group standardization is reported in §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1)\(Table[3](https://arxiv.org/html/2606.29296#S5.T3)\)\.

#### Two instantiations ofsprcs\_\{\\text\{prc\}\}\.

AF is agnostic to howsprcs\_\{\\text\{prc\}\}is sourced\. We highlight two concrete regimes that cover the experiments of §[5](https://arxiv.org/html/2606.29296#S5):

- •PRM\-style\.sprc​\(i,j\)=rprm,iindex​\(j\)s\_\{\\text\{prc\}\}\(i,j\)=r\_\{\\text\{prm\},i\}^\{\\mathrm\{index\}\(j\)\}, a step\-level score from a learned PRM\(Lightmanet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib10)\)\. Used for the math\-reasoning experiments in §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1)\.
- •KL\-style \(on\-policy distillation\)\.Writingℓteacher​\(t\)=log⁡πteacher​\(xt∣x<t\)\\ell\_\{\\text\{teacher\}\}\(t\)=\\log\\pi\_\{\\text\{teacher\}\}\(x\_\{t\}\\mid x\_\{<t\}\)andℓθ​\(t\)=log⁡πθ​\(xt∣x<t\)\\ell\_\{\\theta\}\(t\)=\\log\\pi\_\{\\theta\}\(x\_\{t\}\\mid x\_\{<t\}\)for the token\-level log\-probabilities at positiontt, we setsprc​\(i,j\)=∑t∈step​j\(ℓteacher​\(t\)−ℓθ​\(t\)\)s\_\{\\text\{prc\}\}\(i,j\)=\\sum\_\{t\\in\\text\{step \}j\}\\big\(\\ell\_\{\\text\{teacher\}\}\(t\)\-\\ell\_\{\\theta\}\(t\)\\big\), i\.e\., the \(negated\) token\-level reverse KL against a teacher policy aggregated over tokens inside stepjj\(Agarwalet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib11)\)\. The Generalized On\-Policy Distillation \(G\-OPD\) variant ofYanget al\.\([2026](https://arxiv.org/html/2606.29296#bib.bib17)\)replaces this with a reference policyπbase\\pi\_\{\\text\{base\}\}\(an anchor\) and a reward\-scaling coefficientλ\\lambda, so that, withℓbase​\(t\)=log⁡πbase​\(xt∣x<t\)\\ell\_\{\\text\{base\}\}\(t\)=\\log\\pi\_\{\\text\{base\}\}\(x\_\{t\}\\mid x\_\{<t\}\), sprcG\-OPD​\(i,j\)=−∑t∈step​j\[\(ℓθ​\(t\)−ℓbase​\(t\)\)−λ⋅\(ℓteacher​\(t\)−ℓbase​\(t\)\)\],s\_\{\\text\{prc\}\}^\{\\text\{G\-OPD\}\}\(i,j\)=\-\\sum\_\{t\\in\\text\{step \}j\}\\Big\[\\,\\big\(\\ell\_\{\\theta\}\(t\)\-\\ell\_\{\\text\{base\}\}\(t\)\\big\)\-\\lambda\\cdot\\big\(\\ell\_\{\\text\{teacher\}\}\(t\)\-\\ell\_\{\\text\{base\}\}\(t\)\\big\)\\Big\],\(5\)which reduces to standard OPD whenλ=1\\lambda=1andπbase=πθ\\pi\_\{\\text\{base\}\}=\\pi\_\{\\theta\}\. Used for the multi\-hop QA experiments in §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)\.

### 4\.2Chunk\-by\-Value

Logical progression in long\-form reasoning is rarely uniform where short mechanical derivations are punctuated by a few critical conceptual leaps\(Liet al\.,[2026a](https://arxiv.org/html/2606.29296#bib.bib16)\)\. The intrinsic spatial frequency at which trajectory value changes is rarely matched by the granularity at whichsprcs\_\{\\text\{prc\}\}varies—a PRM emits one score per heuristic step, a token\-level KL fluctuates at every token—so uniformly broadcastings~prc\\tilde\{s\}\_\{\\text\{prc\}\}at either granularity is either too blunt or too noisy\. CV, as the response of PASS to Pathology 2 \(resolution mismatch\), addresses this by deriving the credit\-assignment resolution fromsprcs\_\{\\text\{prc\}\}itself\. We introduce a deterministic boundary detector𝒟\\mathcal\{D\}that segmentsoio\_\{i\}intoKiK\_\{i\}value\-homogeneous chunks with ending token indicesℬi=\{b1,…,bKi\}\\mathcal\{B\}\_\{i\}=\\\{b\_\{1\},\\dots,b\_\{K\_\{i\}\}\\\}\(bKi=Lib\_\{K\_\{i\}\}=L\_\{i\}\), defined as a*numerical\-equality run\-length compressor*: a new boundary is placed wheneversprcs\_\{\\text\{prc\}\}at the current token differs from the previous token by more than a small numerical toleranceη=10−8\\eta=10^\{\-8\}, i\.e\., an equality test up to floating\-point noise\. This threshold\-free rule transfers across process signals with very different numeric ranges\. For PRM\-style signals, which are piecewise constant within a reasoning step,𝒟\\mathcal\{D\}recovers the PRM’s own step boundaries; for token\-level KL\-style signals, PASS applies𝒟\\mathcal\{D\}to the group\-averaged, already\-standardized token signal inside the answer region, a choice that simultaneously filters sample\-level noise \(averaging\), makes an equality\-based compressor well\-posed across scales \(standardization\), and restricts boundaries to optimizable tokens \(answer\-region masking\)\. In both cases𝒟\\mathcal\{D\}is fully determined bysprcs\_\{\\text\{prc\}\}and introduces no signal\-type\-specific hyperparameter; full pseudocode is given in Algorithm[2](https://arxiv.org/html/2606.29296#alg2)\. We then write the chunk\-level fused advantage asA~ibj\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\(evaluated via Equation[4](https://arxiv.org/html/2606.29296#S4.E4)at the boundary token\) and broadcast it to every token within the chunk:

A~i,t=A~ibj,bj−1<t≤bj,\\tilde\{A\}\_\{i,t\}=\\tilde\{A\}\_\{i\}^\{b\_\{j\}\},\\qquad b\_\{j\-1\}<t\\leq b\_\{j\},\(6\)so that tokens inside the same logical step share identical credit and value transitions receive their own dedicated chunks\.

### 4\.3Divide\-Length

CV fixes the local resolution of credit assignment, but the cumulative sum in Equation[3](https://arxiv.org/html/2606.29296#S3.E3)is still insensitive, in a structural way, to whether an appended step is worth its own length\. Under the return\-to\-go sum, any step with a non\-negative marginal contribution is unconditionally rewarded, and any trajectory that continues into a negative\-margin regime is unconditionally penalized at every additional step \(the cumulative trap, Pathology 3\)\. The next theorem, whose full proof is deferred to Appendix[A](https://arxiv.org/html/2606.29296#A1), makes the positive\-margin projection of this pathology precise\.

###### Assumption 1\(Reachability of mediocre redundancy\)\.

There exists a family of “safe, non\-committal” reasoning steps \(tautologies, redundant rewrites, trivial comments\) that the policyπθ\\pi\_\{\\theta\}can readily generate and for which the process signal, after group standardization, is bounded below by some constantϵ\>0\\epsilon\>0\.

###### Theorem 1\(Length collapse in the summation paradigm\)\.

Under Assumption[1](https://arxiv.org/html/2606.29296#Thmassumption1), maximizing the expected initial cumulative advantageJ​\(θ\)=𝔼πθ​\[A^i,1\]J\(\\theta\)=\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\[\\hat\{A\}\_\{i,1\}\]drives the optimal policy to extend the trajectory lengthLiL\_\{i\}toward the environment limitLmaxL\_\{\\max\}, diluting the logical density of the generation\.

*Proof sketch\.*Appendingmmredundant steps to a valid, concise trajectory strictly increases the cumulative advantage by at leastm​ϵm\\epsilonwithout altering the prefix, so the policy gradient is monotonically increasing inmm; the full argument appears in Appendix[A](https://arxiv.org/html/2606.29296#A1)\.

To close this loophole, DL replaces the absolute cumulative sum with a length\-attenuated logical\-density score\. Following the standard process\-supervision aggregation used by GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.29296#bib.bib1)\), writeGij=∑j′≥jA~ibj′G\_\{i\}^\{j\}=\\sum\_\{j^\{\\prime\}\\geq j\}\\tilde\{A\}\_\{i\}^\{b\_\{j^\{\\prime\}\}\}for the per\-chunk return\-to\-go at chunkjjof trajectoryoio\_\{i\}\. DL attenuates eachGijG\_\{i\}^\{j\}by the residual chunk count\(Ki−j\+1\)k\(K\_\{i\}\-j\+1\)^\{k\}rather than by the absolute sequence length:

A~ibj,DL=Gij\(Ki−j\+1\)k=∑j′≥jA~ibj′\(Ki−j\+1\)k,k≥0,\\tilde\{A\}\_\{i\}^\{b\_\{j\},\\,\\text\{DL\}\}=\\frac\{G\_\{i\}^\{j\}\}\{\(K\_\{i\}\-j\+1\)^\{k\}\}=\\frac\{\\sum\_\{j^\{\\prime\}\\geq j\}\\tilde\{A\}\_\{i\}^\{b\_\{j^\{\\prime\}\}\}\}\{\(K\_\{i\}\-j\+1\)^\{k\}\},\\qquad k\\geq 0,\(7\)which, atj=1j=1, recovers a trajectory\-level average\-value\-density scoreS​\(oi\)=\(∑j=1KiA~ibj\)/\(Ki\)kS\(o\_\{i\}\)=\\big\(\\sum\_\{j=1\}^\{K\_\{i\}\}\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\\big\)/\(K\_\{i\}\)^\{k\}\. The DL\-attenuated chunk\-level valueA~ibj,DL\\tilde\{A\}\_\{i\}^\{b\_\{j\},\\,\\text\{DL\}\}is then broadcast back to every token inside the chunk via the same rule as Equation[6](https://arxiv.org/html/2606.29296#S4.E6), so tokens within one chunk continue to share an identical credit\. Working with the residual chunk count rather than the residual token length keeps DL consistent with the chunk\-by\-chunk credit assignment of CV and avoids re\-introducing a token\-position dependence that CV has just eliminated\. The exponentkkinterpolates between no\-penalty \(k→0k\\to 0\) and strict average\-reward MDP normalization \(k=1k=1\): smallkkpreserves the benefit of genuinely long, hard deductions; largekkaggressively discounts redundancy\. All multi\-hop QA runs in this paper fixk=0\.7k=0\.7; the sensitivity of the mathematical\-reasoning results tokkis ablated in Appendix[C](https://arxiv.org/html/2606.29296#A3)\(Table[5](https://arxiv.org/html/2606.29296#A3.T5)\)\.

#### A marginal\-value reading of DL\.

To make explicit what DL does and does not promise, we analyze its effect at the path\-score levelS​\(oi\)=\(∑j=1KiA~ibj\)/\(Ki\)kS\(o\_\{i\}\)=\\big\(\\sum\_\{j=1\}^\{K\_\{i\}\}\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\\big\)/\(K\_\{i\}\)^\{k\}from Equation[7](https://arxiv.org/html/2606.29296#S4.E7)\. Consider appending one extra chunk of signed marginal contributionδ\\deltato a trajectoryoio\_\{i\}withKiK\_\{i\}chunks whose current path score isS​\(oi\)S\(o\_\{i\}\)\. Under the summation objective the resulting change inS​\(oi\)S\(o\_\{i\}\)is simplyδ\\delta, which is always non\-negative wheneverδ≥0\\delta\\geq 0\. Under the DL objective, to first order in1/Ki1/K\_\{i\},

Δ​S​\(oi\)≈1\(Ki\)k​\(δ−k⋅S​\(oi\)Ki\),\\Delta S\(o\_\{i\}\)\\;\\approx\\;\\frac\{1\}\{\(K\_\{i\}\)^\{k\}\}\\left\(\\delta\\;\-\\;k\\cdot\\frac\{S\(o\_\{i\}\)\}\{K\_\{i\}\}\\right\),\(8\)so the sign ofΔ​S​\(oi\)\\Delta S\(o\_\{i\}\)depends on whether the marginal contributionδ\\deltaexceedskktimes the current average value densityS​\(oi\)/KiS\(o\_\{i\}\)/K\_\{i\}\. DL therefore acts as a*two\-sided marginal\-value filter*: it admits additional steps only when they raise the trajectory’s average value density and rejects them otherwise\. Crucially, the sign ofδ\\deltain Equation[8](https://arxiv.org/html/2606.29296#S4.E8)is not fixed by DL itself\. When the regime is dominated by positive\-biased redundant steps, DL suppresses them \(its length\-inflation defusal role noted after Theorem[1](https://arxiv.org/html/2606.29296#Thmtheorem1)\); when the regime is dominated by truncation of a trajectory that is still above its current density threshold, DL admits the depth\-extending step\. The two behaviors are symmetric consequences of the same admission criterion and require no algorithmic change across regimes\. DL does*not*promise that the optimized trajectory must be shorter than a baseline trajectory; a policy that lengthens its output while strictly increasing value density is still improving under DL\. This distinction matters for the interpretation of our multi\-hop QA experiments in §[5](https://arxiv.org/html/2606.29296#S5)\. DL is the PASS response to Pathology 3 \(the cumulative trap\); its pseudocode is given in Algorithm[3](https://arxiv.org/html/2606.29296#alg3)\.

#### From process signal to advantage\.

Putting the three pieces together, PASS transforms an arbitrary process signalsprcs\_\{\\text\{prc\}\}into a per\-token advantageA~i,t\\tilde\{A\}\_\{i,t\}via: \(1\) independent group standardization of the three streams, \(2\) piecewise Advantage Fusion \(Equation[4](https://arxiv.org/html/2606.29296#S4.E4)\), \(3\) chunk\-level broadcasting along𝒟\\mathcal\{D\}\(Equation[6](https://arxiv.org/html/2606.29296#S4.E6)\), \(4\) formation of the per\-chunk return\-to\-goGijG\_\{i\}^\{j\}in the standard GRPO process\-supervision sense, and \(5\) the residual chunk\-count divisor\(Ki−j\+1\)k\(K\_\{i\}\-j\+1\)^\{k\}from DL \(Equation[7](https://arxiv.org/html/2606.29296#S4.E7)\) on top ofGijG\_\{i\}^\{j\}, with the resulting density\-rescaled chunk value re\-broadcast to every token inside the chunk\. Each process signalsprcs\_\{\\text\{prc\}\}must first pass through a group\-level standardization operator𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\)\. Two concrete instantiations of this operator are used in our experiments, Masked\-Norm and Abs\-Max Scaling; the interaction between the choice of𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\)and PASS is analyzed empirically in Appendix[D](https://arxiv.org/html/2606.29296#A4)\. A complete algorithmic specification of the three operators and their end\-to\-end composition is given in Appendix[B](https://arxiv.org/html/2606.29296#A2)\(Algorithms[1](https://arxiv.org/html/2606.29296#alg1)–[4](https://arxiv.org/html/2606.29296#alg4)\)\.

## 5Experiments

We evaluate PASS on two intentionally dissimilar domains to test the claim that PASS is a paradigm\-agnostic middleware, i\.e\., that its effectiveness does not hinge on the specific construction of the upstream process signal\. The two subsections that follow are deliberately orthogonal in what they exercise\. §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1)pairs PASS with a learned PRM on mathematical reasoning: here the process signal is emitted at heuristic step boundaries and is piecewise constant within a step, so this subsection probes whether PASS can consume a discrete, coarse, boundary\-aligned signal produced by an external scorer and convert it into a competitive advantage for the policy\. §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)pairs PASS with a token\-level reverse\-KL signal obtained from on\-policy distillation on multi\-hop question answering: heresprcs\_\{\\text\{prc\}\}fluctuates at every token and has no natural step structure, so this subsection probes whether the same middleware can consume a dense, high\-frequency, continuous signal produced implicitly by a teacher policy\. Taken together, the two subsections are the two ends of the “process\-signal paradigm” axis that the introduction identifies, and an improvement that is visible in both cases is what we read as evidence of paradigm\-agnostic behavior rather than of fitting to a specific signal construction\. A detailed sensitivity study on the DL length exponentkkand on the scale of the upstream PRM is deferred to Appendix[C](https://arxiv.org/html/2606.29296#A3); an alternative group\-standardization operator \(Abs\-Max Scaling\) for the KL\-style setting and a joint length–accuracy analysis across both operators are deferred to Appendix[D](https://arxiv.org/html/2606.29296#A4)\.

### 5\.1Mathematical Reasoning with a PRM Process Signal

#### Setup\.

We instantiatesprcs\_\{\\text\{prc\}\}as a Qwen2\.5\-Math\-PRM\-7B\(Zhanget al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib7)\)step\-level score on Qwen2\.5\-Math\-7B\(Yanget al\.,[2024](https://arxiv.org/html/2606.29296#bib.bib5)\)as the actor, train on DeepScaleR\(Michael Luo and Sijun Tan and Justin Wong and Xiaoxiang Shi and William Tang and Manan Roongta and Colin Cai and Jeffrey Luo and Tianjun Zhang and Erran Li and Raluca Ada Popa and Ion Stoica,[2025](https://arxiv.org/html/2606.29296#bib.bib4)\), and evaluate on AIME24/25, AMC23, GSM8K, MATH, Minerva, and Olympiad\. Metrics are pass@1 / pass@8 \(%\) measured with themath\_verifychecker\(Kydlíček,[2025](https://arxiv.org/html/2606.29296#bib.bib8)\)\. Further dataset and optimizer details followMichael Luo and Sijun Tan and Justin Wong and Xiaoxiang Shi and William Tang and Manan Roongta and Colin Cai and Jeffrey Luo and Tianjun Zhang and Erran Li and Raluca Ada Popa and Ion Stoica \([2025](https://arxiv.org/html/2606.29296#bib.bib4)\)and are omitted for space\.

#### Main results\.

Table[1](https://arxiv.org/html/2606.29296#S5.T1)reports PASS against the untuned base model and a standard outcome\-reward GRPO baseline\. With the PRM assprcs\_\{\\text\{prc\}\}, PASS improves average pass@1 by an absolute5\.95\.9points over GRPO\(ORM\) \(29\.8→35\.729\.8\\to 35\.7\) and lifts pass@8 by1\.91\.9points \(53\.1→55\.053\.1\\to 55\.0\), with especially large gains on competition\-level benchmarks \(AIME25 pass@15\.5→9\.85\.5\\to 9\.8, AMC23 pass@147\.0→53\.847\.0\\to 53\.8\)\.

Table 1:PASS vs\. a base model and outcome\-reward GRPO on mathematical reasoning\. Metrics are pass@1 / pass@8 \(%\)\. Averaged over the seven benchmarks, PASS improves pass@1 by\+5\.9\+5\.9absolute over GRPO\(ORM\)\.
#### Structural ablation: CV and DL\.

Table[2](https://arxiv.org/html/2606.29296#S5.T2)ablates the two downstream PASS operators, which probes Pathology 2 \(resolution mismatch\) and Pathology 3 \(cumulative trap\) jointly\. Removing DL \(PASS w/o DL\) triggers length\-driven reward hacking and collapses average pass@1 by10\.910\.9points \(35\.7→24\.835\.7\\to 24\.8\), exhibiting the positive\-margin projection of the cumulative trap \(Theorem[1](https://arxiv.org/html/2606.29296#Thmtheorem1)\)\. Removing CV \(PASS w/o CV\) preserves DL’s length\-hacking safeguard but sacrifices the high\-signal\-to\-noise credit assignment afforded by chunk\-level aggregation, dropping average pass@1 by3\.13\.1points \(35\.7→32\.635\.7\\to 32\.6\) and pass@8 by4\.64\.6points \(55\.0→50\.455\.0\\to 50\.4\), consistent with the resolution\-mismatch reading of CV\. The full PASS achieves the best pass@1 on all seven benchmarks \(tied with PASS w/o CV on Minerva\) and the best pass@8 on six of seven benchmarks\.

Table 2:CV / DL structural ablation on mathematical reasoning\. PASS w/o DL exhibits length inflation and collapses accuracy; PASS w/o CV retains DL’s safeguard but leaves a visible gap on both pass@1 and pass@8; the full PASS attains the best averages on both metrics\.
#### Advantage Fusion vs\. naive reward shaping\.

A natural alternative to the AF operator of Equation[4](https://arxiv.org/html/2606.29296#S4.E4)is to linearly combine the raw process, outcome, and format rewards before group standardization, which is the default recipe that Pathology 1 \(channel contamination\) warned against\. Table[3](https://arxiv.org/html/2606.29296#S5.T3)compares the two choices while keeping CV and DL fixed atk=1\.0k=1\.0\. Advantage Fusion outperforms the linear reward\-shaping baseline by\+3\.2\+3\.2points on average pass@1 and\+3\.2\+3\.2on pass@8, with the largest gap on AIME25 \(3\.9→9\.83\.9\\to 9\.8pass@1\)\. The gap is consistent with the motivation of §[4\.1](https://arxiv.org/html/2606.29296#S4.SS1): pooling a near\-binary outcome signal with a continuous step\-level signal before standardization inflates the outcome variance and attenuates the gradient contribution ofsprcs\_\{\\text\{prc\}\}, whereas independently standardizing the three streams preserves the step\-level gradient component and additionally makes the format indicator a hard gate rather than a soft summand\.

Table 3:Advantage Fusion \(Equation[4](https://arxiv.org/html/2606.29296#S4.E4)\) versus a naive linear reward\-shaping baseline of the formR=wprm​rprm\+wout​rout\+wfmt​rfmtR=w\_\{\\text\{prm\}\}r\_\{\\text\{prm\}\}\+w\_\{\\text\{out\}\}r\_\{\\text\{out\}\}\+w\_\{\\text\{fmt\}\}r\_\{\\text\{fmt\}\}\. Both variants use CV and DL withk=1\.0k=1\.0\. Metrics are pass@1 / pass@8 \(%\)\.

### 5\.2Multi\-hop Question Answering with a KL\-style Process Signal

#### Setup\.

We evaluate PASS on three multi\-hop question\-answering benchmarks: HotpotQA\(Yanget al\.,[2018](https://arxiv.org/html/2606.29296#bib.bib14)\), 2WikiMultihopQA\(Hoet al\.,[2020](https://arxiv.org/html/2606.29296#bib.bib12)\), and MuSiQue\(Trivediet al\.,[2022](https://arxiv.org/html/2606.29296#bib.bib13)\)\. Training follows the HotpotQA distractor setup on a 5k\-prompt subset of its training split, and evaluation uses the full development splits of all three datasets with1616sampled responses per task\. The actor is Qwen2\.5\-7B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2606.29296#bib.bib15)\)and the teacher is Qwen2\.5\-32B\-Instruct\. We instantiatesprcs\_\{\\text\{prc\}\}in two ways: \(i\)*OPD*, the token\-level negative KL against the teacher\(Agarwalet al\.,[2023](https://arxiv.org/html/2606.29296#bib.bib11)\), and \(ii\)*G\-OPD*, theλ\\lambda\-shifted variant of Equation[5](https://arxiv.org/html/2606.29296#S4.E5)anchored on the actor’s initial weights asπbase\\pi\_\{\\text\{base\}\}, withλ=1\.25\\lambda=1\.25\. Throughout this subsection we use Masked\-Norm as the group\-level standardization operator𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\), i\.e\., we subtract the masked group mean and divide by the masked standard deviation ofsprcs\_\{\\text\{prc\}\}; this is the same operator adopted in the mathematical\-reasoning setup of §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1)\. All eight multi\-hop QA runs—four reported in this subsection and four reported in Appendix[D](https://arxiv.org/html/2606.29296#A4)—share the same optimizer, sampler, and rollout configuration \(global batch size128128, maximum sequence length6,1446\{,\}144,1616rollouts per prompt, learning rate10−610^\{\-6\}, two training epochs\), and evaluation numbers are reported at the end of training\. Per\-run hyperparameter differences are given in Appendix[E](https://arxiv.org/html/2606.29296#A5)\. We report pass@1 \(exact\-match answer accuracy under SQuAD\-style normalization\) on each dataset together with an unweighted average, and defer pass@8 and pass@16 to Appendix[E](https://arxiv.org/html/2606.29296#A5)\.

Table[4](https://arxiv.org/html/2606.29296#S5.T4)summarizes the Masked\-Norm results; the two paired comparisons below probe the effectiveness of PASS on each process\-signal paradigm\.

Table 4:pass@1 \(%\) on three multi\-hop QA benchmarks with the Masked\-Norm standardization operator\. PASS consistently improves the average pass@1 on both the OPD and the G\-OPD process signals\.Δ\\Deltadenotes the absolute improvement over the corresponding non\-PASS baseline\.On the vanilla OPD signal, attaching PASS raises pass@1 from37\.6837\.68to48\.5048\.50on MuSiQue, from63\.2563\.25to66\.8166\.81on 2Wiki, and from62\.0362\.03to64\.1464\.14on HotpotQA, for an average absolute gain of\+5\.50\+5\.50points \(54\.32→59\.8254\.32\\to 59\.82\)\. The largest improvement appears on MuSiQue, the most challenging four\-hop benchmark in the group, which is consistent with chunk\-level credit assignment benefiting tasks whose solution trajectories contain more stable logical segments\. On the G\-OPD variant, PASS again produces a consistent improvement across all three benchmarks \(MuSiQue39\.01→47\.2939\.01\\to 47\.29; 2Wiki63\.69→67\.1963\.69\\to 67\.19; HotpotQA61\.96→64\.3461\.96\\to 64\.34\), yielding an average pass@1 of59\.6159\.61and an absolute gain of\+4\.72\+4\.72points\. The magnitude of the G\-OPD gain tracks that of OPD closely, indicating that the effectiveness of PASS does not depend on the specific construction of the KL\-style signal, and the gains PASS achieves on mathematical reasoning under a PRM signal transfer to multi\-hop QA under a KL\-style signal without algorithmic changes\. The trend on pass@1 carries over to the pass@8 and pass@16 numbers tabulated in Appendix[E](https://arxiv.org/html/2606.29296#A5)\.

#### Robustness to the choice of standardization operator\.

We have reported PASS on top of Masked\-Norm, the same group\-standardization operator adopted in the mathematical\-reasoning setup of §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1)\. The two KL\-style signals, however, differ from PRM scores in one specific respect: OPD and G\-OPD carry an implicit or explicit origin that is not located at the within\-group mean, so a natural question is whether the improvement we report survives a different group\-standardization operator that respects this origin\. We verify that it does\. Specifically, we repeat all four configurations of Table[4](https://arxiv.org/html/2606.29296#S5.T4)with Abs\-Max Scaling, a standardization operator that divides each entry ofsprcs\_\{\\text\{prc\}\}by the group\-wide maximum absolute value and leaves the signal’s zero point in place\. Under Abs\-Max Scaling the baselines are stronger and the residual room for PASS is therefore smaller, but PASS still delivers a non\-negative average pass@1 gain under both OPD and G\-OPD, so the main claim of this subsection—that PASS improves a GRPO baseline with a KL\-style process signal on multi\-hop QA—is not an artifact of the specific standardization operator we use in the main body\. The Abs\-Max Scaling results \(Table[7](https://arxiv.org/html/2606.29296#A4.T7)\) together with a joint analysis of the length, entropy, and teacher\-KL dynamics induced by each operator \(Figures[2](https://arxiv.org/html/2606.29296#A4.F2),[3](https://arxiv.org/html/2606.29296#A4.F3),[4](https://arxiv.org/html/2606.29296#A4.F4), Table[8](https://arxiv.org/html/2606.29296#A4.T8)\), as well as a mechanism\-level reading of how each operator interacts with DL’s two\-sided marginal\-value filter, are deferred to Appendix[D](https://arxiv.org/html/2606.29296#A4)\.

## 6Discussion

#### Beyond the sourcing of the process signal\.

The central value proposition of PASS is that its three operators, Advantage Fusion, Chunk\-by\-Value, and Divide\-Length, act on a generic step\-level scalarsprcs\_\{\\text\{prc\}\}, regardless of whether that scalar is produced by a learned PRM or by a teacher\-anchored KL divergence\. Our mathematical\-reasoning and multi\-hop QA experiments are a first step in validating this scope; extending the same middleware to more diverse process signals \(e\.g\., rubric\-based LLM judges or execution\-based reward models\) is a natural follow\-up that our formulation readily accommodates\.

#### Domain specificity and credit\-assignment boundaries\.

Our strongest evidence comes from mathematical reasoning, where logical steps admit near\-deterministic verification and CV’s value\-homogeneous chunking is therefore especially natural\. On open\-ended or multi\-turn tasks, chunk boundaries and what counts as a “correct” step are softer concepts; deploying PASS in those regimes will benefit from dynamic, rubric\-style signal generators that can still be exposed through thesprcs\_\{\\text\{prc\}\}interface\.

## 7Conclusion

PASS \(Process Advantage Signal Shaping\) is a thin middleware between a generic step\-level process signal and GRPO’s clipped surrogate, whose three operators each intervene at a different stage of GRPO’s advantage pipeline—aggregation, broadcast, and trajectory summation—to close the three structural pathologies identified in §[1](https://arxiv.org/html/2606.29296#S1)\. The same unmodified middleware yields consistent gains over GRPO across two process\-signal paradigms \(a learned PRM for mathematical reasoning; on\-policy\-distillation and G\-OPD KL signals for multi\-hop QA\) and two group\-standardization operators \(Masked\-Norm, Abs\-Max\)\. A natural next step, directly enabled by PASS’s signal\-agnostic interface, is to replace the dedicated PRM with a frozen LLM\-as\-judge at training time, amortizing the critic’s cognitive capacity without the cost of training ever\-larger dedicated scorers\.

## References

- On\-Policy Distillation of Language Models: Learning from Self\-Generated Mistakes\.InProceedings of the Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=3zKtaqxLhW)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p1.1),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px4.p1.1),[§3](https://arxiv.org/html/2606.29296#S3.SS0.SSS0.Px1.p1.5),[2nd item](https://arxiv.org/html/2606.29296#S4.I1.i2.p1.8),[§5\.2](https://arxiv.org/html/2606.29296#S5.SS2.SSS0.Px1.p1.11)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p1.1),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.29296#S3.p1.13)\.
- X\. Ho, A\. Duong Nguyen, S\. Sugawara, and A\. Aizawa \(2020\)Constructing A Multi\-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps\.InProceedings of the 28th International Conference on Computational Linguistics,D\. Scott, N\. Bel, and C\. Zong \(Eds\.\),pp\. 6609–6625\.External Links:[Document](https://dx.doi.org/10.18653/v1/2020.coling-main.580),[Link](https://aclanthology.org/2020.coling-main.580/)Cited by:[§5\.2](https://arxiv.org/html/2606.29296#S5.SS2.SSS0.Px1.p1.11)\.
- M\. Khandoga, R\. Yuan, and V\. K\. Sankarapu \(2026\)Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization\.External Links:[Link](https://arxiv.org/abs/2602.09331v1)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px2.p1.1)\.
- H\. Kydlíček \(2025\)Math\-Verify: Math Verification Library\.Cited by:[§5\.1](https://arxiv.org/html/2606.29296#S5.SS1.SSS0.Px1.p1.1)\.
- G\. Li, Y\. Chen, M\. Lin, and T\. Yang \(2025\)DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization\.External Links:[Link](https://arxiv.org/abs/2510.04474v2)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Li, Y\. Zuo, B\. He, J\. Zhang, C\. Xiao, C\. Qian, T\. Yu, H\. Gao, W\. Yang, Z\. Liu, and N\. Ding \(2026a\)Rethinking On\-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe\.External Links:2604\.13016,[Document](https://dx.doi.org/10.48550/arXiv.2604.13016),[Link](http://arxiv.org/abs/2604.13016)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px4.p1.1),[§4\.2](https://arxiv.org/html/2606.29296#S4.SS2.p1.15)\.
- Z\. Li, J\. Lou, F\. Dong, Z\. Fan, M\. Ren, H\. Lin, X\. Han, D\. Zhang, L\. Sun, Y\. Lu, and X\. Yu \(2026b\)Tackling Length Inflation Without Trade\-offs: Group Relative Reward Rescaling for Reinforcement Learning\.External Links:[Link](https://arxiv.org/abs/2603.10535v1)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p2.3),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px3.p1.1)\.
- H\. Lightman, V\. Kosaraju, Y\. Burda, H\. Edwards, B\. Baker, T\. Lee, J\. Leike, J\. Schulman, I\. Sutskever, and K\. Cobbe \(2023\)Let’s Verify Step by Step\.InProceedings of the Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=v8L0pN6EOi)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p1.1),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.29296#S3.SS0.SSS0.Px1.p1.5),[1st item](https://arxiv.org/html/2606.29296#S4.I1.i1.p1.1)\.
- Q\. Liu, H\. Xu, X\. Chen, W\. Chen, Y\. W\. Teh, and N\. Miao \(2025\)Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey\.External Links:[Link](https://arxiv.org/abs/2510.01925v2)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px3.p1.1)\.
- S\. Liu, X\. Dong, X\. Lu, S\. Diao, P\. Belcak, M\. Liu, M\. Chen, H\. Yin, Y\. F\. Wang, K\. Cheng, Y\. Choi, J\. Kautz, and P\. Molchanov \(2026\)GDPO: Group reward\-Decoupled Normalization Policy Optimization for Multi\-reward RL Optimization\.arXiv\.External Links:2601\.05242,[Document](https://dx.doi.org/10.48550/arXiv.2601.05242)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p2.3),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1)\.
- C\. Ma, S\. Yang, K\. Huang, J\. Lu, H\. Meng, S\. Wang, B\. Ding, S\. Vosoughi, G\. Wang, and J\. Zhou \(2026\)FIPO: Eliciting Deep Reasoning with Future\-KL Influenced Policy Optimization\.External Links:2603\.19835,[Document](https://dx.doi.org/10.48550/arXiv.2603.19835),[Link](http://arxiv.org/abs/2603.19835)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px2.p1.1)\.
- Michael Luo and Sijun Tan and Justin Wong and Xiaoxiang Shi and William Tang and Manan Roongta and Colin Cai and Jeffrey Luo and Tianjun Zhang and Erran Li and Raluca Ada Popa and Ion Stoica \(2025\)DeepScaleR: surpassing o1\-preview with a 1\.5b model by scaling rl\.Technical reportagentica\.Cited by:[§5\.1](https://arxiv.org/html/2606.29296#S5.SS1.SSS0.Px1.p1.1)\.
- Qwen, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 Technical Report\.External Links:2412\.15115,[Document](https://dx.doi.org/10.48550/arXiv.2412.15115),[Link](http://arxiv.org/abs/2412.15115)Cited by:[§5\.2](https://arxiv.org/html/2606.29296#S5.SS2.SSS0.Px1.p1.11)\.
- N\. Razin, Z\. Wang, H\. Strauss, S\. Wei, J\. D\. Lee, and S\. Arora \(2025\)What Makes a Reward Model a Good Teacher? An Optimization Perspective\.arXivarXiv:2503\.15477\.External Links:2503\.15477,[Document](https://dx.doi.org/10.48550/arXiv.2503.15477)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px3.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo \(2024\)DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.arXivarXiv:2402\.03300\.External Links:2402\.03300,[Document](https://dx.doi.org/10.48550/arXiv.2402.03300)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p1.1),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1),[§3](https://arxiv.org/html/2606.29296#S3.p1.13),[§4\.3](https://arxiv.org/html/2606.29296#S4.SS3.p3.5)\.
- P\. Singhal, T\. Goyal, J\. Xu, and G\. Durrett \(2024\)A Long Way to Go: Investigating Length Correlations in RLHF\.External Links:2310\.03716,[Document](https://dx.doi.org/10.48550/arXiv.2310.03716)Cited by:[§4\.1](https://arxiv.org/html/2606.29296#S4.SS1.p1.13)\.
- M\. Sullivan \(2025\)GRPO is Secretly a Process Reward Model\.arXiv\.External Links:2509\.21154,[Document](https://dx.doi.org/10.48550/arXiv.2509.21154)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p2.3),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1)\.
- H\. Trivedi, N\. Balasubramanian, T\. Khot, and A\. Sabharwal \(2022\)MuSiQue: Multihop Questions via Single\-hop Question Composition\.Transactions of the Association for Computational Linguistics10,pp\. 539–554\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00475),[Link](https://aclanthology.org/2022.tacl-1.31/)Cited by:[§5\.2](https://arxiv.org/html/2606.29296#S5.SS2.SSS0.Px1.p1.11)\.
- A\. Yang, B\. Zhang, B\. Hui, B\. Gao, B\. Yu, C\. Li, D\. Liu, J\. Tu, J\. Zhou, J\. Lin, K\. Lu, M\. Xue, R\. Lin, T\. Liu, X\. Ren, and Z\. Zhang \(2024\)Qwen2\.5\-math technical report: toward mathematical expert model via self\-improvement\.arXiv preprint arXiv:2409\.12122\.Cited by:[§5\.1](https://arxiv.org/html/2606.29296#S5.SS1.SSS0.Px1.p1.1)\.
- W\. Yang, W\. Liu, R\. Xie, K\. Yang, S\. Yang, and Y\. Lin \(2026\)Learning beyond Teacher: Generalized On\-Policy Distillation with Reward Extrapolation\.External Links:2602\.12125,[Document](https://dx.doi.org/10.48550/arXiv.2602.12125),[Link](http://arxiv.org/abs/2602.12125)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px4.p1.1),[2nd item](https://arxiv.org/html/2606.29296#S4.I1.i2.p1.8)\.
- Z\. Yang, P\. Qi, S\. Zhang, Y\. Bengio, W\. Cohen, R\. Salakhutdinov, and C\. D\. Manning \(2018\)HotpotQA: A Dataset for Diverse, Explainable Multi\-hop Question Answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),pp\. 2369–2380\.External Links:[Document](https://dx.doi.org/10.18653/v1/D18-1259),[Link](https://aclanthology.org/D18-1259/)Cited by:[§5\.2](https://arxiv.org/html/2606.29296#S5.SS2.SSS0.Px1.p1.11)\.
- C\. Ye, Z\. Yu, Z\. Zhang, H\. Chen, N\. Sadagopan, J\. Huang, T\. Zhang, and A\. Beniwal \(2025\)Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training\.External Links:2509\.03403,[Document](https://dx.doi.org/10.48550/arXiv.2509.03403),[Link](http://arxiv.org/abs/2509.03403)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p2.3),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1)\.
- J\. Yin, H\. Luo, Z\. Li, Y\. Liu, D\. Liu, Z\. Li, and X\. Xu \(2025\)Pinpointing crucial steps: Attribution\-based Credit Assignment for Verifiable Reinforcement Learning\.External Links:[Link](https://arxiv.org/abs/2510.08899v1)Cited by:[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Zhang \(2026\)From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models\.External Links:[Link](https://arxiv.org/abs/2604.09459v2)Cited by:[§1](https://arxiv.org/html/2606.29296#S1.p2.3),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px2.p1.1)\.
- Z\. Zhang, C\. Zheng, Y\. Wu, B\. Zhang, R\. Lin, B\. Yu, D\. Liu, J\. Zhou, and J\. Lin \(2025\)The Lessons of Developing Process Reward Models in Mathematical Reasoning\.arXiv\.External Links:2501\.07301,[Document](https://dx.doi.org/10.48550/arXiv.2501.07301)Cited by:[Appendix C](https://arxiv.org/html/2606.29296#A3.SS0.SSS0.Px2.p1.7),[§1](https://arxiv.org/html/2606.29296#S1.p1.1),[§2](https://arxiv.org/html/2606.29296#S2.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2606.29296#S5.SS1.SSS0.Px1.p1.1)\.

## Appendix AProof of the Length Collapse Theorem

We restate Assumption[1](https://arxiv.org/html/2606.29296#Thmassumption1)and Theorem[1](https://arxiv.org/html/2606.29296#Thmtheorem1)and give the detailed argument deferred from §[4\.3](https://arxiv.org/html/2606.29296#S4.SS3)\.

###### Proof of Theorem[1](https://arxiv.org/html/2606.29296#Thmtheorem1)\.

Letτi\\tau\_\{i\}be a valid, concise reasoning trajectory generated by the current policyπ\\pithat correctly derives the answer and emits an end\-of\-sequence \(EOS\) token at stepk≪Kmaxk\\ll K\_\{\\max\}\. The initial\-token cumulative advantage ofτi\\tau\_\{i\}under Equation[3](https://arxiv.org/html/2606.29296#S3.E3)is

A^i,1​\(τi\)=∑j=1ks~iindex​\(j\)≜Rvalid\.\\hat\{A\}\_\{i,1\}\(\\tau\_\{i\}\)\\;=\\;\\sum\_\{j=1\}^\{k\}\\tilde\{s\}\_\{i\}^\{\\mathrm\{index\}\(j\)\}\\;\\triangleq\\;R\_\{\\text\{valid\}\}\.\(9\)
Consider a perturbed policyπ′\\pi^\{\\prime\}that replicatesτi\\tau\_\{i\}up to stepkkbut suppresses the EOS token and, instead, appendsmmredundant steps satisfying Assumption[1](https://arxiv.org/html/2606.29296#Thmassumption1), withk\+m≤Kmaxk\+m\\leq K\_\{\\max\}\. Write the resulting augmented trajectory asτi′\\tau\_\{i^\{\\prime\}\}\. Its initial\-token cumulative advantage is

A^i′,1​\(τi′\)=∑j=1ks~i′index​\(j\)\+∑j=k\+1k\+ms~i′index​\(j\)\.\\hat\{A\}\_\{i^\{\\prime\},1\}\(\\tau\_\{i^\{\\prime\}\}\)\\;=\\;\\sum\_\{j=1\}^\{k\}\\tilde\{s\}\_\{i^\{\\prime\}\}^\{\\mathrm\{index\}\(j\)\}\+\\sum\_\{j=k\+1\}^\{k\+m\}\\tilde\{s\}\_\{i^\{\\prime\}\}^\{\\mathrm\{index\}\(j\)\}\.\(10\)Because the firstkksteps are shared, the prefix term equalsRvalidR\_\{\\text\{valid\}\}\. By Assumption[1](https://arxiv.org/html/2606.29296#Thmassumption1)the remainingmmterms are each bounded below byϵ\>0\\epsilon\>0, so

A^i′,1​\(τi′\)≥Rvalid\+m⋅ϵ\>A^i,1​\(τi\)for all​m\>0\.\\hat\{A\}\_\{i^\{\\prime\},1\}\(\\tau\_\{i^\{\\prime\}\}\)\\;\\geq\\;R\_\{\\text\{valid\}\}\+m\\cdot\\epsilon\\;\>\\;\\hat\{A\}\_\{i,1\}\(\\tau\_\{i\}\)\\qquad\\text\{for all \}m\>0\.\(11\)By the policy gradient theorem, the objectiveJ​\(θ\)=𝔼πθ​\[A^i,1\]J\(\\theta\)=\\mathbb\{E\}\_\{\\pi\_\{\\theta\}\}\[\\hat\{A\}\_\{i,1\}\]therefore receives a strictly positive gradient in the direction that increasesmm, i\.e\., suppresses EOS and appends additional redundant steps\. Iterating this argument, the optimal policy drivesLiL\_\{i\}towardLmaxL\_\{\\max\}, which proves the claim\. ∎

The chunk\-count\-attenuated path scoreS​\(oi\)=\(∑j=1KiA~ibj\)/\(Ki\)kS\(o\_\{i\}\)=\\big\(\\sum\_\{j=1\}^\{K\_\{i\}\}\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\\big\)/\(K\_\{i\}\)^\{k\}recovered atj=1j=1from Equation[7](https://arxiv.org/html/2606.29296#S4.E7)neutralizes this gradient: fork∈\(0,1\]k\\in\(0,1\], oncemmis large enough the marginal gainm​ϵm\\epsilonis dominated by the\(Ki\)k\(K\_\{i\}\)^\{k\}divisor, so appending further redundant chunks stops improvingSSand can even reduce it \(this is the explicit content of the marginal\-value criterion in Equation[8](https://arxiv.org/html/2606.29296#S4.E8)\)\. This restores a finite optimum forKiK\_\{i\}, and consequently forLiL\_\{i\}, whenever the marginal process signalϵ\\epsilonis small compared with the average density of genuinely logical chunks\.

## Appendix BAlgorithmic Specification of PASS Operators

This appendix gives pseudocode for the three PASS operators introduced in §[4](https://arxiv.org/html/2606.29296#S4)\. All three algorithms take inputs that already come from the upstream preliminaries of §[3](https://arxiv.org/html/2606.29296#S3): a group ofGGrollouts\{o1,…,oG\}\\\{o\_\{1\},\\dots,o\_\{G\}\\\}, a step\-level process signalsprc​\(i,j\)s\_\{\\text\{prc\}\}\(i,j\), sequence\-level outcome and format rewardsrout,i,rfmt,ir\_\{\\text\{out\},i\},r\_\{\\text\{fmt\},i\}, and a group\-level standardization operator𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\)\. The three algorithms are composed in order \(AF→\\toCV→\\toDL\); their composition is summarized in Algorithm[4](https://arxiv.org/html/2606.29296#alg4)\.

Algorithm 1Advantage Fusion \(AF\)\. Independent within\-group standardization of the process, outcome, and format streams, followed by format\-gated piecewise recombination\. Implements Equation[4](https://arxiv.org/html/2606.29296#S4.E4)\.1:Group rollouts

\{o1,…,oG\}\\\{o\_\{1\},\\dots,o\_\{G\}\\\}; per\-step signal

sprc​\(i,j\)s\_\{\\text\{prc\}\}\(i,j\); outcome

rout,ir\_\{\\text\{out\},i\}; format

rfmt,ir\_\{\\text\{fmt\},i\}; format indicator

𝕀fmt,i∈\{0,1\}\\mathbb\{I\}\_\{\\text\{fmt\},i\}\\in\\\{0,1\\\}; mixing weights

wprc,wout,wfmtw\_\{\\text\{prc\}\},w\_\{\\text\{out\}\},w\_\{\\text\{fmt\}\}with

W=wprc\+wout\+wfmtW=w\_\{\\text\{prc\}\}\+w\_\{\\text\{out\}\}\+w\_\{\\text\{fmt\}\}; standardization operator

𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\)\.

2:Per\-step fused advantage

A~i\(t\)\\tilde\{A\}\_\{i\}^\{\(t\)\}for every token

ttin every rollout

oio\_\{i\}\.

3:

𝐒prc←\{sprc​\(i,j\)\}i,j\\mathbf\{S\}\_\{\\text\{prc\}\}\\leftarrow\\\{s\_\{\\text\{prc\}\}\(i,j\)\\\}\_\{i,j\},

𝐒out←\{rout,i\}i\\mathbf\{S\}\_\{\\text\{out\}\}\\leftarrow\\\{r\_\{\\text\{out\},i\}\\\}\_\{i\},

𝐒fmt←\{rfmt,i\}i\\mathbf\{S\}\_\{\\text\{fmt\}\}\\leftarrow\\\{r\_\{\\text\{fmt\},i\}\\\}\_\{i\}⊳\\trianglerightthree independent channel sets

4:foreach rollout

i∈\{1,…,G\}i\\in\\\{1,\\dots,G\\\}and each step

j∈\{1,…,Ki\}j\\in\\\{1,\\dots,K\_\{i\}\\\}do

5:

s~prc\(t\)←𝒩​\(sprc​\(i,j\);𝐒prc\)\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\}\\leftarrow\\mathcal\{N\}\(s\_\{\\text\{prc\}\}\(i,j\);\\mathbf\{S\}\_\{\\text\{prc\}\}\)for

ttinside step

jj⊳\\trianglerightprocess channel

6:endfor

7:foreach rollout

i∈\{1,…,G\}i\\in\\\{1,\\dots,G\\\}do

8:

r~out\(t\)←𝒩​\(rout,i;𝐒out\)\\tilde\{r\}\_\{\\text\{out\}\}^\{\(t\)\}\\leftarrow\\mathcal\{N\}\(r\_\{\\text\{out\},i\};\\mathbf\{S\}\_\{\\text\{out\}\}\)for all

ttin

oio\_\{i\}⊳\\trianglerightoutcome channel, broadcast over tokens

9:

r~fmt\(t\)←𝒩​\(rfmt,i;𝐒fmt\)\\tilde\{r\}\_\{\\text\{fmt\}\}^\{\(t\)\}\\leftarrow\\mathcal\{N\}\(r\_\{\\text\{fmt\},i\};\\mathbf\{S\}\_\{\\text\{fmt\}\}\)for all

ttin

oio\_\{i\}⊳\\trianglerightformat channel, broadcast over tokens

10:endfor

11:foreach rollout

i∈\{1,…,G\}i\\in\\\{1,\\dots,G\\\}and each token

ttin

oio\_\{i\}do

12:if

𝕀fmt,i=1\\mathbb\{I\}\_\{\\text\{fmt\},i\}=1then⊳\\trianglerightformat\-compliant branch

13:

A~i\(t\)←wprc​s~prc\(t\)\+wout​r~out\(t\)\+wfmt​r~fmt\(t\)\\tilde\{A\}\_\{i\}^\{\(t\)\}\\leftarrow w\_\{\\text\{prc\}\}\\,\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\}\+w\_\{\\text\{out\}\}\\,\\tilde\{r\}\_\{\\text\{out\}\}^\{\(t\)\}\+w\_\{\\text\{fmt\}\}\\,\\tilde\{r\}\_\{\\text\{fmt\}\}^\{\(t\)\}
14:else⊳\\trianglerightformat\-violating branch: gate off prc/out, preserve gradient norm viaWW

15:

A~i\(t\)←W⋅r~fmt\(t\)\\tilde\{A\}\_\{i\}^\{\(t\)\}\\leftarrow W\\cdot\\tilde\{r\}\_\{\\text\{fmt\}\}^\{\(t\)\}
16:endif

17:endfor

18:return

\{A~i\(t\)\}i,t\\\{\\tilde\{A\}\_\{i\}^\{\(t\)\}\\\}\_\{i,t\}

Algorithm 2Chunk\-by\-Value \(CV\)\. Numerical\-equality run\-length boundary detection onsprcs\_\{\\text\{prc\}\}, followed by chunk\-level broadcasting of the AF\-fused advantage\. Implements Equation[6](https://arxiv.org/html/2606.29296#S4.E6)\.1:Per\-step signal

sprc​\(i,j\)s\_\{\\text\{prc\}\}\(i,j\)and its already\-standardized form

s~prc\(t\)\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\}from Algorithm[1](https://arxiv.org/html/2606.29296#alg1); fused advantage

A~i\(t\)\\tilde\{A\}\_\{i\}^\{\(t\)\}from Algorithm[1](https://arxiv.org/html/2606.29296#alg1); numerical tolerance

η=10−8\\eta=10^\{\-8\}; signal mode

∈\{PRM,KL\}\\in\\\{\\text\{PRM\},\\text\{KL\}\\\}; for the KL mode, an answer\-region mask

MiM\_\{i\}\.

2:Per\-token advantage

A~i,t\\tilde\{A\}\_\{i,t\}\(chunk\-constant within each chunk\) for every rollout

oio\_\{i\}\.

3:ifsignal mode

=PRM=\\text\{PRM\}then⊳\\trianglerightPRM is piecewise constant within a reasoning step

4:

ut←sprcu\_\{t\}\\leftarrow s\_\{\\text\{prc\}\}at token

tt, padded by PRM step boundaries for rollout

oio\_\{i\}
5:else⊳\\trianglerightKL\-style: smooth and restrict to optimizable tokens

6:

u¯t←1G​∑i′=1Gs~prc\(t\)​\(i′\)\\bar\{u\}\_\{t\}\\leftarrow\\frac\{1\}\{G\}\\sum\_\{i^\{\\prime\}=1\}^\{G\}\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\}\(i^\{\\prime\}\)for each token position

tt⊳\\trianglerightgroup\-averaged, already\-standardized signal

7:

ut←u¯tu\_\{t\}\\leftarrow\\bar\{u\}\_\{t\}if

Mi​\[t\]=1M\_\{i\}\[t\]=1\(inside the answer region\); else treat as out\-of\-scope

8:endif

9:Initialize boundaries

ℬi←\{\}\\mathcal\{B\}\_\{i\}\\leftarrow\\\{\\\}, previous value

vprev←u1v\_\{\\text\{prev\}\}\\leftarrow u\_\{1\}
10:for

t=2,…,Lit=2,\\dots,L\_\{i\}do

11:if

\|ut−vprev\|\>η\|u\_\{t\}\-v\_\{\\text\{prev\}\}\|\>\\etaor

ttcrosses the PRM step boundarythen

12:Append

t−1t\-1to

ℬi\\mathcal\{B\}\_\{i\}⊳\\trianglerightclose previous chunk at tokent−1t\-1

13:

vprev←utv\_\{\\text\{prev\}\}\\leftarrow u\_\{t\}
14:endif

15:endfor

16:Append

LiL\_\{i\}to

ℬi\\mathcal\{B\}\_\{i\}and write

ℬi=\{b1,…,bKi\}\\mathcal\{B\}\_\{i\}=\\\{b\_\{1\},\\dots,b\_\{K\_\{i\}\}\\\}with

bKi=Lib\_\{K\_\{i\}\}=L\_\{i\}
17:foreach chunk index

j=1,…,Kij=1,\\dots,K\_\{i\}do

18:

A~ibj←A~i\(bj\)\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\\leftarrow\\tilde\{A\}\_\{i\}^\{\(b\_\{j\}\)\}⊳\\trianglerightevaluate AF\-fused advantage at the boundary token

19:foreach token

ttwith

bj−1<t≤bjb\_\{j\-1\}<t\\leq b\_\{j\}\(let

b0≜0b\_\{0\}\\triangleq 0\)do

20:

A~i,t←A~ibj\\tilde\{A\}\_\{i,t\}\\leftarrow\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}⊳\\trianglerightbroadcast one value across the chunk

21:endfor

22:endfor

23:return

\{A~i,t\}i,t\\\{\\tilde\{A\}\_\{i,t\}\\\}\_\{i,t\}and chunk structure

\{ℬi\}i\\\{\\mathcal\{B\}\_\{i\}\\\}\_\{i\}

Algorithm 3Divide\-Length \(DL\)\. Trajectory\-level chunk\-count attenuation of the per\-chunk return\-to\-go derived from the CV\-broadcast advantage\. Each chunk’s RTG is divided by the residual chunk count\(Ki−j\+1\)k\(K\_\{i\}\-j\+1\)^\{k\}and the resulting density\-rescaled chunk value is broadcast back to every token inside that chunk\. Implements Equation[7](https://arxiv.org/html/2606.29296#S4.E7)\.1:Per\-token advantage

A~i,t\\tilde\{A\}\_\{i,t\}from Algorithm[2](https://arxiv.org/html/2606.29296#alg2); chunk structure

\{ℬi\}i\\\{\\mathcal\{B\}\_\{i\}\\\}\_\{i\}with

ℬi=\{b1,…,bKi\}\\mathcal\{B\}\_\{i\}=\\\{b\_\{1\},\\dots,b\_\{K\_\{i\}\}\\\}; chunk\-level fused advantage

A~ibj\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}; exponent

k∈\[0,1\]k\\in\[0,1\]\.

2:DL\-reshaped per\-token advantage

A~i,tDL\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}and trajectory\-level path score

S​\(oi\)S\(o\_\{i\}\)\.

3:foreach rollout

i∈\{1,…,G\}i\\in\\\{1,\\dots,G\\\}do

4:foreach chunk index

j=1,…,Kij=1,\\dots,K\_\{i\}do

5:

Gij←∑j′≥jA~ibj′G\_\{i\}^\{j\}\\leftarrow\\sum\_\{j^\{\\prime\}\\geq j\}\\tilde\{A\}\_\{i\}^\{b\_\{j^\{\\prime\}\}\}⊳\\trianglerightper\-chunk return\-to\-go

6:

A~ibj,DL←Gij\(Ki−j\+1\)k\\tilde\{A\}\_\{i\}^\{b\_\{j\},\\,\\text\{DL\}\}\\leftarrow\\dfrac\{G\_\{i\}^\{j\}\}\{\(K\_\{i\}\-j\+1\)^\{k\}\}⊳\\trianglerightEquation[7](https://arxiv.org/html/2606.29296#S4.E7)

7:foreach token

ttwith

bj−1<t≤bjb\_\{j\-1\}<t\\leq b\_\{j\}\(let

b0≜0b\_\{0\}\\triangleq 0\)do

8:

A~i,tDL←A~ibj,DL\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}\\leftarrow\\tilde\{A\}\_\{i\}^\{b\_\{j\},\\,\\text\{DL\}\}⊳\\trianglerightbroadcast one density\-rescaled value per chunk

9:endfor

10:endfor

11:

S​\(oi\)←∑j=1KiA~ibj\(Ki\)kS\(o\_\{i\}\)\\leftarrow\\dfrac\{\\sum\_\{j=1\}^\{K\_\{i\}\}\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\}\{\(K\_\{i\}\)^\{k\}\}⊳\\trianglerighttrajectory\-level density score

12:endfor

13:return

\{A~i,tDL\}i,t\\\{\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}\\\}\_\{i,t\},

\{S​\(oi\)\}i\\\{S\(o\_\{i\}\)\\\}\_\{i\}

Algorithm 4PASS: end\-to\-end composition of AF, CV, and DL\. The outputA~i,tDL\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}is plugged into GRPO’s clipped surrogate in place of the baseline cumulative advantageA^i,t\\hat\{A\}\_\{i,t\}of Equation[3](https://arxiv.org/html/2606.29296#S3.E3)\.1:Group rollouts, raw signals, and hyperparameters from Algorithms[1](https://arxiv.org/html/2606.29296#alg1)–[3](https://arxiv.org/html/2606.29296#alg3)\.

2:PASS\-reshaped per\-token advantage

\{A~i,tDL\}i,t\\\{\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}\\\}\_\{i,t\}for the clipped surrogate\.

3:

\{A~i\(t\)\}←AF​\(\{oi\},sprc,rout,rfmt,𝕀fmt,\{w⋅\},𝒩\)\\\{\\tilde\{A\}\_\{i\}^\{\(t\)\}\\\}\\leftarrow\\text\{AF\}\(\\\{o\_\{i\}\\\},s\_\{\\text\{prc\}\},r\_\{\\text\{out\}\},r\_\{\\text\{fmt\}\},\\mathbb\{I\}\_\{\\text\{fmt\}\},\\\{w\_\{\\cdot\}\\\},\\mathcal\{N\}\)⊳\\trianglerightAlgorithm[1](https://arxiv.org/html/2606.29296#alg1)

4:

\(\{A~i,t\},\{ℬi\}\)←CV​\(\{A~i\(t\)\},sprc,s~prc\(t\),mode,η\)\(\\\{\\tilde\{A\}\_\{i,t\}\\\},\\\{\\mathcal\{B\}\_\{i\}\\\}\)\\leftarrow\\text\{CV\}\(\\\{\\tilde\{A\}\_\{i\}^\{\(t\)\}\\\},s\_\{\\text\{prc\}\},\\tilde\{s\}\_\{\\text\{prc\}\}^\{\(t\)\},\\text\{mode\},\\eta\)⊳\\trianglerightAlgorithm[2](https://arxiv.org/html/2606.29296#alg2)

5:

\{A~i,tDL\}←DL​\(\{A~i,t\},\{ℬi\},\{A~ibj\},k\)\\\{\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}\\\}\\leftarrow\\text\{DL\}\(\\\{\\tilde\{A\}\_\{i,t\}\\\},\\\{\\mathcal\{B\}\_\{i\}\\\},\\\{\\tilde\{A\}\_\{i\}^\{b\_\{j\}\}\\\},k\)⊳\\trianglerightAlgorithm[3](https://arxiv.org/html/2606.29296#alg3)

6:return

\{A~i,tDL\}i,t\\\{\\tilde\{A\}\_\{i,t\}^\{\\text\{DL\}\}\\\}\_\{i,t\}

## Appendix CAdditional Ablations on Mathematical Reasoning

This appendix reports two sensitivity studies on mathematical reasoning that complement the main\-body ablations of §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1): the length\-decay exponentkkof the DL rule \(Equation[7](https://arxiv.org/html/2606.29296#S4.E7)\) and the scale of the upstream PRM\.

#### Length decay exponentkk\.

The DL rule is a one\-parameter family indexed by the length exponentk∈\(0,1\]k\\in\(0,1\], interpolating between strict average\-reward normalization \(k=1k\{=\}1\) and no length attenuation \(k→0k\\to 0\)\. Table[5](https://arxiv.org/html/2606.29296#A3.T5)sweepsk∈\{1\.0,0\.7,0\.5,0\.3\}k\\in\\\{1\.0,0\.7,0\.5,0\.3\\\}\. A relaxation from the strictk=1\.0k\{=\}1\.0towardk=0\.7k\{=\}0\.7protects valid long deductions, lifts average pass@1 from35\.735\.7to36\.936\.9, and moves the ceiling from55\.055\.0to55\.755\.7pass@8\. Further relaxation tok=0\.5k\{=\}0\.5ork=0\.3k\{=\}0\.3does not yield an additional pass@1 improvement overk=0\.7k\{=\}0\.7, although pass@8 continues to rise marginally\. We accordingly setk=0\.7k=0\.7as the default for all subsequent experiments, including the multi\-hop QA runs of §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)\.

Table 5:Sensitivity of PASS to the length\-decay exponentkkin Equation[7](https://arxiv.org/html/2606.29296#S4.E7)\. Metrics are pass@1 / pass@8 \(%\)\. Settingk<1\.0k\{<\}1\.0protects necessary reasoning verbosity and raises the exploratory ceiling;k=0\.7k\{=\}0\.7achieves the best average pass@1 and is used as the default throughout this paper\.
#### PRM scale\.

A natural question is whether the actor’s ceiling is bounded by the resolution of the process signal itself\. Table[6](https://arxiv.org/html/2606.29296#A3.T6)replaces the 7B PRM with the 72B PRM ofZhanget al\.\[[2025](https://arxiv.org/html/2606.29296#bib.bib7)\], keeping all other components of the pipeline \(includingk=1\.0k\{=\}1\.0for a direct match with Table[1](https://arxiv.org/html/2606.29296#S5.T1)\) fixed\. The stronger PRM raises average pass@1 from35\.735\.7to37\.337\.3and pass@8 from55\.055\.0to55\.855\.8, with the largest gains on AIME24 \(14\.8→17\.214\.8\\to 17\.2\) and AMC23 \(53\.8→57\.253\.8\\to 57\.2\) pass@1\. Within the regime we tested, this monotone trend across the 7B\-to\-72B PRM scaling indicates that PASS demonstrates robust performance scaling across PRMs of different capacities, and benefits from the higher resolution of larger reward models without requiring algorithmic modification; we report this as a robustness property of the middleware on the specific scales we evaluated rather than as a strong universal scalability claim, as the present experiments do not extend beyond the 72B PRM tier nor to fundamentally different verifier architectures\.

Table 6:Effect of scaling the PRM from 7B to 72B parameters, with all other components of PASS fixed at the Table[1](https://arxiv.org/html/2606.29296#S5.T1)configuration\. Metrics are pass@1 / pass@8 \(%\)\.

## Appendix DAbs\-Max Scaling and Length–Accuracy Dynamics

This appendix gathers the detailed results for the alternative standardization operator \(Abs\-Max Scaling\) and the joint length–accuracy analysis across both standardization operators\. The main\-body §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)establishes the core comparison under Masked\-Norm; what follows documents the complementary Abs\-Max Scaling runs and the length–accuracy profiles of all eight configurations\.

### D\.1An Alternative Standardization Operator: Abs\-Max Scaling

#### Motivation\.

The Masked\-Norm operator used in §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)is a standard Z\-score normalization that subtracts the within\-group mean and divides by the within\-group standard deviation\. This choice is natural for PRM scores, whose within\-group values are approximately symmetric around a data\-dependent reference level, so subtracting their mean does not displace an intrinsic reference point of the signal\. For KL\-style signals the situation is different\. A token\-level negative reverse KL against a stronger teacher \(the OPD signal\) is non\-positive by construction: the signal has no meaningful origin of its own and is better thought of as a single\-directional friction field, in which zero only means “no penalty” rather than “neutral\.” The G\-OPD variant of Equation[5](https://arxiv.org/html/2606.29296#S4.E5)behaves differently again: by subtracting a base\-policy term, G\-OPD introduces an intrinsic zero that separates tokens on which the teacher is more confident than the base \(a “positive” cognitive increment\), tokens on which teacher and base agree \(a “neutral” region near zero\), and tokens on which the base is more confident than the teacher \(a “negative” region\)\. For both KL\-style signals, Masked\-Norm replaces the implicit or explicit origin of the raw signal with the group mean and recenters the distribution at zero regardless of where the original zero lay\. We therefore study a second standardization operator,*Abs\-Max Scaling*, which divides each entry ofsprcs\_\{\\text\{prc\}\}by the group\-wide maximum absolute value and leaves the mean untouched\. Abs\-Max Scaling bringssprcs\_\{\\text\{prc\}\}to a bounded numeric range comparable to Masked\-Norm while preserving the sign of each entry and the relative position of its zero point, and is the natural candidate for an operator that respects the origin of a KL\-style signal\.

#### Setup\.

We repeat the four configurations of Table[4](https://arxiv.org/html/2606.29296#S5.T4)with𝒩​\(⋅;𝐒\)\\mathcal\{N\}\(\\cdot;\\mathbf\{S\}\)replaced by Abs\-Max Scaling, keeping all other optimizer, sampling, and architectural settings identical\. All eight configurations are summarized in Appendix[E](https://arxiv.org/html/2606.29296#A5)\.

Table 7:pass@1 \(%\) on the three multi\-hop QA benchmarks with the Abs\-Max Scaling standardization operator\. PASS retains a positive improvement on both OPD and G\-OPD; the baselines themselves are stronger than their Masked\-Norm counterparts in Table[4](https://arxiv.org/html/2606.29296#S5.T4), which reduces the absolute headroom for PASS in this setting\.
#### Observation\.

Table[7](https://arxiv.org/html/2606.29296#A4.T7)reports the Abs\-Max Scaling results\. Two features are worth noting\. First, the Abs\-Max baselines are systematically stronger than the Masked\-Norm baselines: GRPO\(\+OPD, scaled\) improves the average over GRPO\(\+OPD\) from54\.3254\.32to57\.2957\.29, and GRPO\(\+G\-OPD, scaled\) improves the average over GRPO\(\+G\-OPD\) from54\.8954\.89to56\.2256\.22\. This confirms that Abs\-Max Scaling is a reasonable choice for KL\-style signals in its own right\. Second, on top of these stronger baselines, PASS still delivers positive average gains under both process signals \(\+0\.13\+0\.13on OPD,\+0\.75\+0\.75on G\-OPD\), though the gains are smaller in absolute terms than those observed under Masked\-Norm\. We therefore report PASS’s effectiveness as consistent across both standardization operators, while deferring the quantitative interpretation of the gain\-gap to §[D\.2](https://arxiv.org/html/2606.29296#A4.SS2), where we analyze it jointly with the response\-length behavior\.

### D\.2Length–Accuracy Dynamics across Standardization Operators

The Masked\-Norm results of §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)and the Abs\-Max Scaling results of §[D\.1](https://arxiv.org/html/2606.29296#A4.SS1)report two different profiles on the same four PASS/baseline pairs: Masked\-Norm achieves larger pass@1 gains, while Abs\-Max Scaling starts from stronger baselines but leaves smaller residual room for PASS\. We now examine these runs jointly through the lens of their response length, first at training time and then at deployment time\. In the terminology of §[4\.3](https://arxiv.org/html/2606.29296#S4.SS3), the DL rule is a two\-sided marginal\-value filter: on the multi\-hop QA panel its admissible region contains trajectories whose marginal step contribution exceeds the current average density of the path, and whether this region corresponds to shorter or longer responses than a baseline trajectory is a regime\-dependent question rather than a design choice\.

![Refer to caption](https://arxiv.org/html/2606.29296v1/figures/length_opd_normed.png)Figure 2:Training\-time mean rollout length \(in tokens\) on HotpotQA across one training epoch, for the four Masked\-Norm configurations of Table[4](https://arxiv.org/html/2606.29296#S5.T4)\.Left:OPD—GRPO\(\+OPD\) vs\. PASS \+ OPD\.Right:G\-OPD—GRPO\(\+G\-OPD\) vs\. PASS \+ G\-OPD\. All four runs start from the same actor at a rollout length of roughly180180tokens\. The two baseline runs drift downward to115115–135135tokens, while the two PASS runs drift upward to235235–260260tokens\. This figure reports rollout\-time tokens and is complementary to the deployment\-time token counts in Table[8](https://arxiv.org/html/2606.29296#A4.T8)\.#### Training\-time rollout length\.

Figure[2](https://arxiv.org/html/2606.29296#A4.F2)shows the mean rollout length of the four Masked\-Norm configurations across one training epoch on HotpotQA training rollouts\. The four runs start from the same actor and therefore share a common starting rollout length of roughly180180tokens\. Over training, the two non\-PASS runs \(GRPO\(\+OPD\) and GRPO\(\+G\-OPD\)\) decrease monotonically to115115–135135tokens, shortening relative to the starting point rather than extending it\. The two PASS runs, by contrast, grow from the same180180\-token starting length to235235–260260tokens over the same number of optimizer steps\. The two trajectories therefore diverge from a common initialization in opposite directions, and the comparison is not between a short baseline and a long PASS output at an arbitrary training snapshot but between two policies that, starting from the same rollout length, are pushed to shrink and to grow respectively by the optimization objective\. The direction of drift is not a free parameter: it is determined by the advantage\-shaping rule that each run optimizes against, with PASS admitting rollout length whenever the marginal contribution is above the current density of the path and the baseline curtailing it whenever cumulative friction outweighs local gains\.

![Refer to caption](https://arxiv.org/html/2606.29296v1/figures/opd_entropy.png)Figure 3:Training\-time token\-level entropy of the actor on HotpotQA across one training epoch, for the same four Masked\-Norm configurations\.Left:OPD\.Right:G\-OPD\. All four runs start from a common actor entropy of roughly0\.350\.35\. The two baseline runs stay near0\.350\.35–0\.450\.45throughout training, whereas the two PASS runs decrease monotonically to0\.070\.07–0\.100\.10\.![Refer to caption](https://arxiv.org/html/2606.29296v1/figures/opd_teacher_kl.png)Figure 4:Training\-time mean token\-level KL divergence against the teacher on HotpotQA across one training epoch, for the same four Masked\-Norm configurations\.Left:OPD\.Right:G\-OPD\. Both the baseline and the PASS runs reduceDKLD\_\{\\mathrm\{KL\}\}relative to the teacher over training, but the PASS runs converge to a higher residualDKLD\_\{\\mathrm\{KL\}\}\(around0\.150\.15\) than the corresponding baseline runs \(≈0\.10\\approx 0\.10on OPD and≈0\.05\\approx 0\.05on G\-OPD\)\.
#### Training\-time entropy and teacher\-KL dynamics\.

Figures[3](https://arxiv.org/html/2606.29296#A4.F3)and[4](https://arxiv.org/html/2606.29296#A4.F4)complement the length profile of Figure[2](https://arxiv.org/html/2606.29296#A4.F2)with two further training\-time observables on the same four Masked\-Norm runs: the actor’s token\-level entropy and its mean token\-level KL against the teacher\. Two regularities are visible and consistent across the OPD and G\-OPD panels\. First, the two PASS runs drive the actor’s entropy down to0\.070\.07–0\.100\.10over training, while the two baseline runs stay near the initial0\.350\.35–0\.450\.45band; taken together with the rollout\-length panel, PASS produces policies that are simultaneously longer and more confident at the token level, which is hard to reconcile with a reading in which the extra length is uncommitted padding\. Second, on the teacher\-KL panel, the PASS runs reduceDKLD\_\{\\mathrm\{KL\}\}against the teacher over training, but plateau at a noticeably higher residual level than the corresponding baseline runs\. The interpretation we adopt is deliberately conservative: read jointly with the downward length drift and the near\-flat entropy of the baselines, the very low baselineDKLD\_\{\\mathrm\{KL\}\}is most parsimoniously attributed to baselines that shorten their rollouts and stay close to the easy, high\-agreement prefix of the teacher’s distribution; the higher plateau of PASS is consistent with a policy that commits to a longer, lower\-entropy rollout while maintaining a non\-degenerate distance to the teacher, rather than collapsing onto it\. We do not claim these three panels isolate a causal mechanism, but the three observables move in a mutually consistent direction under PASS versus the baseline across both process signals\.

Table 8:Average response length in tokens \(Qwen2\.5 tokenizer\) on the three multi\-hop QA benchmarks for all eight configurations of Tables[4](https://arxiv.org/html/2606.29296#S5.T4)and[7](https://arxiv.org/html/2606.29296#A4.T7)\. Under Masked\-Norm, PASS is associated with a sizeable length expansion on top of its base run; under Abs\-Max Scaling, the expansion is substantially smaller\.
#### Two length regimes\.

Table[8](https://arxiv.org/html/2606.29296#A4.T8)exhibits a clear dichotomy\. Under Masked\-Norm, attaching PASS roughly doubles or more the average response length \(OPD:161→313161\\to 313tokens,\+94%\+94\\%; G\-OPD:140→352140\\to 352tokens,\+152%\+152\\%\)\. Under Abs\-Max Scaling the corresponding expansion is limited to about one quarter \(OPD:176→224176\\to 224tokens,\+27%\+27\\%; G\-OPD:185→235185\\to 235tokens,\+27%\+27\\%\)\. A natural question is whether the Masked\-Norm expansion is consistent with what PASS actually promises\. As discussed in §[4](https://arxiv.org/html/2606.29296#S4), DL is a two\-sided marginal\-value filter rather than a length penalty: it admits additional steps only when they raise the path’s average value density, but does not force the final trajectory to be shorter than a baseline trajectory\. The empirical question is therefore whether the observed length growth is accompanied by a commensurate accuracy gain on the same benchmarks, or whether it is closer to free length padding\.

#### Accuracy\-coupled length growth\.

On the hardest benchmark in our multi\-hop panel, MuSiQue \(four hops with adversarial distractors\), the Masked\-Norm rows show that the length growth of PASS is tied to a substantial pass@1 improvement rather than decoupled from accuracy\. For OPD the average length grows from188188to412412tokens, about a factor of two, and pass@1 rises from37\.68%37\.68\\%to48\.50%48\.50\\%, a relative improvement of roughly29%29\\%; for G\-OPD the length grows from171171to477477tokens while pass@1 rises from39\.01%39\.01\\%to47\.29%47\.29\\%\. On 2Wiki and HotpotQA the growth in length is more moderate and pass@1 improves by a smaller but still positive margin\. This pattern is inconsistent with a scenario in which PASS trades length for no additional correctness \(as would be expected from pure step padding\) and is consistent with DL’s two\-sided marginal\-value filter admitting the extra steps precisely because they raise the average value density of the trajectory\.

#### A mechanism\-level reading\.

We describe, without asserting strict causality, how the observed length regimes relate to the interaction between the standardization operator and the downstream DL rule\. Because Masked\-Norm subtracts the group mean, for KL\-style signals whose raw origin does not coincide with the within\-group mean, the operation displaces the zero ofs~prc\\tilde\{s\}\_\{\\text\{prc\}\}relative to the zero ofsprcs\_\{\\text\{prc\}\}, which in turn reshapes the group distribution of marginal values that the DL rule of Equation[8](https://arxiv.org/html/2606.29296#S4.E8)sees\. Under this displacement, some tokens whose raw signal was above the origin can still fall on the negative half ofs~prc\\tilde\{s\}\_\{\\text\{prc\}\}, while others whose raw signal was below the origin can be pushed above it, expanding the dynamic range of the standardized signal that DL integrates\. Under Abs\-Max Scaling the zero ofs~prc\\tilde\{s\}\_\{\\text\{prc\}\}coincides with the zero ofsprcs\_\{\\text\{prc\}\}, so the marginal distribution entering DL is closer to the raw signal’s own profile\. The two operators therefore place PASS at two different operating points of the same two\-sided marginal\-value filter, and it is plausible that the wider dynamic range offered by Masked\-Norm gives the policy more room to lengthen its reasoning whenever those longer trajectories still clear DL’s density threshold\. We stress that this reading is descriptive: it accounts for the observed regimes without an explicit causal intervention on the standardization operator, and isolating the contribution of each mechanism is left to future work\.

#### A caveat on “value density\.”

Dividing pass@1 by average response length gives a very coarse proxy for value density, and on this proxy every PASS row sits below its baseline row in our multi\-hop panel\. We do*not*claim PASS raises this ratio\. DL does not promise that pass@1\-per\-token goes up; it promises that a*single additional step*is rewarded only when its contribution exceeds the current density of the path, which is a local property of the gradient rather than a global property of the deployed trajectory\. The coupling we report above, between large relative pass@1 gains and large relative length growth on the hardest benchmark, is the empirical signal we rely on; the pass@1\-per\-token ratio is not\.

#### Takeaway\.

The main experimental message of Tables[4](https://arxiv.org/html/2606.29296#S5.T4),[7](https://arxiv.org/html/2606.29296#A4.T7), and[8](https://arxiv.org/html/2606.29296#A4.T8)is consistent across the matrix of experiments: PASS delivers a positive pass@1 gain on both KL\-style process signals and under both standardization operators\. The choice of standardization operator shifts the length and accuracy profile along which PASS operates: Masked\-Norm aligns the setup with the mathematical\-reasoning experiments of §[5\.1](https://arxiv.org/html/2606.29296#S5.SS1)and admits longer trajectories that are coupled to larger pass@1 gains on difficult multi\-hop prompts, while Abs\-Max Scaling keeps each entry ofs~prc\\tilde\{s\}\_\{\\text\{prc\}\}on the same side of zero as the corresponding raw signal and yields smaller but length\-controlled gains\. The two operators are better read as two operating points of the same underlying middleware than as a quality ordering\.

## Appendix EMulti\-hop QA Training Configuration and Additional Metrics

All multi\-hop QA runs \(four reported in §[5\.2](https://arxiv.org/html/2606.29296#S5.SS2)and four reported in Appendix[D](https://arxiv.org/html/2606.29296#A4)\) share the following optimizer and sampling configuration: rollout and training global batch sizes are both128128; the maximum sequence length is6,1446\{,\}144; the sampler produces1616rollouts per prompt; we use Adam with learning rate1\.0×10−61\.0\\times 10^\{\-6\}, a constant decay schedule with2020warmup steps,β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999, and weight decay0; the outer loop runs for22training epochs; the GRPO clip ratio is0\.20\.2; and the reference\-KL coefficientβKL\\beta\_\{\\text\{KL\}\}is10−210^\{\-2\}\. Evaluation numbers are reported at the end of training \(roughly7878optimizer updates on the 5k\-prompt subset at the above batch size\)\. The actor is Qwen2\.5\-7B\-Instruct and the teacher is Qwen2\.5\-32B\-Instruct in every run\. The remaining run\-specific fields are summarized in Table[9](https://arxiv.org/html/2606.29296#A5.T9)\.

Table 9:Per\-run hyperparameter differences for the eight multi\-hop QA configurations reported in Tables[4](https://arxiv.org/html/2606.29296#S5.T4),[7](https://arxiv.org/html/2606.29296#A4.T7), and[8](https://arxiv.org/html/2606.29296#A4.T8)\. Entries marked “–” mean the field is not applicable for that configuration\.#### Signal\-aggregation weights\.

All eight configurations set the mixing weightswout=wfmt=wprc=1\.0w\_\{\\text\{out\}\}=w\_\{\\text\{fmt\}\}=w\_\{\\text\{prc\}\}=1\.0for the outcome, format, and process\-signal streams, so that the total magnitudeW=3\.0W=3\.0in Equation[4](https://arxiv.org/html/2606.29296#S4.E4)is shared across runs\.

#### Chunk\-by\-Value and Divide\-Length\.

For the four PASS configurations we enable both the CV and DL sub\-modules and fix the length exponent tok=0\.7k=0\.7\. The non\-PASS configurations do not invoke these sub\-modules\.

#### G\-OPD anchoring\.

The four G\-OPD configurations use the student’s initial weights asπbase\\pi\_\{\\text\{base\}\}and the shift coefficientλ=1\.25\\lambda=1\.25; the four OPD configurations reduce to the\(πθ,πteach\)\(\\pi\_\{\\theta\},\\pi\_\{\\text\{teach\}\}\)formulation and do not consumeλ\\lambda\.

#### Additional metrics: pass@8 and pass@16\.

Table[10](https://arxiv.org/html/2606.29296#A5.T10)reports pass@8 and pass@16 for the same eight configurations on the three multi\-hop QA benchmarks, complementing the pass@1 numbers in the main body\. The ordering of the methods atk∈\{8,16\}k\\in\\\{8,16\\\}is broadly consistent with the pass@1 ordering; in particular, the PASS variants retain their non\-negative margin on the unweighted average under both Masked\-Norm and Abs\-Max Scaling\.

Table 10:pass@8 and pass@16 \(%\) on the three multi\-hop QA benchmarks for the eight configurations of Table[9](https://arxiv.org/html/2606.29296#A5.T9)\.