Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

# Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning
Source: [https://arxiv.org/html/2608.21501](https://arxiv.org/html/2608.21501)
August 2026

###### Abstract

Credit assignment in large\-language\-model reinforcement learning \(LLM RL\) can be separated into three objects:*evidence*about success, a*transport operator*that converts this evidence into token\-level advantages, and an*update geometry*that turns advantages into policy changes\. Recent work has greatly improved evidence, sampling, and update geometry, but the transport operator is usually architecture\-agnostic\. Fixed\-discount GAE applies a stationary geometric kernel along token time; group\-relative methods broadcast an outcome statistic across an entire response\. Neither operator represents the trajectory\-specific computation used by the Transformer policy itself\.

We introduce*computation\-conditioned credit transport*\(CCT\), a general framework in which a detached statistic of the behavior policy’s internal computation parameterizes the causal kernel that transports downstream value through a rollout\. Our concrete algorithm,CompPO, maps native attention concentration to a bounded per\-token retention gate, uses the gate in both the one\-step bootstrap and a path\-dependent generalized\-advantage trace \(Comp\-GAE\), and co\-designs a transport\-aligned critic \(TAC\) that reuses the actor’s hidden states and routing information without a second same\-scale Transformer\. The task reward and clipped PPO policy objective remain unchanged; a constant gate recovers fixed\-coefficient GAE\.

Across five Qwen3\-4B seeds,CompPOreaches61\.4%61\.4\\%final held\-out development accuracy \(95% CI\[60\.8,62\.0\]\[60\.8,62\.0\]\), versus53\.8%53\.8\\%\[52\.9,54\.7\]\[52\.9,54\.7\]for a separately tuned GRPO baseline\. A controlled×22\\\!\\times\\\!2gate\-by\-critic experiment shows that neitherComp\-GAEwith a standard critic \(55\.2%55\.2\\%\) norTACwith a mean\-matched fixed gate \(56\.4%56\.4\\%\) reproduces the full result; the seed\-matched final interaction is\+2\.4\+2\.4points\[1\.9,2\.9\]\[1\.9,2\.9\]\. Gate shuffling and position\-only controls show that trajectory\-specific alignment matters beyond a favorable marginal schedule\. In a matched PPO stress grid,CompPOis stable in10/1210/12runs versus3/123/12for PPO\. Final\-checkpoint frozen evaluation improves over GRPO by4\.34\.3and3\.93\.9greedy pass@1 macro points on Qwen3\-4B and Llama\-3\.1\-8B\-Instruct, respectively\. These results establish policy\-internal computation as a useful estimator variable and open a broader research program on architecture\-aware credit kernels, traces, and critics\.

Keywords:large language models, reinforcement learning, credit assignment, actor–critic, attention, generalized advantage estimation

## 1Introduction

Reinforcement learning with verifiable or preference\-derived rewards has become a primary route to stronger large\-language\-model \(LLM\) reasoning and decision making\. The algorithmic landscape has advanced quickly: PPO supplies bootstrapped token\-level advantages\([Schulman et al\. 2017](https://arxiv.org/html/2608.21501#bib.bib19);[Schulman et al\. 2015](https://arxiv.org/html/2608.21501#bib.bib18)\); RLOO and GRPO replace the critic with response\-level comparison\([Ahmadian et al\. 2024](https://arxiv.org/html/2608.21501#bib.bib1);[Shao et al\. 2024](https://arxiv.org/html/2608.21501#bib.bib22)\); DAPO, Dr\. GRPO, and GSPO improve clipping, normalization, sampling, and importance\-ratio geometry\([Yu et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib32);[Liu et al\. 2025b](https://arxiv.org/html/2608.21501#bib.bib14);[Zheng et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib35)\); process\-reward and rollout\-based methods generate denser evidence about intermediate decisions\([Lightman et al\. 2023](https://arxiv.org/html/2608.21501#bib.bib12);[Uesato et al\. 2022](https://arxiv.org/html/2608.21501#bib.bib25);[Cui et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib3);[Kazemnejad et al\. 2024](https://arxiv.org/html/2608.21501#bib.bib7)\)\. Yet one part of the learning stack remains comparatively implicit:*the operator that transports delayed evidence through the generated trajectory*\.

This omission matters because a Transformer policy is not a homogeneous token chain\. At each position it performs a content\-dependent computation over the prefix: heads retrieve different events, representations compose them, and the active dependency pattern changes across trajectories\. A late mathematical conclusion may use an early substitution; a tool action may depend on an observation many turns earlier; a code token may be constrained by a distant declaration\. The policy is therefore architecture\-rich and history\-structured\. Its credit estimator is usually not\. Fixed\-discount GAE uses one continuation coefficient at every transition; group\-relative methods assign a response\-level statistic to all valid tokens in the response\. We call this discrepancy thearchitecture–credit mismatch\.

To make the mismatch precise, we factor a token\-level policy\-gradient pipeline into three conceptually distinct objects:

1. 1\.credit evidence: what indicates that a decision sequence was good or bad—terminal verifiers, process rewards, judges, counterfactual continuations, or critic residuals;
2. 2\.credit transport: the causal operator that moves that evidence across positions or states to produce token\-level advantages; and
3. 3\.update geometry: importance ratios, clipping, KL constraints, entropy terms, and normalization that convert advantages into a policy update\.

This factorization clarifies why many powerful LLM\-RL advances are complementary rather than interchangeable\. Better verifiers change evidence\. GRPO changes the baseline and sampling statistic\. GSPO changes update geometry\. A process reward changes where evidence enters\. None of these choices uniquely determines how downstream evidence should be transported through the policy’s own computation\.

For standard GAE, the transport can be written as a stationary upper\-triangular kernel\. If𝜹\\boldsymbol\{\\delta\}is the vector of TD residuals, then

𝐀=𝐊γ,λ𝜹,Kt,uγ,λ=𝟏\[u≥t\]\(λγ\)u−t\.\\mathbf\{A\}=\\mathbf\{K\}^\{\\gamma,\\lambda\}\\boldsymbol\{\\delta\},\\qquad K^\{\\gamma,\\lambda\}\_\{t,u\}=\\mathbf\{1\}\[u\\geq t\]\(\\lambda\\gamma\)^\{u\-t\}\.\(1\)The kernel depends only on temporal separation\. A canonical group\-relative estimator instead produces𝐀\(i\)=bi​𝟏\\mathbf\{A\}^\{\(i\)\}=b\_\{i\}\\mathbf\{1\}within responseii: a rank\-one broadcast geometry with no within\-response transport structure\. Both are coherent estimators; both discard the realized internal computation of the policy\.

Our thesis is:

> *The policy already computes a trajectory\-specific dependency structure; credit transport should be allowed to condition on that computation\.*

We introducecomputation\-conditioned credit transport\(CCT\)\. LetΦθ−​\(ht\)\\Phi\_\{\\theta^\{\-\}\}\(h\_\{t\}\)be a statistic extracted from the behavior policy’s internal computation at prefixhth\_\{t\}, and let a bounded map produce a retention gateκt\\kappa\_\{t\}\. The resulting path kernel is

Kt,uκ,λ=𝟏\[u≥t\]λu−t∏j=tu−1κj\.K^\{\\kappa,\\lambda\}\_\{t,u\}=\\mathbf\{1\}\[u\\geq t\]\\lambda^\{u\-t\}\\prod\_\{j=t\}^\{u\-1\}\\kappa\_\{j\}\.\(2\)Unlike[Equation1](https://arxiv.org/html/2608.21501#S1.E1), it is nonstationary and trajectory\-specific: the sensitivity ofAtA\_\{t\}to a later residualδu\\delta\_\{u\}is the realized product of gates along the intervening computation\. The gate is computed by the old policy, detached, and stored with the rollout\. ThusCCTchanges the estimator\-side transport kernel, not the environment reward or task objective\.

We instantiate the framework asComputation\-Conditioned Policy Optimization\(CompPO\)\.111The anonymous submission used the working nameNMPOand denoted the gate byαt\\alpha\_\{t\}\. We rename the framework and algorithm to foreground the more general contribution and to avoid conflating estimator\-side retention with task discounting\. The implemented mechanism and objective experimental results are unchanged\.CompPOuses native attention concentration as an inexpensive scalar summary of whether the current computation is dominated by a sparse subset of history or supported diffusely\. The gate enters a one\-step computation\-conditioned bootstrap and a path\-dependent generalized\-advantage trace,Comp\-GAE\. A transport\-aligned critic \(TAC\) pools value\-relevant history from the actor’s existing hidden states and attention and uses the same gate to couple local and routed\-history value estimates\. The outer clipped PPO objective, verifier, rollout format, and reward are unchanged\.

The method is deliberately more than an attention\-weighted loss\. Attention does not directly multiply the policy gradient\. It parameterizes the bootstrap coefficient, every downstream residual’s trace weight, and the critic state geometry used to estimate those targets\. Nor do we claim that attention is causal token attribution\. Concentration is a policy\-native structural proxy, and its usefulness must be established through controlled alignment tests\.

The empirical evidence is organized to test the main alternative explanations directly\. First, GRPO andCompPOare screened over the same×33\\\!\\times\\\!3actor\-learning\-rate/KL grid and confirmed with five seeds\.CompPOreaches61\.4%61\.4\\%final held\-out development accuracy, compared with53\.8%53\.8\\%for GRPO, and everyCompPOseed exceeds every GRPO seed at both best and final endpoints\. Second, a five\-seed×22\\\!\\times\\\!2experiment crosses dynamic versus mean\-matched fixed transport with standard versus aligned critics\. Dynamic transport with the standard critic reaches55\.2%55\.2\\%final accuracy; the aligned critic with a fixed gate reaches56\.4%56\.4\\%; fullCompPOreaches61\.4%61\.4\\%, with a positive final interaction of\+2\.4\+2\.4points\[1\.9,2\.9\]\[1\.9,2\.9\]\. The standard critic has negative explained variance and weak TD\-target rank correlation, whereas the aligned critic tracks the induced targets substantially better\. Third, global gate shuffling and a frozen position\-only schedule reduce performance, showing that the result is not explained by marginal coefficient scale or position alone\. Fourth, a matched PPO stress grid supports a bounded robustness claim:CompPOis stable in10/1210/12runs versus3/123/12for PPO\. Finally, final\-checkpoint frozen evaluation improves over GRPO by4\.34\.3and3\.93\.9greedy macro points on Qwen3\-4B and Llama\-3\.1\-8B\-Instruct\.

The deeper implication is that LLM RL need not remain architecture\-agnostic\. The dominant critic\-free trend may reflect a contingent mismatch between conventional value representations and long\-horizon targets, rather than an intrinsic impossibility of actor–critic learning for language models\. More broadly, attention, retrieval, memory access, MoE routing, tool dependencies, or cross\-modal information flow could parameterize future Bellman\-style operators, traces, critics, and trust regions\.

Our contributions are:

- •A factorization and problem abstraction\.We separate credit evidence, credit transport, and update geometry, identify the architecture–credit mismatch, and characterize fixed\-discount GAE and group\-relative broadcast as two architecture\-agnostic transport geometries\.
- •A general framework\.CCTturns detached policy\-internal computation into a path\-dependent causal transport kernel\. We derive its matrix form, sensitivity interpretation, reduction to fixed GAE, contraction, and bounded\-trace properties\.
- •A concrete algorithm\.CompPOcouples an attention\-derived retention gate,Comp\-GAE, and a transport\-aligned actor\-feature critic while preserving the external reward and clipped PPO update\.
- •Mechanism\-level evidence\.Matched tuning, five\-seed factorials, critic diagnostics, shuffling, position\-only schedules, a PPO stress grid, a stronger group baseline, and two\-backbone frozen evaluation isolate what the mechanism does and where the evidence stops\.
- •A research agenda\.The work elevates policy\-internal computation to a first\-class RL estimator variable and motivates architecture\-aware credit transport beyond the scalar attention instantiation studied here\.

## 2Credit Assignment as Evidence, Transport, and Update

credit evidencereward, value, process signaltransport operator𝐊\\mathbf\{K\}evidence→\\rightarrowtoken advantageupdate geometry𝒰\\mathcal\{U\}ratio, clipping, KL, entropyThis work changes the middle object and co\-designs the value representation used to construct it\.Temporal coordinatePPO / fixed\-discount GAEt1t\_\{1\}t2t\_\{2\}t3t\_\{3\}t4t\_\{4\}t5t\_\{5\}γ\\gammaγ\\gammaγ\\gammaγ\\gammastationary geometric kernelindexed only by token distanceOutcome/group coordinateGRPO familyy\(1\)y^\{\(1\)\}bib\_\{i\}y\(2\)y^\{\(2\)\}bib\_\{i\}y\(3\)y^\{\(3\)\}bib\_\{i\}one response\-level statisticbroadcast to every valid tokenComputational coordinateCompPOt1t\_\{1\}t2t\_\{2\}t3t\_\{3\}t4t\_\{4\}t5t\_\{5\}κ1\\kappa\_\{1\}κ2\\kappa\_\{2\}κ3\\kappa\_\{3\}κ4\\kappa\_\{4\}trajectory\-specific path kernelconditioned on policy computationFigure 1:Credit assignment factorized into evidence, transport, and update geometry \(top\), with three transport coordinates used in LLM RL \(bottom\)\. Fixed GAE uses a stationary temporal kernel; group\-relative methods broadcast response\-level evidence;CompPOlets behavior\-policy computation parameterize the causal transport path\.### 2\.1Trajectory setup

Letxxbe a prompt andy1:Ty\_\{1:T\}a response sampled fromπθ​\(yt∣ht\)\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{t\}\), whereht=\(x,y<t\)h\_\{t\}=\(x,y\_\{<t\}\)\. Letrtr\_\{t\}denote externally specified reward evidence at positiontt; the main experiments use a sparse terminal verifier\. A policy\-gradient method ultimately constructs token coefficientsAtA\_\{t\}and applies them through an update rule such as PPO\. We write this process abstractly as

𝐞⏟evidence\\displaystyle\\underbrace\{\\mathbf\{e\}\}\_\{\\text\{evidence\}\}→𝐊𝐀⏟transported credit→𝒰Δ​θ⏟policy update,\\displaystyle\\xrightarrow\{\\;\\mathbf\{K\}\\;\}\\underbrace\{\\mathbf\{A\}\}\_\{\\text\{transported credit\}\}\\xrightarrow\{\\;\\mathcal\{U\}\\;\}\\underbrace\{\\Delta\\theta\}\_\{\\text\{policy update\}\},\(3\)𝐀=𝐊𝐞\.\\displaystyle\\mathbf\{A\}=\\mathbf\{K\}\\mathbf\{e\}\.Here𝐞\\mathbf\{e\}may contain rewards, TD residuals, process scores, counterfactual estimates, or other local evidence\.𝐊\\mathbf\{K\}is a causal transport operator, and𝒰\\mathcal\{U\}contains likelihood ratios, clipping, KL, entropy, and normalization\. The factorization is analytic rather than exclusive: an algorithm may jointly modify all three terms\. Its purpose is to identify which object a contribution changes\.

### 2\.2Fixed\-discount GAE is stationary temporal transport

With a value functionVψV\_\{\\psi\}, standard GAE uses

δtγ\\displaystyle\\delta\_\{t\}^\{\\gamma\}=rt\+γ​Vψ​\(ht\+1\)​mt\+1−Vψ​\(ht\)​mt,\\displaystyle=r\_\{t\}\+\\gamma V\_\{\\psi\}\(h\_\{t\+1\}\)m\_\{t\+1\}\-V\_\{\\psi\}\(h\_\{t\}\)m\_\{t\},\(4\)Atγ,λ\\displaystyle A\_\{t\}^\{\\gamma,\\lambda\}=δtγ\+γ​λ​At\+1γ,λ​mt\+1,\\displaystyle=\\delta\_\{t\}^\{\\gamma\}\+\\gamma\\lambda A\_\{t\+1\}^\{\\gamma,\\lambda\}m\_\{t\+1\},\(5\)wheremtm\_\{t\}masks invalid or padded positions\. Ignoring masks for notation and stacking residuals gives

𝐀=\(𝐈−λ​γ​𝐒\)−1​𝜹,\\mathbf\{A\}=\(\\mathbf\{I\}\-\\lambda\\gamma\\mathbf\{S\}\)^\{\-1\}\\boldsymbol\{\\delta\},\(6\)where\(𝐒𝐯\)t=vt\+1\(\\mathbf\{S\}\\mathbf\{v\}\)\_\{t\}=v\_\{t\+1\}\. Therefore

Kt,uγ,λ=𝟏\[u≥t\]\(λγ\)u−t\.K^\{\\gamma,\\lambda\}\_\{t,u\}=\\mathbf\{1\}\[u\\geq t\]\(\\lambda\\gamma\)^\{u\-t\}\.\(7\)The transport kernel is stationary and Toeplitz: it depends only on token distance\. This is not equivalent to saying that GAE has no local signal; TD bootstrapping can makeδt\\delta\_\{t\}informative even with sparse reward\. The structural limitation is narrower: the continuation geometry contains no variable representing how the policy computed the realized action\.

### 2\.3Group\-relative credit is response\-level broadcast

Fornnresponses to the same prompt, a canonical group\-relative baseline is

bi=Ri−meanj⁡Rjstdj⁡Rj\+ε,𝐀\(i\)=bi​𝟏Ti\.b\_\{i\}=\\frac\{R\_\{i\}\-\\operatorname\{mean\}\_\{j\}R\_\{j\}\}\{\\operatorname\{std\}\_\{j\}R\_\{j\}\+\\varepsilon\},\\qquad\\mathbf\{A\}^\{\(i\)\}=b\_\{i\}\\mathbf\{1\}\_\{T\_\{i\}\}\.\(8\)Within a response, this is a rank\-one broadcast geometry: every valid token receives the same outcome\-derived coefficient\. DAPO, Dr\. GRPO, and GSPO materially improve clipping, normalization, sampling, or sequence\-ratio behavior\([Yu et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib32);[Liu et al\. 2025b](https://arxiv.org/html/2608.21501#bib.bib14);[Zheng et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib35)\), but do not by themselves insert a trajectory\-specific Transformer\-computation variable into a Bellman/GAE transport kernel\.

When a sampled group has identical rewards, the standardized group evidence may vanish; when rewards differ, the estimator still does not distinguish tokens that were pivotal from routine tokens\. These properties help explain why fine\-grained credit has become a major LLM\-RL research direction\([Zhang 2026](https://arxiv.org/html/2608.21501#bib.bib34)\)\.

### 2\.4Architecture–credit mismatch

A causal Transformer generates each action through a realized internal computation graph\. Two responses can have identical lengths, rewards, and group statistics yet route information through different historical tokens, heads, memories, or experts\. An estimator that assigns them the same transport kernel discards this difference\.

###### Definition 1\(Architecture–credit mismatch\)\.

A credit estimator has an architecture–credit mismatch when its transport operator is invariant to changes in the policy’s realized internal computation that may alter which history is used to produce the optimized actions\.

This definition does not imply that internal computation is a faithful causal explanation\. It identifies an unused information channel\. Whether that channel improves learning is an empirical question requiring controls for critic capacity, position, marginal gate distribution, and training engineering\.

### 2\.5Three coordinates, not three mutually exclusive algorithms

[Figure1](https://arxiv.org/html/2608.21501#S2.F1)visualizes temporal, group/outcome, and computational coordinates\. They are not mutually exclusive sources of evidence\. A process reward can be transported temporally or computationally; a group baseline can be combined with a token trace; a sequence\-level importance ratio can consume an architecture\-aware advantage\. The contribution of this paper is to make the transport coordinate explicit and optimizable rather than treating fixed time or response broadcast as unavoidable defaults\.

## 3Computation\-Conditioned Credit Transport

### 3\.1Policy\-internal computation as an estimator variable

Letθ−\\theta^\{\-\}denote the behavior policy that generated the rollout\. Its forward pass exposes a computation object𝒢θ−​\(ht\)\\mathcal\{G\}\_\{\\theta^\{\-\}\}\(h\_\{t\}\), such as attention, retrieval scores, memory access, expert routing, or cross\-modal flow\. A statisticΦ\\Phiand bounded mapggproduce

zt=Φ⁡\(𝒢θ−​\(ht\)\),κt=g⁡\(zt\),0≤κt≤κ¯<1\.z\_\{t\}=\\Phi\(\\mathcal\{G\}\_\{\\theta^\{\-\}\}\(h\_\{t\}\)\),\\qquad\\kappa\_\{t\}=g\(z\_\{t\}\),\\qquad 0\\leq\\kappa\_\{t\}\\leq\\bar\{\\kappa\}<1\.\(9\)The gate is stored with the rollout and detached\. The same realizedκt\\kappa\_\{t\}is used throughout the optimization epochs for that batch\. This behavior\-policy conditioning is central:CCTdoes not differentiate through a policy\-dependent task discount and does not redefine the environment’s utility\. It constructs a rollout\-conditioned credit estimator\.

### 3\.2Computation\-conditioned bootstrap and trace

For a value function over a computation\-aware state representationstCs\_\{t\}^\{\\mathrm\{C\}\}, define

δtκ\\displaystyle\\delta\_\{t\}^\{\\kappa\}=rt\+κt​Vψ​\(st\+1C\)​mt\+1−Vψ​\(stC\)​mt,\\displaystyle=r\_\{t\}\+\\kappa\_\{t\}V\_\{\\psi\}\(s\_\{t\+1\}^\{\\mathrm\{C\}\}\)m\_\{t\+1\}\-V\_\{\\psi\}\(s\_\{t\}^\{\\mathrm\{C\}\}\)m\_\{t\},\(10\)Atκ,λ\\displaystyle A\_\{t\}^\{\\kappa,\\lambda\}=δtκ\+λ​κt​At\+1κ,λ​mt\+1\.\\displaystyle=\\delta\_\{t\}^\{\\kappa\}\+\\lambda\\kappa\_\{t\}A\_\{t\+1\}^\{\\kappa,\\lambda\}m\_\{t\+1\}\.\(11\)We call[Equation11](https://arxiv.org/html/2608.21501#S3.E11)Comp\-GAE\. The gate has two linked effects: it changes the one\-step bootstrap target and the retention of all later residuals across the current transition\.

With𝐃κ=diag⁡\(κ1,…,κT\)\\mathbf\{D\}\_\{\\kappa\}=\\diag\(\\kappa\_\{1\},\\ldots,\\kappa\_\{T\}\)and shift matrix𝐒\\mathbf\{S\}, the valid\-token recursion is

𝐀\\displaystyle\\mathbf\{A\}=𝜹\+λ​𝐃κ​𝐒𝐀,\\displaystyle=\\boldsymbol\{\\delta\}\+\\lambda\\mathbf\{D\}\_\{\\kappa\}\\mathbf\{S\}\\mathbf\{A\},\(12\)𝐀\\displaystyle\\mathbf\{A\}=𝐊κ,λ𝜹,𝐊κ,λ=\(𝐈−λ𝐃κ𝐒\)−1\.\\displaystyle=\\mathbf\{K\}^\{\\kappa,\\lambda\}\\boldsymbol\{\\delta\},\\qquad\\mathbf\{K\}^\{\\kappa,\\lambda\}=\(\\mathbf\{I\}\-\\lambda\\mathbf\{D\}\_\{\\kappa\}\\mathbf\{S\}\)^\{\-1\}\.Because𝐒\\mathbf\{S\}is nilpotent on a finite response, the inverse is a finite causal series\. Its entries are

Kt,uκ,λ=𝟏\[u≥t\]λu−t∏j=tu−1κj\.K^\{\\kappa,\\lambda\}\_\{t,u\}=\\mathbf\{1\}\[u\\geq t\]\\lambda^\{u\-t\}\\prod\_\{j=t\}^\{u\-1\}\\kappa\_\{j\}\.\(13\)Thus the exact local sensitivity is

∂At∂δu=Kt,uκ,λ\.\\frac\{\\partial A\_\{t\}\}\{\\partial\\delta\_\{u\}\}=K^\{\\kappa,\\lambda\}\_\{t,u\}\.\(14\)This equation is the core conceptual object\. Standard GAE uses one stationary kernel for all rollouts\.CCTuses a policy\-computation\-conditioned causal kernel whose paths differ across positions and trajectories\.

### 3\.3Latent continuation interpretation

An estimator\-side latent variablect∈\{0,1\}c\_\{t\}\\in\\\{0,1\\\}can be used for intuition\. If

Pr⁡\(ct=1∣𝒢θ−​\(ht\)\)=κt,\\Pr\(c\_\{t\}=1\\mid\\mathcal\{G\}\_\{\\theta^\{\-\}\}\(h\_\{t\}\)\)=\\kappa\_\{t\},\(15\)then marginalizingctc\_\{t\}yields the one\-step rule

\(𝒯κ​V\)​\(ht\)=𝔼⁡\[rt\+κt​V​\(ht\+1\)∣ht\]\.\(\\mathcal\{T\}\_\{\\kappa\}V\)\(h\_\{t\}\)=\\mathbb\{E\}\[r\_\{t\}\+\\kappa\_\{t\}V\(h\_\{t\+1\}\)\\mid h\_\{t\}\]\.\(16\)This does not claim that the environment terminates with probability1−κt1\-\\kappa\_\{t\}\. It says that the estimator retains an expected fractionκt\\kappa\_\{t\}of downstream value across the current computational transition\.

Transition\-dependent discounting is well established as a way to specify task horizon or temporal abstraction\([White 2017](https://arxiv.org/html/2608.21501#bib.bib29);[Harutyunyan et al\. 2019](https://arxiv.org/html/2608.21501#bib.bib5)\); state\-dependent discounting can also be learned with additional consistency controls\([Wang et al\. 2026](https://arxiv.org/html/2608.21501#bib.bib28)\)\.CCTdiffers in role and coupling: the external task remains fixed, the gate is read from behavior\-policy computation, and the same coordinate is carried through the trace and value representation\. We therefore use “retention gate” rather than treatingκt\\kappa\_\{t\}as a new normative task discount\.

### 3\.4Outer update and scope

The actor uses the unchanged clipped surrogate

ℒclip\(θ;θ−\)=𝔼t\[min\(\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{clip\}\}\(\\theta;\\theta^\{\-\}\)=\\mathbb\{E\}\_\{t\}\\\!\\big\[\\min\\big\(ρt​\(θ\)​A^t,\\displaystyle\\rho\_\{t\}\(\\theta\)\\widehat\{A\}\_\{t\},\(17\)clip\(ρt\(θ\),1−ϵ,1\+ϵ\)A^t\)\]\.\\displaystyle\\clip\(\\rho\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\widehat\{A\}\_\{t\}\\big\)\\big\]\.whereρt=πθ​\(yt∣ht\)/πθ−​\(yt∣ht\)\\rho\_\{t\}=\\pi\_\{\\theta\}\(y\_\{t\}\\mid h\_\{t\}\)/\\pi\_\{\\theta^\{\-\}\}\(y\_\{t\}\\mid h\_\{t\}\)andA^t\\widehat\{A\}\_\{t\}is the normalizedComp\-GAEestimate\. Reward computation, rollout sampling, and KL regularization are unchanged\.

This scope avoids two overclaims\. First,CCTis not an attention\-weighted policy loss:κt\\kappa\_\{t\}acts inside the bootstrap and trace before the policy objective is formed\. Second, we do not claim an unbiased gradient theorem for a current\-policy\-dependent discounted\-return objective\. The implemented object is a detached, rollout\-conditioned surrogate\. Gate staleness across multiple update epochs is a real approximation analyzed as a limitation\.

### 3\.5The general design space

The scalar gate studied here is the smallest member of a larger family\. A computation\-conditioned transport design specifies:

1. 1\.an internal computation object𝒢\\mathcal\{G\}and summaryΦ\\Phi;
2. 2\.a bounded transport mapggor a graph\-valued kernel;
3. 3\.a local evidence vector, such as TD residuals, process rewards, tree values, or counterfactual scores;
4. 4\.a transport\-aligned baseline or critic; and
5. 5\.an outer update geometry, potentially including computation\-conditioned clipping or KL\.

Attention concentration is one deployable instantiation, not a claim of optimality\. Future variants can use full attention\-flow graphs, retrieval provenance, tool\-observation edges, MoE expert assignments, recurrent memory gates, or modality\-specific routing\.

## 4CompPO: An Attention\-Routed Instantiation

old policy forwardπθ−\(⋅∣ht\)\\pi\_\{\\theta^\{\-\}\}\(\\cdot\\mid h\_\{t\}\)native outputs\{ht\(ℓ\)\},at​i\\\{h\_\{t\}^\{\(\\ell\)\}\\\},\\;a\_\{ti\}routing sharpnessHt→ct→κtH\_\{t\}\\rightarrow c\_\{t\}\\rightarrow\\kappa\_\{t\}credit transportδtκ,Atκ,λ\\delta\_\{t\}^\{\\kappa\},\\;A\_\{t\}^\{\\kappa,\\lambda\}unchanged clippedPPO objectivecomputation\-aligned criticcross\-layer fusion \+ routed historylocal/global valueVt=κt​VtG\+\(1−κt\)​VtLV\_\{t\}=\\kappa\_\{t\}V\_\{t\}^\{G\}\+\(1\-\\kappa\_\{t\}\)V\_\{t\}^\{L\}actor updatecomputation\-conditioned credit\-estimation path

Figure 2:CompPOchanges the credit\-estimation path while preserving the task reward and outer clipped policy update\. A native policy\-computation statistic yields a detached gateκt\\kappa\_\{t\}\. The same gate controls one\-step bootstrap, eligibility\-trace continuation, and the local/global geometry of a critic that reuses actor features\.### 4\.1Attention concentration as routing sharpness

For each valid response tokentt, letℐt\\mathcal\{I\}\_\{t\}be the set of valid historical positions exposed to the selected causal\-attention layer and letnt=\|ℐt\|n\_\{t\}=\|\\mathcal\{I\}\_\{t\}\|\. We aggregate final\-layer GQA attention across query heads and renormalize overℐt\\mathcal\{I\}\_\{t\}to obtainat​i≥0a\_\{ti\}\\geq 0with∑i∈ℐtat​i=1\\sum\_\{i\\in\\mathcal\{I\}\_\{t\}\}a\_\{ti\}=1\. The routing\-sharpness statistic is the Herfindahl concentration

Ht=∑i∈ℐtat​i2,Ht∈\[1/nt,1\]\.H\_\{t\}=\\sum\_\{i\\in\\mathcal\{I\}\_\{t\}\}a\_\{ti\}^\{2\},\\qquad H\_\{t\}\\in\[1/n\_\{t\},1\]\.\(18\)Uniform attention givesHt=1/ntH\_\{t\}=1/n\_\{t\}and a point mass givesHt=1H\_\{t\}=1\. Because the raw lower bound changes with available history, we normalize it as

ct=log⁡\(nt​Ht\)log⁡nt∈\[0,1\],c\_\{t\}=\\frac\{\\log\(n\_\{t\}H\_\{t\}\)\}\{\\log n\_\{t\}\}\\in\[0,1\],\(19\)fornt\>1n\_\{t\}\>1, withct=1/2c\_\{t\}=1/2for the first valid position\. This transformation maps the two position\-dependent extremes exactly to zero and one while preserving ordering\. It is not a learned position schedule\.

The gate is

κt=κlo\+\(κhi−κlo\)​σ​\(τ⁡\(ct−1/2\)\),\\kappa\_\{t\}=\\kappa\_\{\\mathrm\{lo\}\}\+\(\\kappa\_\{\\mathrm\{hi\}\}\-\\kappa\_\{\\mathrm\{lo\}\}\)\\sigma\\\!\\left\(\\tau\(c\_\{t\}\-1/2\)\\right\),\(20\)with configured envelope parameters\(κlo,κhi,τ\)=\(0\.1,0\.9,4\)\(\\kappa\_\{\\mathrm\{lo\}\},\\kappa\_\{\\mathrm\{hi\}\},\\tau\)=\(0\.1,0\.9,4\)in all main experiments\. Because the sigmoid has finite temperature, the attained range is the strict subinterval\[g⁡\(0\),g⁡\(1\)\]\[g\(0\),g\(1\)\]\. Largerctc\_\{t\}means that the current policy computation is dominated by fewer historical inputs and therefore retains more downstream value across this transition\.

The semantic claim is intentionally limited\.HtH\_\{t\}does not identify which token causally earned the reward\. It summarizes the*shape*of upstream support for the current decision\. The question answered byκt\\kappa\_\{t\}is “how much downstream value should cross this transition under the chosen transport model?”, not “which past token is a causal explanation?” The shuffle and position\-only controls in[Section7\.3](https://arxiv.org/html/2608.21501#S7.SS3)test whether the realized signal carries information beyond a generic position\-dependent gate\.

### 4\.2Computation\-aligned critic

The value estimator must predict targets generated by[Equation10](https://arxiv.org/html/2608.21501#S3.E10)while remaining inexpensive enough for 16K\-token rollouts\. Rather than train a second Transformer, we reuse actor activations and add a small critic with fewer than0\.5%0\.5\\%new parameters\.

#### Cross\-layer actor feature fusion\.

For a selected set ofL′L^\{\\prime\}upper layers,

h¯t=∑ℓ=1L′ωℓ​ht\(ℓ\),ω=softmax⁡\(η\),\\bar\{h\}\_\{t\}=\\sum\_\{\\ell=1\}^\{L^\{\\prime\}\}\\omega\_\{\\ell\}h\_\{t\}^\{\(\\ell\)\},\\qquad\\omega=\\softmax\(\\eta\),\(21\)so the critic can combine representations at different abstraction levels\.

#### Value\-refined history routing\.

Native attention was trained for next\-token computation, not scalar value prediction\. We therefore modulate each edge with a learned bilinear relevance gate,

βt​i\\displaystyle\\beta\_\{ti\}=⟨Wβ,q​h¯t,Wβ,k​h¯i⟩d,\\displaystyle=\\frac\{\\langle W\_\{\\beta,q\}\\bar\{h\}\_\{t\},W\_\{\\beta,k\}\\bar\{h\}\_\{i\}\\rangle\}\{\\sqrt\{d\}\},\(22\)a~t​i\\displaystyle\\widetilde\{a\}\_\{ti\}=at​i​σ​\(βt​i\)\.\\displaystyle=a\_\{ti\}\\,\\sigma\(\\beta\_\{ti\}\)\.\(23\)A second nonnegative token\-relevance score

mt​i=ReLU⁡\(⟨Wr,q​h¯t,Wr,k​h¯i⟩d\)m\_\{ti\}=\\operatorname\{ReLU\}\\\!\\left\(\\frac\{\\langle W\_\{r,q\}\\bar\{h\}\_\{t\},W\_\{r,k\}\\bar\{h\}\_\{i\}\\rangle\}\{\\sqrt\{d\}\}\\right\)\(24\)filters superficial routing correlations\. The earlier version called this module “TokenMI”; because no explicit mutual\-information bound is optimized, the present paper uses the operationally accurate name*token\-relevance gate*\. The final causal pooling score and routed\-history representation are

st​i\\displaystyle s\_\{ti\}=a~t​imt​i𝟏\[i∈ℐt\],\\displaystyle=\\widetilde\{a\}\_\{ti\}m\_\{ti\}\\mathbf\{1\}\[i\\in\\mathcal\{I\}\_\{t\}\],\(25\)htG\\displaystyle h\_\{t\}^\{G\}=∑i∈ℐtst​i∑j∈ℐtst​j\+ε​h¯i\.\\displaystyle=\\sum\_\{i\\in\\mathcal\{I\}\_\{t\}\}\\frac\{s\_\{ti\}\}\{\\sum\_\{j\\in\\mathcal\{I\}\_\{t\}\}s\_\{tj\}\+\\varepsilon\}\\bar\{h\}\_\{i\}\.\(26\)

#### Gate\-aligned local/global value\.

Two lightweight heads produce a local valueVL​\(h¯t\)V^\{L\}\(\\bar\{h\}\_\{t\}\)and a routed\-history valueVG​\(htG\)V^\{G\}\(h\_\{t\}^\{G\}\)\. We combine them using the same transition regime that generated the bootstrap target:

Vψ​\(stC\)=clip⁡\(κt​VG​\(htG\)\+\(1−κt\)​VL​\(h¯t\),−1,1\)\.V\_\{\\psi\}\(s\_\{t\}^\{\\mathrm\{C\}\}\)=\\clip\\\!\\left\(\\kappa\_\{t\}V^\{G\}\(h\_\{t\}^\{G\}\)\+\(1\-\\kappa\_\{t\}\)V^\{L\}\(\\bar\{h\}\_\{t\}\),\-1,1\\right\)\.\(27\)This is a co\-design hypothesis, not a universal theorem about all critics\. Under concentrated routing, the value estimate relies more on explicitly pooled history; under diffuse routing, it relies more on the local actor state\. The factorial evidence in[Section7\.2](https://arxiv.org/html/2608.21501#S7.SS2)tests whether this coupling matters empirically while holding the stability stack fixed\.

### 4\.3Actor and critic objectives

The actor uses[Equation17](https://arxiv.org/html/2608.21501#S3.E17)with normalizedComp\-GAEadvantages\. The critic regresses to the detachedλ\\lambda\-return

G^tκ,λ=Atκ,λ\+Vψ​\(stC\)​mt\\widehat\{G\}\_\{t\}^\{\\kappa,\\lambda\}=A\_\{t\}^\{\\kappa,\\lambda\}\+V\_\{\\psi\}\(s\_\{t\}^\{\\mathrm\{C\}\}\)m\_\{t\}\(28\)with clipped value loss\. The stored attention\-derived gate is detached and cannot be changed by the critic loss within the current rollout batch\. The critic consumes actor features from the same forward pass; all reported variants use the same feature\-sharing convention so that the controlled contrasts isolate only the gate and critic architecture\. The exact backbone\-gradient routing is an implementation fact that must be copied from the released code and is flagged in the accompanying author\-verification checklist\.

### 4\.4Stability and systems implementation

History\-conditioned bootstrapping introduces the familiar actor–critic risks of early value error and policy–critic co\-divergence\. The reported system uses three controls:

1. 1\.a fixed value\-domain clamp\[−1,1\]\[\-1,1\], matched to the bounded reward scale;
2. 2\.zero\-output initialization and a brief critic warm\-up, so early advantages are not dominated by arbitrary bootstrap predictions; and
3. 3\.a closed\-loop phase controller that adjusts actor learning rate, clip range, value\-loss clip, and critic multiplier only when jointly monitored stability statistics cross predefined thresholds\.

The core×22\\\!\\times\\\!2experiment holds all three controls identical across cells, so they cannot explain the gate\-by\-critic interaction\.

The implementation avoids materializing the fullT×TT\\times Tattention matrix: only the final layer exposes chunked attention; GQA heads are aggregated by group; buffers are restricted to response positions; online\-softmax recomputation avoids dense HBM writes; and top\-KKcausal pooling reduces critic work toO⁡\(T​K\)O\(TK\)withK=64K=64\. These choices avoid a separate same\-scale value Transformer and its optimizer state; the additional work is confined to attention extraction and the lightweight critic\. Exact measured wall\-clock, peak\-memory, and throughput records are unavailable in the retained experiment logs\. We therefore restrict the present efficiency claim to the algorithmic structure—actor\-feature reuse, no second same\-scale value Transformer, andO⁡\(T​K\)O\(TK\)routed pooling—rather than claiming an unmeasured runtime advantage\.

## 5Properties, Identifiability, and Claim Boundary

The results below condition on a fixed rollout batch\. The gates were computed byπθ−\\pi\_\{\\theta^\{\-\}\}and are detached, which is the regime implemented by the algorithm\.

###### Proposition 1\(Strict reduction\)\.

Ifκt=γ\\kappa\_\{t\}=\\gammafor every valid transition and the standard critic state is restored, then the computation\-conditioned bootstrap,Comp\-GAE, and clipped policy update reduce exactly to the standard fixed\-γ\\gammaGAE/PPO estimator path\.

The fixed\-gate \+ aligned\-critic ablation is intentionally not vanilla PPO: it preserves the proposed value representation while removing trajectory\-specific transport\. This is necessary to isolate the gate rather than replacing the entire system at once\.

###### Proposition 2\(Causal transport kernel\)\.

For a finite response,[Equation11](https://arxiv.org/html/2608.21501#S3.E11)has the unique solution

Atκ,λ=∑u=tTλu−t​\(∏j=tu−1κj\)​δuκ\.A\_\{t\}^\{\\kappa,\\lambda\}=\\sum\_\{u=t\}^\{T\}\\lambda^\{u\-t\}\\left\(\\prod\_\{j=t\}^\{u\-1\}\\kappa\_\{j\}\\right\)\\delta\_\{u\}^\{\\kappa\}\.\(29\)Equivalently,𝐀=𝐊κ,λ​𝛅\\mathbf\{A\}=\\mathbf\{K\}^\{\\kappa,\\lambda\}\\boldsymbol\{\\delta\}with entries in[Equation13](https://arxiv.org/html/2608.21501#S3.E13)\. Ifκt=γ\\kappa\_\{t\}=\\gamma, the kernel is stationary Toeplitz and equals[Equation7](https://arxiv.org/html/2608.21501#S2.E7); otherwise it is generally path\-dependent and nonstationary\.

The proposition formalizes “let credit follow computation\.” It does not imply nonlocal token attribution: the method still transports evidence through a scalar sequential kernel\. What changes is the local retention assigned to each transition, and therefore the sensitivity of every earlier advantage to every later residual\.

###### Proposition 3\(Well\-posed detached backup\)\.

For a fixed policy and a gate satisfying0≤κt≤κ¯<10\\leq\\kappa\_\{t\}\\leq\\bar\{\\kappa\}<1, the operator in[Equation16](https://arxiv.org/html/2608.21501#S3.E16)is a sup\-norm contraction with modulus at mostκ¯\\bar\{\\kappa\}:

‖𝒯κ​V−𝒯κ​W‖∞≤κ¯​‖V−W‖∞\.\\\|\\mathcal\{T\}\_\{\\kappa\}V\-\\mathcal\{T\}\_\{\\kappa\}W\\\|\_\{\\infty\}\\leq\\bar\{\\kappa\}\\\|V\-W\\\|\_\{\\infty\}\.\(30\)It therefore has a unique bounded fixed point\.

This is a fixed\-policy evaluation statement\. It does not establish global convergence of PPO while the behavior policy, critic, and future gates change\.

###### Proposition 4\(Bounded trace\)\.

If\|δtκ\|≤D\|\\delta\_\{t\}^\{\\kappa\}\|\\leq Dandλ​κ¯<1\\lambda\\bar\{\\kappa\}<1, then

\|Atκ,λ\|≤D1−λ​κ¯\.\|A\_\{t\}^\{\\kappa,\\lambda\}\|\\leq\\frac\{D\}\{1\-\\lambda\\bar\{\\kappa\}\}\.\(31\)

The trace is therefore controlled by the largest realized gate, not by sequence length directly\. TD bootstrapping still provides local residuals: even withrt=0r\_\{t\}=0,δtκ\\delta\_\{t\}^\{\\kappa\}can be nonzero whenever the critic detects a value change\. We do not claim that terminal reward is transmitted without decay through an arbitrarily long chain\.

#### Transport identifiability\.

A performance gain from dynamic gates could arise from at least four explanations: a better average horizon, a useful position schedule, trajectory\-specific alignment, or critic capacity\. The experimental design explicitly separates them\. The fixed gate is set to the empirical mean0\.610\.61; a position\-only control freezes the mean gate in 20 equal\-width position bins; global shuffling preserves the gate values but destroys token correspondence; and the×22\\\!\\times\\\!2factorial crosses transport with critic architecture\. These controls do not establish causal token attribution, but they make the trajectory\-specific transport claim falsifiable\.

#### Old\-policy dependence and staleness\.

The gate is computed fromθ−\\theta^\{\-\}and reused during the actor epochs\. Asθ\\thetamoves, its current internal computation may differ from the stored coordinate\. This resembles other rollout\-derived quantities in PPO but introduces a new source of estimator staleness\. The present implementation uses two actor epochs and clipping; a future analysis should bound the error in terms of policy\-ratio drift, gate drift, and critic error, or recompute the gate per epoch\.

#### Configured envelope versus realized range\.

The attention instantiation uses the smooth map in[Equation20](https://arxiv.org/html/2608.21501#S4.E20)\. With a finite temperature, the configured parameters\(κlo,κhi\)\(\\kappa\_\{\\mathrm\{lo\}\},\\kappa\_\{\\mathrm\{hi\}\}\)are asymptotic envelope parameters; the actual mathematical range is\[g⁡\(0\),g⁡\(1\)\]⊂\(κlo,κhi\)\[g\(0\),g\(1\)\]\\subset\(\\kappa\_\{\\mathrm\{lo\}\},\\kappa\_\{\\mathrm\{hi\}\}\)\. All propositions use the actual supremumκ¯=supcg⁡\(c\)<1\\bar\{\\kappa\}=\\sup\_\{c\}g\(c\)<1\. Descriptive gate statistics reported later focus on mean scale, dispersion, position, and outcome stratification\.

#### What is not proved\.

We do not prove that attention is a causal explanation, that the chosen concentration statistic is optimal, that every standard critic fails, or thatCompPOdominates tuned PPO in every regime\. The theoretical contribution is the transport formulation and its fixed\-batch properties; the algorithmic claims are supported by controlled experiments in the stated domain\.

## 6Experimental Design

The experiments are organized around five falsifiable questions rather than a flat list of baselines\.

1. 1\.Performance:does computation\-conditioned credit improve over a credible group\-relative baseline under a matched search budget?
2. 2\.Mechanism:are the dynamic gate and computation\-aligned critic jointly necessary, or is the result explained by either component alone?
3. 3\.Signal specificity:does trajectory\-specific gate alignment matter beyond position or marginal regularization?
4. 4\.Robustness:does the method occupy a wider stable region than PPO in a matched stress grid?
5. 5\.Transfer and baseline strength:do the gains survive frozen evaluation, a second backbone, and a larger GRPO group?

### 6\.1Task, models, and evaluation

Training uses hard mathematical\-reasoning prompts from DAPO\-Math\-17K with a maximum response length of 16K\. The primary backbone is Qwen3\-4B\([Yang et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib31)\); Llama\-3\.1\-8B\-Instruct\([AI at Meta 2024](https://arxiv.org/html/2608.21501#bib.bib2)\)is used as a transfer control\. A deterministic answer extractor supplies terminal correctness reward\. Malformed reasoning tags receive a negative terminal reward, and the length penalty is applied only to correct, format\-valid responses\.

The in\-distribution \(ID\) development/validation set contains 500 held\-out problems and is distinct from frozen MATH500\. We use the following endpoint conventions throughout:

- •Best: maximum ID validation accuracy within the predeclared training budget;
- •Final: the last checkpoint \(step 200 for the performance and factorial experiments; step 150 for the PPO stress test\);
- •Last\-20: mean over the final 20 checkpoints;
- •AULC: validation accuracy integrated over training steps and divided by the observed step range\.

Final is the primary endpoint\. Frozen evaluation always uses the final checkpoint and is never consulted for hyperparameter selection\.

Frozen benchmarks are AIME25, AIME2024, MATH500, GSM8K, LiveBench, GPQA, and an Olympiad\-style set\. We report greedy pass@1 and majority@8\. Macro\-averages are unweighted across benchmarks and should be interpreted alongside per\-benchmark values, especially because AIME sets contain only 30 items\.

### 6\.2Common training configuration

All primary Qwen runs use batch size 4, four responses per prompt, PPO/GRPO minibatch size 4, 16K maximum response length, BF16, AdamW with\(β1,β2\)=\(0\.9,0\.999\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.999\), zero weight decay, gradient clipping at 1, two update epochs, and sampling temperature/top\-ppof1\.0/0\.71\.0/0\.7\. The actor learning\-rate candidates are\{×10−7,10−6,×10−6\}\\\{5\\\!\\times\\\!10^\{\-7\},10^\{\-6\},2\\\!\\times\\\!10^\{\-6\}\\\}and KL candidates are\{0,10−3,×10−3\}\\\{0,10^\{\-3\},2\\\!\\times\\\!10^\{\-3\}\\\}for both GRPO andCompPO\. The selected GRPO configuration is\(10−6,10−3\)\(10^\{\-6\},10^\{\-3\}\); the selectedCompPOconfiguration is\(10−6,×10−3\)\(10^\{\-6\},2\\\!\\times\\\!10^\{\-3\}\)\. TheCompPOcritic uses learning rate10−510^\{\-5\},λ=0\.95\\lambda=0\.95, configured gate\-envelope parameters\(0\.1,0\.9\)\(0\.1,0\.9\)withτ=4\\tau=4, and value clamp\[−1,1\]\[\-1,1\]\.

The same×33\\\!\\times\\\!3grid is screened once on ID validation for each method\. The selected configuration is then rerun with five independent seeds\{17,42,123,256,2026\}\\\{17,42,123,256,2026\\\}\. The submitted seed is 42\. At every learning rate, the selected\-KLCompPOendpoint exceeds the selected\-KL GRPO endpoint; full grids are reported in[AppendixB](https://arxiv.org/html/2608.21501#A2)\.

### 6\.3Core coefficient\-by\-critic factorial

We cross two transport rules with two critic classes:

TransportCriticInterpretationfixedκ=0\.61\\kappa=0\.61standardmean\-matched fixed\-coordinate controlκt\\kappa\_\{t\}/Comp\-GAEstandarddynamic transport without aligned criticfixedκ=0\.61\\kappa=0\.61alignedaligned critic without trajectory gateκt\\kappa\_\{t\}/Comp\-GAEalignedfullCompPOThe constant 0\.61 equals the empirical mean gate of fullCompPO\. In the fixed/aligned cell, it replacesκt\\kappa\_\{t\}both in the GAE recursion and in the local/global value mixture, removing trajectory dependence while preserving all other architecture and stability components\. All four cells share the same Qwen checkpoint, hard\-DAPO prompt stream, 500\-problem ID set, data seeds, response budget, reward, optimizer, KL/clip configuration, stability system, and 200\-step budget\.

We report policy endpoints and critic diagnostics on a shared held\-out token sample: explained variance \(EV\), Spearman correlation with TD targets, value MAE, and clamp saturation\. The seed\-matched factorial interaction is

ℐ=Full−\(Gate\+Std\)−\(Fixed\+Aligned\)\+\(Fixed\+Std\)\.\\mathcal\{I\}=\\text\{Full\}\-\(\\text\{Gate\+Std\}\)\-\(\\text\{Fixed\+Aligned\}\)\+\(\\text\{Fixed\+Std\}\)\.\(32\)A positiveℐ\\mathcal\{I\}indicates complementarity beyond additive main effects\.

### 6\.4Gate controls

We use two negative controls\.

#### Global shuffle\.

Attention and gates are computed normally, then realizedκt\\kappa\_\{t\}values are randomly permuted across valid response positions before bothComp\-GAEand critic mixing\. This preserves the global marginal distribution but destroys token\-wise alignment\.

#### Position\-only schedule\.

For each seed, full\-training rollout gates are averaged within 20 equal\-width bins of normalized response positiont/Tt/T\. The schedule is frozen and used in bothComp\-GAEand critic mixing; no validation or frozen benchmark data are used to construct it\. This preserves the mean positional profile but removes trajectory\-specific variation\.

We also report the realized gate distribution from the final checkpoints, including mean, median, interquartile range, response\-position thirds, and correct/incorrect trajectories\.

### 6\.5Matched PPO stress grid

The PPO comparison is a robustness stress test, not a claim of universally tuned superiority\. PPO andCompPOare evaluated for 150 steps over actor learning rates\{×10−7,×10−7,10−6\}\\\{2\\\!\\times\\\!10^\{\-7\},5\\\!\\times\\\!10^\{\-7\},10^\{\-6\}\\\}and KL coefficients\{0,10−3\}\\\{0,10^\{\-3\}\\\}, with two seeds per cell\. No annealing, extra entropy/KL constraint, critic warm\-up, or adaptive controller is used\. Both methods use the same fixed value\-domain bound\[−1,1\]\[\-1,1\]; separate no\-clamp checks show that it acts only during a comparable early critic transient\.

A run is declared stable when final accuracy is at least the 42\.4% base accuracy, peak\-to\-final drop is below 15 points, and there is no sequence of five consecutive post\-peak checkpoints below base\. We report stable runs, mean best/final, and peak\-to\-final drop\. The permitted inference is a wider stable region in this tested grid; it is not that PPO cannot be stabilized by method\-specific tuning\.

### 6\.6Stronger GRPO group control

The submitted GRPO uses four prompts and four responses, i\.e\., 16 trajectories per update\. We additionally evaluate a requested×88\\\!\\times\\\!8single\-run GRPO control with 64 trajectories per update and all other settings matched\. This result is a point estimate without training\-seed confidence intervals; it tests baseline sensitivity rather than replacing the five\-seed matched\-budget comparison\.

## 7Results

GRPOCompPO50505555606056\.256\.261\.761\.753\.853\.861\.461\.4ID validation accuracy \(%\)BestFinal\(a\)Five\-seed selected configurations; bars show means and 95% Student\-ttintervals\.Fixed\+StdGate\+StdFixed\+AlignedFull50505555606052\.652\.655\.255\.256\.456\.461\.461\.4Final ID accuracy \(%\)\(b\)Gate\-by\-critic factorial\. The final interaction is\+2\.4\+2\.4points\[1\.9,2\.9\]\[1\.9,2\.9\]\.
Figure 3:Main training evidence\.CompPOimproves both best and final endpoints over the selected GRPO baseline, and the factorial experiment shows that the trajectory\-specific gate and aligned critic are complementary rather than interchangeable\.### 7\.1Five\-seed performance against GRPO

The selected GRPO configuration obtains56\.256\.2% best accuracy \(95% CI\[55\.6,56\.9\]\[55\.6,56\.9\]\) and53\.853\.8% final accuracy\[52\.9,54\.7\]\[52\.9,54\.7\]\.CompPOobtains61\.761\.7% best\[61\.3,62\.1\]\[61\.3,62\.1\]and61\.461\.4% final\[60\.8,62\.0\]\[60\.8,62\.0\]\. EveryCompPOseed exceeds every GRPO seed at both endpoints\. The method’s best\-to\-final drop is only0\.30\.3point, compared with2\.42\.4points for GRPO\.

The difference is not an isolated selected cell\. At actor learning rates×10−75\\\!\\times\\\!10^\{\-7\},10−610^\{\-6\}, and×10−62\\\!\\times\\\!10^\{\-6\}, each method’s preferred\-KL endpoint yieldsCompPO–GRPO gains of\+6\.8/\+7\.2\+6\.8/\+7\.2,\+5\.4/\+7\.6\+5\.4/\+7\.6, and\+6\.8/\+8\.0\+6\.8/\+8\.0best/final points, respectively\. The complete screen appears in[Table4](https://arxiv.org/html/2608.21501#A2.T4)\. This pattern argues against a single lucky learning\-rate/KL choice\.

### 7\.2The gate and critic form a coupled mechanism

[Figures3\(b\)](https://arxiv.org/html/2608.21501#S7.F3.sf2)and[1](https://arxiv.org/html/2608.21501#S7.T1)summarize the controlled×22\\\!\\times\\\!2\. A fixed mean\-matched gate with a standard critic reaches52\.652\.6% final\. Replacing only the trace byComp\-GAEraises the result to55\.255\.2%, but the tested standard critic still has negative EV \(−0\.08\-0\.08\), weak TD\-target Spearman correlation \(0\.160\.16\), and never reaches the submitted GRPO peak of56\.456\.4%\. Replacing only the critic raises final accuracy to56\.456\.4%, with EV0\.450\.45, but remains five points below fullCompPO\. The complete method reaches61\.461\.4%, EV0\.520\.52, and TD\-target Spearman0\.570\.57\.

VariantBestFinalLast\-20AULCPeak–finalEVTDρs\\rho\_\{s\}MAEClamp sat\.Step to 56\.4Fixed gate \+ standard critic53\.452\.652\.047\.40\.8−0\.15\-0\.150\.110\.410\.24neverComp\-GAE\+ standard critic56\.055\.254\.750\.20\.8−0\.08\-0\.080\.160\.370\.19neverFixed gate \+ aligned critic57\.256\.455\.953\.10\.80\.450\.520\.200\.05112FullCompPO61\.761\.460\.858\.90\.30\.520\.570\.180\.0333

Table 1:Five\-seed gate\-by\-critic factorial\. Best and final means have 95% Student\-ttintervals: Fixed\+Std53\.4​\[52\.9,53\.9\]/52\.6​\[52\.0,53\.2\]53\.4\[52\.9,53\.9\]/52\.6\[52\.0,53\.2\];Comp\-GAE\+Std56\.0​\[55\.6,56\.5\]/55\.2​\[54\.6,55\.8\]56\.0\[55\.6,56\.5\]/55\.2\[54\.6,55\.8\]; Fixed\+Aligned57\.2​\[56\.6,57\.8\]/56\.4​\[55\.8,57\.0\]57\.2\[56\.6,57\.8\]/56\.4\[55\.8,57\.0\]; Full61\.7​\[61\.3,62\.1\]/61\.4​\[60\.8,62\.0\]61\.7\[61\.3,62\.1\]/61\.4\[60\.8,62\.0\]\.Seed\-matched differences sharpen the attribution\. FullCompPOexceedsComp\-GAE\+standard critic by\+5\.7\+5\.7best and\+6\.2\+6\.2final points, and exceeds fixed\-gate\+aligned critic by\+4\.5\+4\.5and\+5\.0\+5\.0\. The interaction in[Equation32](https://arxiv.org/html/2608.21501#S6.E32)is\+1\.9​\[1\.3,2\.4\]\+1\.9\[1\.3,2\.4\]for best and\+2\.4​\[1\.9,2\.9\]\+2\.4\[1\.9,2\.9\]for final\. Thus the components are super\-additive under the controlled configuration: the aligned critic is not a free\-standing replacement for the gate, and the gate cannot realize its full benefit through the tested standard value head\.

The critic diagnostics support the proposed mechanism rather than merely restating endpoint accuracy\. Standard\-critic cells have negative EV, large MAE, and high clamp saturation\. Aligned\-critic cells explain target variation, rank TD targets more faithfully, and saturate less\. We therefore interpret the standard critic result as an empirical target/representation mismatch in this setting, not as a theorem that every conventional critic must fail\.

### 7\.3Trajectory\-specific alignment matters

ShufflePositionFull565658586060626258\.758\.759\.759\.761\.461\.4Final ID accuracy \(%\)\(a\)Gate controls\. Full uses trajectory\-specific alignment\.PPOCompPO00551010331010Stable runs \(out of 12\)\(b\)Matched LR–KL stress grid; this is a bounded robustness claim, not tuned PPO ranking\.
Figure 4:Specificity and robustness controls\. Destroying token\-wise alignment or replacing it with a fixed positional profile reduces performance, whileCompPOoccupies a wider stable region in the tested PPO grid\.Global shuffling reduces best/final accuracy from61\.7/61\.461\.7/61\.4to59\.4/58\.759\.4/58\.7, with a2\.72\.7\-point final penalty\. The position\-only schedule recovers part of the effect, reaching60\.2/59\.760\.2/59\.7\. It exceeds the matched fixed gate by\+3\.0/\+3\.3\+3\.0/\+3\.3best/final points and exceeds global shuffle by\+0\.8/\+1\.0\+0\.8/\+1\.0, showing that the mean positional profile is informative\. Nevertheless, fullCompPOexceeds position\-only by a paired\+1\.5​\[1\.3,1\.7\]\+1\.5\[1\.3,1\.7\]best and\+1\.7​\[1\.5,1\.8\]\+1\.7\[1\.5,1\.8\]final points\. The remaining gain therefore depends on trajectory\-specific variation rather than position alone\.

The realized gate is not constant\. Across five final checkpoints, its mean/median are0\.61/0\.630\.61/0\.63and its interquartile range is0\.180\.18\. Mean gates in the early, middle, and late response thirds are0\.52/0\.63/0\.680\.52/0\.63/0\.68, and correct trajectories have higher mean gate than incorrect trajectories \(0\.640\.64versus0\.570\.57\)\. These descriptive differences do not establish causality; they show that the retained coordinate varies meaningfully across positions, trajectories, and outcomes\.

### 7\.4A wider stable region than PPO in the tested grid

Across 12 runs, PPO is stable in3/123/12, with mean best/final/drop47\.7/34\.4/13\.347\.7/34\.4/13\.3\.CompPOis stable in10/1210/12, with52\.9/45\.6/7\.352\.9/45\.6/7\.3\. The largest difference occurs at aggressive learning rates: at10−610^\{\-6\}with KL10−310^\{\-3\}, PPO obtains49\.8/22\.649\.8/22\.6best/final whileCompPOobtains57\.2/44\.657\.2/44\.6; in the originally submitted cell, the corresponding single\-seed endpoints were PPO47\.6/15\.047\.6/15\.0andCompPO58\.4/44\.258\.4/44\.2\.

The grid also bounds the conclusion\. At the conservative×10−72\\\!\\times\\\!10^\{\-7\}learning rate, PPO is stable in one of two seeds for each KL value and can finish above base\. At10−610^\{\-6\}with no KL, oneCompPOcell is also unstable\. The evidence therefore supports a wider tested stable region and lower average degradation, not universal dominance over a method\-specifically tuned PPO\.

### 7\.5Frozen transfer across benchmarks and backbones

BackboneBaseGRPOCompPOΔ\\DeltaQwen3\-4B, greedy58\.257\.561\.7\+4\.3Qwen3\-4B, majority@869\.768\.771\.0\+2\.3Llama\-3\.1\-8B, greedy31\.139\.042\.8\+3\.9Llama\-3\.1\-8B, majority@836\.547\.852\.0\+4\.2Table 2:Unweighted frozen\-benchmark macro\-averages\.Δ\\DeltaisCompPOminus GRPO\. These are final\-checkpoint point estimates; complete per\-benchmark results are in[Tables10](https://arxiv.org/html/2608.21501#A2.T10)and[11](https://arxiv.org/html/2608.21501#A2.T11)\.On Qwen3\-4B,CompPOexceeds GRPO by4\.34\.3greedy and2\.32\.3majority@8 macro points\. AIME25 greedy is the exception: GRPO scores50\.050\.0whileCompPOand base score43\.343\.3; majority@8 ties at66\.766\.7\. The strongest Qwen gains occur on AIME2024 \(\+16\.7\+16\.7greedy\), GPQA \(\+11\.6\+11\.6\), LiveBench \(\+3\.0\+3\.0\), and MATH500 \(\+3\.2\+3\.2\)\. On Llama\-3\.1\-8B\-Instruct, all seven greedy differences favorCompPO, for a\+3\.9\+3\.9macro improvement\.

The Qwen GRPO frozen greedy macro is0\.70\.7below base despite a\+11\.4\+11\.4\-point final improvement on ID validation\. This is a small transfer trade\-off, not failed optimization\. The Llama control further shows that GRPO is a legitimate learning baseline: it improves frozen greedy macro from31\.131\.1to39\.039\.0\.

### 7\.6Stronger prompt–group GRPO control

Increasing GRPO from×44\\\!\\times\\\!4to×88\\\!\\times\\\!8improves ID best/final from56\.4/53\.856\.4/53\.8to58\.0/56\.258\.0/56\.2and frozen greedy macro from57\.557\.5to60\.260\.2\. The corresponding submitted\-seedCompPOreference is61\.8/61\.461\.8/61\.4ID and61\.761\.7frozen macro\. Thus the stronger group closes the frozen gap to1\.51\.5points but leaves descriptive gaps of\+3\.8\+3\.8best and\+5\.2\+5\.2final ID points\. Because this control has one training run, we present it as baseline\-sensitivity evidence rather than a significance\-tested comparison\.

#### Summary\.

The evidence chain is consistent across levels: matched tuning establishes performance; the factorial isolates coupling; shuffled and position\-only controls isolate trajectory\-specific alignment; target metrics support the critic interpretation; the stress grid bounds robustness; and frozen/transfer controls show that the effect is not confined to a single ID curve\. No individual experiment establishes universal generality, but together they support computation\-conditioned credit as a substantive algorithmic variable rather than an attention\-themed engineering add\-on\.

## 8Related Work

#### Fixed and transition\-dependent credit in RL\.

TD learning, eligibility traces, GAE, and PPO provide the classical basis for the estimator used here\([Schulman et al\. 2015](https://arxiv.org/html/2608.21501#bib.bib18);[Schulman et al\. 2017](https://arxiv.org/html/2608.21501#bib.bib19)\)\. Generalized RL task formulations allow transition\-dependent discounting\([White 2017](https://arxiv.org/html/2608.21501#bib.bib29)\), and option\-level work has studied per\-decision discount factors as temporal\-abstraction and bias–variance controls\([Harutyunyan et al\. 2019](https://arxiv.org/html/2608.21501#bib.bib5)\)\. AdaGamma, developed concurrently, learns state\-dependent discounts with a return\-consistency objective and analyzes the resulting operator\([Wang et al\. 2026](https://arxiv.org/html/2608.21501#bib.bib28)\)\.CompPOdiffers in interpretation and source: the task reward is unchanged, the gate is a detached statistic of the LLM policy’s own computation, and the contribution is its joint use in bootstrap, trace, and critic representation\.

#### LLM policy optimization\.

PPO\-based RLHF trains actor and value models with clipped policy updates and GAE\([Ziegler et al\. 2020](https://arxiv.org/html/2608.21501#bib.bib36);[Stiennon et al\. 2020](https://arxiv.org/html/2608.21501#bib.bib23);[Ouyang et al\. 2022](https://arxiv.org/html/2608.21501#bib.bib16)\)\. RLOO and GRPO remove the learned value model through leave\-one\-out or group\-relative baselines\([Ahmadian et al\. 2024](https://arxiv.org/html/2608.21501#bib.bib1);[Shao et al\. 2024](https://arxiv.org/html/2608.21501#bib.bib22)\)\. DAPO, Dr\. GRPO, and GSPO improve large\-scale training through asymmetric clipping, dynamic sampling, bias correction, loss normalization, or sequence\-level importance ratios\([Yu et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib32);[Liu et al\. 2025b](https://arxiv.org/html/2608.21501#bib.bib14);[Zheng et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib35)\)\. These methods primarily change baseline statistics, sampling, or update geometry\. Computation\-conditioned credit changes the Bellman/advantage transport path and can in principle be combined with those advances\.

#### Fine\-grained LLM credit\.

Process supervision supplies step\-level evidence but requires labels or learned reward models\([Uesato et al\. 2022](https://arxiv.org/html/2608.21501#bib.bib25);[Lightman et al\. 2023](https://arxiv.org/html/2608.21501#bib.bib12)\); PRIME learns implicit process rewards online from outcome labels\([Cui et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib3)\)\. VinePPO estimates intermediate state values through additional continuations and documents severe weaknesses of conventional value networks on reasoning trajectories\([Kazemnejad et al\. 2024](https://arxiv.org/html/2608.21501#bib.bib7)\)\. VC\-PPO attributes long\-CoT PPO collapse to value initialization and reward\-signal decay and uses value pretraining plus decoupled GAE\([Yuan et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib33)\)\. TEMPO constructs prefix trees and provides critic\-free TD corrections at branching points\([Tran et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib24)\); GRPO\-λ\\lambdaintroduces a critic\-free eligibility\-trace approximation\([Parthasarathi et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib17)\)\. OAR reshapes group advantages using token perturbation or gradient\-based outcome influence\([Li et al\. 2026c](https://arxiv.org/html/2608.21501#bib.bib11)\)\. High\-entropy\-token methods update only likely reasoning forks\([Wang et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib27)\)\. OPPO derives a Bayesian token\-level value recursion using an oracle\-conditioned likelihood ratio\([Li et al\. 2026b](https://arxiv.org/html/2608.21501#bib.bib10)\)\. S\-trace introduces critic\-free selective eligibility traces\([Mou et al\. 2026](https://arxiv.org/html/2608.21501#bib.bib15)\)\. These works reinforce the importance of nonuniform token credit, but use external rollouts, branch statistics, outcome influence, entropy, oracle scores, or critic\-free trace surrogates rather than one policy\-internal statistic coupled across bootstrap, trace, and critic\. A recent survey organizes this rapidly expanding landscape across reasoning and agentic settings\([Zhang 2026](https://arxiv.org/html/2608.21501#bib.bib34)\)\.

#### Critics for LLM reasoning\.

The difficulty and cost of LLM value models motivated the shift toward critic\-free methods\. Recent work reopens this design space from complementary directions\. AsyPPO uses lightweight prompt\-sharded mini\-critics\([Liu et al\. 2025a](https://arxiv.org/html/2608.21501#bib.bib13)\); Generative Actor–Critic replaces one\-shot scalar prediction with a reasoning critic and in\-context conditioning\([Shan et al\. 2026](https://arxiv.org/html/2608.21501#bib.bib21)\); VC\-PPO improves initialization and target construction\([Yuan et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib33)\)\.CompPOinstead asks whether the critic’s*state geometry*matches the policy\-internal transport rule\. Its factorial result supports the narrower claim that the tested standard head does not track the computation\-conditioned target, while an aligned actor\-feature critic does\.

#### Attention and internal signals as optimization primitives\.

Attention is a central Transformer mechanism\([Vaswani et al\. 2017](https://arxiv.org/html/2608.21501#bib.bib26)\), but its faithfulness as a causal explanation is debated\([Jain and Wallace 2019](https://arxiv.org/html/2608.21501#bib.bib6);[Wiegreffe and Pinter 2019](https://arxiv.org/html/2608.21501#bib.bib30);[Serrano and Smith 2019](https://arxiv.org/html/2608.21501#bib.bib20)\)\. We use it as an observable routing signal, not as proof of causal attribution\. HICRA identifies attention\-based planning and anchor tokens and targets their updates\([Li et al\. 2025](https://arxiv.org/html/2608.21501#bib.bib9)\); Reinforced Attention Learning directly optimizes internal attention policies for multimodal grounding\([Li et al\. 2026a](https://arxiv.org/html/2608.21501#bib.bib8)\)\. FlowTracer, posted after the initial submission, builds an answer\-targeted attention\-flow DAG and uses throughput for reward shaping\([Dong et al\. 2026](https://arxiv.org/html/2608.21501#bib.bib4)\)\. These methods are close in spirit because they treat internal computation as useful for RL\. The distinctive object inCompPOis the*transition\-level credit\-retention gate*: the same native statistic parameterizes the one\-step value backup, eligibility trace, and critic’s local/global state representation while leaving reward and outer policy loss unchanged\.

FamilyEvidence or baselineTransport geometryUpdate or value mechanismRelation to this workPPO/GAElearned scalar valuestationary temporal tracetoken\-ratio clipping; criticfixed\-kernel ancestorGRPO/RLOOgroup or leave\-one\-out outcomeresponse\-level broadcastcritic\-free token updateoutcome\-coordinate controlDAPO /Dr\. GRPO /GSPOimproved sampling and normalizationmainly response/group creditclipping, loss normalization, or sequence ratiosorthogonal optimizer advancesPRIME/PRMprocess or implicit step evidencereward placement plus downstream estimatorprocess/reward modelcomplementary evidence axisVinePPO /TEMPOcontinuation or prefix\-tree valuesrollout\- or tree\-based TDMonte Carlo or nonparametric valuealternative fine\-grained transportOAR /S\-trace /GRPO\-λ\\lambdainfluence, entropy, or trace proxyreshaping or selective tracescritic\-free updateneighboring token\-credit methodsCompPOnative policy computationpath\-dependent Bellman/GAE kernelactor\-featureTAC; clipped PPOcoupled architecture\-aware transportTable 3:Positioning by the evidence–transport–update factorization\. Categories identify the primary intervention, not an exclusive taxonomy; many systems combine multiple axes\.
#### Novelty boundary\.

We therefore do not claim the first use of attention or the first token\-level credit method\. The contribution is the coupled construction and the broader abstraction it instantiates: policy\-internal computation can be an explicit coordinate of Bellman\-style credit transport\. Concurrent work makes this boundary more, not less, important by showing that internal\-signal credit is becoming a distinct research axis\.

## 9Discussion: Toward Architecture\-Aware RL

### 9\.1The scientific claim is larger than an attention heuristic

The most conservative reading ofCompPOis a new credit estimator that improves long\-chain mathematical RL on two backbones\. The more consequential reading is that it exposes a design variable that conventional LLM RL leaves implicit\. A Transformer is not merely a large black\-box policy with token actions; it computes a trajectory\-specific internal routing structure at every decision\. Once that structure is made available to the estimator, fixed temporal distance and response\-level outcome are no longer the only viable coordinates for transporting credit\.

This perspective changes the algorithm\-design question from

> “Which black\-box policy optimizer should be applied to an LLM?”

to

> “Which internal computation of this policy should parameterize its value transport, baseline, trust region, and update?”

The latter is an architecture\-aware RL question\. Attention concentration is one answer, chosen because it is native, bounded after normalization, inexpensive to extract, and falsifiable through shuffling\. It is unlikely to be the final answer\.

### 9\.2Reinterpreting the apparent failure of actor–critic LLM RL

The rise of critic\-free methods has often been read as evidence that value learning is a poor fit for LLM reasoning\. The evidence in this paper suggests a more specific diagnosis\. The standard critic in the factorial experiment fails to track the computation\-conditioned target: EV is negative, TD\-target correlation is low, and policy gains stall\. The aligned critic tracks the target and enables the full algorithm\. This does not prove that one architecture is universally necessary, but it supports a field\-level hypothesis:

> Conventional LLM critics may fail partly because their representation and training target are mismatched to the policy’s history routing, not because token\-level actor–critic learning is intrinsically unsuitable\.

This distinction matters\. If the critic\-free turn is driven by a contingent representation failure, then improved value models, generative critics, retrieval\-aware critics, and computation\-aligned heads may recover the sample\-efficiency benefits of TD learning without a second same\-scale model\.

### 9\.3An orthogonal primitive rather than another monolithic optimizer

LLM RL algorithms often combine several axes in one recipe\. It is useful to separate them:

- •reward evidence: outcome verifiers, process rewards, preference models, or judges;
- •sampling and baseline: groups, leave\-one\-out statistics, dynamic sampling, or trees;
- •credit transport: fixed GAE, counterfactual values, tree TD, oracle recursion, or computation\-conditioned traces;
- •update geometry: token/sequence ratios, clipping, KL, entropy, and normalization;
- •systems: rollout scheduling, kernels, memory, and parallelism\.

CompPOintervenes primarily in the third axis and co\-designs the critic representation needed by that intervention\. This makes it potentially composable with DAPO\-style clipping, GSPO\-style sequence ratios, process rewards, prefix\-tree values, or larger group baselines\. The present experiments deliberately keep the outer objective unchanged to isolate the new axis\.

### 9\.4From interpretability signals to optimization primitives

Most uses of attention ask whether it explains a prediction\. Computation\-conditioned credit asks a different question: whether an internal signal can improve the estimator used to train the policy\. The burden of proof is therefore empirical alignment, not human interpretability\. A signal may be useful for optimization even when it is not a faithful natural\-language explanation, provided negative controls show that its trajectory\-specific structure matters and its failure modes are bounded\.

This shift extends beyond attention\. Future policies expose increasingly explicit internal mechanisms:

- •MoE models choose experts and routing weights;
- •retrieval\-augmented models select documents and spans;
- •agents read memory, tool observations, and environment states;
- •multimodal models route between text, image, audio, and video tokens;
- •state\-space and recurrent architectures expose gates and memory updates\.

Each mechanism can define a candidate computational coordinate\. A mature architecture\-aware RL stack may use different coordinates for value transport, exploration, KL control, and critic state construction\.

### 9\.5From a scalar gate to a computational Bellman graph

The scalarκt\\kappa\_\{t\}is intentionally minimal: it compresses the geometry of the upstream computation into one transition coefficient\. A richer extension would transport credit directly over a graph\. LetGtG\_\{t\}contain historical tokens or latent states as nodes and normalized policy\-computation weights as edges\. A graph Bellman operator could propagate value along multiple nonlocal routes, while a graph critic could estimate node\- or subgraph\-level value\. Multi\-head traces could maintain separate horizons for specialist heads; hierarchical traces could operate at token, reasoning\-step, and tool\-call scales\. The present result should be interpreted as feasibility evidence for this larger program, not as proof that scalar concentration is optimal\.

### 9\.6Where the current evidence stops

The experiments establish the mechanism on mathematical reasoning, two Transformer backbones up to 8B, terminal verifiers, and 16K responses\. They do not establish gains in dialogue, tool use, code, retrieval, multimodal trajectories, dense\-reward tasks, or larger models\. Those domains are motivated because their dependencies are sparse and nonlocal, but they remain untested\.

Attention concentration is a structural proxy\. Attention sinks, positional bias, induction heads, and output\-layer calibration can corrupt it\. The learned relevance gate and negative controls reduce but do not eliminate these risks\. The gate is also stale across actor epochs, and the current theory does not provide a global policy\-improvement bound for policy\-generated gates\. Finally, the stronger×88\\\!\\times\\\!8GRPO control and frozen benchmark values are point estimates, while exact measured hardware and wall\-clock records are unavailable\. These limitations motivate targeted follow\-up rather than weakening the central observation that policy\-internal computation can carry useful credit information\.

## 10Conclusion

LLM policies compute with history, but their credit estimators usually do not\. This paper identifies that architecture–credit mismatch and introduces computation\-conditioned credit: a detached statistic of the policy’s own internal computation controls how downstream value crosses a transition\.CompPOinstantiates the idea with attention concentration, a path\-dependent GAE trace, and a computation\-aligned actor\-feature critic, while preserving the external reward and clipped PPO update\.

The empirical evidence supports the mechanism at several levels\. Under matched tuning and five seeds,CompPOsubstantially exceeds GRPO; a controlled gate\-by\-critic factorial reveals a positive interaction and better critic target tracking; shuffle and position\-only controls show that trajectory\-specific alignment matters; a PPO stress grid shows a wider stable region; and frozen evaluation transfers across Qwen3\-4B and Llama\-3\.1\-8B\-Instruct\. The contribution is therefore not that attention is universally causal or that one optimizer replaces all others\. It is that internal computation can become a first\-class variable in RL credit transport\. This opens a broader agenda in which attention, routing, retrieval, memory, and modality structure shape Bellman operators, eligibility traces, critics, and trust regions for the policies that actually use them\.

## References

- Ahmadian et al\. \(2024\)Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker\.Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms\.*arXiv preprint arXiv:2402\.14740*, 2024\.
- AI at Meta \(2024\)AI at Meta\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Cui et al\. \(2025\)Ganqu Cui, Lifan Yuan, Zefan Wang, et al\.Process reinforcement through implicit rewards\.*arXiv preprint arXiv:2502\.01456*, 2025\.
- Dong et al\. \(2026\)Zhichen Dong, Yang Li, Yuhan Sun, et al\.How does reasoning flow? tracing attention\-induced information flow for targeted rl in llms\.*arXiv preprint arXiv:2606\.10646*, 2026\.
- Harutyunyan et al\. \(2019\)Anna Harutyunyan, Peter Vrancx, Philippe Hamel, Ann Nowe, and Doina Precup\.Per\-decision option discounting\.In*Proceedings of the 36th International Conference on Machine Learning*, volume 97 of*Proceedings of Machine Learning Research*, pages 2644–2652, 2019\.
- Jain and Wallace \(2019\)Sarthak Jain and Byron C\. Wallace\.Attention is not explanation\.In*Proceedings of NAACL\-HLT*, 2019\.
- Kazemnejad et al\. \(2024\)Amirhossein Kazemnejad et al\.Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment\.*arXiv preprint arXiv:2410\.01679*, 2024\.
- Li et al\. \(2026a\)Bangzheng Li, Jianmo Ni, Chen Qu, Ian Miao, Liu Yang, Xingyu Fu, Muhao Chen, and Derek Zhiyuan Cheng\.Reinforced attention learning\.*arXiv preprint arXiv:2602\.04884*, 2026a\.
- Li et al\. \(2025\)Yang Li, Zhichen Dong, Yuhan Sun, et al\.Attention illuminates llm reasoning: The preplan\-and\-anchor rhythm enables fine\-grained policy optimization\.*arXiv preprint arXiv:2510\.13554*, 2025\.
- Li et al\. \(2026b\)Yu Li et al\.Oracle\-guided proximal policy optimization for fine\-grained credit assignment\.*arXiv preprint arXiv:2605\.21851*, 2026b\.
- Li et al\. \(2026c\)Ziheng Li, Liu Kang, Feng Xiao, Luxi Xing, Qingyi Si, Zhuoran Li, Weikang Gong, Deqing Yang, Yanghua Xiao, and Hongcheng Guo\.Outcome\-grounded advantage reshaping for fine\-grained credit assignment in mathematical reasoning\.*arXiv preprint arXiv:2601\.07408*, 2026c\.
- Lightman et al\. \(2023\)Hunter Lightman, Vineet Kosaraju, Yura Burda, et al\.Let’s verify step by step\.*arXiv preprint arXiv:2305\.20050*, 2023\.
- Liu et al\. \(2025a\)Jiashun Liu, Johan Obando\-Ceron, Han Lu, Yancheng He, Weixun Wang, Wenbo Su, Bo Zheng, Pablo Samuel Castro, Aaron Courville, and Ling Pan\.Asymmetric proximal policy optimization: Mini\-critics boost llm reasoning\.*arXiv preprint arXiv:2510\.01656*, 2025a\.
- Liu et al\. \(2025b\)Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin\.Understanding r1\-zero\-like training: A critical perspective\.*arXiv preprint arXiv:2503\.20783*, 2025b\.
- Mou et al\. \(2026\)Chaoli Mou, Zhan Zhuang, Xinning Chen, and Yu Zhang\.Beyond uniform credit assignment: Selective eligibility traces for rlvr\.*arXiv preprint arXiv:2605\.05965*, 2026\.
- Ouyang et al\. \(2022\)Long Ouyang, Jeffrey Wu, Xu Jiang, et al\.Training language models to follow instructions with human feedback\.*Advances in Neural Information Processing Systems*, 2022\.
- Parthasarathi et al\. \(2025\)Prasanna Parthasarathi, Mathieu Reymond, Boxing Chen, Yufei Cui, and Sarath Chandar\.Grpo\-λ\\lambda: Credit assignment improves llm reasoning\.*arXiv preprint arXiv:2510\.00194*, 2025\.
- Schulman et al\. \(2015\)John Schulman, Philipp Moritz, Sergey Levine, Michael I\. Jordan, and Pieter Abbeel\.High\-dimensional continuous control using generalized advantage estimation\.*arXiv preprint arXiv:1506\.02438*, 2015\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*arXiv preprint arXiv:1707\.06347*, 2017\.
- Serrano and Smith \(2019\)Sofia Serrano and Noah A\. Smith\.Is attention interpretable?In*Proceedings of ACL*, 2019\.
- Shan et al\. \(2026\)Zikang Shan, Han Zhong, Liwei Wang, and Li Zhao\.Bringing value models back: Generative critics for value modeling in llm reinforcement learning\.*arXiv preprint arXiv:2604\.10701*, 2026\.
- Shao et al\. \(2024\)Zhihong Shao, Peiyi Wang, Qihao Zhu, et al\.Deepseekmath: Pushing the limits of mathematical reasoning in open language models\.*arXiv preprint arXiv:2402\.03300*, 2024\.
- Stiennon et al\. \(2020\)Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F\. Christiano\.Learning to summarize with human feedback\.In*Advances in Neural Information Processing Systems*, 2020\.
- Tran et al\. \(2025\)Hieu Tran, Zonghai Yao, and Hong Yu\.Exploiting tree structure for credit assignment in rl training of llms\.*arXiv preprint arXiv:2509\.18314*, 2025\.
- Uesato et al\. \(2022\)Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins\.Solving math word problems with process\- and outcome\-based feedback\.*arXiv preprint arXiv:2211\.14275*, 2022\.
- Vaswani et al\. \(2017\)Ashish Vaswani, Noam Shazeer, Niki Parmar, et al\.Attention is all you need\.In*Advances in Neural Information Processing Systems*, 2017\.
- Wang et al\. \(2025\)Shenzhi Wang, Le Yu, Chang Gao, et al\.Beyond the 80/20 rule: High\-entropy minority tokens drive effective reinforcement learning for llm reasoning\.*arXiv preprint arXiv:2506\.01939*, 2025\.
- Wang et al\. \(2026\)Yaomin Wang, Jianting Pan, Ran Tian, Xiaoyang Li, Yu Zhang, Hengle Qin, and Tianshu Yu\.Adagamma: State\-dependent discounting for temporal adaptation in reinforcement learning\.*arXiv preprint arXiv:2605\.06149*, 2026\.
- White \(2017\)Martha White\.Unifying task specification in reinforcement learning\.*arXiv preprint arXiv:1609\.01995*, 2017\.
- Wiegreffe and Pinter \(2019\)Sarah Wiegreffe and Yuval Pinter\.Attention is not not explanation\.In*Proceedings of EMNLP\-IJCNLP*, 2019\.
- Yang et al\. \(2025\)An Yang et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- Yu et al\. \(2025\)Qiying Yu, Zheng Zhang, Ruofei Zhu, et al\.Dapo: An open\-source llm reinforcement learning system at scale\.*arXiv preprint arXiv:2503\.14476*, 2025\.
- Yuan et al\. \(2025\)Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan\.What’s behind ppo’s collapse in long\-cot? value optimization holds the secret\.*arXiv preprint arXiv:2503\.01491*, 2025\.
- Zhang \(2026\)Chenchen Zhang\.From reasoning to agentic: Credit assignment in reinforcement learning for large language models\.*arXiv preprint arXiv:2604\.09459*, 2026\.
- Zheng et al\. \(2025\)Chujie Zheng, Shixuan Liu, Mingze Li, Xiong\-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin\.Group sequence policy optimization\.*arXiv preprint arXiv:2507\.18071*, 2025\.
- Ziegler et al\. \(2020\)Daniel M\. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B\. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving\.Fine\-tuning language models from human preferences\.*arXiv preprint arXiv:1909\.08593*, 2020\.

## Appendix AFull Algorithm

Algorithm 1Computation\-Conditioned Policy Optimization \(CompPO\)1:actor

πθ\\pi\_\{\\theta\}, reference policy

πref\\pi\_\{\\rm ref\}, aligned critic

VψV\_\{\\psi\}, verifier

RR, gate parameters

\(κlo,κhi,τ\)\(\\kappa\_\{\\mathrm\{lo\}\},\\kappa\_\{\\mathrm\{hi\}\},\\tau\), GAE

λ\\lambda, clip

ϵ\\epsilon
2:forrollout iteration

k=1,2,…k=1,2,\\ldotsdo

3:Set behavior parameters

θ−←θ\\theta^\{\-\}\\leftarrow\\theta
4:Sample prompts and responses

y1:T∼πθ−y\_\{1:T\}\\sim\\pi\_\{\\theta^\{\-\}\}; compute rewards

r1:Tr\_\{1:T\}and valid\-token masks

m1:Tm\_\{1:T\}
5:foreach valid response token

ttdo

6:Extract final\-layer GQA\-group attention

at​ia\_\{ti\}over valid history

ℐt\\mathcal\{I\}\_\{t\}and actor hidden states

\{ht\(ℓ\)\}\\\{h\_\{t\}^\{\(\\ell\)\}\\\}
7:Renormalize

at​ia\_\{ti\}over

i∈ℐti\\in\\mathcal\{I\}\_\{t\}; set

nt=\|ℐt\|n\_\{t\}=\|\\mathcal\{I\}\_\{t\}\|
8:

Ht←∑i∈ℐtat​i2H\_\{t\}\\leftarrow\\sum\_\{i\\in\\mathcal\{I\}\_\{t\}\}a\_\{ti\}^\{2\}
9:

ct←log⁡\(nt​Ht\)/log⁡ntc\_\{t\}\\leftarrow\\log\(n\_\{t\}H\_\{t\}\)/\\log n\_\{t\}if

nt\>1n\_\{t\}\>1, else

1/21/2
10:

κt←κlo\+\(κhi−κlo\)​σ​\(τ⁡\(ct−1/2\)\)\\kappa\_\{t\}\\leftarrow\\kappa\_\{\\mathrm\{lo\}\}\+\(\\kappa\_\{\\mathrm\{hi\}\}\-\\kappa\_\{\\mathrm\{lo\}\}\)\\sigma\(\\tau\(c\_\{t\}\-1/2\)\)
11:Detach and store

κt\\kappa\_\{t\}, attention summaries, and required actor features

12:Fuse actor features

h¯t←∑ℓωℓ​ht\(ℓ\)\\bar\{h\}\_\{t\}\\leftarrow\\sum\_\{\\ell\}\\omega\_\{\\ell\}h\_\{t\}^\{\(\\ell\)\}
13:Compute value\-refined scores

st​is\_\{ti\}by[Equations23](https://arxiv.org/html/2608.21501#S4.E23)and[24](https://arxiv.org/html/2608.21501#S4.E24)

14:Pool routed history

htGh\_\{t\}^\{G\}by[Equation26](https://arxiv.org/html/2608.21501#S4.E26)

15:Evaluate

Vt←clip⁡\(κt​VG​\(htG\)\+\(1−κt\)​VL​\(h¯t\),−1,1\)V\_\{t\}\\leftarrow\\clip\(\\kappa\_\{t\}V^\{G\}\(h\_\{t\}^\{G\}\)\+\(1\-\\kappa\_\{t\}\)V^\{L\}\(\\bar\{h\}\_\{t\}\),\-1,1\)
16:Set

AT\+1=0A\_\{T\+1\}=0and

VT\+1=0V\_\{T\+1\}=0
17:for

t=T,T−1,…,1t=T,T\-1,\\ldots,1do

18:

δt←rt\+κt​Vt\+1​mt\+1−Vt​mt\\delta\_\{t\}\\leftarrow r\_\{t\}\+\\kappa\_\{t\}V\_\{t\+1\}m\_\{t\+1\}\-V\_\{t\}m\_\{t\}
19:

At←δt\+λ​κt​At\+1​mt\+1A\_\{t\}\\leftarrow\\delta\_\{t\}\+\\lambda\\kappa\_\{t\}A\_\{t\+1\}m\_\{t\+1\}
20:

Gt←At\+Vt​mtG\_\{t\}\\leftarrow A\_\{t\}\+V\_\{t\}m\_\{t\}
21:Normalize valid\-token advantages

A^t\\widehat\{A\}\_\{t\}
22:foreach optimization epochdo

23:Update actor with clipped objective in[Equation17](https://arxiv.org/html/2608.21501#S3.E17)plus reference\-policy KL

24:Update critic by clipped regression to detached

GtG\_\{t\}
25:Apply the predeclared stability phase controller if enabled

#### Gradient convention\.

The behavior\-policy attention andκt\\kappa\_\{t\}are detached for the whole rollout batch\. The critic consumes actor features from the same forward computation and cannot alter the detached, stored transport gate for the current batch\. The exact backbone\-gradient routing must match the released implementation; all controlled variants use the same convention\. The outer actor update consumes detached normalized advantages, as in ordinary PPO\.

#### Fixed\-gate controls\.

For a fixed\-gate ablation, every occurrence ofκt\\kappa\_\{t\}in the TD residual, trace, and value mixture is replaced by the same constant\. The main factorial usesκ=0\.61\\kappa=0\.61, the empirical mean of the full method, to avoid confounding trajectory dependence with mean scale\.

## Appendix BComplete Experimental Tables

### B\.1Common tuning screen

MethodActor LRKLBestFinalGRPO5​e−75\\mathrm\{e\}\{\-7\}052\.851\.2GRPO5​e−75\\mathrm\{e\}\{\-7\}1​e−31\\mathrm\{e\}\{\-3\}54\.052\.4GRPO5​e−75\\mathrm\{e\}\{\-7\}2​e−32\\mathrm\{e\}\{\-3\}53\.251\.8GRPO1​e−61\\mathrm\{e\}\{\-6\}054\.652\.8GRPO1​e−61\\mathrm\{e\}\{\-6\}1​e−31\\mathrm\{e\}\{\-3\}56\.453\.8GRPO1​e−61\\mathrm\{e\}\{\-6\}2​e−32\\mathrm\{e\}\{\-3\}55\.053\.2GRPO2​e−62\\mathrm\{e\}\{\-6\}053\.851\.6GRPO2​e−62\\mathrm\{e\}\{\-6\}1​e−31\\mathrm\{e\}\{\-3\}54\.252\.0GRPO2​e−62\\mathrm\{e\}\{\-6\}2​e−32\\mathrm\{e\}\{\-3\}53\.051\.4CompPO5​e−75\\mathrm\{e\}\{\-7\}059\.858\.4CompPO5​e−75\\mathrm\{e\}\{\-7\}1​e−31\\mathrm\{e\}\{\-3\}60\.459\.2CompPO5​e−75\\mathrm\{e\}\{\-7\}2​e−32\\mathrm\{e\}\{\-3\}60\.859\.6CompPO1​e−61\\mathrm\{e\}\{\-6\}060\.659\.0CompPO1​e−61\\mathrm\{e\}\{\-6\}1​e−31\\mathrm\{e\}\{\-3\}61\.260\.2CompPO1​e−61\\mathrm\{e\}\{\-6\}2​e−32\\mathrm\{e\}\{\-3\}61\.861\.4CompPO2​e−62\\mathrm\{e\}\{\-6\}059\.658\.0CompPO2​e−62\\mathrm\{e\}\{\-6\}1​e−31\\mathrm\{e\}\{\-3\}60\.859\.4CompPO2​e−62\\mathrm\{e\}\{\-6\}2​e−32\\mathrm\{e\}\{\-3\}61\.060\.0Table 4:Single\-seed common×33\\\!\\times\\\!3screen used to select the five\-seed configurations\. Frozen benchmarks were not used\.
### B\.2Five\-seed confirmation

SeedGRPO BestGRPO FinalCompPOBestCompPOFinal1755\.652\.961\.260\.84256\.453\.861\.861\.412355\.853\.461\.561\.225656\.654\.261\.961\.6202656\.854\.762\.162\.0Mean56\.253\.861\.761\.495% CI\[55\.6,56\.9\]\[52\.9,54\.7\]\[61\.3,62\.1\]\[60\.8,62\.0\]Table 5:Five\-seed selected\-configuration confirmation\.
### B\.3Per\-seed factorial results

Fixed\+StdComp\-GAE\+StdFixed\+AlignedFullSeedBestFinalBestFinalBestFinalBestFinal1752\.851\.955\.654\.656\.655\.761\.260\.84253\.853\.156\.655\.857\.856\.961\.861\.412353\.252\.455\.854\.957\.056\.161\.561\.225653\.652\.856\.255\.457\.456\.661\.961\.6202653\.652\.856\.055\.357\.256\.762\.162\.0Table 6:Seed\-level gate\-by\-critic factorial endpoints\.
### B\.4Gate controls and descriptive statistics

VariantBestFinalLast\-20AULCGlobal shuffle59\.4 \[58\.9,59\.9\]58\.7 \[58\.0,59\.4\]58\.157\.2Position\-only60\.2 \[59\.8,60\.6\]59\.7 \[59\.2,60\.2\]––FullCompPO61\.7 \[61\.3,62\.1\]61\.4 \[60\.8,62\.0\]60\.858\.9Table 7:Five\-seed gate controls\. Brackets are 95% Student\-ttintervals\.StatisticValueStatisticValueMean / median0\.61 / 0\.63IQR0\.18Early / middle / late0\.52 / 0\.63 / 0\.68Correct / incorrect0\.64 / 0\.57Table 8:Realized gate statistics from five final full\-method checkpoints\. Archived tail and boundary\-proximity summaries are excluded pending reconciliation of the exact implementation map with the printed smooth\-map equation\.
### B\.5Matched PPO stress grid

MethodActor LRKLBestFinalStable seedsPPO2​e−72\\mathrm\{e\}\{\-7\}048\.244\.81/2PPO2​e−72\\mathrm\{e\}\{\-7\}1​e−31\\mathrm\{e\}\{\-3\}49\.046\.21/2PPO5​e−75\\mathrm\{e\}\{\-7\}046\.831\.00/2PPO5​e−75\\mathrm\{e\}\{\-7\}1​e−31\\mathrm\{e\}\{\-3\}48\.443\.61/2PPO1​e−61\\mathrm\{e\}\{\-6\}044\.218\.40/2PPO1​e−61\\mathrm\{e\}\{\-6\}1​e−31\\mathrm\{e\}\{\-3\}49\.822\.60/2CompPO2​e−72\\mathrm\{e\}\{\-7\}052\.448\.62/2CompPO2​e−72\\mathrm\{e\}\{\-7\}1​e−31\\mathrm\{e\}\{\-3\}53\.049\.22/2CompPO5​e−75\\mathrm\{e\}\{\-7\}051\.647\.02/2CompPO5​e−75\\mathrm\{e\}\{\-7\}1​e−31\\mathrm\{e\}\{\-3\}52\.247\.82/2CompPO1​e−61\\mathrm\{e\}\{\-6\}050\.836\.40/2CompPO1​e−61\\mathrm\{e\}\{\-6\}1​e−31\\mathrm\{e\}\{\-3\}57\.244\.62/2Table 9:Two seeds per LR–KL cell\. Across 12 runs, PPO is stable in 3 andCompPOin 10\.
### B\.6Frozen Qwen3\-4B evaluation

Greedy pass@1Majority@8BenchmarkNNBaseGRPOCompPOBaseGRPOCompPOAIME253043\.350\.043\.366\.766\.766\.7AIME20243063\.353\.370\.076\.773\.380\.0MATH50050067\.266\.669\.879\.879\.281\.0GSM8K131991\.793\.194\.193\.594\.394\.5LiveBench20046\.550\.553\.561\.061\.567\.0GPQA19847\.040\.452\.057\.654\.555\.6Olympiad67448\.548\.249\.453\.051\.652\.1Macro–58\.257\.561\.769\.768\.771\.0Table 10:Frozen Qwen3\-4B final\-checkpoint point estimates\.
### B\.7Frozen Llama\-3\.1\-8B\-Instruct evaluation

Greedy pass@1Majority@8BenchmarkBaseGRPOCompPOBaseGRPOCompPOAIME253\.313\.316\.76\.723\.330\.0AIME202413\.320\.023\.316\.733\.336\.7MATH50053\.464\.269\.664\.476\.680\.6GSM8K81\.386\.487\.685\.690\.491\.2LiveBench22\.535\.039\.029\.044\.549\.0GPQA26\.832\.840\.432\.339\.447\.0Olympiad17\.121\.123\.321\.126\.729\.8Macro31\.139\.042\.836\.547\.852\.0Table 11:Frozen Llama\-3\.1\-8B\-Instruct final\-checkpoint point estimates\.
### B\.8Stronger GRPO group control

MethodTrajectories/updateID BestID FinalFrozen greedy macroGRPO×44\\\!\\times\\\!41656\.453\.857\.5GRPO×88\\\!\\times\\\!86458\.056\.260\.2CompPOreference1661\.861\.461\.7Table 12:Single\-run stronger\-group control\. TheCompPOrow is the submitted\-seed reference\.

## Appendix CReproducibility Details

### C\.1Primary configuration

ItemGRPOCompPOBackboneQwen3\-4BsameTransfer backboneLlama\-3\.1\-8B\-InstructsameMaximum response16K16KBatch / responses / minibatch4 / 4 / 4sameActor learning rate10−610^\{\-6\}10−610^\{\-6\}Critic learning rate–10−510^\{\-5\}KL coefficient10−310^\{\-3\}2×10−32\\times 10^\{\-3\}Policy clip0\.200\.20Credit estimatorgroup\-relativeκt\\kappa\_\{t\}backup;λ=0\.95\\lambda=0\.95Gate map–configured envelope\(0\.1,0\.9\)\(0\.1,0\.9\),τ=4\\tau=4Value clamp–\[−1,1\]\[\-1,1\]OptimizerAdamW\(0\.9,0\.999\)\(0\.9,0\.999\)sameWeight decay / grad clip0 / 1sameUpdate epochs22Sampling temperature / top\-pp1\.0 / 0\.7samePrecisionBF16BF16Table 13:Selected primary settings\. Both methods were screened on the same actor\-LR/KL grid\.
### C\.2Phase controller

The full performance experiments and all four factorial cells use the same closed\-loop controller\. It changes only after the predeclared stability trigger persists; escalation requires two consecutive trigger events and de\-escalation requires five clear steps\.

PhaseKLClipActor LRValue\-loss clipCritic LR mult\.Stable0\.00200\.2010−610^\{\-6\}0\.51\.0Warning0\.00350\.167×10−77\\times 10^\{\-7\}0\.40\.9Strong0\.00450\.145×10−75\\times 10^\{\-7\}0\.30\.7Hard\-stop0\.00700\.092×10−72\\times 10^\{\-7\}0\.20\.5Table 14:Adaptive stability phases\. The same controller is held fixed across the factorial, so it cannot induce the gate\-by\-critic interaction\.The monitored signals are policy KL, gradient norm, clip fraction, entropy, and ID validation accuracy\. Because validation participates in phase control, the 500\-problem set is a development/held\-out\-ID set rather than a final test set\. Frozen benchmarks are never used by the controller, checkpoint selection, or hyperparameter search\.

#### Controller inputs and hysteresis\.

The controller maintains exponential moving averages with coefficient0\.40\.4for policy clip fraction, PPO KL, KL loss, and gradient norm, and coefficient0\.20\.2for entropy\. It also reads raw response\-clip ratio and ID development accuracy when evaluated\. Entropy decline is measured against the oldest four values in an eight\-step window\. The target phase is the maximum severity triggered by the following rules:

- •entropy EMA below8\.0/7\.5/6\.58\.0/7\.5/6\.5requests warning/strong/hard\-stop;
- •clip fraction at least0\.14/0\.18/0\.250\.14/0\.18/0\.25requests warning/strong/hard\-stop;
- •PPO KL at least0\.08/0\.12/0\.220\.08/0\.12/0\.22requests warning/strong/hard\-stop;
- •KL loss at least2\.85/3\.20/4\.002\.85/3\.20/4\.00requests warning/strong/hard\-stop;
- •gradient norm at least40/8040/80requests warning/strong;
- •development accuracy drop from the running best of at least0\.04/0\.08/0\.200\.04/0\.08/0\.20requests warning/strong/hard\-stop;
- •an entropy drop of at least1\.5/3\.01\.5/3\.0, when clip fraction or PPO KL is already in warning territory, requests strong/hard\-stop; and
- •response\-clip ratio at least0\.650\.65requests hard\-stop\.

Escalation requires two consecutive steps with a higher target phase; de\-escalation requires five consecutive clear steps\. Fresh runs start in the stable phase\. On fresh phase\-0 runs, steps 25–40 linearly interpolate stable to warning knobs\. If a screened launch KL differs from0\.0020\.002, later phases preserve the same multipliers1\.75×1\.75\\times,2\.25×2\.25\\times, and3\.5×3\.5\\timesrelative to that launch value\.

#### Additional phase\-controlled quantities\.

The stable/warning/strong/hard\-stop phases use advantage clips5\.0/4\.0/3\.5/2\.55\.0/4\.0/3\.5/2\.5, actor gradient clips1\.0/0\.8/0\.7/0\.51\.0/0\.8/0\.7/0\.5, actor epochs2/2/2/12/2/2/1, and critic gradient clips1\.0/0\.8/0\.7/0\.51\.0/0\.8/0\.7/0\.5\. A reward schedule, applied in parallel rather than triggered by phase, ramps the format\-penalty coefficient from0\.20\.2to1\.01\.0over steps 0–40; the length penalty is enabled at step 20 and ramps from3×10−53\\times 10^\{\-5\}to8×10−58\\times 10^\{\-5\}over steps 20–60\. The PPO stress experiment disables critic warm\-up and this controller; the fixed value\-domain clamp remains enabled for the applicable value path\.

### C\.3Value clamp control

The clamp is a fixed output\-domain constraint, not a method\-specific adaptive intervention\. In the most collapse\-prone cell \(Qwen3\-4B, actor LR10−610^\{\-6\}, KL 0\), PPO andCompPOwere each repeated three times without the clamp\. The last out\-of\-range critic prediction occurred around step 37 for PPO and step 32 forCompPOon average\. In the bounded grid, activation was therefore confined to comparable early critic transients and then became the identity\. The stress\-grid comparison applies the same bound to both methods\.

### C\.4Attention extraction and memory

The implementation uses final\-layer GQA\-group attention\. Only the final layer exposes chunked attention; earlier layers retain fused attention kernels\. Response\-window\-only buffers avoid prompt\-heavy storage, online softmax/recomputation avoids writing dense attention to HBM, and prior\-guided causal top\-KKpooling usesK=64K=64\. The resulting critic adds fewer than0\.5%0\.5\\%trainable parameters and does not require a second same\-scale Transformer or a second full optimizer state\.

Exact GPU model/count, wall\-clock time, peak allocated/reserved HBM, generated\-token count, and throughput were not retained in the experiment record available for this version\. They cannot be reconstructed reliably from public model specifications and are therefore not estimated\. Consequently, the present paper does not make a measured wall\-clock or peak\-memory superiority claim\.

### C\.5Statistical conventions

Training intervals are two\-sided 95% Student\-ttintervals over five independent runs\. The same seed set is used across factorial cells, enabling seed\-matched interaction intervals\. Best is reported for completeness, but final is primary and last\-20/AULC are secondary\. Frozen results are currently point estimates from final checkpoints; they reflect finite benchmark sampling and checkpoint variance that is not quantified by the training\-seed intervals\.

### C\.6Data and result availability

Editable CSV versions of every numerical table are included with the source archive\. The public code release should additionally include the rollout configuration, evaluator scripts, checkpoint\-selection rules, and commands needed to reproduce the training and frozen\-evaluation tables\.

## Appendix DProofs and Limiting Cases

### D\.1Strict reduction

Ifκt=γ\\kappa\_\{t\}=\\gammafor all validtt, then[Equation16](https://arxiv.org/html/2608.21501#S3.E16)becomes

\(𝒯κ​V\)​\(ht\)=𝔼⁡\[rt\+γ​V​\(ht\+1\)∣ht\]=\(𝒯γ​V\)​\(ht\)\.\(\\mathcal\{T\}\_\{\\kappa\}V\)\(h\_\{t\}\)=\\mathbb\{E\}\[r\_\{t\}\+\\gamma V\(h\_\{t\+1\}\)\\mid h\_\{t\}\]=\(\\mathcal\{T\}\_\{\\gamma\}V\)\(h\_\{t\}\)\.Substitution into[Equations10](https://arxiv.org/html/2608.21501#S3.E10)and[11](https://arxiv.org/html/2608.21501#S3.E11)yields the standard TD residual and GAE recursion in[Equation5](https://arxiv.org/html/2608.21501#S2.E5)\. The outer objective[Equation17](https://arxiv.org/html/2608.21501#S3.E17)is unchanged by construction, so the fixed\-gate estimator path recovers the standard GAE/PPO path when the critic is also restored to the standard control\.

### D\.2Contraction of the detached operator

For bounded functionsV,WV,Wand any historyhth\_\{t\},

\|𝒯κ​V​\(ht\)−𝒯κ​W​\(ht\)\|\\displaystyle\|\\mathcal\{T\}\_\{\\kappa\}V\(h\_\{t\}\)\-\\mathcal\{T\}\_\{\\kappa\}W\(h\_\{t\}\)\|=\|𝔼⁡\[κt​\(V⁡\(ht\+1\)−W⁡\(ht\+1\)\)∣ht\]\|\\displaystyle=\\left\|\\mathbb\{E\}\\left\[\\kappa\_\{t\}\(V\(h\_\{t\+1\}\)\-W\(h\_\{t\+1\}\)\)\\mid h\_\{t\}\\right\]\\right\|≤𝔼⁡\[κt​\|V⁡\(ht\+1\)−W⁡\(ht\+1\)\|∣ht\]\\displaystyle\\leq\\mathbb\{E\}\\left\[\\kappa\_\{t\}\|V\(h\_\{t\+1\}\)\-W\(h\_\{t\+1\}\)\|\\mid h\_\{t\}\\right\]≤κhi​‖V−W‖∞\.\\displaystyle\\leq\\kappa\_\{\\mathrm\{hi\}\}\\\|V\-W\\\|\_\{\\infty\}\.Taking the supremum overhth\_\{t\}proves the contraction\. Banach’s fixed\-point theorem gives a unique bounded fixed point for a fixed policy and detached gate\.

### D\.3Bounded trace

Unrolling[Equation11](https://arxiv.org/html/2608.21501#S3.E11)gives[Equation29](https://arxiv.org/html/2608.21501#S5.E29)\. If\|δtκ\|≤D\|\\delta\_\{t\}^\{\\kappa\}\|\\leq Dandκt≤κhi\\kappa\_\{t\}\\leq\\kappa\_\{\\mathrm\{hi\}\}, then

\|Atκ,λ\|\\displaystyle\|A\_\{t\}^\{\\kappa,\\lambda\}\|≤∑ℓ=0T−tλℓ​\(∏j=0ℓ−1κt\+j\)​D\\displaystyle\\leq\\sum\_\{\\ell=0\}^\{T\-t\}\\lambda^\{\\ell\}\\left\(\\prod\_\{j=0\}^\{\\ell\-1\}\\kappa\_\{t\+j\}\\right\)D≤D​∑ℓ=0∞\(λ​κhi\)ℓ=D1−λ​κhi\.\\displaystyle\\leq D\\sum\_\{\\ell=0\}^\{\\infty\}\(\\lambda\\kappa\_\{\\mathrm\{hi\}\}\)^\{\\ell\}=\\frac\{D\}\{1\-\\lambda\\kappa\_\{\\mathrm\{hi\}\}\}\.

### D\.4Length normalization of attention concentration

For a probability vector overntn\_\{t\}valid historical positions,

1nt≤Ht=∑iat​i2≤1,\\frac\{1\}\{n\_\{t\}\}\\leq H\_\{t\}=\\sum\_\{i\}a\_\{ti\}^\{2\}\\leq 1,where the lower bound follows from Cauchy–Schwarz and is achieved by the uniform vector, while the upper bound is achieved by a point mass\. Thereforent​Ht∈\[1,nt\]n\_\{t\}H\_\{t\}\\in\[1,n\_\{t\}\], so

0≤log⁡\(nt​Ht\)log⁡nt≤1\.0\\leq\\frac\{\\log\(n\_\{t\}H\_\{t\}\)\}\{\\log n\_\{t\}\}\\leq 1\.At uniform attention,nt​Ht=1n\_\{t\}H\_\{t\}=1andct=0c\_\{t\}=0; at a point mass,nt​Ht=ntn\_\{t\}H\_\{t\}=n\_\{t\}andct=1c\_\{t\}=1\. Because logarithm is monotone, the transformation preserves the ordering ofHtH\_\{t\}at eachntn\_\{t\}while aligning the extremes across positions\.

### D\.5Why the coefficient is not a post\-hoc loss weight

Multiplying a policy loss by an importance score changes the magnitude of the gradient at tokenttbut leaves the value target and propagation path unchanged\. InCompPO,κt\\kappa\_\{t\}enters both terms that define the estimator:

δtκ=rt\+κt​Vt\+1−Vt,Atκ,λ=δtκ\+λ​κt​At\+1κ,λ\.\\delta\_\{t\}^\{\\kappa\}=r\_\{t\}\+\\kappa\_\{t\}V\_\{t\+1\}\-V\_\{t\},\\qquad A\_\{t\}^\{\\kappa,\\lambda\}=\\delta\_\{t\}^\{\\kappa\}\+\\lambda\\kappa\_\{t\}A\_\{t\+1\}^\{\\kappa,\\lambda\}\.It therefore changes the one\-step bootstrap and every later residual’s retention weight in[Equation29](https://arxiv.org/html/2608.21501#S5.E29)\. Its additional use in[Equation27](https://arxiv.org/html/2608.21501#S4.E27)changes the state representation used to estimate the corresponding target\. The policy loss consumes the resulting advantage without a direct multiplicative attention weight\.

## Appendix ELimitations, Open Questions, and Broader Impact

#### Attention is a proxy, not causal attribution\.

Concentration can be distorted by attention sinks, positional biases, delimiter tokens, induction heads, or final\-layer calibration\. The method does not establish that highly attended tokens caused the outcome\. A future version should compare layers and heads, mask known sinks, test intervention\-based relevance, and use graph\-valued rather than scalar signals\.

#### Policy dependence and staleness\.

The gate is computed by the behavior policy and held fixed through multiple actor epochs\. This is operationally simple and avoids differentiating the estimator through attention, but it creates staleness as the actor moves\. Theoretical work should quantify the interaction among gate drift, PPO ratios, critic error, and clipping; empirical work should compare recomputation, fewer epochs, and explicit consistency regularization\.

#### Choice of concentration statistic\.

Herfindahl concentration is bounded, monotone, inexpensive, and length\-normalizable, but it is one point in a larger space\. Shannon/Rényi entropy, top\-kkcoverage, head\-specialist statistics, multi\-layer aggregation, learned bounded maps, and attention\-flow measures may be stronger\. The present paper establishes that one trajectory\-specific policy\-internal statistic is useful, not that it is optimal\.

#### Reward quality\.

Computation\-conditioned transport can amplify a bad reward just as easily as a good one\. Checker false negatives, formatting artifacts, or answer conventions may cause the algorithm to transport spurious evidence\. Because the gate is correlated with the policy computation, reward artifacts that themselves dominate attention could be reinforced more strongly\. Robust verifiers and audit sets remain essential\.

#### Scope\.

The current evidence is mathematical reasoning on Qwen3\-4B and Llama\-3\.1\-8B\-Instruct, with sparse terminal reward and responses up to 16K\. Dialogue, tools, code, retrieval, multimodal generation, agents, dense rewards, and larger models are not experimentally established\. The framework applies syntactically to these settings, but usefulness depends on whether the chosen internal statistic captures value\-relevant computation\.

#### Statistical and systems limitations\.

Training uncertainty is reported for the principal Qwen comparisons, but frozen evaluations and the×88\\\!\\times\\\!8GRPO control are point estimates\. Exact hardware, wall\-clock, throughput, and peak\-memory records are unavailable, so the current evidence does not support a measured systems\-efficiency comparison\.

#### Potential positive impact\.

More accurate credit transport may reduce rollout and training requirements, improve the viability of small actor\-feature critics, and make long\-horizon reasoning or agent training more sample efficient\. The architecture\-aware framing may also encourage modular combinations of credit estimators with better sampling, verifiers, and update geometry\.

#### Potential risks\.

The method is a general optimizer for capable generative models and can improve downstream systems with both beneficial and harmful uses\. Internal\-signal optimization may also create new forms of reward hacking: a policy could learn attention or routing patterns that manipulate the estimator without improving task behavior\. Detached gates and current controls do not rule out such co\-adaptation over many training iterations\. Future deployments should monitor gate distributions, test counterfactual stability, retain external behavioral evaluation, and avoid treating internal routing as a safety guarantee\.