Batched Contextual Reinforcement II: Does RLVR Need Long Rollout Windows?

# Batched Contextual Reinforcement II:Does RLVR Need Long Rollout Windows?
Source: [https://arxiv.org/html/2610.22887](https://arxiv.org/html/2610.22887)
1\]University of Illinois at Urbana\-Champaign 2\]Tsinghua University 3\]Rutgers University\\contribution\[\*\]Equal contribution\\contribution\[†\]Corresponding author\\correspondenceGe Liu,

Hongbo MaRuihan GuoJiajun FanXi ZhuGe LiuAffiliation:\[Affiliation:\[Affiliation:\[Email:[geliu@illinois\.edu](mailto:[email protected])

October 16, 2026

###### Abstract

Long rollout windows make reinforcement learning with verifiable rewards \(RLVR\) expensive, yet simply shortening them can remove the correctness signal: when every single\-problem rollout truncates before an answer, group\-relative accuracy advantages vanish\. As a technical continuation of Batched Contextual Reinforcement \(BCR\), we study how concurrent rewards relax this dependence on long windows\. Concurrent Constraint Reinforcement \(CCR\) places three independently verifiable problems in one capped training completion\. Correct answers completed before truncation retain credit, while a shared budget encourages shorter useful reasoning without an explicit length penalty\. We connect this mechanism to conditional reward\- and update\-variance analyses and study 4B and 8B mathematical reasoning models\. Against single\-problem GRPO with a 32k window, CCR with an 8,000\-token window reduces tokens per training step from 975,464 to 174,957 on 4B and from 793,832 to 165,788 on 8B, saving 82\.1% and 79\.1%, respectively\. An 8B training trace shows increasing accuracy reward alongside shorter completions\. Single\-problem evaluations provide supporting evidence of retained reasoning quality\. These results identify reward\-preserving training structure as a practical route to lower per\-step cost in short\-window RLVR\.

## 1Introduction

Reinforcement learning with verifiable rewards \(RLVR\) improves mathematical reasoning by repeatedly generating candidate solutions, checking their answers, and updating the policy\[[1](https://arxiv.org/html/2610.22887#bib.bib4),[2](https://arxiv.org/html/2610.22887#bib.bib5),[3](https://arxiv.org/html/2610.22887#bib.bib18)\]\. Long reasoning therefore raises the cost of collecting training experience\. The*rollout window*caps each completion\. Large windows let models finish long solutions, but permit expensive rollouts throughout training\.

Simply shrinking the window can destroy the signal needed to learn\. In single\-problem GRPO, truncation before an answer yields zero accuracy reward\. If every candidate for a prompt is truncated this way, relative accuracy advantages vanish\. Repeated across groups, this*reward starvation*can stall learning\. Larger windows provide room to finish answers at a higher token budget\. This creates the central challenge:*can RLVR retain useful, stable training signals with a smaller rollout budget?*

Figure 1:Standard GRPO versus CCR\.Top: each candidate solves one problem with a long window and receives a terminal correctness reward\. Bottom: each CCR candidate attempts grouped problems sequentially under a shorter shared cap\. Correct completed answers retain credit despite suffix truncation; correctness is averaged over all assigned problems for the update\. Checks, crosses, and slashes denote correct answers, incorrect answers, and truncation\. Shorter ribbons illustrate training\-token efficiency, not measured ratios\. Reward averaging motivates the conditional stability analysis in Section[4](https://arxiv.org/html/2610.22887#S4)\.*Concurrent Constraint Reinforcement*\(CCR; Figure[1](https://arxiv.org/html/2610.22887#S1.F1)\) addresses this challenge by changing the unit of reward\. Building on batched contextual reinforcement\[[4](https://arxiv.org/html/2610.22887#bib.bib8)\], it packs independently verifiable problems into one prompt and generates their solutions sequentially under a shared completion cap\. Correctness is averaged over all assigned problems\. An unfinished suffix contributes zero without erasing earlier correct answers\. Candidates can thus receive different rewards even when all are truncated, allowing a short window to retain comparative feedback\.

The shared budget connects this feedback to efficiency: unnecessary reasoning on one problem can displace another correct answer\. CCR therefore encourages useful compression without an explicit token penalty\. Our theory makes the link to stability precise\. Different counts of correct completed answers guarantee nonzero centered accuracy advantages despite suffix truncation\. Conditional independence yields anO⁡\(1/N\)O\(1/N\)reward\-noise bound, while an approximate update analysis identifies the non\-shared variance reduced by averaging\. These guarantees concern the training signal within a feasible regime; they require the stated dependence assumptions and do not guarantee convergence for arbitrary windows\.

We evaluate this training design on 4B and 8B models at fixed concurrencyN=3N=3, using an 8,000\-token window against conventional single\-problem, 32k\-window GRPO\. Our evidence combines per\-step token cost, an 8B training trace, and ordinary single\-problem evaluations\. The method retains GRPO and requires no grouped questions at deployment\. Cumulative token cost to matched accuracy remains a separate measurement\.

Our contributions address three aspects of this objective:

- •Reward preservation\.We formalize correctness feedback after suffix truncation and the shared\-budget incentive for useful compression\.
- •Conditional stability\.We establish reward\-noise and approximate update\-variance bounds, separating signal availability from stability within a feasible window\.
- •Training efficiency\.CCR reduces tokens per step by 82\.1% on 4B and 79\.1% on 8B, with a learning trace and single\-problem evaluations supporting retained reasoning quality\.

## 2Rollout Windows and Reward Starvation

Letqqbe one mathematical problem and lety1,…,ySy\_\{1\},\\ldots,y\_\{S\}beS≥2S\\geq 2candidate completions sampled for that prompt\. A completion contains at mostBBgenerated tokens, including reasoning and answer tokens; prompt tokens are outside this cap\. We distinguish this completion limit from the model’s total context capacity\. Under terminal verification,

rj=𝟏​\{yj​contains a correct answer\}\.r\_\{j\}=\\mathbf\{1\}\\\{y\_\{j\}\\text\{ contains a correct answer\}\\\}\.\(1\)Missing or unverifiable answers receive zero\. The centered accuracy advantage is

aj=rj−1S​∑s=1Srs\.a\_\{j\}=r\_\{j\}\-\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}r\_\{s\}\.\(2\)Dividing by a stabilized group standard deviation, as in GRPO, does not change the fact that equal rewards produce zero advantages\.

If every candidate reaches the cap before its answer, thenrj=aj=0r\_\{j\}=a\_\{j\}=0for alljj\. This is*accuracy\-reward starvation*: those candidates cannot be ranked by correctness\. Groups with different rewards retain task\-directed feedback; format and KL terms can contribute separately to the update\. The issue addressed here is the missing correctness signal in groups whose answers are all truncated\.

A longer window can let more trajectories reach verifiable answers\. The maximum window is an upper bound, however, rather than the number of tokens actually consumed\. Early stopping, problem difficulty, and the policy all affect realized rollout length\. We therefore measure tokens per training step for each configuration\. This describes the realized token workload under each training configuration\. We use a 32k completion window for the standard GRPO comparison\.

## 3Concurrent Rewards for Short Windows

### 3\.1Training unit and reward

For a training groupG=\(q1,…,qN\)G=\(q\_\{1\},\\ldots,q\_\{N\}\), the policy generates one ordered completion containing the solutions to allNNproblems, subject toL⁡\(y\)≤BL\(y\)\\leq B\. We useN=3N=3in the reported CCR experiments\. This is concurrency within a completion, not GPU minibatching: all three problems compete for the same generated\-token budget\.

For eachGG, sampleSSjoint completions\. LetXi,jX\_\{i,j\}indicate that completionjjcontains a complete, correct answer to problemii\. Missing, malformed, or incorrect answers haveXi,j=0X\_\{i,j\}=0\. Define

racc,j\\displaystyle r\_\{\\mathrm\{acc\},j\}=1N​∑i=1NXi,j,\\displaystyle=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}X\_\{i,j\},\(3\)Rj\\displaystyle R\_\{j\}=wacc​racc,j\+wfmt​rfmt,j,\\displaystyle=w\_\{\\mathrm\{acc\}\}r\_\{\\mathrm\{acc\},j\}\+w\_\{\\mathrm\{fmt\}\}r\_\{\\mathrm\{fmt\},j\},\(4\)A~j\\displaystyle\\widetilde\{A\}\_\{j\}=Rj−1S​∑s=1SRs\.\\displaystyle=R\_\{j\}\-\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}R\_\{s\}\.\(5\)The reported CCR reward weights arewacc=2w\_\{\\mathrm\{acc\}\}=2andwfmt=1w\_\{\\mathrm\{fmt\}\}=1\. The format term checks the required answer structure\. Crucially, accuracy is computed independently for each recoverable answer: a malformed or truncated suffix must not invalidate a correctly completed prefix\. The denominator remainsNN, including unfinished problems, so omitting difficult answers cannot inflate the average over attempted problems\.

Group\-relative comparisons are made across candidates for the*same*GG\. In particular, a candidate solving one answer is not compared against a candidate generated for an unrelated set of easier questions\. CCR supplies these rewards to a GRPO\-style policy update\[[1](https://arxiv.org/html/2610.22887#bib.bib4)\]\. For clarity, the idealized, unnormalized policy\-gradient term is

g^=1S​∑j=1SA~j​∇θ​log⁡πθ​\(yj∣G\)\.\\widehat\{g\}=\\frac\{1\}\{S\}\\sum\_\{j=1\}^\{S\}\\widetilde\{A\}\_\{j\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{j\}\\mid G\)\.\(6\)The training implementation may include probability\-ratio clipping, advantage scaling, and KL regularization\. Equation \([6](https://arxiv.org/html/2610.22887#S3.E6)\) isolates the accuracy\-driven signal for the analysis below\.

#### Operational training procedure\.

CCR samples joint candidates, verifies each recoverable answer, and computes within\-group advantages from the combined accuracy and format reward\. Verification stops at the last recoverable answer, while normalization includes all assigned problems\. Appendix[C](https://arxiv.org/html/2610.22887#A3)gives the procedure and an illustrative answer\-recovery interface\.

### 3\.2Completed answers survive truncation

###### Proposition 1\(Accuracy signal after suffix truncation\)\.

LetCj=∑iXi,jC\_\{j\}=\\sum\_\{i\}X\_\{i,j\}be the number of correct completed answers in candidatejj\. The centered accuracy advantages satisfy

aj=Cj−C¯N,C¯=1S​∑s=1SCs\.a\_\{j\}=\\frac\{C\_\{j\}\-\\bar\{C\}\}\{N\},\\qquad\\bar\{C\}=\\frac\{1\}\{S\}\\sum\_\{s=1\}^\{S\}C\_\{s\}\.\(7\)ForN=1N=1, truncation before every candidate’s sole answer givesaj=0a\_\{j\}=0for alljj\. ForN\>1N\>1, any variation inCjC\_\{j\}gives a nonzero accuracy advantage, even if every candidate has an unfinished suffix\.

Substitutingracc,j=Cj/Nr\_\{\\mathrm\{acc\},j\}=C\_\{j\}/Ninto Eq\. \([2](https://arxiv.org/html/2610.22887#S2.E2)\) proves the result\. In Table[1](https://arxiv.org/html/2610.22887#S3.T1), four candidates complete zero, one, two, and one correct answers\. Although none finishes the three\-problem group, their rewards yield both positive and negative advantages\.

Table 1:IllustrativeN=3N=3,S=4S=4group\. All candidates truncate before completing the group\. Their mean accuracy reward is1/31/3; completed prefix answers preserve relative feedback\.Preserving positive rewards alone is insufficient: candidates must differ in reward to create a group\-relative signal\. CCR can still starve if the window is too short to finish the first answer, or if every candidate completes the same number of correct answers\. Moreover, retaining a parsed reward is useful only if the trainer also retains the corresponding policy\-loss contribution\. Discarding every truncated completion would remove the very feedback the construction is intended to preserve\. The mechanism therefore requires prefix\-tolerant verification and a compatible truncation mask\.

#### How often is a group informative?

The reward resolution also admits an exact finite\-sample description\. For a fixed problem group and cap, letpc=Pr⁡\(Cj=c∣G,B\)p\_\{c\}=\\Pr\(C\_\{j\}=c\\mid G,B\)forc=0,…,Nc=0,\\ldots,N\. If theSScandidates are sampled independently, then

Pr\(∃j:aj≠0∣G,B\)=1−∑c=0NpcS\.\\Pr\(\\exists j:a\_\{j\}\\neq 0\\mid G,B\)=1\-\\sum\_\{c=0\}^\{N\}p\_\{c\}^\{S\}\.\(8\)Indeed, all centered accuracy advantages vanish exactly when all candidates have the same correct\-answer count\. The all\-zero case has probabilityp0Sp\_\{0\}^\{S\}, but equal positive rewards can also be uninformative\. This expression requires independence across sampled candidates, not across problems inside a candidate\.

Concurrency supplies additional possible reward levels, but additional levels help only if the policy places probability on them\. Whenp0=1p\_\{0\}=1, Eq\. \([8](https://arxiv.org/html/2610.22887#S3.E8)\) is zero regardless ofNNorSS\. Conversely, spreading mass across distinct completed\-answer counts can preserve useful comparisons even when every rollout is truncated\. This identifies a concrete diagnostic for short\-window training: log the fraction of groups with nonzero accuracy\-reward variance, rather than using average reward or truncation rate alone as a proxy for the learning signal\.

### 3\.3Why the same constraint encourages compression

LetK⁡\(y\)K\(y\)be the completed prefix length in an ordered trajectory\. For a finished problem,ℓi​\(y\)\\ell\_\{i\}\(y\)counts its reasoning and answer tokens; setℓi​\(y\)=∞\\ell\_\{i\}\(y\)=\\inftywhen problemiiis unfinished\. Then

K⁡\(y\)=max⁡\{k≤N:∑i=1kℓi​\(y\)≤B\},K\(y\)=\\max\\left\\\{k\\leq N:\\sum\_\{i=1\}^\{k\}\\ell\_\{i\}\(y\)\\leq B\\right\\\},\(9\)where the maximum is zero if the first answer is unfinished\. Unfinished problems contribute zero to Eq\. \([3](https://arxiv.org/html/2610.22887#S3.E3)\)\. Under this simplified prefix model, reward is the sum of correctness indicators for completed problems divided byNN\.

###### Proposition 2\(Opportunity cost of reasoning tokens\)\.

Suppose concise and verbose trajectories completeKc\>KvK\_\{c\}\>K\_\{v\}problems and have equal success probabilities on their shared completed prefix\. Ifpip\_\{i\}is the success probability on each additional completed problem, then

𝔼⁡\[rc−rv\]=1N​∑i=Kv\+1Kcpi≥0\.\\mathbb\{E\}\[r\_\{c\}\-r\_\{v\}\]=\\frac\{1\}\{N\}\\sum\_\{i=K\_\{v\}\+1\}^\{K\_\{c\}\}p\_\{i\}\\geq 0\.\(10\)The inequality is strict when an additional problem has positive success probability\.

The statement does not reward brevity by itself: it assumes that shortening preserves correctness on the shared prefix\. Useful extra reasoning remains valuable\. If an edit increases problemii’s success probability byΔ​pi\\Delta p\_\{i\}without displacing an answer, its expected accuracy reward gain isΔ​pi/N\\Delta p\_\{i\}/N\. If it displaces a setDDof otherwise completed answers, the gain becomes

Δ​𝔼​\[racc\]=1N​\(Δ​pi−∑k∈Dpk\)\.\\Delta\\mathbb\{E\}\[r\_\{\\mathrm\{acc\}\}\]=\\frac\{1\}\{N\}\\left\(\\Delta p\_\{i\}\-\\sum\_\{k\\in D\}p\_\{k\}\\right\)\.\(11\)In contrast, an explicit rewardracc−λ​Lr\_\{\\mathrm\{acc\}\}\-\\lambda Lsubtractsλ​Δ​ℓi\\lambda\\Delta\\ell\_\{i\}for every added token\. CCR imposes no such fixed tax\. It optimizes expected correctness under a hard shared budget, making unnecessary reasoning costly when that reasoning crowds out another answer\.

These effects support a common training mechanism\. Completed prefixes offer a route to comparative feedback before all problems fit within the cap\. When that feedback favors trajectories that finish more correct answers, the policy has an incentive to allocate its limited tokens more effectively\. Additional useful reasoning can increase both reward and length; the shared budget favors productive allocation rather than uniformly shorter responses\.

## 4Stability Within the Feasible Regime

Partial credit explains how a short\-window group can remain informative\. We next analyze the complementary variance\-reduction mechanism: when multiple answers are feasible, averaging their correctness can reduce observation noise, and averaging problem\-level updates can reduce non\-shared variance\. We state the relevant independence and coupling assumptions explicitly\.

### 4\.1Reward noise and accuracy advantages

For completionjj, letpi,jp\_\{i,j\}be the latent probability that problemiiyields an observed correct answer and setp¯j=N−1​∑ipi,j\\bar\{p\}\_\{j\}=N^\{\-1\}\\sum\_\{i\}p\_\{i,j\}\. Write

racc,j=p¯j\+ϵj\.r\_\{\\mathrm\{acc\},j\}=\\bar\{p\}\_\{j\}\+\\epsilon\_\{j\}\.\(12\)We analyze the*accuracy component*of the centered advantage\.

###### Theorem 1\(Conditional reward\-noise reduction\)\.

IfX1,j,…,XN,jX\_\{1,j\},\\ldots,X\_\{N,j\}are independent Bernoulli variables conditional on their latent probabilities, then

Var⁡\(ϵj∣\{pi,j\}\)=1N2​∑ipi,j​\(1−pi,j\)≤14​N\.\\operatorname\{Var\}\(\\epsilon\_\{j\}\\mid\\\{p\_\{i,j\}\\\}\)=\\frac\{1\}\{N^\{2\}\}\\sum\_\{i\}p\_\{i,j\}\(1\-p\_\{i,j\}\)\\leq\\frac\{1\}\{4N\}\.\(13\)With independent candidate completions, the centered accuracy advantage decomposes as

aj=\(p¯j−p¯¯\)\+\(ϵj−ϵ¯\),a\_\{j\}=\(\\bar\{p\}\_\{j\}\-\\overline\{\\bar\{p\}\}\)\+\(\\epsilon\_\{j\}\-\\bar\{\\epsilon\}\),\(14\)and its noise term has varianceO⁡\(1/N\)O\(1/N\)for fixedSS\.

A proof and the exact baseline\-centering factor appear in Appendix[A\.2](https://arxiv.org/html/2610.22887#A1.SS2)\. A linear signal\-to\-noise improvement additionally requires nonvanishing latent\-quality variation across candidates and reward\-noise variance of order1/N1/N\. Under these stronger conditions,SNR⁡\(a\)=Θ⁡\(N\)\\operatorname\{SNR\}\(a\)=\\Theta\(N\)\.

The independence condition also matters\. For correlated correctness indicators, the conditional variance contains an additional term,

2N2​∑i<kCov⁡\(Xi,j,Xk,j∣p\)\.\\frac\{2\}\{N^\{2\}\}\\sum\_\{i<k\}\\operatorname\{Cov\}\(X\_\{i,j\},X\_\{k,j\}\\mid p\)\.\(15\)A shared token cap and sequential generation can induce exactly this dependence\. The covariance term characterizes how dependence modifies the stability benefit of reward averaging\.

### 4\.2Averaged updates and residual coupling

An analogous calculation applies if the accuracy\-driven joint update is well approximated by an average of problem\-level updates\. Letgig\_\{i\}denote the centered update associated with problemiiand suppose

gacc≈waccN​∑igi\.g\_\{\\mathrm\{acc\}\}\\approx\\frac\{w\_\{\\mathrm\{acc\}\}\}\{N\}\\sum\_\{i\}g\_\{i\}\.\(16\)For a unit directionuu, writeZi=u⊤​giZ\_\{i\}=u^\{\\top\}g\_\{i\}\. If allZiZ\_\{i\}have varianceσu2\\sigma\_\{u\}^\{2\}and pairwise correlationρu\\rho\_\{u\}, then

Var⁡\(u⊤​gacc\)≈wacc2​σu2​\(ρu\+1−ρuN\)\.\\operatorname\{Var\}\(u^\{\\top\}g\_\{\\mathrm\{acc\}\}\)\\approx w\_\{\\mathrm\{acc\}\}^\{2\}\\sigma\_\{u\}^\{2\}\\left\(\\rho\_\{u\}\+\\frac\{1\-\\rho\_\{u\}\}\{N\}\\right\)\.\(17\)At fixed covariance structure, averaging reduces the non\-shared component whenρu<1\\rho\_\{u\}<1; the shared component remains\. Appendix[A\.3](https://arxiv.org/html/2610.22887#A1.SS3)derives the cross\-problem residual omitted by Eq\. \([16](https://arxiv.org/html/2610.22887#S4.E16)\)\. A shared budget does not by itself make that residual small\.

### 4\.3Choosing a feasible shared budget

Stability and reward preservation both depend on the availability of usable answers\. If a model needs roughlybbtokens per solution, then fitting allNNsolutions requires a budget on the order ofN​bNb, although actual lengths vary by problem and trajectory\. IncreasingNNcan therefore make the cap infeasible and destroy the signal it was intended to enrich\. We fixN=3N=3and study its training behavior for two backbones\.

## 5Experiments

### 5\.1Training and evaluation setup

We study Qwen3\-4B\-Thinking and Qwen3\-8B\-ODA\-Math with training concurrencyN=3N=3and a shared completion window of 8,000 tokens\. The experiments use a reported subset of 3,000 problems from DeepMath\-103K\[[5](https://arxiv.org/html/2610.22887#bib.bib6)\], a TRL\-based training stack\[[6](https://arxiv.org/html/2610.22887#bib.bib7)\], and eight NVIDIA A100 GPUs\. CCR combines accuracy and format rewards as in Eq\. \([4](https://arxiv.org/html/2610.22887#S3.E4)\), with no explicit length penalty or auxiliary process reward\. Appendix[B](https://arxiv.org/html/2610.22887#A2)summarizes the configuration and log\-processing procedure\.

For training cost, we compare CCR with standard single\-problem GRPO using a 32k completion window, matching the remaining training parameters and hardware\. Fewer than 10% of the 8B baseline completions are truncated, so the long\-window baseline operates with a low truncation rate\. We measure tokens consumed per training step under each configuration\.

For training dynamics, we use the 8B accuracy\-reward and completion\-length logs over steps 1–1,000\. For downstream quality, we evaluate the base and CCR checkpoints on nine mathematical reasoning benchmarks\. Every evaluation prompt contains one problem \(Neval=1N\_\{\\mathrm\{eval\}\}=1\): concurrent training does not require grouped inference at deployment\. The base checkpoints in these evaluations are the starting policies, distinct from the trained GRPO cost baseline\.

### 5\.2Training\-token efficiency at 4B and 8B

Figure[2](https://arxiv.org/html/2610.22887#S5.F2)compares the measured per\-step token workload\. On 4B, CCR reduces token consumption from 975,464 to 174,957, an 82\.1% reduction\. On 8B, it reduces consumption from 793,832 to 165,788, a 79\.1% reduction\. The baseline\-to\-CCR ratios are 5\.58 and 4\.79, respectively: CCR uses approximately one fifth of the baseline’s tokens per step on both backbones\.

Figure 2:CCR reduces tokens per training step by 82\.1% on 4B and 79\.1% on 8B\.Filled berry markers denote CCR with three problems and an 8,000\-token shared window; hollow clay markers denote single\-problem GRPO with a 32k window\. The common linear axis shows the absolute per\-step counters\. Both comparisons use eight A100 GPUs and matched remaining training parameters\.This comparison captures the combined training design: multiple verifiable answers share a smaller completion budget\. The window is an upper bound on generation, while the per\-step counter records realized token use\. Prefix credit makes the restricted budget useful for learning, and the shared cap creates pressure to allocate those tokens to answers that can earn reward\.

### 5\.3Reward improves while completions shorten

Figure[3](https://arxiv.org/html/2610.22887#S5.F3)shows the 8B CCR trajectory\. Faint lines display the raw per\-step observations, and solid lines show a trailing 50\-step mean\. Accuracy reward rises from 0\.142 over the first 50 steps to 0\.509 over the last 50 steps\. Over the same intervals, mean joint\-completion length falls from 6,867 to 5,034 tokens, a 26\.7% decrease\.

Figure 3:Accuracy improves as training completions shorten\.The 8B CCR run usesN=3N=3and an 8,000\-token shared cap\. Faint traces show raw observations over steps 1–1,000; solid traces show trailing 50\-step means of accuracy reward and joint\-completion length\.The completion cap stays fixed throughout this run, so the shortening reflects changes in the policy’s generated trajectories rather than a progressively smaller configured window\. Completion length declines through much of training and increases modestly near the end, while accuracy reward continues to improve\. This pattern is consistent with the opportunity\-cost mechanism in Eq\. \([11](https://arxiv.org/html/2610.22887#S3.E11)\): extra reasoning remains valuable when it helps produce correct answers within the shared budget\.

The cost comparison and learning trajectory describe complementary aspects of efficiency\. The former measures the token workload of the short\-window configuration; the latter shows continued learning as the policy uses that budget more compactly\. Proposition[1](https://arxiv.org/html/2610.22887#Thmproposition1)explains how completed prefixes can preserve the relative feedback needed for this process\.

### 5\.4Reasoning quality after short\-window training

Table[2](https://arxiv.org/html/2610.22887#S5.T2)reports single\-problem evaluation results\. Average accuracy increases from 75\.3% to 77\.7% on 4B and from 68\.8% to 69\.7% on 8B, with variation across individual benchmarks\. These results show that the CCR checkpoints retain useful mathematical reasoning after short\-window training\. Evaluation lengths also decrease, providing supporting context for the training\-efficiency results\.

Table 2:Reasoning quality after short\-window training\.CCR is trained withN=3N=3and evaluated on single\-problem prompts\. Accuracy is final\-answer correctness \(%\); tokens are evaluation lengths\. Base denotes the starting checkpoint\.

## 6Related Work

#### RLVR and rollout design\.

GRPO uses relative rewards among candidate completions to train mathematical reasoning policies\[[1](https://arxiv.org/html/2610.22887#bib.bib4)\]; DeepSeek\-R1 demonstrates the broader role of verifiable RL in eliciting reasoning\[[2](https://arxiv.org/html/2610.22887#bib.bib5)\]\. DAPO addresses long\-chain RL training with dynamic sampling and overlong reward shaping, among other changes\[[7](https://arxiv.org/html/2610.22887#bib.bib9)\]\. Verifier noise can corrupt these rewards even when answers are checkable\[[3](https://arxiv.org/html/2610.22887#bib.bib18)\]; SocraticPO adds corrective teacher guidance to failed rollouts\[[8](https://arxiv.org/html/2610.22887#bib.bib12)\]\. Heterogeneous agent collaboration offers another efficiency route by sharing verified rollouts across agents\[[9](https://arxiv.org/html/2610.22887#bib.bib17)\]\. These approaches highlight that sampling and reward handling affect the usefulness of generated trajectories\. CCR changes the number of independently verifiable answers available within a trajectory, so an unfinished suffix need not eliminate all accuracy feedback\.

#### Grouped reasoning and implicit compression\.

Batched contextual reinforcement already introduced multiple problems in a shared context as a structural incentive for concise reasoning\[[4](https://arxiv.org/html/2610.22887#bib.bib8)\]\. We build on that construction, rather than claiming a new optimizer or the first use of grouped prompts\. Our focus is its consequence for RLVR training: the coupling between completion caps, reward availability, and measured step cost\. The conditional stability analysis supports this motivation without turning fixed\-concurrency results into a task\-scaling claim\.

#### Reasoning and inference compute\.

Chain\-of\-thought prompting and self\-consistency improve reasoning by eliciting or sampling intermediate trajectories\[[10](https://arxiv.org/html/2610.22887#bib.bib2),[11](https://arxiv.org/html/2610.22887#bib.bib3)\]\. Test\-time compute allocation studies how additional inference can improve individual predictions\[[12](https://arxiv.org/html/2610.22887#bib.bib1)\]\. Related efficiency work studies when reasoning should stop\[[13](https://arxiv.org/html/2610.22887#bib.bib16)\]and how latent guidance can transfer reasoning to smaller models\[[14](https://arxiv.org/html/2610.22887#bib.bib11)\]\. In medical vision\-language tasks, adaptive causal reflection and latent diagnostic memory provide distinct ways to shape reasoning trajectories\[[15](https://arxiv.org/html/2610.22887#bib.bib14),[16](https://arxiv.org/html/2610.22887#bib.bib15)\]\. Here the scarce resource is repeated on\-policy generation during training\. Shorter evaluation traces are useful supporting observations, but our primary measurements concern training steps under a restricted rollout window\.

#### Other feedback and evaluation settings\.

Robust preference optimization handles noisy preference labels\[[17](https://arxiv.org/html/2610.22887#bib.bib10)\]; positive\-unlabeled RL distillation constructs preference signals for small locally deployed models\[[18](https://arxiv.org/html/2610.22887#bib.bib19)\]; and am\-ELO addresses ranking stability in arena\-based evaluation\[[19](https://arxiv.org/html/2610.22887#bib.bib13)\]\. These studies concern different feedback or evaluation regimes from CCR’s verified multi\-problem rewards\.

## 7Conclusion

BCR II develops the training\-efficiency perspective of batched contextual reinforcement\. CCR preserves credit for completed answers under suffix truncation and rewards productive use of a shared token budget\. Our analysis connects prefix credit to group\-relative feedback and reward averaging to conditional stability\. WithN=3N=3and an 8,000\-token window, CCR uses 82\.1% fewer tokens per step on 4B and 79\.1% fewer on 8B than single\-problem, 32k\-window GRPO\. Increasing reward, shorter training completions, and retained single\-problem reasoning quality support a common conclusion: structuring rollouts around multiple verifiable answers can make short\-window RLVR effective and efficient\.

## References

- \[1\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models\.External Links:2402\.03300,[Link](https://arxiv.org/abs/2402.03300)Cited by:[§1](https://arxiv.org/html/2610.22887#S1.p1.1),[§3\.1](https://arxiv.org/html/2610.22887#S3.SS1.p3.1),[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px1.p1.1)\.
- \[2\]D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)DeepSeek\-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning\.External Links:2501\.12948,[Link](https://arxiv.org/abs/2501.12948)Cited by:[§1](https://arxiv.org/html/2610.22887#S1.p1.1),[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px1.p1.1)\.
- \[3\]X\. Cai, W\. Wang, F\. Liu, T\. Liu, G\. Niu, and M\. Sugiyama\(2025\)Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers\.arXiv preprint arXiv:2510\.00915\.External Links:[Link](https://arxiv.org/abs/2510.00915)Cited by:[§1](https://arxiv.org/html/2610.22887#S1.p1.1),[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px1.p1.1)\.
- \[4\]B\. Yang, H\. Ma, J\. Fan, and G\. Liu\(2026\)Batched contextual reinforcement\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=8Oc3Mx754M)Cited by:[§1](https://arxiv.org/html/2610.22887#S1.p3.1),[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px2.p1.1)\.
- \[5\]Z\. He, T\. Liang, J\. Xu, Q\. Liu, X\. Chen, Y\. Wang, L\. Song, D\. Yu, Z\. Liang, W\. Wang, Z\. Zhang, R\. Wang, Z\. Tu, H\. Mi, and D\. Yu\(2025\)DeepMath\-103K: A Large\-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning\.External Links:2504\.11456,[Link](https://arxiv.org/abs/2504.11456)Cited by:[§5\.1](https://arxiv.org/html/2610.22887#S5.SS1.p1.1)\.
- \[6\]L\. von Werra, Y\. Belkada, L\. Tunstall, E\. Beeching, T\. Thrush, N\. Lambert, S\. Huang, K\. Rasul, and Q\. Gallouedec\(2020\)TRL: Transformers Reinforcement Learning\.Note:GitHub repositoryApache\-2\.0 licenseExternal Links:[Link](https://github.com/huggingface/trl)Cited by:[§5\.1](https://arxiv.org/html/2610.22887#S5.SS1.p1.1)\.
- \[7\]Q\. Yu, Z\. Zhang, R\. Zhu, Y\. Yuan, X\. Zuo, Y\. Yue, W\. Dai, T\. Fan, G\. Liu, L\. Liu, X\. Liu, H\. Lin, Z\. Lin, B\. Ma, G\. Sheng, Y\. Tong, C\. Zhang, M\. Zhang, W\. Zhang, H\. Zhu, J\. Zhu, J\. Chen, J\. Chen, C\. Wang, H\. Yu, Y\. Song, X\. Wei, H\. Zhou, J\. Liu, W\. Ma, Y\. Zhang, L\. Yan, M\. Qiao, Y\. Wu, and M\. Wang\(2025\)DAPO: An Open\-Source LLM Reinforcement Learning System at Scale\.External Links:2503\.14476,[Document](https://dx.doi.org/10.48550/arXiv.2503.14476),[Link](https://arxiv.org/abs/2503.14476)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px1.p1.1)\.
- \[8\]Z\. Liu, T\. Pan, J\. Ouyang, Q\. Liu, X\. Wang, J\. Liu, Q\. Li, J\. Sha, Z\. Huang, S\. Wang, and E\. Chen\(2026\)SocraticPO: policy optimization via interactive guidance\.External Links:2606\.09887,[Link](https://arxiv.org/abs/2606.09887)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px1.p1.1)\.
- \[9\]Z\. Zhang, Z\. Huang, X\. Xia, D\. Wang, F\. Zhuang, S\. Ma, N\. Ding, Y\. Yang, J\. Li, and Y\. Ban\(2026\)Heterogeneous agent collaborative reinforcement learning\.arXiv preprint arXiv:2603\.02604\.External Links:[Link](https://arxiv.org/abs/2603.02604)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px1.p1.1)\.
- \[10\]J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, F\. Xia, E\. H\. Chi, Q\. V\. Le, and D\. Zhou\(2022\)Chain\-of\-Thought Prompting Elicits Reasoning in Large Language Models\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[11\]X\. Wang, J\. Wei, D\. Schuurmans, Q\. V\. Le, E\. H\. Chi, S\. Narang, A\. Chowdhery, and D\. Zhou\(2023\)Self\-Consistency Improves Chain of Thought Reasoning in Language Models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=1PL1NIMMrw)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[12\]C\. Snell, J\. Lee, K\. Xu, and A\. Kumar\(2025\)Scaling LLM Test\-Time Compute Optimally can be More Effective than Scaling Model Parameters\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=4FWAwZtd2n)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[13\]Z\. Huang, X\. Xia, Y\. Ren, J\. Zheng, X\. Wang, Z\. Zhang, H\. Xie, S\. Liang, Z\. Chen, X\. Xiao,et al\.\(2026\)Does your reasoning model implicitly know when to stop thinking?\.arXiv preprint arXiv:2602\.08354\.External Links:[Link](https://arxiv.org/abs/2602.08354)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[14\]H\. Chen, L\. Yang, J\. Wang, J\. Yan, Z\. Wang, X\. Liang, and J\. Hao\(2026\)Latent\-guided reasoning: empowering small llms with large\-model thinking\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2026/hash/6958e9d0f5a76d54ff97da8c45f4d52e-Abstract-Conference.html)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[15\]J\. Lin, C\. Zhu, P\. J\. Kneuertz, Y\. Bai, and Y\. Xue\(2026\)When models learn to ask why: adaptive causal reasoning for trustworthy medical vision\-language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 5556–5568\.External Links:[Link](https://arxiv.org/abs/2603.23085)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[16\]C\. Zhu, J\. Zeng, J\. Jiang, J\. Lin, and Y\. Wang\(2026\)MedSynapse\-v: bridging visual perception and clinical intuition via latent memory evolution\.External Links:2604\.26283,[Link](https://arxiv.org/abs/2604.26283)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px3.p1.1)\.
- \[17\]X\. Liang, C\. Chen, S\. Qiu, J\. Wang, Y\. Wu, Z\. Fu, H\. Chen, F\. Wu, and J\. Ye\(2025\)ROPO: robust preference optimization for large language models\.InInternational Conference on Machine Learning,pp\. 37131–37161\.External Links:[Link](https://proceedings.mlr.press/v267/liang25d.html)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px4.p1.1)\.
- \[18\]Z\. Kou, J\. Chen, X\. Cai, X\. Xia, M\. Xie, D\. Wu, B\. Liu, Y\. Jia, X\. Geng, M\. Sugiyama, and T\. Chua\(2026\)Positive–unlabeled reinforcement learning distillation for on\-premise small models\.InForty\-third International Conference on Machine Learning,External Links:[Link](https://proceedings.mlr.press/v306/kou26a.html)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px4.p1.1)\.
- \[19\]Z\. Liu, J\. Li, Y\. Zhuang, Q\. Liu, S\. Shen, J\. Ouyang, M\. Cheng, and S\. Wang\(2025\)am\-ELO: a stable framework for arena\-based llm evaluation\.InInternational Conference on Machine Learning,pp\. 38857–38868\.External Links:[Link](https://proceedings.mlr.press/v267/liu25ak.html)Cited by:[§6](https://arxiv.org/html/2610.22887#S6.SS0.SSS0.Px4.p1.1)\.

## Appendix ADerivations and Assumptions

### A\.1Prefix rewards and opportunity cost

For Proposition[1](https://arxiv.org/html/2610.22887#Thmproposition1), substitutingrj=Cj/Nr\_\{j\}=C\_\{j\}/Ninto the centered advantage givesaj=\(Cj−C¯\)/Na\_\{j\}=\(C\_\{j\}\-\\bar\{C\}\)/N\. If someCjC\_\{j\}differ, they cannot all equal their mean, so some advantages are nonzero\. In fact, the finite\-group accuracy variance is

1S​∑jaj2=1N2​S​∑j\(Cj−C¯\)2\.\\frac\{1\}\{S\}\\sum\_\{j\}a\_\{j\}^\{2\}=\\frac\{1\}\{N^\{2\}S\}\\sum\_\{j\}\(C\_\{j\}\-\\bar\{C\}\)^\{2\}\.\(18\)This equality makes the necessary reward\-diversity condition explicit\. It concerns the accuracy component; the total reward also contains the format term\.

For Proposition[2](https://arxiv.org/html/2610.22887#Thmproposition2), the expected correctness contributions on the common completed prefix cancel\. Only the additional completed problems remain, giving Eq\. \([10](https://arxiv.org/html/2610.22887#S3.E10)\)\. For Eq\. \([11](https://arxiv.org/html/2610.22887#S3.E11)\), add the expected gainΔ​pi/N\\Delta p\_\{i\}/Nand subtract the expected correctness of each displaced answer divided byNN\. These are comparisons of expected rewards under the stated preservation assumptions\.

### A\.2Reward variance and centering

Conditional independence gives

𝔼⁡\[rj∣p\]\\displaystyle\\mathbb\{E\}\[r\_\{j\}\\mid p\]=p¯j,\\displaystyle=\\bar\{p\}\_\{j\},\(19\)Var⁡\(rj∣p\)\\displaystyle\\operatorname\{Var\}\(r\_\{j\}\\mid p\)=1N2​∑ipi,j​\(1−pi,j\)\.\\displaystyle=\\frac\{1\}\{N^\{2\}\}\\sum\_\{i\}p\_\{i,j\}\(1\-p\_\{i,j\}\)\.\(20\)Each Bernoulli variance is at most1/41/4, proving Eq\. \([13](https://arxiv.org/html/2610.22887#S4.E13)\)\. With

νj=1N​∑ipi,j​\(1−pi,j\),\\nu\_\{j\}=\\frac\{1\}\{N\}\\sum\_\{i\}p\_\{i,j\}\(1\-p\_\{i,j\}\),\(21\)the conditional noise variance isνj/N\\nu\_\{j\}/N\. Under independent candidate noise, a baseline computed from the sameSSsamples yields

Var⁡\(ϵj−ϵ¯∣p\)\\displaystyle\\operatorname\{Var\}\(\\epsilon\_\{j\}\-\\bar\{\\epsilon\}\\mid p\)=1N​\[\(1−2S\)​νj\+1S2​∑sνs\]\.\\displaystyle\\quad=\\frac\{1\}\{N\}\\left\[\\left\(1\-\\frac\{2\}\{S\}\\right\)\\nu\_\{j\}\+\\frac\{1\}\{S^\{2\}\}\\sum\_\{s\}\\nu\_\{s\}\\right\]\.\(22\)It isO⁡\(1/N\)O\(1/N\)for fixedS≥2S\\geq 2\. When candidate noise has the same unconditional varianceν/N\\nu/N, this reduces to\(1−1/S\)​ν/N\(1\-1/S\)\\nu/N\.

If candidates’ latent mean qualities are independent and identically distributed with varianceσp2\\sigma\_\{p\}^\{2\}, their centered signal variance is\(1−1/S\)​σp2\(1\-1/S\)\\sigma\_\{p\}^\{2\}\. The conditional noise has zero mean, so it is uncorrelated with the latent signal\. In this setting,

SNR⁡\(a\)=N​σp2ν\.\\operatorname\{SNR\}\(a\)=\\frac\{N\\sigma\_\{p\}^\{2\}\}\{\\nu\}\.\(23\)Thus linear orderΘ⁡\(N\)\\Theta\(N\)requires bothσp2=Θ⁡\(1\)\\sigma\_\{p\}^\{2\}=\\Theta\(1\)andν=Θ⁡\(1\)\\nu=\\Theta\(1\), with positive lower bounds\. A noise upper bound alone only gives a one\-sided statement when the signal remains nonzero\. If all rewards coincide because all answers truncate, there is no informative latent correctness signal to which this argument can apply\. Conditional covariance between problem outcomes adds Eq\. \([15](https://arxiv.org/html/2610.22887#S4.E15)\) and can prevent1/N1/Ndecay\.

### A\.3Update averaging and cross\-problem terms

Write the score of candidatejjas

sj=∇θ​log​πθ​\(yj∣G\)=∑k=1Nsk,j\+ej\.s\_\{j\}=\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(y\_\{j\}\\mid G\)=\\sum\_\{k=1\}^\{N\}s\_\{k,j\}\+e\_\{j\}\.\(24\)Heresk,js\_\{k,j\}denotes the component associated with problemkkandeje\_\{j\}collects any residual under the chosen segmentation\. Define centered correctness indicators

ai,j=Xi,j−1S​∑sXi,s,a\_\{i,j\}=X\_\{i,j\}\-\\frac\{1\}\{S\}\\sum\_\{s\}X\_\{i,s\},\(25\)so the centered accuracy advantage isN−1​∑iai,jN^\{\-1\}\\sum\_\{i\}a\_\{i,j\}\. Its unnormalized gradient contribution becomes

gacc\\displaystyle g\_\{\\mathrm\{acc\}\}=waccN​S​∑i,jai,j​sj\\displaystyle=\\frac\{w\_\{\\mathrm\{acc\}\}\}\{NS\}\\sum\_\{i,j\}a\_\{i,j\}s\_\{j\}=waccN​∑igi\+Ccross\+Cres,\\displaystyle=\\frac\{w\_\{\\mathrm\{acc\}\}\}\{N\}\\sum\_\{i\}g\_\{i\}\+C\_\{\\mathrm\{cross\}\}\+C\_\{\\mathrm\{res\}\},\(26\)where

gi\\displaystyle g\_\{i\}=1S​∑jai,j​si,j,\\displaystyle=\\frac\{1\}\{S\}\\sum\_\{j\}a\_\{i,j\}s\_\{i,j\},\(27\)Ccross\\displaystyle C\_\{\\mathrm\{cross\}\}=waccN​S​∑j∑i≠kai,j​sk,j,\\displaystyle=\\frac\{w\_\{\\mathrm\{acc\}\}\}\{NS\}\\sum\_\{j\}\\sum\_\{i\\neq k\}a\_\{i,j\}s\_\{k,j\},\(28\)Cres\\displaystyle C\_\{\\mathrm\{res\}\}=waccN​S​∑i,jai,j​ej\.\\displaystyle=\\frac\{w\_\{\\mathrm\{acc\}\}\}\{NS\}\\sum\_\{i,j\}a\_\{i,j\}e\_\{j\}\.\(29\)Equation \([16](https://arxiv.org/html/2610.22887#S4.E16)\) assumes both residual contributions are small\. Sequential conditioning and a shared budget may violate this assumption; the constraint alone does not suppress cross terms\.

For the averaged component, ifZi=u⊤​giZ\_\{i\}=u^\{\\top\}g\_\{i\}has common varianceσu2\\sigma\_\{u\}^\{2\}and common pairwise correlationρu\\rho\_\{u\}, then

Var⁡\(1N​∑iZi\)\\displaystyle\\operatorname\{Var\}\\\!\\left\(\\frac\{1\}\{N\}\\sum\_\{i\}Z\_\{i\}\\right\)=N​σu2\+N⁡\(N−1\)​ρu​σu2N2\\displaystyle=\\frac\{N\\sigma\_\{u\}^\{2\}\+N\(N\-1\)\\rho\_\{u\}\\sigma\_\{u\}^\{2\}\}\{N^\{2\}\}=σu2​\(ρu\+1−ρuN\)\.\\displaystyle=\\sigma\_\{u\}^\{2\}\\left\(\\rho\_\{u\}\+\\frac\{1\-\\rho\_\{u\}\}\{N\}\\right\)\.\(30\)Multiplying bywacc2w\_\{\\mathrm\{acc\}\}^\{2\}yields Eq\. \([17](https://arxiv.org/html/2610.22887#S4.E17)\)\. The equal\-correlation model must have a valid covariance matrix\. This comparison holdsρu\\rho\_\{u\}fixed and isolates the averaged accuracy\-gradient component\. Format\-gradient effects, normalization, and clipping enter through the full GRPO update\.

## Appendix BExperimental Configuration and Log Processing

Table[3](https://arxiv.org/html/2610.22887#A2.T3)summarizes the settings used for the comparisons\. The training\-token measurements use the recorded per\-step counters for each method with matched remaining training parameters and hardware\.

Table 3:Training and evaluation configuration\.#### Training dynamics\.

The two W&B exports each contain 1,008 observations with matching global\-step identifiers\. We join the accuracy\-reward and mean\-completion\-length columns by global step and select steps 1–1,000\. For observationsztz\_\{t\}, the displayed trailing mean is

z¯t=1min⁡\(t,50\)​∑u=max⁡\(1,t−49\)tzu\.\\bar\{z\}\_\{t\}=\\frac\{1\}\{\\min\(t,50\)\}\\sum\_\{u=\\max\(1,t\-49\)\}^\{t\}z\_\{u\}\.\(31\)The smoother uses the available prefix for the first 49 steps\. First\-50 and last\-50 summaries are arithmetic means of the raw observations\. The faint traces in Figure[3](https://arxiv.org/html/2610.22887#S5.F3)display the raw values from this run\. Source CSVs, plotting scripts, and data hashes accompany the figure assets\.

## Appendix CTraining Procedure and Reward Interface

The following procedure specifies the CCR training structure\. Candidate count, optimizer settings, and decoding hyperparameters are inputs to the training loop\.

1. 1\.Form an ordered groupG=\(q1,…,qN\)G=\(q\_\{1\},\\ldots,q\_\{N\}\)of independently verifiable problems; useN=3N=3for the reported runs\.
2. 2\.Build one prompt with distinct problem identifiers and request an ordered sequence of solutions and recoverable answers\.
3. 3\.SampleSScandidate completions for this same prompt, each with the shared generation capB=8,000B=8\{,\}000\.
4. 4\.For each candidate, extract every completed answer that can be unambiguously assigned to its problem\. Set the correctness indicator to zero for missing or incorrect answers\. An unfinished suffix does not revoke earlier valid answers\.
5. 5\.Computeracc,j=N−1​∑iXi,jr\_\{\\mathrm\{acc\},j\}=N^\{\-1\}\\sum\_\{i\}X\_\{i,j\}, retain the fixed denominatorNN, and add the format term with the reported reward weights\.
6. 6\.Compute group\-relative advantages among the candidates forGGand apply the configured GRPO loss\. Loss masking must retain the policy contribution associated with the reward\-bearing prefix of a truncated trajectory\.
7. 7\.Repeat with the updated policy\. Record completion length, per\-step token consumption, and accuracy\-reward statistics separately\.

#### Illustrative prompt interface\.

A minimal interface presentsProblem 1,Problem 2, andProblem 3and asks for a separately identifiable answer after each solution\. For example,Answer 1:followed by a boxed expression identifies the first answer\. This illustrates the answer\-recovery interface\. The implementation should specify delimiters, verifier normalization, ambiguous\-answer handling, and how the final answer is associated with each problem\.

#### Partial credit and format checking\.

The format reward and the accuracy reward serve distinct roles\. A suffix that violates a formatting convention may lose format credit while leaving earlier correctness indicators valid\. Requiring the entire response to pass a format gate before evaluating any answer would reintroduce all\-or\-nothing terminal feedback\. Similarly, dividing by the number of attempted answers would allow the policy to inflate reward by omitting difficult problems; the fixed denominator avoids that incentive\.

#### Truncation and the policy loss\.

A verifier that preserves prefix credit is insufficient if a trainer subsequently masks the whole truncated response\. Conversely, broadcasting a rollout advantage to every generated token can introduce credit\-assignment noise for an unfinished suffix\. These choices should be documented separately: the reward definition determines whether correctness information exists, whereas token masking and policy\-loss aggregation determine how it reaches the update\. The analytical results isolate the first issue and state explicit approximations for the second\.

## Appendix DAdditional Reward and Budget Derivations

### D\.1Heterogeneous answer probabilities

The reward\-noise calculation can retain the heterogeneity across problems\. For a candidate with latent correctness probabilitiesp1,…,pNp\_\{1\},\\ldots,p\_\{N\}, letp¯=N−1​∑ipi\\bar\{p\}=N^\{\-1\}\\sum\_\{i\}p\_\{i\}andvp=N−1​∑i\(pi−p¯\)2v\_\{p\}=N^\{\-1\}\\sum\_\{i\}\(p\_\{i\}\-\\bar\{p\}\)^\{2\}\. Under conditional independence,

Var⁡\(racc∣p\)\\displaystyle\\operatorname\{Var\}\(r\_\{\\mathrm\{acc\}\}\\mid p\)=1N2​∑ipi​\(1−pi\)\\displaystyle=\\frac\{1\}\{N^\{2\}\}\\sum\_\{i\}p\_\{i\}\(1\-p\_\{i\}\)=p¯​\(1−p¯\)−vpN\.\\displaystyle=\\frac\{\\bar\{p\}\(1\-\\bar\{p\}\)\-v\_\{p\}\}\{N\}\.\(32\)The identity follows by expandingvp=N−1​∑ipi2−p¯2v\_\{p\}=N^\{\-1\}\\sum\_\{i\}p\_\{i\}^\{2\}\-\\bar\{p\}^\{2\}\. It explains why the1/\(4​N\)1/\(4N\)bound is loose when some problems are almost certainly correct or incorrect\. It also separates Bernoulli observation noise from variation in mean trajectory quality across candidates\. The latter provides the signal that group\-relative centering must retain\.

### D\.2Explicit length penalties versus a shared cap

Consider a local edit that addsΔ​ℓi\>0\\Delta\\ell\_\{i\}\>0tokens and increases problemii’s success probability byΔ​pi\\Delta p\_\{i\}, holding the completed set fixed\. With an explicit rewardracc−λ​Lr\_\{\\mathrm\{acc\}\}\-\\lambda L, the expected change is

Δ​𝔼​\[Rλ\]=Δ​piN−λ​Δ​ℓi\.\\Delta\\mathbb\{E\}\[R\_\{\\lambda\}\]=\\frac\{\\Delta p\_\{i\}\}\{N\}\-\\lambda\\Delta\\ell\_\{i\}\.\(33\)Useful reasoning is discouraged whenever its accuracy gain is smaller than this fixed token tax\. Under CCR, the same edit changes expected accuracy byΔ​pi/N\\Delta p\_\{i\}/Nif it displaces no completed answer\. If it displaces a setDD, the change becomes\(Δ​pi−∑k∈Dpk\)/N\(\\Delta p\_\{i\}\-\\sum\_\{k\\in D\}p\_\{k\}\)/N\. This comparison assumes that the local edit has no other effects on the remaining solutions; the shared autoregressive context can violate that simplification\.

The two objectives assign different costs to extra reasoning\. A shared cap assigns an opportunity cost through displaced correctness, while an explicit penalty assigns a prescribed cost to every additional token\.

### D\.3Feasible completion and conditional signal

Letbib\_\{i\}be the tokens required to finish problemiialong a particular successful trajectory\. Completing the whole group requires∑ibi≤B\\sum\_\{i\}b\_\{i\}\\leq B\. Partial credit relaxes the need to complete the entire group before receiving any accuracy reward, but does not remove the need to finish at least one answer\. If all candidates fail that condition,p0=1p\_\{0\}=1and Eq\. \([8](https://arxiv.org/html/2610.22887#S3.E8)\) yields zero probability of an informative accuracy group\. If the candidate count distribution has at least two values with positive mass, independently sampled groups can have nonzero accuracy advantages even when no sample completes allNNtasks\.

This makes ordering relevant: an unusually long first solution may block every later answer\. Heterogeneous difficulty, cross\-problem interference, and ordering rules should be measured along with completion rate\. A fixedB/NB/Nis only a rough allocation heuristic because CCR does not impose equal per\-problem sub\-budgets\.

## Appendix EQualitative Trajectory Summaries

We present base andN=3N=3trajectory summaries for Qwen3\-4B\-Thinking\-2507\. The examples below are ordinary single\-problem evaluations\. Their lengths may exceed the training cap because the cap applies to RL rollouts, not to these evaluation traces\. The descriptions are summaries, not verbatim complete model outputs\. Cases were selected for illustration, so their correctness rates and length reductions should not be averaged into a new benchmark result\.

### E\.1Torus–Sphere Tangency \(AIME 2024\)

#### Problem\.

A torus with major radius66and minor radius33is tangent to a sphere of radius1111along circles of radiirir\_\{i\}andror\_\{o\}\. Ifri−ro=mnr\_\{i\}\-r\_\{o\}=\\frac\{m\}\{n\}, findm\+nm\+n\. Ground Truth: 127

#### Base checkpoint \(32,312 tokens\)\.

Searches over possible sphere\-center offsetsddand repeatedly checks special cases\. Several candidate radii are rejected as invalid or irrational, but the trace never derives the correct tangency\-circle formula\. The reasoning remains dominated by coordinate trial\-and\-error and finally reports an incorrect value\.

Answer:352×\\text\{Answer: \}\\boxed\{352\}\\ \\times

#### CCR,N=3N=3\(17,837 tokens\)\.

Keeps the essential tangency geometry while compressing the derivation and avoids the long offset search\. Uses the inner and outer tangency\-circle formulas directly, then computes the difference cleanly\.

ri−ro=334−337=9928\.r\_\{i\}\-r\_\{o\}=\\frac\{33\}\{4\}\-\\frac\{33\}\{7\}=\\frac\{99\}\{28\}\.
Answer:127✓\\text\{Answer: \}\\boxed\{127\}\\ \\checkmark

### E\.2Date Digit Parity \(AMC 2023\)

#### Problem\.

A date in 2023 is displayed as an 8\-digit integer YYYYMMDD\. For how many dates in 2023 does every digit appear an even number of times? Ground Truth: 9

#### Base checkpoint \(25,269 tokens\)\.

Sets up the correct parity condition from the fixed year 2023: digits 0 and 3 must appear oddly in the month\-day part, while all other digits must appear evenly\. Performs a long exhaustive enumeration over valid dates and recovers all nine satisfying dates\. The reasoning is correct but token\-heavy\.

Answer:9✓\\text\{Answer: \}\\boxed\{9\}\\ \\checkmark

#### CCR,N=3N=3\(15,316 tokens\)\.

Uses month\-by\-month parity vectors rather than broad enumeration\. Finds the valid dates by month: Jan has 2, Feb has 1, Mar has 2, Oct has 2, and Nov has 2\.

Answer:9✓\\text\{Answer: \}\\boxed\{9\}\\ \\checkmark

### E\.3Party Graph with Enemy Closure

#### Problem\.

Bruno invitesnnguests\. Each pair of guests are either friends or enemies\. Each guest has exactly1212enemies, and all guests believe that the friend of an enemy is an enemy\. Find the sum of all possible values ofnn\. Ground Truth: 100

#### Base checkpoint \(26,146 tokens\)\.

Works in the enemy graph and correctly finds two extremal examples: everyone mutually enemies, givingn=13n=13, and two equal friend groups, givingn=24n=24\. However, it treats these examples as exhaustive after only informal testing\. It misses the full disjoint\-clique structure of the friend graph, so it loses the intermediate values14,15,16,1814,15,16,18\.

Answer:37×\\text\{Answer: \}\\boxed\{37\}\\ \\times

#### CCR,N=3N=3\(14,139 tokens\)\.

Uses the global clique structure in a concise derivation\. Lists the divisor cases directly:

k−1∈\{1,2,3,4,6,12\}\.k\-1\\in\\\{1,2,3,4,6,12\\\}\.
Hence

n∈\{13,14,15,16,18,24\}n\\in\\\{13,14,15,16,18,24\\\}13\+14\+15\+16\+18\+24=100\.13\+14\+15\+16\+18\+24=100\.
Answer:100✓\\text\{Answer: \}\\boxed\{100\}\\ \\checkmark

### E\.4Vector Projection \(MATH\-500\)

#### Problem\.

Find the projection of𝐚\\mathbf\{a\}onto𝐛=\(263\)\\mathbf\{b\}=\\begin\{pmatrix\}2\\\\ 6\\\\ 3\\end\{pmatrix\}if𝐚⋅𝐛=8\\mathbf\{a\}\\cdot\\mathbf\{b\}=8\. Ground Truth:\(16/4948/4924/49\)\\begin\{pmatrix\}16/49\\\\ 48/49\\\\ 24/49\\end\{pmatrix\}

#### Base checkpoint \(3,717 tokens\)\.

Treats “projection” as the scalar projection rather than the vector projection\. Computes‖𝐛‖=7\\\|\\mathbf\{b\}\\\|=7and applies𝐚⋅𝐛‖𝐛‖=87\\frac\{\\mathbf\{a\}\\cdot\\mathbf\{b\}\}\{\\\|\\mathbf\{b\}\\\|\}=\\frac\{8\}\{7\}\. The reasoning notices the ambiguity but resolves it in the wrong direction\.

Answer:87×\\text\{Answer: \}\\boxed\{\\dfrac\{8\}\{7\}\}\\ \\times

#### CCR,N=3N=3\(1,546 tokens\)\.

Also identifies that the requested projection should be the vector projection\. Keeps the derivation compact:

849​\(263\)=\(16/4948/4924/49\)\.\\frac\{8\}\{49\}\\begin\{pmatrix\}2\\\\ 6\\\\ 3\\end\{pmatrix\}=\\begin\{pmatrix\}16/49\\\\ 48/49\\\\ 24/49\\end\{pmatrix\}\.
Despite briefly considering the scalar interpretation, it selects the correct vector\-valued answer\.

Answer:\(16/4948/4924/49\)✓\\text\{Answer: \}\\boxed\{\\begin\{pmatrix\}16/49\\\\ 48/49\\\\ 24/49\\end\{pmatrix\}\}\\ \\checkmark

### E\.5Polytropic Atmosphere Hydrostatic Equilibrium \(Minerva\)

#### Problem\.

A thin stellar atmosphere obeysP=K​ρ5/3P=K\\rho^\{5/3\}with constant gravitational accelerationgg\. Using hydrostatic equilibrium, deriveP⁡\(z\)P\(z\)above the surface, withP⁡\(0\)=P0P\(0\)=P\_\{0\}\. Ground Truth:\[P02/5−25gK−3/5z\]5/2\\left\[P\_\{0\}^\{2/5\}\-\\frac\{2\}\{5\}gK^\{\-3/5\}z\\right\]^\{5/2\}

#### Base checkpoint \(5,683 tokens\)\.

Starts from the correct hydrostatic equationd​P/d​z=−ρ​gdP/dz=\-\\rho gand the correct substitutionρ=\(P/K\)3/5\\rho=\(P/K\)^\{3/5\}\. However, it mishandles the factor ofKKduring separation, effectively movingK3/5K^\{3/5\}into the numerator\. This yields the wrong pressure profile

P⁡\(z\)=\[P02/5−2​g​K3/55​z\]5/2\.P\(z\)=\\left\[P\_\{0\}^\{2/5\}\-\\frac\{2gK^\{3/5\}\}\{5\}z\\right\]^\{5/2\}\.
\[P02/5−2​g​K3/55​z\]5/2×\\boxed\{\\left\[P\_\{0\}^\{2/5\}\-\\frac\{2gK^\{3/5\}\}\{5\}z\\right\]^\{5/2\}\}\\ \\times

#### CCR,N=3N=3\(2,930 tokens\)\.

Keeps the essential hydrostatic derivation while compressing the solution\. After substitutingρ=\(P/K\)3/5\\rho=\(P/K\)^\{3/5\}, it explicitly writes the correct differential equation withK−3/5K^\{\-3/5\}\.

P2/5=P02/5−2​g5​K3/5​z\.P^\{2/5\}=P\_\{0\}^\{2/5\}\-\\frac\{2g\}\{5K^\{3/5\}\}z\.
A dimensional check reinforces the correct dependence onK3/5K^\{3/5\}in the denominator\.

\[P02/5−2​g5​K3/5​z\]5/2​✓\\boxed\{\\left\[P\_\{0\}^\{2/5\}\-\\frac\{2g\}\{5K^\{3/5\}\}z\\right\]^\{5/2\}\}\\ \\checkmark

### E\.6Three Non\-overlapping Squares

#### Problem\.

Three non\-overlapping squares of positive integer side lengths each have one vertex at the origin and sides parallel to the coordinate axes\. Together, the three squares enclose a region whose area is4141\. Compute the largest possible perimeter of the region\. Ground Truth: 32

#### Base checkpoint \(24,806 tokens\)\.

Finds the two possible side\-length triples\(1,2,6\)\(1,2,6\)and\(3,4,4\)\(3,4,4\), but repeatedly changes the perimeter formula\. Even after computing3232for both triples, it overrides that result using an unsupported “more precise” boundary correction\. The final answer is therefore driven by a late inconsistent shortcut rather than the geometric boundary calculation\.

30×\\boxed\{30\}\\ \\times

#### CCR,N=3N=3\(17,823 tokens\)\.

Keeps the same finite case analysis but compresses the search\. Correctly minimizes the total shared boundary: two shared edges of the smallest square\.

\(1,2,6\):4⋅9−4=32,\(1,2,6\):\\ 4\\cdot 9\-4=32,\(3,4,4\):4⋅11−12=32\.\(3,4,4\):\\ 4\\cdot 11\-12=32\.
32​✓\\boxed\{32\}\\ \\checkmark

### E\.7Sphere Tangent to Six Tetrahedra

#### Problem\.

Six solid regular tetrahedra are placed on a flat surface so that their bases form a regular hexagonℋ\\mathcal\{H\}with side length11, and so that the top vertices are coplanar\. A spherical ball of radiusrrhas center directly above the center of the hexagon, rests on the tetrahedra, and is tangent to one edge from each tetrahedron\. If the ball’s center is coplanar with the top vertices, computerr\. Ground Truth:23\\frac\{\\sqrt\{2\}\}\{3\}

#### Base checkpoint \(31,054 tokens\)\.

Sets up reasonable coordinates for the hexagon and the top\-vertex heighth=2/3h=\\sqrt\{2/3\}\. Then selects a lateral edge from an outer base vertex to the top vertex, rather than the symmetric edge adjacent to the hexagon center\. The point\-to\-line distance therefore becomes11/6\\sqrt\{11\}/6, a plausible but wrong radius\.

116×\\boxed\{\\frac\{\\sqrt\{11\}\}\{6\}\}\\ \\times

#### CCR,N=3N=3\(17,030 tokens\)\.

Makes the closest\-edge choice explicit: the tangent edge isO​TkOT\_\{k\}, from the hexagon center to a top vertex\. Uses a direct cross\-product distance

r=∥C​O→×Tk​O→∥∥Tk​O→∥\.r=\\frac\{\\lVert\\vec\{CO\}\\times\\vec\{T\_\{k\}O\}\\rVert\}\{\\lVert\\vec\{T\_\{k\}O\}\\rVert\}\.
The computation yields2/3\\sqrt\{2\}/3without the baseline’s lateral\-edge detour\.

23​✓\\boxed\{\\frac\{\\sqrt\{2\}\}\{3\}\}\\ \\checkmark

### E\.8Trigonometric Cubic

#### Problem\.

For real numbersα,B,C\\alpha,B,C, the zeros ofT⁡\(x\)=x3\+x2\+B​x\+CT\(x\)=x^\{3\}\+x^\{2\}\+Bx\+Caresin2⁡α\\sin^\{2\}\\alpha,cos2⁡α\\cos^\{2\}\\alpha, and−csc2⁡α\-\\csc^\{2\}\\alpha\. ComputeT⁡\(5\)T\(5\)\. Ground Truth:5674\\frac\{567\}\{4\}

#### Base checkpoint \(32,631 tokens\)\.

Introduces an auxiliary variable and derives a long expression forT⁡\(5\)T\(5\), but never proves it is constant\. Switches to numerical sampling arounds≈1\.9s\\approx 1\.9and observes values near129129\. The final answer is a rounded numerical guess rather than a consequence of Vieta’s formulas\.

129×\\boxed\{129\}\\ \\times

#### CCR,N=3N=3\(21,204 tokens\)\.

Keeps the product form

T⁡\(5\)=\(5−r1\)​\(5−r2\)​\(5−r3\)T\(5\)=\(5\-r\_\{1\}\)\(5\-r\_\{2\}\)\(5\-r\_\{3\}\)and substitutes the root triple\(1/2,1/2,−2\)\(1/2,1/2,\-2\)\. The arithmetic is direct and yields5674\\frac\{567\}\{4\}\.

5674​✓\\boxed\{\\frac\{567\}\{4\}\}\\ \\checkmark

### E\.9Product of Factorials

#### Problem\.

Given positive integersa,b,c,da,b,c,dsuch thata\!⋅b\!⋅c\!=d\!a\!\\cdot b\!\\cdot c\!=d\!anda\+b\+c\+d=37a\+b\+c\+d=37, computea​b​c​dabcd\. Ground Truth: 2240

#### Base checkpoint \(32,643 tokens\)\.

Performs a long unstructured enumeration of many quadruples and known factorial identities\. Finds several near misses but fails to reorganize the search around the ratiod\!/c\!d\!/c\!\. The trace reaches the token limit without a final answer\.

No final answer×\\text\{No final answer\}\\ \\times

#### CCR,N=3N=3\(15,458 tokens\)\.

Checksd=16d=16early and computes

16\!14\!=240=2\!⋅5\!\.\\frac\{16\!\}\{14\!\}=240=2\!\\cdot 5\!\.
Verifies both constraints

2\!​5\!​14\!=16\!,2\+5\+14\+16=37\.2\!5\!14\!=16\!,\\qquad 2\+5\+14\+16=37\.
Then computesa​b​c​d=2⋅5⋅14⋅16=2240abcd=2\\cdot 5\\cdot 14\\cdot 16=2240\.

2240​✓\\boxed\{2240\}\\ \\checkmark