ROSS：通过选择性监督重学自生成轨迹

# ROSS: Relearning from Self-Generated Rollouts through Selective Supervision
Source: [https://arxiv.org/html/2609.35954](https://arxiv.org/html/2609.35954)
###### Abstract

Large language model post\-training generates self\-generated rollouts through reinforcement learning and on\-policy distillation, yet this experience is often treated as stale once the policy advances\. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably\. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer\-grained selective supervision\. We introduce*ROSS*\(Relearning from Self\-GeneratedRollouts throughSelectiveSupervision\), which preserves the full historical trajectory as context while applying loss only to selected model\-generated continuations\. Across domain\-specific reinforcement learning, multi\-teacher on\-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering\. On Qwen3\.6\-35B\-A3B, ROSS improves the six\-benchmark MOPD average from 58\.40% to 62\.20% and SWE\-bench Verified from 64\.20% to 68\.40%\. These results show that self\-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine\-tuning \(SFT\), without additional policy rollouts\.

\\setcitestyle

numbers,square,comma,sortcompress

\\fancyhead

\\outfitROSS:Relearning from Self\-Generated Rollouts throughSelectiveSupervisionAllSpark Team\\outfitDate:September 28, 2026

\(a\)Historical experience preserves discovered behaviors\.![Refer to caption](https://arxiv.org/html/2609.35954v1/panel_c_rollout_example.png)\(b\)Successful rollouts contain both useful and uninformative segments\.

Figure 1:Relearning from self\-generated historical rollouts\.\(a\)As the policy evolves, some useful behaviors may become underrepresented in its current rollout distribution\.\(b\)Even when a rollout reaches a successful outcome, it may include redundant reasoning, or unnecessary detours, making full\-trajectory replay suboptimal\.## 1Introduction

Large language models \(LLMs\)\\citepguo2025deepseek,lambert2024tulu,yuan2023scaling,yue2025does are increasingly trained from experience generated by the models themselves\. Reinforcement learning \(RL\)\\citepyue2025does,guo2025deepseek,hou2026single,zheng2025group repeatedly samples rollouts from the current policy and updates it based on the resulting feedback, while on\-policy distillation \(OPD\)\\citepagarwal2024policy, gu2024minillm,lu2025onpolicydistillation,yang2025qwen3,xiao2026mimo,team2026kimi collects student trajectories and applies teacher supervision to the states they visit\. Over training, these procedures produce not only a sequence of model checkpoints, but also a growing record of self\-generated experience, including successful reasoning paths, alternative strategies, recoveries, and other behaviors discovered along the way\. Yet as training advances, historical rollouts are often treated as stale and discarded\. This raises a fundamental question:can a later checkpoint continue to learn from what the model discovered earlier?

This possibility stems from how the policy evolves during training\. As the rollout distribution shifts with the policy, behaviors discovered at earlier stages need not remain reliably expressed at later checkpoints\. As illustrated in Figure[1](https://arxiv.org/html/2609.35954#S0.F1)\(a\), some valid behaviors may become underrepresented even though they remain useful\\citepzhang2026on,wang2025octothinker,yuan2023scaling,dong2023raft\. Historical rollouts can therefore retain behaviors that the current policy no longer reliably recovers, providing complementary training signal beyond its current rollout distribution\\citepzheng2026swe,xiong2024watch,slinko2026step\. Yet this complementarity is not uniform across a trajectory: as Figure[1](https://arxiv.org/html/2609.35954#S0.F1)\(b\) illustrates, even successful rollouts may interleave useful reasoning with mistakes, redundant steps, and unnecessary detours\. Outcome\-level success alone therefore does not guarantee that every intermediate step provides desirable supervision\\citepwang2025octothinker,lightman2024let\. The challenge is thus to identify not only which historical rollouts remain valuable, but also which parts of those rollouts are worth relearning\.

We propose*ROSS*,Relearning from Self\-GeneratedRollouts throughSelectiveSupervision, to address these two levels of selection\. ROSS identifies valuable historical rollouts and selectively supervises useful segments within successful trajectories while preserving the complete trajectory as context\. In this way, the current policy can consolidate previously discovered behaviors without indiscriminately imitating entire trajectories or reinforcing mistakes and unnecessary steps\. We evaluate ROSS across domain\-specific RL, multi\-teacher on\-policy distillation \(MOPD\), and agentic RL, spanning reasoning, coding, instruction following, and software engineering\. Across these settings, ROSS consistently improves the corresponding upstream checkpoints, including a gain from 58\.40% to 62\.20% on the six\-benchmark MOPD average and from 64\.20% to 68\.40% on SWE\-bench Verified\. These results show that self\-rollout training leaves behind not only a stronger policy, but also reusable experience that can be selectively consolidated by later checkpoints\.

Our contributions are as follows:

- •We establish historical self\-generated rollouts as a reusable source of training experience, showing that they can retain useful behaviors under an evolving policy\.
- •We introduce*ROSS*, which selectively consolidates historical experience by supervising informative segments while preserving full\-trajectory context\.
- •We validate ROSS as an additional offline SFT stage after domain\-specific RL, MOPD, and agentic RL, with further gains in math, coding, instruction following, and long\-horizon agentic tasks\.

## 2Preliminary Analysis

To characterize when historical experience is worth revisiting, we consider two complementary dimensions:*compatibility*and*complementarity*\.Compatibilitymeasures how well a historical behavior aligns with the current policy, whilecomplementaritymeasures whether it provides behavioral coverage that the current policy cannot reliably produce\. As illustrated in Figure[2](https://arxiv.org/html/2609.35954#S2.F2), their combination distinguishes useful historical experience from behavior that is either misaligned or already well covered by the current policy\. We therefore examine compatibility through response\-level predictability in Sec\.[2\.1](https://arxiv.org/html/2609.35954#S2.SS1)and complementarity through under\-consolidated success on historically solved problems in Sec\.[2\.2](https://arxiv.org/html/2609.35954#S2.SS2)\.

Figure 2:When is historical experience worth revisiting?Its value depends on*compatibility*with the current policy and*complementarity*to its behavioral coverage\.Figure 3:Historical rollouts contain compatible but under\-consolidated behaviors\.\(a\)Historical rollouts have response\-level NLL comparable to the current policy and lower than external\-teacher trajectories\.\(b\)These historically successful behaviors remain unreliable under the current policy, with meanpass​@​1=37\.7%\\mathrm\{pass@1\}=37\.7\\%\.\(c\)Repeated sampling raises success topass​@​32=99\.0%\\mathrm\{pass@32\}=99\.0\\%, showing that the behaviors remain available but are not reliably expressed\.### 2\.1Compatibility Analysis

#### Historical behaviors remain compatible with the current policy\.

We first ask whether historical rollouts remain compatible with the current policy\. We select 100 mathematical problems and collect one response per problem from three sources: historical rollouts from earlier checkpoints, rollouts from the current policy, and an external teacher, GLM\-5\.2\\citepzeng2026glm\. We measure the token\-level negative log\-likelihood \(NLL\) of each response under the current policy, with lower NLL indicating greater consistency with its output distribution\. As shown in Figure[3](https://arxiv.org/html/2609.35954#S2.F3)\(a\), historical rollouts have substantially lower NLL than external\-teacher responses and remain close to current\-policy rollouts\. Thus, despite being generated by earlier checkpoints, historical behaviors can remain naturally compatible with the current policy, making them plausible sources of supervision\\citepslinko2026step,yuan2023scaling\.

### 2\.2Complementarity Analysis

#### Historical experience preserves underrepresented behaviors\.

We next ask whether historically discovered behaviors remain reliably expressed by the current policy\. We select the100100most difficult training problems and collect3232current\-policy rollouts for each, retaining the9898problems for which at least one historical rollout was successful\. Shown in Figure[3](https://arxiv.org/html/2609.35954#S2.F3)\(c\), the current policy solves9797of these problems, with an averagepass​@​32\\mathrm\{pass@32\}of99\.0%99\.0\\%, showing that the corresponding behaviors remain within its behavioral support\. However, the averagepass​@​1\\mathrm\{pass@1\}is only37\.7%37\.7\\%, and7474of9898problems have a per\-problem success rate of at most50%50\\%\. Historical rollouts therefore preserve behaviors that are compatible with the current policy but not yet reliably expressed\. Combined with their high compatibility, this indicates a*near\-on\-policy*source of complementary experience for consolidating underrepresented behaviors\.

#### Complementarity can also exist at the segment level\.

Useful signal need not span an entire rollout\. Appendix[D](https://arxiv.org/html/2609.35954#A4)provides a concrete example in which a historical rollout reaches the correct count,10071007, by recovering from an intermediate210=10242^\{10\}=1024counting error\. The recovery segment identifies the overlooked constraint and corrects the count, whereas the preceding error is not a desirable imitation target\. Thus, useful and undesirable reasoning can coexist within the same successful rollout, and trajectory\-level correctness alone does not determine which segments should be learned\. Complementarity is therefore not only a property of which rollout to revisit, but also of which segments within it provide useful signal\.

Taken together, historical experience can remain*compatible*with the current policy while providing*complementary*behavioral coverage, sometimes only within specific segments\. These observations motivate ROSS’s two\-level selective supervision\.

## 3Method

ROSS relearns from historical self\-generated rollouts through selective supervision\. Starting from the final checkpoint of the original training run, an outcome verifier first identifies positive trajectories\. A staged annotation procedure then uses an LLM reviewer to assess each candidate\. If a reliable imitation target can be extracted, the reviewer identifies the model\-generated tokens that should receive imitation loss\. Figure[4](https://arxiv.org/html/2609.35954#S3.F4)summarizes the procedure\.

### 3\.1Problem Formulation

Consider a self\-rollout training procedure that produces a sequence of policy checkpoints\{πθk\}k=0T\\\{\\pi\_\{\\theta\_\{k\}\}\\\}\_\{k=0\}^\{T\}and rollout batches\{𝒟k\}k=0T−1\\\{\\mathcal\{D\}\_\{k\}\\\}\_\{k=0\}^\{T\-1\}, whereτ∼πθk\\tau\\sim\\pi\_\{\\theta\_\{k\}\}for everyτ∈𝒟k\\tau\\in\\mathcal\{D\}\_\{k\}\. The original training may use reinforcement learning, on\-policy distillation, or agentic reinforcement learning; ROSS requires only the resulting checkpoints and saved rollouts\. We define the historical experience pool as

ℋ<T=⋃k<T𝒟k,\\mathcal\{H\}\_\{<T\}=\\bigcup\_\{k<T\}\\mathcal\{D\}\_\{k\},\(1\)and initialize relearning from the final checkpointπθT\\pi\_\{\\theta\_\{T\}\}\. Althoughℋ<T\\mathcal\{H\}\_\{<T\}is no longer strictly on\-policy forπθT\\pi\_\{\\theta\_\{T\}\}, its rollouts were generated by earlier checkpoints from the same training run and may contain behaviors thatπθT\\pi\_\{\\theta\_\{T\}\}does not reliably express\.

We represent each rollout asτ=\(z1,…,zL\)\\tau=\(z\_\{1\},\\ldots,z\_\{L\}\), which may encode a single\-turn exchange or an agentic interaction history\. Letat∈\{0,1\}a\_\{t\}\\in\\\{0,1\\\}indicate whetherztz\_\{t\}is a policy\-generated token eligible for imitation; prompts, environment observations, and other exogenous context haveat=0a\_\{t\}=0\. ROSS applies a trajectory\-level selectorg⁡\(τ\)∈\{0,1\}g\(\\tau\)\\in\\\{0,1\\\}and, for retained rollouts, a token\-level supervision maskmt∈\{0,1\}m\_\{t\}\\in\\\{0,1\\\}\. Staged LLM review followed by deterministic boundary and token\-alignment checks produces a mask satisfyingmt≤atm\_\{t\}\\leq a\_\{t\}\.

![Refer to caption](https://arxiv.org/html/2609.35954v1/method.png)Figure 4:Overview of ROSS\.Starting from the final checkpoint, ROSS filters saved rollouts by outcome verification, reviews the positive trajectories, and applies imitation loss only to selected model\-generated tokens while preserving the full history as context\.
### 3\.2Selective Supervision over Historical Rollouts

#### Trajectory\-level selection\.

Letv⁡\(τ\)v\(\\tau\)denote the task outcome verifier, such as an exact\-answer check, executable checker, unit\-test suite, or environment success signal\. It supplies trajectory\-level feedback only\. We retain

ℋ\+=\{τ∈ℋ<T:g\(τ\)=1\},g\(τ\)=𝟏\[v\(τ\)=1\]\.\\mathcal\{H\}^\{\+\}=\\\{\\tau\\in\\mathcal\{H\}\_\{<T\}:g\(\\tau\)=1\\\},\\qquad g\(\\tau\)=\\mathbf\{1\}\[v\(\\tau\)=1\]\.\(2\)

#### Within\-trajectory selection\.

For eachτ∈ℋ\+\\tau\\in\\mathcal\{H\}^\{\+\}, a staged LLM\-based annotation procedureAAuses an LLM reviewer to examine the task, complete trajectory, and verifier or environment evidence\. It returns an audit statusqτq\_\{\\tau\}and disjoint, ordered intervals𝒦τ=\{\[bj,ej\)\}j=1Jτ\\mathcal\{K\}\_\{\\tau\}=\\\{\[b\_\{j\},e\_\{j\}\)\\\}\_\{j=1\}^\{J\_\{\\tau\}\}that contain locally correct, self\-contained, and behaviorally useful model outputs\. We convert these intervals into a token mask

mt\(τ\)=at⋅\[t∈⋃\[b,e\)∈𝒦τ\[b,e\)\]\.m\_\{t\}\(\\tau\)=a\_\{t\}\\cdot\\mathbf\{1\}\\\!\\left\[t\\in\\bigcup\_\{\[b,e\)\\in\\mathcal\{K\}\_\{\\tau\}\}\[b,e\)\\right\]\.\(3\)Proposed targets are independently audited\. A positive verifier outcome alone does not guarantee a valid imitation target: a response may reach the correct final answer despite an invalid derivation\. A candidate is rejected when no valid target can be recovered from the recorded trajectory without retaining a confirmed error or introducing missing reasoning\. The resulting training subset is

ℋROSS=\{τ∈ℋ\+:qτ=accept,𝒦τ≠∅,Validate\(τ,m\(τ\)\)=1\}\.\\mathcal\{H\}\_\{\\mathrm\{ROSS\}\}=\\left\\\{\\tau\\in\\mathcal\{H\}^\{\+\}:q\_\{\\tau\}=\\textsc\{accept\},\\;\\mathcal\{K\}\_\{\\tau\}\\neq\\varnothing,\\;\\text\{\{Validate\}\}\(\\tau,m\(\\tau\)\)=1\\right\\\}\.Here,Validatedenotes the deterministic checks in Algorithm[1](https://arxiv.org/html/2609.35954#alg1)that ensure source consistency and token alignment\. Single\-turn responses use positive\-span extraction, whereas agentic histories use defect localization; both are ultimately compiled into supervised token spans and the same token\-level mask\. Appendix[E](https://arxiv.org/html/2609.35954#A5)details the annotation protocols and core reviewer prompts\.

#### Full context, selective targets\.

Settingmt=0m\_\{t\}=0removes a token from the loss, not from the sequence\. A selected tokenztz\_\{t\}is therefore predicted from the complete original prefixz<tz\_\{<t\}, including earlier mistakes, abandoned attempts, and environment feedback, preserving the state in which the continuation originally occurred\.

### 3\.3ROSS Training Objective

Starting fromθT\\theta\_\{T\}, ROSS performs masked teacher\-forcing on the selected historical rollouts\. Its objective is

ℒROSS​\(θ\)=−∑τ∈ℋROSS∑t=1\|τ\|mt​\(τ\)​log⁡πθ​\(zt∣z<t\)∑τ∈ℋROSS∑t=1\|τ\|mt​\(τ\)\.\\mathcal\{L\}\_\{\\mathrm\{ROSS\}\}\(\\theta\)=\-\\frac\{\\sum\_\{\\tau\\in\\mathcal\{H\}\_\{\\mathrm\{ROSS\}\}\}\\sum\_\{t=1\}^\{\|\\tau\|\}m\_\{t\}\(\\tau\)\\log\\pi\_\{\\theta\}\(z\_\{t\}\\mid z\_\{<t\}\)\}\{\\sum\_\{\\tau\\in\\mathcal\{H\}\_\{\\mathrm\{ROSS\}\}\}\\sum\_\{t=1\}^\{\|\\tau\|\}m\_\{t\}\(\\tau\)\}\.\(4\)Settingmt=atm\_\{t\}=a\_\{t\}for everyτ∈ℋROSS\\tau\\in\\mathcal\{H\}\_\{\\mathrm\{ROSS\}\}recovers ROSS w/o mask, which uses the same post\-annotation subset as ROSS but supervises all eligible policy\-generated tokens\. Positive\-Rollout SFT instead supervises all eligible policy\-generated tokens in all verifier\-positive trajectories inℋ\+\\mathcal\{H\}^\{\+\}\. Appendix[B](https://arxiv.org/html/2609.35954#A2)states the full procedure as Algorithm[1](https://arxiv.org/html/2609.35954#alg1)\.

## 4Experiments

### 4\.1Experimental Setup

#### Models and training\.

We study two forms of self\-generated experience with Qwen3\.6\-35B\-A3B: single\-turn rollouts and agentic trajectories\.For single\-turn experiments, we reuse historical rollouts from independent mathematics and code RL runs, as well as multi\-teacher on\-policy distillation \(MOPD\) spanning mathematics, code, and instruction following\. ROSS starts from the final checkpoint and applies masked SFT to these rollouts\. All single\-turn experiments use a no\-thinking configuration and report results after three SFT epochs, with optimization settings matched within each setting\. Rollout review and mask annotation use GLM\-5\.2 in high\-thinking mode as the LLM judge, which reviews the saved rollouts without generating replacement targets\.For agentic scenarios, we train the model with agentic RL on OpenSWE tasks from daVinci\-Env\\citepfu2026davinci, using a Codex CLI agent\\citepopenai2025codex to interact with containerized repositories through Harbor\\citepharbor2026framework\. The task verifier provides terminal rewards\. Both agentic training and relearning retain thinking traces and use a separate long\-context schedule\. Detailed configurations, rollout windows, and annotation protocols are provided in Appendices[C](https://arxiv.org/html/2609.35954#A3)and[E](https://arxiv.org/html/2609.35954#A5)\.

#### Comparisons\.

We compare against four baselines:*\(i\) Base*, the open\-source checkpoint used to initialize the original RL or MOPD training;*\(ii\) Upstream*, the final checkpoint of the original RL or MOPD run;*\(iii\) Continued RL/MOPD*, which follows the original training procedure for two additional epochs in Math and Code RL, or for 200 additional steps from the 200\-step MOPD checkpoint; and*\(iv\) Positive\-Rollout SFT*, which applies SFT to verifier\-positive historical rollouts\.

#### Evaluation\.

Single\-turn evaluation covers mathematics \(AIME 2025, AIME 2026, HMMT\-November 2025\), code generation \(LCB Gen, OJBench\), and instruction following \(IFBench\)\. Agentic coding is evaluated on SWE\-bench Verified\\citepjimenez2024swebench in thinking mode, with the evaluation scaffold, task environments, tool interfaces, and execution budgets fixed across policies\. Details of the evaluation setup are given in Appendix[C](https://arxiv.org/html/2609.35954#A3)\.

### 4\.2Main Results

Table 1:Main results with Qwen3\.6\-35B\-A3B\.All scores are higher\-is\-better\. \(a\) Math and Code use independent domain\-specific RL runs; Math Avg\. and Code Avg\. are the corresponding domain means\. \(b\) MOPD spans all three domains, and Avg\. is the mean of its six benchmarks\. All relearning methods start from the corresponding Upstream checkpoint; bold marks the best result among them, including ties\.\(a\) Domain\-specific RL

Math RLCode RLMethodAIME 25AIME 26HMMT\-Nov\.Math Avg\.LCB GenOJBenchCode Avg\.Base71\.2576\.5670\.6272\.8157\.2422\.2039\.72Upstream74\.8477\.5073\.2375\.1958\.9525\.2242\.09Continued RL74\.6478\.8572\.8175\.4357\.2427\.3742\.31Positive\-Rollout SFT75\.4779\.0673\.4475\.9958\.1923\.4940\.84ROSS \(ours\)76\.4679\.9074\.5876\.9861\.0527\.3744\.21

\(b\) MOPD

MathCodeIFMethodAIME 25AIME 26HMMT\-Nov\.LCB GenOJBenchIFBenchAvg\.Base71\.2576\.5670\.6257\.2422\.2034\.1055\.33Upstream72\.8177\.6071\.2557\.4326\.7244\.5858\.40Continued MOPD73\.3978\.2372\.6057\.7127\.5945\.4559\.16Positive\-Rollout SFT74\.0677\.6669\.1756\.8626\.9441\.4957\.70ROSS \(ours\)77\.7680\.4276\.7760\.7629\.7447\.7662\.20

We evaluate ROSS on historical rollouts from domain\-specific RL and MOPD, comparing it with the upstream checkpoints and baseline training strategies\. Table[1](https://arxiv.org/html/2609.35954#S4.T1)summarizes the results\. Across both settings, ROSS consistently achieves the best performance among all relearning methods\. After domain\-specific RL, ROSS achieves the highest average on both Math and Code, improving from 75\.19 to 76\.98 and from 42\.09 to 44\.21, respectively\. After MOPD, ROSS improves the six\-benchmark average from 58\.40 to 62\.20, outperforming Continued MOPD and Positive\-Rollout SFT\. These results show that historical rollouts retain useful learning signal beyond their original training stage\. While continued RL/MOPD further explores the policy’s behavioral frontier, ROSS complements this process by consolidating useful behaviors accumulated throughout training that may remain unreliably expressed by the current checkpoint\.

We further examine whether ROSS introduces degradation outside the domain targeted by domain\-specific RL\. Table[9](https://arxiv.org/html/2609.35954#A7.T9)in Appendix[G](https://arxiv.org/html/2609.35954#A7)compares each ROSS\-retrained checkpoint with its corresponding upstream RL checkpoint on mathematics, code generation, instruction following, and agentic tool use\. ROSS preserves the gains of the domain\-specific checkpoints while improving several off\-domain capabilities\. After Math RL, it raises the Avg\. Code from 41\.21 to 44\.24 and IFBench from 33\.30 to 34\.60; after Code RL, it improves Avg\. Math from 74\.19 to 75\.03\. ROSS also improves multi\-turn agentic tool use, raising the overall BFCL score from 44\.12 to 45\.62 and from 46\.25 to 49\.38, respectively\. These cross\-domain gains are consistent with the near\-on\-policy compatibility of historical rollouts demonstrated in Section[2\.1](https://arxiv.org/html/2609.35954#S2.SS1), suggesting that relearning from them can consolidate useful experience without sacrificing capabilities acquired by the current policy\.

### 4\.3Relearning from Agentic Trajectories

Table 2:Agentic trajectory reuse\.Δ\\Deltais relative to Upstream\.MethodSWE\-bench Verified↑\(Δ\)\\uparrow\(\\Delta\)Base60\.80Upstream64\.20Positive\-Rollout SFT65\.20\(\+1\.00\)\(\+1\.00\)ROSS \(ours\)68\.40\(\+4\.20\)\\mathbf\{\(\+4\.20\)\}The results above establish the value of historical experience in single\-turn rollouts\. We next examine whether this benefit extends to long\-horizon agent–environment interaction\. We reuse successful OpenSWE trajectories from the agentic RL run and evaluate on SWE\-bench Verified, with the training protocol described in Appendix[C\.4](https://arxiv.org/html/2609.35954#A3.SS4)\. As shown in Table[2](https://arxiv.org/html/2609.35954#S4.T2), ROSS improves the resolved\-issue rate from 64\.20 to 68\.40, yielding a 4\.20\-point gain over the Upstream RL checkpoint\. The gain shows that ROSS generalizes beyond single\-turn rollouts to long\-horizon agentic trajectories, where useful behaviors are embedded in extended interaction histories\.

### 4\.4What Drives Relearning from Historical Rollouts?

Having shown that historical rollouts remain useful across both single\-turn and agentic settings, we analyze two factors behind these gains: whether token\-level masking adds value beyond trajectory\-level filtering, and whether relearning depends on inheriting the Upstream parameter updates\.

#### The Role of Trajectory Filtering and Selective Supervision

We disentangle trajectory filtering from token\-level selective supervision\. Table[3](https://arxiv.org/html/2609.35954#S4.T3)compares Positive\-Rollout SFT on all verifier\-positive rollouts, ROSS w/o mask on the rollouts retained by ROSS with all eligible tokens supervised, and full ROSS\. We find that:\(i\)Trajectory filtering already provides a clear benefit\. ROSS w/o mask reverses the degradation of Positive\-Rollout SFT on Code, improving over Upstream by 0\.19 points on LCB Gen and 1\.72 points on OJBench, while raising the MOPD average from 57\.70 to 58\.49, slightly above Upstream checkpoint\.\(ii\)Selective supervision accounts for the remaining gains\. With the replay set fixed, ROSS further improves over ROSS w/o mask by 3\.71 points on MOPD, 1\.91 on LCB Gen, 0\.43 on OJBench, and 0\.66 on Math, showing that selectively supervising informative segments is more effective than imitating all eligible tokens\.

Table 3:Effect of trajectory filtering and token\-level masking\.Subscripts show percentage\-point changes from Upstream; green/red denote gains/losses and bold marks the best score\. ROSS w/o mask and ROSS use identical examples and differ only by token\-level masking\.\(a\) Domain\-specific RL

Math RLCode RLMethodAIME 25AIME 26HMMT\-Nov\.LCB GenOJBenchUpstream74\.8477\.5073\.2358\.9525\.22Positive\-Rollout SFT75\.47\+0\.63\+0\.6379\.06\+1\.56\+1\.5673\.44\+0\.21\+0\.2158\.19−0\.76\-0\.7623\.49−1\.73\-1\.73ROSS w/o mask76\.67\+1\.83\+1\.8379\.06\+1\.56\+1\.5673\.23\+0\.00\+0\.0059\.14\+0\.19\+0\.1926\.94\+1\.72\+1\.72ROSS \(ours\)76\.46\+1\.62\+1\.6279\.90\+2\.40\+2\.4074\.58\+1\.35\+1\.3561\.05\+2\.10\+2\.1027\.37\+2\.15\+2\.15
\(b\) Multi\-teacher on\-policy distillation

MethodAIME 25AIME 26HMMT\-Nov\.LCB GenOJBenchIFBenchAvg\.Upstream72\.8177\.6071\.2557\.4326\.7244\.5858\.40Positive\-Rollout SFT74\.06\+1\.25\+1\.2577\.66\+0\.06\+0\.0669\.17−2\.08\-2\.0856\.86−0\.57\-0\.5726\.94\+0\.22\+0\.2241\.49−3\.09\-3\.0957\.70−0\.70\-0\.70ROSS w/o mask73\.28\+0\.47\+0\.4777\.55−0\.05\-0\.0571\.67\+0\.42\+0\.4258\.38\+0\.95\+0\.9527\.59\+0\.87\+0\.8742\.45−2\.13\-2\.1358\.49\+0\.09\+0\.09ROSS \(ours\)77\.76\+4\.95\+4\.9580\.42\+2\.82\+2\.8276\.77\+5\.52\+5\.5260\.76\+3\.33\+3\.3329\.74\+3\.02\+3\.0247\.76\+3\.18\+3\.1862\.20\+3\.80\+3\.80

Table 4:ROSS reaches similar scores from different initializations\.Within each setting, the two runs use identical historical trajectories, masks, and SFT configurations\. Math is Avg\. Math, Code is LCB Gen, and IF is IFBench\.Math RLCode RLMOPDMath↑\\uparrowCode↑\\uparrowMath↑\\uparrowCode↑\\uparrowIF↑\\uparrowBase72\.8157\.2472\.8157\.2434\.10Upstream75\.1958\.9573\.8957\.4344\.58After ROSS on identical trajectories and masksInitialized from Base77\.0961\.3377\.9561\.2447\.18Initialized from Upstream76\.98−0\.11\\mathbf\{\-0\.11\}61\.05−0\.28\\mathbf\{\-0\.28\}78\.32\+0\.37\\mathbf\{\+0\.37\}60\.76−0\.48\\mathbf\{\-0\.48\}47\.76\+0\.58\\mathbf\{\+0\.58\}
Table 5:Lower entropy does not imply less excluded supervision\.Math \(blue\) and Code \(green\) over 100 RL rollout steps: \(a\) logged entropy; \(b\) TER among retained trajectories\. Thin curves show per\-step values; thick curves show eight\-step moving averages\.
#### Relearning Across Initializations

The previous comparison shows that ROSS’s token\-level mask adds value beyond trajectory\-level filtering\. We further examine whether the value of historical rollouts depends on starting from the Upstream checkpoint, or whether the same recorded experience can transfer useful behaviors to a different initialization\. To test this, we apply identical historical trajectories, supervision masks, and SFT configurations starting from either Base, before the original post\-training run, or the corresponding Upstream checkpoint\. As shown in Table[5](https://arxiv.org/html/2609.35954#S4.T5), Base\-initialized ROSS achieves performance comparable to Upstream\-initialized ROSS across all evaluated settings, despite inheriting none of the original RL/MOPD parameter updates\. This result suggests that historical rollouts are useful not only for further refining the policy that generated them; more importantly, they preserve behaviors discovered during training that can transfer across initializations and be consolidated into a different checkpoint\. The rollout history therefore provides reusable learning value across initializations, beyond what is carried forward through the Upstream parameters alone\.

### 4\.5Characterizing Excluded Supervision

The controlled comparisons in Section[4\.4](https://arxiv.org/html/2609.35954#S4.SS4.SSS0.Px1)show that token\-level masking contributes beyond trajectory\-level filtering\. We therefore examine what ROSS removes from the training loss and whether such excluded supervision diminishes as RL progresses\. We analyze both the semantic content of masked regions and their prevalence over the course of training\.

#### The excluded content differs across domains\.

We use GLM\-5\.2 to classify masked excerpts from 1,185 sampled trajectories, assigning one primary category to each trajectory\. In Math, mistakes followed by a visible correction dominate both early and late training samples \(72\.7%72\.7\\%and68\.3%68\.3\\%\)\. In Code, redundant exploration is the largest category and becomes more prevalent from early to late training \(67\.9%67\.9\\%to76\.6%76\.6\\%\)\. These patterns highlight two common forms of undesirable supervision within globally successful rollouts: Math trajectories often recover from explicit intermediate errors, whereas Code trajectories more often reach successful outcomes through exploratory detours\. Appendix[F](https://arxiv.org/html/2609.35954#A6)provides the full category distribution, definitions, and sampling protocol\.

#### Excluded supervision persists as RL progresses\.

We next examine whether the need for masking naturally diminishes as the policy evolves\. Over the first 100 rollout steps of the Math and Code RL runs, we measure the token\-weighted*target exclusion rate*\(TER\) among retained, verifier\-positive responses𝒟s\\mathcal\{D\}\_\{s\}at stepss, defined asTER⁡\(s\)=∑τ∈𝒟s∑t=1Lτ\(1−mτ,t\)∑τ∈𝒟sLτ\\mathrm\{TER\}\(s\)=\\frac\{\\sum\_\{\\tau\\in\\mathcal\{D\}\_\{s\}\}\\sum\_\{t=1\}^\{L\_\{\\tau\}\}\(1\-m\_\{\\tau,t\}\)\}\{\\sum\_\{\\tau\\in\\mathcal\{D\}\_\{s\}\}L\_\{\\tau\}\}, whereLτL\_\{\\tau\}counts response\-content tokens andmτ,tm\_\{\\tau,t\}indicates whether a token overlaps a selected span; prompt and chat\-template tokens are excluded\. Policy entropy decreases throughout both RL runs, but the amount of excluded supervision does not vanish with training\. From the early to late window, TER decreases only modestly from11\.60%11\.60\\%to9\.27%9\.27\\%in Math, while increasing from22\.65%22\.65\\%to28\.07%28\.07\\%in Code \(Figure[5](https://arxiv.org/html/2609.35954#S4.T5)\)\. Thus, increasing policy certainty does not make successful rollouts uniformly suitable for imitation\. For outcome\-level RL objectives such as GRPO, positive trajectory\-level feedback can still reinforce local mistakes, abandoned reasoning, or redundant exploration, motivating retrospective masking when historical rollouts are reused\.

Appendix[H](https://arxiv.org/html/2609.35954#A8)provides complementary evidence from a separate 4B Math RL run, where Upstream develops a repetitive\-answer mode\. ROSS suppresses this behavior while improving Avg\. Math from52\.05%52\.05\\%to63\.65%63\.65\\%, showing that selective relearning can both consolidate useful historical behaviors and avoid reinforcing undesirable modes\.

## 5Related Work

#### Learning from Self\-Generated Experience

Recent work studies improving language models through self\-generated supervision\\citepzhang2024rest, yuan2023scaling, zelikman2022star, dong2023raft\. Rejection Sampling Fine\-Tuning \(RFT\)\\citepyuan2023scaling generates candidate solutions and fine\-tunes on correct reasoning paths, while RAFT\\citepdong2023raft ranks samples with a reward model and fine\-tunes on high\-quality responses\. Iterative self\-training extends this paradigm: ReST\-MCTS⋆\\citepzhang2024rest combines process\-reward\-guided tree search with iterative self\-training, SCoRe learns self\-correction from multi\-turn self\-generated interactions through online reinforcement learning\\citepkumar2025training, and RLoop reuses successful trajectories from intermediate policies to initialize later RL stages\\citepzhiyuan2025rloop\. RIFT reweights positive and negative self\-generated trajectories with scalar rewards rather than discarding negative samples\\citepliu2026rift\. Despite these advances, existing methods largely treat a rollout as an indivisible supervision unit, selecting, refining, or reweighting trajectories by overall outcomes or rewards\. ROSS instead asks whether behaviors within a successful self\-generated rollout should be learned uniformly, motivating finer\-grained selective supervision\.

#### Selecting Useful Supervision from Rollouts

Beyond selecting successful trajectories as a whole, recent work explores how to identify which parts of a rollout are useful for learning\. Process supervision provides step\-level feedback for reasoning\\citeplightman2024let, while ProcessBench\\citepzheng2025processbench and ThinkPRM\\citepkhalifa2025process study identifying and verifying erroneous reasoning steps\. For interactive agents, IPR\\citepxiong2024watch performs step\-level process refinement by comparing generated actions with expert trajectories\. STeP\\citepchen2025training and Step Rejection Fine\-Tuning \(SRFT\)\\citepslinko2026step further use loss masking to avoid imitating erroneous or undesirable steps\. Most closely related, SWE\-Prime\\citepzheng2026swe explicitly selects useful segments from successful software\-engineering trajectories based on their contribution, learnability, and risk, while keeping the full trajectory as context during SFT\\citepzheng2026swe\. These works demonstrate the value of selective supervision within rollouts\. ROSS focuses on a complementary setting: successful historical rollouts generated by intermediate policies during upstream training, where the central question is not only whether a rollout is worth keeping, but which behaviors within it are worth reinforcing\. ROSS therefore applies supervision selectively to useful portions of successful self\-generated rollouts, avoiding uniform imitation of the full trajectory\.

## 6Conclusion

In this work, we show that historical self\-generated rollouts from LLM post\-training are not merely transient artifacts to be discarded, but a reusable source of behavioral experience\. To avoid imitating intermediate mistakes and redundant exploration present in successful rollouts, we proposeROSS, which preserves the complete historical trajectory as context while supervising only informative, verified segments\. Across domain\-specific RL, MOPD, and agentic RL, ROSS consistently outperforms both upstream checkpoints and competing relearning baselines, while preserving or even enhancing out\-of\-domain capabilities\. These results highlight a broader post\-training paradigm: systematically consolidating the reusable behavioral footprint accumulated during policy evolution\.

## References

- \[Agarwal et al\.\(2024\)Agarwal, Vieillard, Zhou, Stanczyk, Ramos Garea, Geist, and Bachem\]Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem\.On\-policy distillation of language models: Learning from self\-generated mistakes\.In*International Conference on Learning Representations*, volume 2024, pp\. 21246–21263, 2024\.
- \[Chen et al\.\(2025\)Chen, Xu, Wang, Zhang, and Mao\]Yihan Chen, Benfeng Xu, Xiaorui Wang, Yongdong Zhang, and Zhendong Mao\.Training llm\-based agents with synthetic self\-reflected trajectories and partial masking\.*arXiv preprint arXiv:2505\.20023*, 2025\.
- \[Dong et al\.\(2023\)Dong, Xiong, Goyal, Zhang, Chow, Pan, Diao, Zhang, Shum, and Zhang\]Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang\.Raft: Reward ranked finetuning for generative foundation model alignment\.*arXiv preprint arXiv:2304\.06767*, 2023\.
- \[Fu et al\.\(2026\)Fu, Wu, Wu, Peng, Huang, Sun, Zeng, Jiang, Zhang, Li, Hu, Liu, Hou, and Liu\]Dayuan Fu, Shenyu Wu, Yunze Wu, Zerui Peng, Yaxing Huang, Jie Sun, Ji Zeng, Mohan Jiang, Lin Zhang, Yukun Li, Jiarui Hu, Liming Liu, Jinlong Hou, and Pengfei Liu\.daVinci\-Env: Open SWE environment synthesis at scale\.*arXiv preprint arXiv:2603\.13023*, 2026\.URL[https://arxiv\.org/abs/2603\.13023](https://arxiv.org/abs/2603.13023)\.
- \[Gu et al\.\(2024\)Gu, Dong, Wei, and Huang\]Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang\.Minillm: Knowledge distillation of large language models\.In*International Conference on Learning Representations*, volume 2024, pp\. 32694–32717, 2024\.
- \[Guo et al\.\(2025\)Guo, Yang, Zhang, Song, Wang, Zhu, Xu, Zhang, Ma, Bi, et al\.\]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al\.Deepseek\-r1: Incentivizing reasoning capability in llms via reinforcement learning\.*arXiv preprint arXiv:2501\.12948*, 2025\.
- \[Harbor Framework Team\(2026\)\]Harbor Framework Team\.Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026\.URL[https://www\.harborframework\.com/](https://www.harborframework.com/)\.
- \[Hou et al\.\(2026\)Hou, Li, Tang, and Dong\]Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong\.Single\-rollout asynchronous optimization for agentic reinforcement learning\.*arXiv preprint arXiv:2607\.07508*, 2026\.
- \[Jimenez et al\.\(2024\)Jimenez, Yang, Wettig, Yao, Pei, Press, and Narasimhan\]Carlos E\. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R\. Narasimhan\.SWE\-bench: Can language models resolve real\-world GitHub issues?In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=VTF8yNQM66](https://openreview.net/forum?id=VTF8yNQM66)\.
- \[Khalifa et al\.\(2025\)Khalifa, Agarwal, Logeswaran, Kim, Peng, Lee, Lee, and Wang\]Muhammad Khalifa, Rishabh Agarwal, Lajanugen Logeswaran, Jaekyeom Kim, Hao Peng, Moontae Lee, Honglak Lee, and Lu Wang\.Process reward models that think\.*arXiv preprint arXiv:2504\.16828*, 2025\.
- \[Kumar et al\.\(2025\)Kumar, Zhuang, Agarwal, Su, Co\-Reyes, Singh, Baumli, Iqbal, Bishop, Roelofs, Zhang, McKinney, Shrivastava, Paduraru, Tucker, Precup, Behbahani, and Faust\]Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co\-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust\.Training language models to self\-correct via reinforcement learning\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=CjwERcAU7w](https://openreview.net/forum?id=CjwERcAU7w)\.
- \[Lambert et al\.\(2024\)Lambert, Morrison, Pyatkin, Huang, Ivison, Brahman, Miranda, Liu, Dziri, Lyu, et al\.\]Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al\.Tulu 3: Pushing frontiers in open language model post\-training\.*arXiv preprint arXiv:2411\.15124*, 2024\.
- \[Li et al\.\(2025\)Li, Guo, Yang, Xu, Wu, and He\]Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He\.CodeI/O: Condensing reasoning patterns via code input\-output prediction\.*arXiv preprint arXiv:2502\.07316*, 2025\.URL[https://arxiv\.org/abs/2502\.07316](https://arxiv.org/abs/2502.07316)\.
- \[Lightman et al\.\(2024\)Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, and Cobbe\]Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe\.Let’s verify step by step\.In*International Conference on Learning Representations*, volume 2024, pp\. 39578–39601, 2024\.
- \[Liu et al\.\(2026\)Liu, Liu, Zhong, and Yuan\]Zehua Liu, Shuqi Liu, Tao Zhong, and Mingxuan Yuan\.Rift: Repurposing negative samples via reward\-informed fine\-tuning\.In*Findings of the Association for Computational Linguistics: ACL 2026*, pp\. 14399–14415, 2026\.
- \[Lu & Lab\(2025\)Lu and Lab\]Kevin Lu and Thinking Machines Lab\.On\-policy distillation\.*Thinking Machines Lab: Connectionism*, 2025\.doi:10\.64434/tml\.20251026\.https://thinkingmachines\.ai/blog/on\-policy\-distillation\.
- \[OpenAI\(2025\)\]OpenAI\.Codex CLI\.GitHub repository, 2025\.URL[https://github\.com/openai/codex](https://github.com/openai/codex)\.
- \[Slinko et al\.\(2026\)Slinko, Zavidnyi, Bogomolov, and Zharov\]Igor Slinko, Ilia Zavidnyi, Egor Bogomolov, and Yaroslav Zharov\.Step rejection fine\-tuning: A practical distillation recipe\.*arXiv preprint arXiv:2605\.10674*, 2026\.
- \[Team et al\.\(2026\)Team, Bai, Bai, Bao, Cai, Cai, Cao, Cao, Chai, Charles, et al\.\]Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, Jianfeng Cai, Xinyuan Cai, Peizhou Cao, Yuxuan Cao, Ziwei Chai, Y Charles, et al\.Kimi k3: Open frontier intelligence\.*arXiv preprint arXiv:2607\.24653*, 2026\.
- \[Wang et al\.\(2025\)Wang, Zhou, Li, and Liu\]Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu\.Octothinker: Mid\-training incentivizes reinforcement learning scaling\.*arXiv preprint arXiv:2506\.20512*, 2025\.
- \[Xiao et al\.\(2026\)Xiao, Xia, Yang, Gao, Shen, Zhang, He, Lou, Luo, Wang, et al\.\]Bangjun Xiao, Bingquan Xia, Bo Yang, Bofei Gao, Bowen Shen, Chen Zhang, Chenhong He, Chiheng Lou, Fuli Luo, Gang Wang, et al\.Mimo\-v2\-flash technical report\.*arXiv preprint arXiv:2601\.02780*, 2026\.
- \[Xiong et al\.\(2024\)Xiong, Song, Zhao, Wu, Wang, Wang, Li, Peng, and Li\]Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li\.Watch every step\! llm agent learning via iterative step\-level process refinement\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp\. 1556–1572, 2024\.
- \[Yang et al\.\(2025\)Yang, Li, Yang, Zhang, Hui, Zheng, Yu, Gao, Huang, Lv, et al\.\]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al\.Qwen3 technical report\.*arXiv preprint arXiv:2505\.09388*, 2025\.
- \[Yang et al\.\(2026\)Yang, Liu, Chen, Dai, Wang, Lin, Lee, Chen, Jiang, He, et al\.\]Zhuolin Yang, Zihan Liu, Yang Chen, Wenliang Dai, Boxin Wang, Sheng\-Chieh Lin, Chankyu Lee, Yangyi Chen, Dongfu Jiang, Jiafan He, et al\.Nemotron\-cascade 2: Post\-training LLMs with cascade RL and multi\-domain on\-policy distillation\.*arXiv preprint arXiv:2603\.19220*, 2026\.URL[https://arxiv\.org/abs/2603\.19220](https://arxiv.org/abs/2603.19220)\.
- \[Yu et al\.\(2025\)Yu, Zhang, Zhu, Yuan, Zuo, Yue, Dai, Fan, Liu, Liu, et al\.\]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al\.DAPO: An open\-source LLM reinforcement learning system at scale\.*arXiv preprint arXiv:2503\.14476*, 2025\.URL[https://arxiv\.org/abs/2503\.14476](https://arxiv.org/abs/2503.14476)\.
- \[Yuan et al\.\(2023\)Yuan, Yuan, Li, Dong, Lu, Tan, Zhou, and Zhou\]Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou\.Scaling relationship on learning mathematical reasoning with large language models\.*arXiv preprint arXiv:2308\.01825*, 2023\.
- \[Yue et al\.\(2025\)Yue, Chen, Lu, Zhao, Wang, Yue, Song, and Huang\]Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang\.Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?In*2nd AI for Math Workshop @ ICML 2025*, 2025\.URL[https://openreview\.net/forum?id=upehLVgq1b](https://openreview.net/forum?id=upehLVgq1b)\.
- \[Zelikman et al\.\(2022\)Zelikman, Wu, Mu, and Goodman\]Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman\.Star: Bootstrapping reasoning with reasoning\.*Advances in Neural Information Processing Systems*, 35:15476–15488, 2022\.
- \[Zeng et al\.\(2026\)Zeng, Lv, Hou, Du, Zheng, Chen, Yin, Ge, Huang, Xie, et al\.\]Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al\.Glm\-5: from vibe coding to agentic engineering\.*arXiv preprint arXiv:2602\.15763*, 2026\.
- \[Zhang et al\.\(2026\)Zhang, Neubig, and Yue\]Charlie Zhang, Graham Neubig, and Xiang Yue\.On the interplay of pre\-training, mid\-training, and RL on reasoning language models\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=TBaUfO9znF](https://openreview.net/forum?id=TBaUfO9znF)\.
- \[Zhang et al\.\(2024\)Zhang, Zhoubian, Hu, Yue, Dong, and Tang\]Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang\.Rest\-mcts\*: Llm self\-training via process reward guided tree search\.*Advances in Neural Information Processing Systems*, 37:64735–64772, 2024\.
- \[Zheng et al\.\(2025a\)Zheng, Liu, Li, Chen, Yu, Gao, Dang, Liu, Men, Yang, et al\.\]Chujie Zheng, Shixuan Liu, Mingze Li, Xiong\-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al\.Group sequence policy optimization\.*arXiv preprint arXiv:2507\.18071*, 2025a\.
- \[Zheng et al\.\(2025b\)Zheng, Zhang, Zhang, Lin, Lu, Yu, Liu, Zhou, and Lin\]Chujie Zheng, Zhenru Zhang, Beichen Zhang, Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin\.Processbench: Identifying process errors in mathematical reasoning\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 1009–1024, 2025b\.
- \[Zheng et al\.\(2026\)Zheng, Ye, Wang, Ye, Zhang, Shi, Liu, Ma, Yu, and Zheng\]Dewu Zheng, Ruizhe Ye, Yanlin Wang, Yang Ye, Hongyu Zhang, Ensheng Shi, Xilin Liu, Yuchi Ma, Jianxing Yu, and Zibin Zheng\.Swe\-prime: Fewer trajectories, better performance\.*arXiv preprint arXiv:2608\.27449*, 2026\.
- \[Zhiyuan et al\.\(2025\)Zhiyuan, Liu, Yin, Zhang, Huang, and Qiu\]Zeng Zhiyuan, Jiashuo Liu, Zhangyue Yin, Ge Zhang, Wenhao Huang, and Xipeng Qiu\.Rloop: An self\-improving framework for reinforcement learning with iterative policy initialization\.*arXiv preprint arXiv:2511\.04285*, 2025\.
- \[Zhu et al\.\(2025\)Zhu, Xie, Lv, and slime Contributors\]Zilin Zhu, Chengxing Xie, Xin Lv, and slime Contributors\.slime: An LLM post\-training framework for RL scaling\.GitHub repository, 2025\.URL[https://github\.com/THUDM/slime](https://github.com/THUDM/slime)\.

## Appendix AContributors

Authors are listed in order of contribution\.

Zhiwei Zhang111Equal contribution\., Huayu Deng111Equal contribution\., Fei Zhao222Project lead\., Jiayan Fu, Bin Liang, Kam\-Fai Wong, Mu Chuan\.33footnotetext:Corresponding authors\.

## Appendix BROSS Procedure

Algorithm[1](https://arxiv.org/html/2609.35954#alg1)summarizes the relearning stage described in Section[3](https://arxiv.org/html/2609.35954#S3)\. The task verifier defines the trajectory\-level selectorgg, the LLM reviewerAAproposes and audits within\-trajectory imitation targets, and all source\-boundary resolution and mask compilation steps are deterministic\.

Algorithm 1ROSS: Relearning from Self\-Generated Rollouts through Selective Supervision1:Checkpoints

\{πθk\}k=0T\\\{\\pi\_\{\\theta\_\{k\}\}\\\}\_\{k=0\}^\{T\}, saved rollouts

\{𝒟k\}k<T\\\{\\mathcal\{D\}\_\{k\}\\\}\_\{k<T\}, outcome selector

gg, reviewer

AA
2:

𝒮←∅\\mathcal\{S\}\\leftarrow\\varnothing
3:for

τ∈⋃k<T𝒟k\\tau\\in\\bigcup\_\{k<T\}\\mathcal\{D\}\_\{k\}do

4:if

g⁡\(τ\)=0g\(\\tau\)=0then

5:continue

6:endif

7:

𝒦τ,qτ←A⁡\(τ\)\\mathcal\{K\}\_\{\\tau\},q\_\{\\tau\}\\leftarrow A\(\\tau\)⊳\\trianglerightpropose and audit imitation targets

8:if

qτ=acceptq\_\{\\tau\}=\\textsc\{accept\}and

𝒦τ≠∅\\mathcal\{K\}\_\{\\tau\}\\neq\\varnothingthen

9:

m⁡\(τ\)←CompileMask​\(τ,𝒦τ\)m\(\\tau\)\\leftarrow\\textsc\{CompileMask\}\(\\tau,\\mathcal\{K\}\_\{\\tau\}\)
10:if

Validate​\(τ,m​\(τ\)\)\\textsc\{Validate\}\(\\tau,m\(\\tau\)\)then

11:

𝒮←𝒮∪\{\(τ,m⁡\(τ\)\)\}\\mathcal\{S\}\\leftarrow\\mathcal\{S\}\\cup\\\{\(\\tau,m\(\\tau\)\)\\\}
12:endif

13:endif

14:endfor

15:

θ←θT\\theta\\leftarrow\\theta\_\{T\}
16:Update

θ\\thetaon

𝒮\\mathcal\{S\}using Eq\.[4](https://arxiv.org/html/2609.35954#S3.E4)

17:return

πθROSS\\pi\_\{\\theta\_\{\\mathrm\{ROSS\}\}\}

## Appendix CTraining and Evaluation Details

### C\.1Historical Data and Controlled Comparisons

The main experiments use Qwen3\.6\-35B\-A3B, a mixture\-of\-experts model with approximately 3B active parameters\. The single\-turn upstream training pools contain 3,000 Math prompts from DAPO\-Math\-17K\\citepyu2025dapo, 12,000 Code prompts from CodeI/O\\citepli2025codeio, and 10,000 instruction\-following prompts from the data released with Nemotron\-Cascade 2\\citepyang2026nemotroncascade2\. Domain\-specific RL uses the corresponding Math or Code pool, whereas MOPD spans all three domains\. ROSS reuses the saved model responses rather than obtaining replacement solutions from the annotation model\. Table[6](https://arxiv.org/html/2609.35954#A3.T6)specifies the rollout windows and dataset sizes\. The additional Qwen3\.5\-4B case is described in Appendix[H](https://arxiv.org/html/2609.35954#A8)\.

Table 6:Historical data used for relearning\.ROSS w/o mask and ROSS share the retained examples; only the token\-level loss mask differs\. Positive\-Rollout SFT uses all verifier\-positive examples in the corresponding window\.SettingInitialization / rollout windowPositive\-RolloutSFT examplesROSS examplesMath RLIteration 47 / steps 1–4835,70027,848Code RLIteration 187 / steps 1–187152,454144,702MOPDStep 199 / first 200 steps42,55831,228The comparisons in Section[4\.4](https://arxiv.org/html/2609.35954#S4.SS4.SSS0.Px1)follow three sequential contrasts: replay \(Positive\-Rollout SFT minus Upstream\), filtering \(ROSS w/o mask minus Positive\-Rollout SFT\), and masking \(ROSS minus ROSS w/o mask\)\. The masking contrast fixes the examples and input context; filtering changes both dataset composition and size\. Dataset and mask sizes also determine the number of supervised tokens per epoch\.

### C\.2Optimization and Infrastructure

We use the slime training stack\\citepslime2025framework\. Upstream Math RL runs asynchronous rollout generation and policy updates with GRPO advantages, no group\-standard\-deviation normalization, a PPO\-clipped surrogate, and IcePop mismatch correction\. Its recorded configuration uses 128 prompts per rollout step, eight responses per prompt, an update batch of 256, temperature 1\.0, and a 10,240\-token response limit\. Adam uses a constant learning rate of10−610^\{\-6\},\(β1,β2\)=\(0\.9,0\.98\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.98\), weight decay 0\.1, and policy clipping at 0\.2\. The launch allocates four eight\-GPU training nodes and two eight\-GPU rollout nodes\. These upstream settings are distinct from the subsequent SFT configuration\.

Table 7:35B relearning configurations\.Single\-turn Math, Code, and MOPD rollouts use no\-thinking responses; agentic rollouts retain thinking traces and use a longer context and separate schedule\. Within each setting, ROSS and its matched full\-supervision control share the listed configuration\.ParameterSingle\-turn rollouts\(no\-thinking\)Agentic trajectories\(thinking\)OptimizerAdam,β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.98\\beta\_\{2\}=0\.98Peak / minimum learning rate4×10−64\\times 10^\{\-6\}/2×10−72\\times 10^\{\-7\}Schedule / warmup fractionCosine / 0\.05Weight decay0\.1Global batch size128Maximum sequence length \(tokens\)32,76865,536Configured epochs38Tensor / pipeline / context parallelism2 / 2 / 12 / 1 / 8Expert / expert\-tensor parallelism2 / 18 / 1Relearning preserves the saved response text and applies the accepted supervision mask to eligible assistant positions\. The annotation model supplies selection decisions, not new target responses; Appendix[E](https://arxiv.org/html/2609.35954#A5)describes proposal, audit, and mask compilation\. For the single\-turn rollout experiments, we report epoch\-three endpoints consistently rather than selecting each method’s best test score\. The agentic configuration is described separately below\.

### C\.3Single\-Turn Rollout Evaluation Protocol

The Math, Code, and MOPD evaluations use the no\-thinking inference configuration and a 16,384\-token output limit\. AIME 2025 and AIME 2026 average accuracy over 64 generations per problem; HMMT\-November 2025 averages over 32\. Avg\. Math is their unweighted mean\. LCB Gen is the code\-generation pass@1 score averaged over six runs, excluding execution and output\-prediction subtasks\. OJBench reports the overall C\+\+/Python aggregate\. IFBench measures instruction following\. The additional BFCL multi\-turn evaluation reports overall and subset scores in Appendix[G](https://arxiv.org/html/2609.35954#A7)\. Scores are percentages and differences are percentage points\. The 32K diagnostic for the 4B model is an explicitly separate output\-budget control\.

### C\.4Agentic Training and Evaluation Protocol

#### Upstream agentic RL\.

OpenSWE and SWE\-bench Verified play different roles in our protocol: OpenSWE supplies the upstream RL tasks, whereas SWE\-bench Verified is reserved for evaluation\. Before training, we remove tasks with exposed\.githistories or other artifacts that could enable reward hacking, together with low\-quality examples, yielding 4,048 training tasks\. We train Qwen3\.6\-35B\-A3B on this filtered OpenSWE set from the daVinci\-Env project\\citepfu2026davinci\. A Codex CLI agent\\citepopenai2025codex executes repository\-level actions in the containerized environments provided by Harbor\\citepharbor2026framework; the task verifier converts the final repository state into a terminal reward\. Rollout generation is thinking\-enabled, and each saved trajectory interleaves model\-generated assistant content with tool results and other environment observations\.

Table 8:A segment\-level complementarity case\.The earlier historical rollout recovers from a shared overcounting error, whereas the later rollout retains it\.Setting\.The task asks how many distinct tuples\(f⁡\(1\),…,f⁡\(2014\)\)\(f\(1\),\\ldots,f\(2014\)\)can be induced byf:ℕ→ℕf:\\mathbb\{N\}\\to\\mathbb\{N\}satisfyingf⁡\(1\)=1f\(1\)=1, monotonicity, andf⁡\(2​a\)=f⁡\(a\)\+1f\(2a\)=f\(a\)\+1for everya∈ℕa\\in\\mathbb\{N\}\.Earlier historical rolloutcorrect: 1007Later rolloutincorrect: 1024Shared construction\.LetTkT\_\{k\}count entries taking the lower valuek\+1k\+1in\[2k,2k\+1−1\]\[2^\{k\},2^\{k\+1\}\-1\]\. Both rollouts deriveT0=1T\_\{0\}=1andTk\+1∈\{2​Tk−1,2​Tk\}T\_\{k\+1\}\\in\\\{2T\_\{k\}\-1,2T\_\{k\}\\\}, then count the admissible paths\.Same overcounting error\.“So all2102^\{10\}sequences are valid\. Thus, there are10241024possible tuples\. …”Same overcounting error\.“Since210=1024≤2014<2048=2112^\{10\}=1024\\leq 2014<2048=2^\{11\}, the number of choices is …10241024\.”Recovery\.“Let’s double check ifT10T\_\{10\}being larger than990990creates any ambiguity or if we missed a constraint\. …”No corresponding recovery\.The rollout never uses the observation limitn≤2014n\\leq 2014to test whether distinct threshold sequences induce the same tuple\.Corrected count\.EachT9∈\{1,…,512\}T\_\{9\}\\in\\\{1,\\ldots,512\\\}uniquely fixes the prefix\. ForT9≤495T\_\{9\}\\leq 495, its two admissibleT10T\_\{10\}values yield distinct observed suffixes; for the1717valuesT9≥496T\_\{9\}\\geq 496, they yield the same suffix\. Hence495×2\+17×1=1007495\\times 2\+17\\times 1=1007\.Terminal claim\.The rollout retains the uncorrected count and returns10241024\.Interpretation\.The green recovery is complementary behavior\. The earlier rollout is correct overall, but its red overcount is an undesirable imitation target\. ROSS preserves the full prefix as context while supervising only the selected continuation\.Excerpts are abridged; ellipses mark omissions\. Colors indicate analytical roles, not additional automatic annotations\.
#### Trajectory reuse\.

The task verifier first identifies successful interaction histories\. Positive\-Rollout SFT treats the eligible assistant tokens in these histories as full\-trajectory SFT targets\. ROSS instead applies the agentic annotation procedure in Appendix[E\.3](https://arxiv.org/html/2609.35954#A5.SS3): the complete history remains visible to the model, while assistant spans associated with identified errors are excluded from the loss\. User messages, tool results, and environment observations serve only as context in both methods\. Neither method replaces the recorded model actions with responses generated by the annotation model\.

#### Long\-context trajectory SFT\.

Agentic relearning remains thinking\-enabled and uses a 65,536\-token sequence limit to accommodate multi\-turn interaction histories, compared with 32,768 tokens for single\-turn rollouts\. Both ROSS and Positive\-Rollout SFT start from upstream RL iteration 29 and use trajectory datasets filtered to the longer limit\. They are configured for eight epochs on 64 GPUs across eight nodes, with the parallelism settings in Table[7](https://arxiv.org/html/2609.35954#A3.T7)\. Thus, “full\-trajectory” supervision refers to eligible assistant targets across the retained interaction, rather than to a loss on every token in the sequence\.

#### SWE\-bench Verified evaluation\.

We evaluate Base, Upstream, Positive\-Rollout SFT, and ROSS on SWE\-bench Verified\\citepjimenez2024swebench\. Evaluation also enables thinking and measures the percentage of repository issues resolved under the benchmark verifier\. Across all four policies, we hold fixed the Codex agent scaffold, Harbor task environments, tool interfaces, and execution budgets, so only the policy checkpoint changes\. Section[4\.3](https://arxiv.org/html/2609.35954#S4.SS3)reports the resulting comparison\.

## Appendix DA Segment\-Level Complementarity Case

The aggregate analysis in Section[2\.2](https://arxiv.org/html/2609.35954#S2.SS2)shows that historically discovered solutions can remain available but unreliable under the later policy\. Table[8](https://arxiv.org/html/2609.35954#A3.T8)localizes this complementarity within a trajectory\. The two rollouts solve the same functional\-equation problem and use the same threshold construction, but only the earlier rollout contains the recovery needed to obtain the correct count\.

#### Longitudinal context\.

The displayed late failure is representative of a broader pattern rather than an isolated unsuccessful sample\. We draw eight rollouts for this problem at each of training steps 2, 42, 67, 86, and 109; the numbers of correct responses are respectively11,33,22,00, and00\. The solution is therefore expressed only sparsely earlier in training and is absent from all 16 sampled rollouts at the final two observed steps\. Across the five steps, the problem’s success rate is6/40=15\.0%6/40=15\.0\\%, compared with74\.3%74\.3\\%averaged over the full training set, and it remains within the hardest11–11%11\\%of problems at every sampled step\. The late rollout in Table[8](https://arxiv.org/html/2609.35954#A3.T8)illustrates what is missing after this longitudinal change; the change itself is established by the repeated samples\.

## Appendix ESelective\-Supervision Annotation Protocols

This appendix specifies the staged reviewerAAin Eq\.[3](https://arxiv.org/html/2609.35954#S3.E3)\. Outcome verification first defines the candidate poolℋ\+\\mathcal\{H\}^\{\+\}; annotation then determines whether a candidate yields a valid supervision mask and entersℋROSS\\mathcal\{H\}\_\{\\mathrm\{ROSS\}\}\. The two data regimes use complementary procedures\. For single\-turn Math and Code rollouts, the reviewer extracts a self\-contained correct solution from the response; verifier\-positive instruction\-following responses retain full\-response supervision\. For agentic trajectories, the reviewer first diagnoses errors at the assistant\-step level and then localizes the source text responsible for each error\. In both cases, deterministic code resolves the returned source references and maps them to the stored training tokens\. The prompt boxes below present the core decision rules and output requirements; service wrappers and task\-specific demonstrations are omitted\.

### E\.1Shared Annotation Contract

Every annotation is grounded in exact text from the saved rollout\. The reviewer neither rewrites a response nor supplies missing reasoning or actions\. Only model\-generated assistant tokens withat=1a\_\{t\}=1can receive imitation loss; user messages, verifier evidence, tool observations, and protocol wrappers remain context\. Settingmt=0m\_\{t\}=0changes the loss mask while leaving the original sequence intact\.

The two procedures differ in which regions they return\. Single\-turn annotation identifies text to keep, whereas agentic annotation identifies text to exclude\. The compiler resolves these regions against the original source, maps them onto the stored tokenization, and verifies their order, ownership, overlap, and alignment\. It never constructs the training example by concatenating selected text or retokenizing an edited response\. A candidate is withheld when its semantic decision or source boundary cannot be resolved reliably\.

### E\.2Single\-Turn Rollouts: Positive\-Span Selection

This procedure is used for single\-turn Math and Code rollouts\. Each candidate contains the problem, complete model response, verifier evidence, and a reference answer when available\. Instruction\-following rollouts use the outcome verifier directly because responses in this training pool are typically short answers with little extended reasoning, leaving limited benefit from span\-level annotation or masking\. A positive response therefore keeps all eligible assistant tokens, whereas a failed response is discarded\.

#### Proposal, audit, and repair\.

The first LLM judge chooses among three outcomes: supervise the complete response, retain an ordered sequence of clean spans, or reject the candidate\. Partial selections use verbatim start and end anchors\. Exact matching converts them into half\-open character intervals; a missing, repeated, ambiguous, or out\-of\-order anchor invalidates the proposal\. The selected text must include the complete final answer and, when its spans are read in order, retain the reasoning needed to support that answer\. A separate judge audits this exact selected text rather than the proposal’s summary\. If the audit finds a separable error, a revision call updates the anchors and repeats the source and completeness checks\. The candidate is dropped when a clean solution cannot be recovered solely from text already present in the rollout\.

System Prompt: Single\-Turn Rollout Span SelectionRole\.You are a strict segment\-level credit\-assignment annotator\. Convert one verified model rollout into masked SFT supervision by selecting only contiguous regions of the*original response*that are worth imitating\.Keep\.Retain correct problem setup used by the final solution, locally valid intermediate reasoning, the complete self\-contained derivation, valid verification, and the explicit final answer\. A genuine proof by contradiction is valid when its temporary assumption is stated and correctly refuted\.Exclude\.Do not supervise false assumptions, arithmetic or logical errors, abandoned attempts, discarded code, hard\-coded shortcuts, or correction\-trigger text that is inseparable from a wrong branch\. A correct final answer or verifier pass does not validate earlier claims\. Recompute incompatible counts, signs, states, and quantified claims locally\.Do not penalize correct checking\.Redundancy alone is not an error\. Keep a second calculation or verification when it is correct\. Expressions such as “let us verify” are not masking signals by themselves; correctness of the resulting claim is decisive\.Self\-containment\.Read the selected spans in order as a standalone solution\. Every variable, definition, case split, and dependency used later must be introduced within selected text\. If the only required setup is inseparable from an error, declare that no clean solution is extractable rather than inventing a bridge\.Boundary rule\.Copy unique start and end anchors verbatim from the response\. Multiple spans are allowed\. Preserve complete sentences and equations, and split spans around wrong text instead of describing an internal hole\.Output\.Return structured fields indicating whether the whole response is clean, whether a clean final solution exists, the ordered keep spans and their reasons, a summary of excluded regions, and risk tags\. If the outcome verifier fails or no complete correct derivation can be extracted, return no keep spans\.

System Prompt: Independent Audit of Selected Single\-Turn SpansAudit the*actual text selected by the current proposal*, not the proposal’s explanation\. Treat the concatenated spans as the candidate training target\.Check that \(i\) every retained statement is locally correct; \(ii\) all definitions and dependencies required by later spans are present; \(iii\) the selected text contains the derivation rather than only setup and answer; \(iv\) the final answer is complete and consistent with the verifier evidence; and \(v\) no contradicted assertion, invalid implication, abandoned branch, or answer\-format corruption remains\.Scan the entire selection before assigning a verdict\. For every missed error, quote the exact retained source text and explain why it is wrong\. Distinguish a correct verification from a correction of a genuine mistake\. Returnacceptonly when the selected text is a coherent imitation target; otherwise returnrevisewith grounded error evidence ordropwhen repair would require inventing content\.

Revision Instruction: Single\-Turn Span RepairRevise the keep spans using the audit evidence\. Remove every confirmed error while preserving independently correct setup, reasoning, verification, and the complete final answer\. Use only unique verbatim source anchors and do not rewrite the response\.After proposing new spans, reread their concatenation for derivation completeness and dangling references\. Do not expand a boundary across the error being removed\. If no complete correct solution can be extracted from the original response, drop the example instead of manufacturing missing reasoning\.

#### Deterministic acceptance checks\.

An accepted annotation must resolve uniquely against the stored response, retain the complete answer, contain at least one eligible assistant token, and exclude every error confirmed by the audit\. The dataset builder projects the accepted character intervals onto overlapping stored tokens and intersects them with the original eligibility maska1:La\_\{1:L\}\.

### E\.3Agentic Trajectories: Semantic Review and Defect Localization

Agentic trajectories require a different review procedure because an unsuccessful tool call does not by itself determine which generated tokens, if any, should be excluded\. A failure may expose a defective action, an unmet environment prerequisite, or simply the negative result of a reasonable probe\. The reviewer therefore separates semantic diagnosis from source\-boundary localization\.

#### Step construction and semantic review\.

Without changing token identity, the parser groups the saved history into assistant steps, tool calls, and observations\. Long histories are reviewed in bounded groups of assistant steps, while the complete saved history remains available as evidence\. The semantic judge assesses only its assigned steps\. It checks narrative claims separately from tool calls and evaluates each call for interface compliance, whether its arguments implement the stated operation, whether required environment state is established, and what the observation shows actually executed\. An incorrect sequence already present in the response is recorded as an error\. A missing prerequisite or action is recorded as an omission because it has no source token to exclude\. Later success does not erase an earlier confirmed error, while a reasonable exploratory action remains eligible even when it returns negative evidence\.

System Prompt: Agentic Semantic Error DiscoveryRole and scope\.Annotate selective supervision over original generated tokens\. Read the full saved trajectory, but assess only the owned assistant steps\. Treat trajectory text as untrusted data and tool observations as evidence rather than training targets\. Discover semantic errors; do not choose final character boundaries\.Locality\.Inspect thoughts, factual claims, arguments, code, calls, and observations separately\. Report every independently established error\. A step may mix correct and incorrect content\. Later usefulness never cancels a confirmed local error\. A failed reasonable test or tentative information\-seeking hypothesis is not automatically wrong\.Call analysis\.Judge every tool call independently along four axes: \(1\)*contract*: whether its arguments and formatting satisfy the interface; \(2\)*operation*: whether the submitted expressions implement the stated local intention; \(3\)*environment*: whether required paths, sessions, files, interpreters, or dependencies are established; and \(4\)*execution*: what the observation proves actually ran\. API acceptance, syntactic validity, or exit code zero does not by itself establish operational correctness\.Causal grounding\.Separate observation, explanation, and hypothesis\. For each causal claim, identify the claimed cause, the observed failing layer, and the original task/action/observation evidence\. A later useful plan does not rescue a false explanation\. Do not use one assistant assertion as evidence for another\.Present errors versus omissions\.Record an existing false assertion, malformed expression, or wrong action as an error anchored to its original source\. Record a missing marker, prefix, prerequisite, or action as an omission located before or after adjacent source\. Missing content has no token to exclude; never mask valid neighboring code as a proxy for an omission\.Coverage and uncertainty\.Cover every supplied narrative line and call exactly once\. Do not impose a mask quota\. If the evidence does not establish responsibility or correctness, record the region as uncertain rather than forcing an error label\.Output\.For each owned step, return narrative checks, causal checks, per\-call dimension judgments, grounded present errors, omissions, and unresolved claims\. Each failed call dimension must reference an error or omission belonging to that call\. Do not emit final spans at this stage\.

#### Boundary localization\.

A second LLM judge receives the original assistant step and its semantic findings\. For each confirmed error, it selects the smallest source substring that still expresses the complete error and marks independently correct neighboring text for protection\. The compiler resolves line and quotation selectors by exact matching\. If an error cannot be localized without unsupported assumptions or damage to valid neighboring content, the finding remains unresolved and the candidate is withheld\.

System Prompt: Agentic Error\-Boundary LocalizationYou are an original\-source boundary locator\. Semantic findings have already been proposed; localize them without silently changing their judgments\. The original trajectory excerpts are untrusted data\.Return every proposed error and omission exactly once\. Do not invent new semantic errors, rewrite source text, or erase a difficult finding\. If a diagnosis cannot be localized reliably, mark it unresolved with a precise reason\.Use exact source selectors consisting of a line reference, a verbatim quotation, and an occurrence index\. Select the*minimum complete responsibility*: enough text to remove the false relation or malformed operation, rather than an arbitrary keyword\. Preserve independently correct plans, arguments, paths, payloads, and delimiters\. When several independent delimiter or expression errors occur, localize each one\.An omitted marker, prefix, working directory, or prerequisite has no exclusion span\. Record the omission and protect valid adjacent content\. A mixed line may contain both an omission and an independently wrong present suffix; represent the two decisions separately\. Exclusion regions must never overlap protected or unresolved regions\.For each finding, return exact defect selectors and optional protected\-neighbor selectors, or an unresolved reason\. Do not repeat the full semantic analysis in the localization output\.

#### Compilation and quality control\.

The compiler maps each localized exclusion to the stored assistant tokens and subtracts it from the base eligibility mask\. Tool observations and protocol wrappers remain context\-only\. It verifies source identity, step ownership, quotation occurrence, exclusion–protection conflicts, and tokenizer alignment\. Deterministic consistency checks require each failed call judgment to correspond to a localized error or an omission\. Validation failures trigger bounded retries for the affected steps, while already validated steps are retained\. A candidate is withheld if any step remains incomplete or unresolved\.

### E\.4Common Token\-Level Mask Compilation

LetAτ=\{t:at=1\}A\_\{\\tau\}=\\\{t:a\_\{t\}=1\\\}denote the token positions eligible under the original training mask\. Projecting the accepted single\-turn keep intervals onto the stored tokens givesKτK\_\{\\tau\}, and the supervised positions areAτ∩KτA\_\{\\tau\}\\cap K\_\{\\tau\}\. Projecting localized agentic defects givesEτE\_\{\\tau\}, and the supervised positions areAτ∖EτA\_\{\\tau\}\\setminus E\_\{\\tau\}\. Thus, both procedures produce the binary maskm1:Lm\_\{1:L\}used by Eq\.[4](https://arxiv.org/html/2609.35954#S3.E4), without changing the sequencez1:Lz\_\{1:L\}\.

## Appendix FAnalysis of Excluded Supervision

This section supplements Section[4\.5](https://arxiv.org/html/2609.35954#S4.SS5)with the sampling protocol, category taxonomy, and full distribution used to characterize excluded supervision\. It then records the tokenization and alignment details for the step\-wise TER diagnostic\.

#### Sampling and LLM classification\.

For each domain, we sample up to 15 retained masked trajectories per step from steps 1–20 and 81–100, using a fixed random seed\. Eligibility requires an excluded character fraction above0\.5%0\.5\\%and at least one non\-whitespace excluded interval of 12 or more characters\. The resulting sample contains 300 early and 300 late Math trajectories, and 290 early and 295 late Code trajectories\. This analysis covers two windows of the observed run, not a separately sampled middle stage\.

GLM\-5\.2 receives up to the first four qualifying masked intervals, together with the original annotation summary, reasons, and risk tags for each interval\. The judge assigns one primary category using the definitions below; optional secondary labels are not included in the plotted distribution\. Medium\-confidence and unclear assignments undergo a further LLM review\. Figure[5](https://arxiv.org/html/2609.35954#A6.F5)reports the final trajectory\-level proportions\.

#### Category taxonomy\.

The categories characterize the supplied excluded text and its visible continuation:

CategoryDefinitionCorrected mistakeAn incorrect attempt is explicitly recognized, abandoned, or repaired in the visible continuation\. This label takes precedence over incorrect reasoning when a recovery is evident\.Redundant explorationA detour, excessive planning, or abandoned exploration that the judge considers unnecessary without finding a clearly false statement\. This descriptive label does not imply that all verification or exploration should be masked\.Incorrect reasoning or implementationA false derivation, invalid mathematical step, buggy algorithm or code, or mistaken factual claim without an explicit subsequent correction in the supplied context\.Protocol or format violationBroken output conventions, stray control tags, malformed answer or code fences, irrelevant metadata, or format\-only corruption\.Repetition or reward\-hacking patternDegenerate repetition, duplicated answers, grader\-targeting text, or similar surface patterns\. The label identifies observable behavior, not proven exploitation of the reward function\.Invalid conclusionA local or final claimed result that is unsupported, inconsistent, or invalid, where the conclusion itself is the principal defect\.Other or unclearInsufficient evidence for the preceding categories, or excluded content outside their scope\.
![Refer to caption](https://arxiv.org/html/2609.35954v1/mask_category_appendix_heatmap.png)Figure 5:Full distribution of masked\-content categories\.Each column gives the percentage of sampled trajectories assigned to each primary category\. Math early/late haven=300/300n=300/300; Code early/late haven=290/295n=290/295\. Early and late denote rollout steps 1–20 and 81–100\. Percentages sum to approximately 100 within each column, subject to rounding\.
#### Aggregation\.

The classifier assigns one primary label to each sampled trajectory\. The reported proportions therefore measure trajectory frequency, while TER in Section[4\.5](https://arxiv.org/html/2609.35954#S4.SS5)measures the fraction of excluded tokens\. Together, the two views describe both the amount and the semantic composition of excluded supervision\.

#### Token\-level diagnostic details\.

The first 100 rollout steps contain 61,032 retained Math trajectories and 75,145 retained Code trajectories\. We tokenize saved response text with the training tokenizer and project accepted character spans onto token offsets; any overlap with a selected span makes a token supervised\. We remove the trailing rollout end marker and exclude chat\-template wrappers\. Across all 100 steps, token\-weighted TER is9\.98%9\.98\\%for Math and26\.28%26\.28\\%for Code, with window\-level TER computed by pooling excluded and total token counts\. For Figure[5](https://arxiv.org/html/2609.35954#S4.T5), rollout stepssis aligned with the mean loggedtrain/entropy\_lossover optimizer updates4​s4sthrough4​s\+34s\+3\.

## Appendix GCross\-Domain Performance after Domain\-Specific Relearning

#### Evaluation protocol\.

The main RL table reports performance within each training domain\. Here we examine capabilities outside that domain, comparing each Qwen3\.6\-35B\-A3B Upstream checkpoint with its corresponding third\-epoch ROSS model: iteration 47 for Math and iteration 187 for Code\. Math ROSS is evaluated on code generation and instruction following; Code ROSS is evaluated on mathematics and instruction following\. We use the same benchmark definitions as the main experiments, including generation\-only LiveCodeBench and IFBench\. Here, “cross\-domain” denotes benchmarks outside the ROSS training corpus\. All changes are measured against Upstream rather than Base\.

We additionally evaluate multi\-turn tool use with BFCL, reporting its multi\-turn overall score and the Base, Missing Function, Missing Parameter, and Long Context subsets\. The evaluated ROSS checkpoints for this comparison are Math iteration 650 and Code iteration 3389, paired with Math RL iteration 47 and Code RL iteration 187, respectively\. BFCL Base denotes a benchmark subset, not the pretrained base model\.

#### Math ROSS improves all measured cross\-domain scores\.

Math ROSS increases LCB Gen by 2\.19 points, OJBench by 3\.88, and IFBench by 1\.30 \(Table[9](https://arxiv.org/html/2609.35954#A7.T9)\)\. Its in\-domain gain is therefore accompanied by improvements on all measured code and instruction\-following benchmarks\.

#### Code ROSS preserves aggregate cross\-domain performance\.

Code ROSS decreases AIME 2025 by 1\.77 points while increasing AIME 2026 and HMMT\-November by 1\.61 and 2\.70\. These changes raise Avg\. Math from 74\.19 to 75\.03\. IFBench remains close to Upstream, changing from 35\.45 to 35\.07 \(−0\.38\-0\.38points\)\.

#### Multi\-turn tool use improves overall\.

Table[9](https://arxiv.org/html/2609.35954#A7.T9)shows that BFCL multi\-turn overall rises from 44\.12 to 45\.62 after Math ROSS and from 46\.25 to 49\.38 after Code ROSS\. All four Math subcategories increase\. For Code, the largest improvement is on Missing Function \(\+11\.00\+11\.00points\), followed by Missing Parameter \(\+4\.50\+4\.50\), while Base and Long Context decrease by 1\.00 and 2\.00 points\. The category breakdown reveals which interaction types drive the overall tool\-use gain\.

Table 9:Cross\-domain retention and transfer after domain\-specific ROSS\.Scores are percentages, andΔ\\Deltais ROSS minus Upstream\. The first seven columns use the corresponding third\-epoch ROSS models; the BFCL columns use Math i650 and Code i3389\. Avg\. Math averages AIME 2025, AIME 2026, and HMMT\-November 2025\. BFCL Base denotes the standard multi\-turn subset\.SettingCheckpointAIME25AIME26HMMT\-Nov\.Avg\.MathLCBGenOJBenchIFBenchBFCLOverallBFCLBaseMiss\.Func\.Miss\.Param\.LongContextMath RLUpstream \(s47\)––––56\.7625\.6533\.3044\.1257\.5029\.5039\.0050\.50ROSS––––58\.9529\.5334\.6045\.6258\.5032\.5040\.5051\.00Δ\\Delta––––\+2\.19\+2\.19\+3\.88\+3\.88\+1\.30\+1\.30\+1\.50\+1\.50\+1\.00\+1\.00\+3\.00\+3\.00\+1\.50\+1\.50\+0\.50\+0\.50Code RLUpstream \(s187\)75\.2176\.9370\.4274\.19––35\.4546\.2560\.5032\.5040\.5051\.50ROSS73\.4478\.5473\.1275\.03––35\.0749\.3859\.5043\.5045\.0049\.50Δ\\Delta−1\.77\-1\.77\+1\.61\+1\.61\+2\.70\+2\.70\+0\.84\+0\.84––−0\.38\-0\.38\+3\.13\+3\.13−1\.00\-1\.00\+11\.00\+11\.00\+4\.50\+4\.50−2\.00\-2\.00

#### Comparison with full replay\.

For the Math run, Positive\-Rollout SFT scores 60\.76 on LCB Gen, 29\.96 on OJBench, and 36\.78 on IFBench, exceeding ROSS on all three cross\-domain benchmarks\. The corresponding Code Positive\-Rollout SFT evaluation is unavailable\. ROSS therefore improves over Upstream across the measured Math cross\-domain tasks, while full replay produces larger gains in this particular comparison\.

## Appendix HMitigating Repetitive Answer Emission after 4B Math RL

#### Setting\.

We examine an earlier Qwen3\.5\-4B mathematics run using the same mathematics dataset and a similar RL recipe to the 35B setting, with architecture\-specific settings such as those for mixture\-of\-experts training omitted\. The resulting Upstream checkpoint exhibits repeated answer emission near the end of its responses\. We test whether ROSS can mitigate this behavior while improving task performance\. From rollout steps 1–48, the annotation pipeline retains 25,869 verifier\-positive trajectories for masked SFT\. Our primary comparison initializes ROSS from the corresponding Upstream checkpoint \(iteration 47\), while a secondary variant uses Base with the same annotated data\. We report the third SFT epoch with a 16,384\-token output limit\. Accuracy is evaluated on AIME 2025, AIME 2026, and HMMT\-November 2025, with their unweighted mean denoted Avg\. Math\.

#### Failure pattern and behavioral recovery\.

Upstream frequently continues emitting boxed answers after reaching a conclusion\. One recorded suffix repeatedly emits106\\boxed\{106\}until the output budget is exhausted\. Across the mathematical predictions, the median output length reaches the 16,384\-token cap, and84\.5%84\.5\\%of responses contain at least five boxed\-answer markers\. This marker count serves as a proxy for repetitive output rather than requiring every emitted answer to be identical\. Table[10](https://arxiv.org/html/2609.35954#A8.T10)shows that Upstream\-initialized ROSS reduces the repetition rate to21\.5%21\.5\\%, lowers the mean marker count from 628\.3 to 63\.0, and shortens mean output length by approximately30%30\\%\. ROSS therefore substantially suppresses the repetitive mode\.

Table 10:Reduced output degeneration after ROSS on 4B Math RL\.Measurements use the 16K output budget\. Repetition denotes at least five boxed\-answer markers per response\.MetricUpstreamROSS \(from Upstream\)ROSS \(from Base\)Mean output tokens↓\\downarrow13,8959,7789,606Median output tokens16,3849,8209,850Cap\-hit rate \(%\)↓\\downarrow68–74≈15\\approx 1515–17Mean boxed\-answer count↓\\downarrow628\.363\.091\.6Repetition rate \(%\)↓\\downarrow84\.521\.524\.5
#### Task performance improves alongside output behavior\.

Upstream\-initialized ROSS raises Avg\. Math from52\.05%52\.05\\%to63\.65%63\.65\\%\(Table[11](https://arxiv.org/html/2609.35954#A8.T11)\), with the largest gains on AIME\. The accuracy improvement accompanies the substantial reduction in repetitive and length\-limited outputs\. Base\-initialized ROSS reaches65\.59%65\.59\\%on the same annotated data\.

Table 11:Mathematical accuracy and the output\-budget control\.All scores are percentages\. Only Upstream is reevaluated with a 32K limit; both ROSS variants use 16K\. Avg\. Math is the unweighted mean of the three displayed benchmarks\.MethodBudgetAIME 25AIME 26HMMT\-Nov\.Avg\. MathUpstream16K45\.4248\.5462\.1952\.05Upstream32K44\.1748\.4962\.1951\.62ROSS \(from Upstream\)16K60\.9468\.3361\.6763\.65ROSS \(from Base\)16K62\.2471\.3063\.2365\.59
#### A larger output budget does not resolve the failure\.

Doubling Upstream’s output limit to 32,768 tokens does not improve Avg\. Math \(52\.05%52\.05\\%versus51\.62%51\.62\\%\)\. Inspected outputs continue to show repeated boxed answers after a correct derivation, indicating that additional tokens can prolong the repetitive suffix rather than support further useful reasoning\. This control rules out the 16K output budget as the primary explanation for Upstream’s low mathematical accuracy\.