ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

Hugging Face Daily Papers Papers
reinforcement-learning fine-tuning language-models selective-supervision off-policy-training academic-research

Summary

ROSS introduces selective supervision on historical rollouts to improve language model post-training, enhancing performance in math, code, and software engineering tasks without additional policy rollouts.

Large language model post-training generates self-generated rollouts through reinforcement learning and on-policy distillation, yet this experience is often treated as stale once the policy advances. Historical rollouts can remain compatible with a later policy while preserving behaviors that the policy no longer expresses reliably. However, they may also contain mistakes, abandoned attempts, and redundant actions that should not be imitated, motivating finer-grained selective supervision. We introduce ROSS (Relearning from Self-Generated Rollouts through Selective Supervision), which preserves the full historical trajectory as context while applying loss only to selected model-generated continuations. Across domain-specific reinforcement learning, multi-teacher on-policy distillation, and agentic reinforcement learning, ROSS consistently improves upstream checkpoints and outperforms baselines across mathematics, code generation, instruction following, and software engineering. On Qwen3.6-35B-A3B, ROSS improves the six-benchmark MOPD average from 58.40% to 62.20% and SWE-bench Verified from 64.20% to 68.40%. These results show that self-rollout training leaves behind reusable behavioral experience that can yield further gains through offline supervised fine-tuning (SFT), without additional policy rollouts.
Original Article
View Cached Full Text

Cached at: 09/30/26, 04:17 AM

Paper page - ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

Source: https://huggingface.co/papers/2609.35954

Abstract

Largelanguagemodelpost-traininggeneratesself-generatedrolloutsthroughreinforcementlearningandon-policydistillation,yetthisexperienceisoftentreatedasstaleoncethepolicyadvances.Historicalrolloutscanremaincompatiblewithalaterpolicywhilepreservingbehaviorsthatthepolicynolongerexpressesreliably.However,theymayalsocontainmistakes,abandonedattempts,andredundantactionsthatshouldnotbeimitated,motivatingfiner-grainedselectivesupervision.WeintroduceROSS(RelearningfromSelf-GeneratedRolloutsthroughSelectiveSupervision),whichpreservesthefullhistoricaltrajectoryascontextwhileapplyinglossonlytoselectedmodel-generatedcontinuations.Acrossdomain-specificreinforcementlearning,multi-teacheron-policydistillation,andagenticreinforcementlearning,ROSSconsistentlyimprovesupstreamcheckpointsandoutperformsbaselinesacrossmathematics,codegeneration,instructionfollowing,andsoftwareengineering.OnQwen3.6-35B-A3B,ROSSimprovesthesix-benchmarkMOPDaveragefrom58.40%to62.20%andSWE-benchVerifiedfrom64.20%to68.40%.Theseresultsshowthatself-rollouttrainingleavesbehindreusablebehavioralexperiencethatcanyieldfurthergainsthroughofflinesupervisedfine-tuning(SFT),withoutadditionalpolicyrollouts.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.35954

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.35954 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.35954 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.35954 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

ROSS: Relearning from Self-Generated Rollouts through Selective Supervision

arXiv cs.LG

ROSS (Relearning from Self-Generated Rollouts through Selective Supervision) recycles historical self-generated rollouts by applying loss only to selected continuations while preserving full trajectories as context, improving Qwen3.6-35B-A3B across math, code, and SWE-bench benchmarks via offline SFT without new policy rollouts.

Multi-Rollout On-Policy Distillation via Peer Successes and Failures

arXiv cs.LG

Introduces Multi-Rollout On-Policy Distillation (MOPD), a method that conditions the teacher on both successful and failed peer rollouts to provide denser token-level supervision for language model post-training, improving performance across multiple benchmarks.

Better Supervision Is Nearby: Neighborhood On-Policy Self-Distillation

Hugging Face Daily Papers

The paper introduces Neighborhood On-Policy Self-Distillation (N-OPSD), which exploits local parameter perturbations of a privileged teacher to pool complementary reference-aligned corrections and route them to student-visited states, improving math reasoning benchmarks (AIME 2024/2025, HMMT Feb 2025) across Qwen3 1.7B/4B/8B models.

Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

Hugging Face Daily Papers

This paper presents Single-rollout Asynchronous Optimization (SAO) to address stability and off-policy challenges in asynchronous RL for agentic tasks, outperforming GRPO and its variants on coding and reasoning benchmarks. SAO is deployed in the GLM-5.2 model's agentic RL pipeline.