AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers Papers
reinforcement-learning self-distillation agentic-rl credit-assignment bayesian arxiv

Summary

AgentOPSD proposes a critic-free recursive method for turn-level credit assignment in agentic RL, reweighting outcomes via Bayesian belief updates in log-odds space. It achieves 89.1% success on ALFWorld with Qwen2.5-7B, outperforming GRPO and self-distillation baselines.

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.
Original Article
View Cached Full Text

Cached at: 08/07/26, 05:55 AM

Paper page - AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Source: https://huggingface.co/papers/2608.05987 Published on Aug 6

#1 Paper of the day Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

Reinforcementlearning(RL)withverifiablerewardsconstructstrajectory-leveladvantageestimates,yetitoftenfailstocreditthefewpivotaldecisionsthatdetermineoutcomesinlong-horizon,multi-turnagentictasks.Recentworkintroducesprivilegedself-distillationforcreditassignment,providingdensersupervision,butitremainsunclearhowsuchlocalsignalsshouldrepresentsequentialcredit.WeproposeAgentOPSD,acritic-free,recursivemethodforturn-levelcreditassignmentinagenticreinforcementlearning.AgentOPSDaggregatestoken-levelteacher-studentlog-probabilitygapsintoturn-levelevidenceandrecursivelyupdatesaBayesianbeliefstateinlog-oddsspace.Thisyieldsaprincipledreweightingschemethatconvertssparseoutcomesupervisionintoturn-levelcreditsignalsandidentifiespivotalturnsthroughthemarginalbeliefrevisionbetweenconsecutivestates.Themethodisfullycompatiblewithstandardpolicyoptimizationandrequiresneitheranadditionalcriticnorextrarollouts.WeevaluateAgentOPSDonALFWorld,WebShop,andSearch-QAusingQwen2.5modelsattwoscales(3Band7B).AgentOPSDoutperformsGRPOandstrongself-distillationbaselines,achieving89.1%successonALFWorldwithQwen2.5-7B.Ablationstudiesattributethegainstoturn-levelaggregationandhistory-dependentrecursivebeliefupdates.

View arXiv pageView PDFGitHub2Add to collection

Get this paper in your agent:

hf papers read 2608\.05987

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.05987 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.05987 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.05987 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

OPID: On-Policy Skill Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers

OPID proposes an on-policy skill distillation framework that extracts dense hindsight supervision from completed trajectories, combining outcome-based RL with token-level self-distillation to improve language agent training efficiency and performance on multi-turn tasks.

Self-Distilled Agentic Reinforcement Learning

Hugging Face Daily Papers

SDAR enhances multi-turn agent training by integrating self-distillation with a sigmoid gate to selectively strengthen positive token-level guidance while mitigating negative teacher rejections, achieving significant improvements over GRPO across multiple benchmarks.