PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers Papers
agentic-rl self-distillation reinforcement-learning language-model-agents grpo hidden-state-alignment

Summary

提出特权表示在策略自蒸馏（PR-OPD），通过让策略为轨迹生成事后技能并以自身作为停止梯度教师对齐各层隐藏状态，在 ALFWorld 与 WebShop 上显著优于 GRPO，最高提升 4.7 和 14.0 个百分点。

Language-model agents are usually trained by reinforcement learning from one reward per episode, and privileged self-distillation enriches it by letting the same policy, given a skill, teach its skill-free self through token probabilities. However, we identify two phenomena that question this channel. Invisible Advantage: a skill in context lifts WebShop success from 42.2% to 56.2%, yet changes the probabilities of fewer than a quarter of the sampled tokens. Much to Align: a skill changes the hidden states of over 80% of response tokens, in a way that linear probes can trace back to the specific skill. To exploit this, we propose Privileged Representation On-policy Self-Distillation (PR-OPD). After a GRPO warm start, the policy writes a hindsight skill for each trajectory, re-reads its own responses with that skill as a stop-gradient teacher, and aligns its projected hidden states to the teacher's at every layer alongside the reward objective, with no external skill library, separate teacher, or inference overhead. On ALFWorld and WebShop with two backbones, PR-OPD achieves the best overall results in every setting, improving over GRPO by up to 4.7 points in ALFWorld success and 14.0 points in WebShop accuracy. Code is available at https://github.com/balibata/PR-OPD.
Original Article
View Cached Full Text

Cached at: 10/13/26, 10:48 PM

Paper page - PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning

Source: https://huggingface.co/papers/2609.36642

Abstract

Language-modelagentsareusuallytrainedbyreinforcementlearningfromonerewardperepisode,andprivilegedself-distillationenrichesitbylettingthesamepolicy,givenaskill,teachitsskill-freeselfthroughtokenprobabilities.However,weidentifytwophenomenathatquestionthischannel.InvisibleAdvantage:askillincontextliftsWebShopsuccessfrom42.2%to56.2%,yetchangestheprobabilitiesoffewerthanaquarterofthesampledtokens.MuchtoAlign:askillchangesthehiddenstatesofover80%ofresponsetokens,inawaythatlinearprobescantracebacktothespecificskill.Toexploitthis,weproposePrivilegedRepresentationOn-policySelf-Distillation(PR-OPD).AfteraGRPOwarmstart,thepolicywritesahindsightskillforeachtrajectory,re-readsitsownresponseswiththatskillasastop-gradientteacher,andalignsitsprojectedhiddenstatestotheteacher’sateverylayeralongsidetherewardobjective,withnoexternalskilllibrary,separateteacher,orinferenceoverhead.OnALFWorldandWebShopwithtwobackbones,PR-OPDachievesthebestoverallresultsineverysetting,improvingoverGRPObyupto4.7pointsinALFWorldsuccessand14.0pointsinWebShopaccuracy.Codeisavailableathttps://github.com/balibata/PR-OPD.

View arXiv pageView PDFGitHubAdd to collection

Get this paper in your agent:

hf papers read 2609\.36642

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.36642 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.36642 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.36642 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

GraphOPD: Graph-Augmented On-Policy Distillation for LLM Agents

arXiv cs.LG

GraphOPD 提出首个将图结构增强引入 on-policy 蒸馏的 agent 训练方法，利用环境状态变化构建步骤依赖图并结合随机游走平稳分布进行信用分配，在 ALFWorld、WebShop、SearchQA 上较最强基线提升最高 5.8 个百分点。

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers

AgentOPSD proposes a critic-free recursive method for turn-level credit assignment in agentic RL, reweighting outcomes via Bayesian belief updates in log-odds space. It achieves 89.1% success on ALFWorld with Qwen2.5-7B, outperforming GRPO and self-distillation baselines.