RL 训练 LLM Agent 有个经典难题：

一次长任务失败后，模型到底该从哪里学起？

最终奖励通常只能告诉 Agent「成功或失败」，却很难指出中间哪些判断值得保留，哪些动作把整条轨迹带偏了。

这篇论文提出 SEED，用「自进化在线蒸馏」解决这个问题。

训练时，当前策略先进入环境完成任务，再复盘完整轨迹，生成自然语言形式的「事后技能」。这些技能会总结有效工作流、关键观察和避错经验。

随后，SEED 比较加入技能前后的行为差异，把这种变化转成更细粒度的训练信号，再蒸馏回策略本身。

这样，轨迹中的经验不会只留在外部提示或记忆里，而是逐渐内化进模型参数。部署时也不需要额外加载技能。

实验中，SEED 在 ALFWorld、WebShop 和搜索问答等长时 Agent 任务上都优于多种强化学习与技能蒸馏方法。在 ALFWorld 未见任务上，成功率从 GRPO 的 70.9% 提升到 86.2%。

这篇工作让 Agent 的「复盘」真正进入了训练循环。

过去，Agent 写下的反思往往只是下一轮的临时参考。SEED 更进一步，把复盘带来的行为变化直接压回模型，让分析能力和行动能力一起更新。

当 Agent 开始把自己的完整经历变成可学习的技能，训练曲线就有了一点人类复盘的影子。

arxiv: https://arxiv.org/abs/2607.14777https://arxiv.org/pdf/2607.14777…

Seed: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

Source: https://arxiv.org/html/2607.14777 Jinyang Wu1,Shuo Yang111footnotemark:1,Zhengxi Lu2,Fan Zhang3,Yuhao Shen2,Lang Feng4, Haoran Luo4,Zheng Lian5,Shuai Zhang1,Zhengqi Wen1,Jianhua Tao1

1

Tsinghua University2Zhejiang University3The Chinese University of Hong Kong 4Nanyang Technological University5Tongji University Corresponding to: [email protected]

Abstract

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We proposeSeed(SElf-Evolving On-PolicyDistillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model.Seedfirst fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy.Seedthen re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show thatSeedconsistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available atjinyangwu/Seed.

Figure 1:Overall performance overview.Compared with powerful baseline methods,Seedachieves the strongest average performance across three representative agentic benchmarks.## 1Introduction

Recent large language model (LLM) systems are moving beyond single-turn response generation toward multi-turn agentic interaction, where a model repeatedly reasons, acts, uses tools, and incorporates feedback from its environment(Liuet al.,2023; Schicket al.,2023; Patilet al.,2024; Luoet al.,2025a; Xiet al.,2025; Wuet al.,2026b; Xuet al.,2026). Such settings require an agent to make sequential decisions whose consequences may only become visible after many interaction steps. The model must learn when to gather information, when to call a tool, how to interpret feedback, and how to revise a plan after partial progress or failure. Reinforcement learning (RL) has therefore become an important post-training paradigm for LLM-based agents, since it directly optimizes policies against task-level feedback from environments, simulators, or verifiers(Shaoet al.,2024; Wanget al.,2025; Luoet al.,2025b; Wuet al.,2026a).

Despite this progress, outcome-based agentic RL provides only coarse supervision. In long-horizon environments, rewards are often sparse, delayed, and assigned at the trajectory level: they indicate whether an episode succeeds but not which intermediate observations, actions, or tool calls should be reinforced or corrected(Andrychowiczet al.,2017; Arjona-Medinaet al.,2019; Uesatoet al.,2022; Lightmanet al.,2024). This leaves a supervision gap between episode-level outcomes and token-level policy learning. A failed trajectory may contain useful partial behaviors but fail because of a few local mistakes, whereas a successful trajectory may contain reusable strategies that the scalar reward never identifies. As a result, outcome-only optimization provides limited guidance for fine-grained, decision-level credit assignment over long interaction histories.

A key observation is that completed trajectories reveal hindsight unavailable during online decision making. Once an episode terminates, the full interaction history reveals which subgoals were achieved, where the agent deviated from an effective strategy, which observations were decisive, and which behavioral patterns may transfer to future attempts. This view is related to hindsight learning in RL, where completed experience can be reinterpreted to improve learning under sparse feedback(Andrychowiczet al.,2017), and to language-agent methods based on verbal reflection, episodic memory, or experience summaries(Shinnet al.,2023; Zhaoet al.,2024; Wanget al.,2023; Madaanet al.,2023). However, many such methods treat hindsight as static experience, inference-time context, or retrieved memory. For practical agentic RL, hindsight supervision should not remain fixed: as the policy improves and encounters new states, strategies, and failure modes, the hindsight extracted from its trajectories must adapt to its current behavior(Andrychowiczet al.,2017; Zhanget al.,2026a). Our goal is therefore to convert policy-generated hindsight into parametric supervision, enabling the policy to internalize reusable behavioral guidance without external memory or additional deployment-time prompts.

On-policy distillation (OPD) offers a natural mechanism for converting hindsight information into decision-level learning signals. Classical knowledge distillation transfers teacher behavior into a student through token- or sequence-level supervision(Hintonet al.,2015; Kim and Rush,2016), while on-policy variants reduce distribution mismatch by supervising outputs sampled from the student policy itself(Rosset al.,2011; Agarwalet al.,2024). Recent on-policy self-distillation methods further avoid a separate external teacher by comparing the same model under different contexts, such as privileged reasoning traces or feedback-conditioned prompts(Zhaoet al.,2026; Hübotteret al.,2026). Related agentic methods have explored skill- or feedback-conditioned distillation for multi-turn interaction and tool use(Wanget al.,2026; Luet al.,2026a; Zhonget al.,2026; Koet al.,2026; Yanget al.,2026b). Together, these advances point to three requirements for effective hindsight supervision in agentic RL.

First, the supervision should beon-policy, because useful corrections depend on the states, actions, and failure modes induced by the current policy. Second, it should bedense, allowing trajectory-level hindsight to guide individual decision tokens rather than only the final outcome. Third, it should beself-evolving: as the policy improves, its decision-making and trajectory-analysis capabilities should advance together. Fixed teachers, static skill datasets, and one-time distillation cannot continually adapt to the policy’s evolving capabilities and trajectory distribution.

We proposeSeed(SElf-Evolving On-PolicyDistillation), a framework that turns completed on-policy trajectories into hindsight skills and distills their behavioral effect back into the policy model through a self-evolving training loop. During RL, the current policy collects trajectories and also serves as the analyzer that extracts natural-language skills describing reusable workflows, decisive observations, or failure-avoidance rules. Because both roles share the same model, policy updates improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. Given an extracted skill,Seedkeeps the sampled actions fixed and re-scores them under ordinary and skill-augmented contexts. The resulting probability shift provides a dense token-level OPD signal, which is jointly optimized with the outcome-based RL objective. Specifically,Seedconsists of two stages. First,hindsight-skill supervised fine-tuningequips the model to analyze completed interaction histories and generate reusable trajectory-level skills. Second,self-evolving OPDrepeatedly uses the latest policy checkpoint for both trajectory collection and skill analysis, then updates the policy with the joint RL and OPD objective. The generated skills act only as privileged supervision during training and require neither external memory nor additional prompts at inference time.

We evaluateSeedacross embodied interaction, web navigation, search-based QA, and visual perception and planning.Seedachieves superior task performance, sample efficiency, and robustness. Taken together, our work makes the following contributions:

• •We proposeSeed, a self-evolving OPD framework that continually transforms the policy’s completed trajectories into hindsight skills and internalizes their behavioral guidance during agentic RL, allowing decision-making and skill analysis to improve together.
• •We introduce a policy-synchronized hindsight OPD mechanism that converts skill-induced log-probability shifts on sampled actions into dense token-level supervision and jointly optimizes this signal with outcome-based RL.
• •Extensive experiments across diverse long-horizon agentic benchmarks show thatSeedimproves task performance, sample efficiency, and robustness over representative baselines.

2Related Work

Reinforcement learning for agentic LLMs.

LLMs are increasingly trained as interactive agents that reason, use tools, and act over long horizons(Liuet al.,2023; Schicket al.,2023; Patilet al.,2024; Luoet al.,2025a; Xiet al.,2025; Wuet al.,2026b; Xuet al.,2026). Reinforcement learning provides a natural post-training paradigm for such agents by directly optimizing task-level outcomes(Shaoet al.,2024; Wanget al.,2025; Luoet al.,2025b; Wuet al.,2026a; Luet al.,2026b). However, outcome-based RL typically assigns sparse and delayed rewards at the trajectory level, offering limited guidance on which intermediate observations, actions, or tool calls should be reinforced. SEED retains outcome-based RL as the optimization backbone while supplementing it with dense supervision derived from completed trajectories.

Hindsight learning for language agents.

Completed trajectories expose reusable strategies, decisive observations, and failure causes that are unavailable during online decision making. Prior work exploits such information through hindsight relabeling, return decomposition, process supervision, verbal reflection, and experience memory(Andrychowiczet al.,2017; Arjona-Medinaet al.,2019; Uesatoet al.,2022; Lightmanet al.,2024; Shinnet al.,2023; Zhaoet al.,2024; Wanget al.,2023; Madaanet al.,2023). Nevertheless, hindsight knowledge is often stored as static experience or introduced as additional inference-time context. SEED instead converts completed on-policy trajectories into natural-language skills and internalizes their behavioral guidance into the policy, requiring neither external memory nor skill prompts at inference time.

On-policy self-distillation for agentic RL.

Knowledge distillation transfers teacher behavior through token- or sequence-level supervision, while on-policy variants reduce distribution mismatch by training on outputs sampled from the learner itself(Hintonet al.,2015; Kim and Rush,2016; Rosset al.,2011; Agarwalet al.,2024). Recent methods construct privileged self-teachers from reasoning traces, feedback, or skills and use them to provide token-level guidance for agentic RL(Zhaoet al.,2026; Wanget al.,2026; Luet al.,2026a; Zhonget al.,2026; Yanget al.,2026b). However, the source of privileged supervision is often static, externally generated, or updated independently of the policy. As the policy improves and encounters new states and failure modes, such supervision can become stale or mismatched with its current behavior. SEED addresses this limitation through a self-evolving loop in which the latest policy checkpoint serves simultaneously as the rollout actor and the trajectory analyzer. After each policy update, the shared checkpoint is refreshed for both roles, allowing decision making and hindsight supervision to co-evolve.

Figure 2:Overview ofSeed.Stage 1(Hindsight Skill SFT) equips the policy to extract hindsight skills from completed trajectories.Stage 2(Self-Evolving On-Policy Distillation) jointly optimizes outcome-based RL and skill-conditioned OPD in a self-evolving agentic loop.

3Method

We presentSeed, a self-evolving OPD framework for agentic RL.Seedis motivated by the observation that completed agent trajectories contain rich hindsight information: even when reward is sparse, a full trajectory often reveals useful behavioral patterns, failure causes, and reusable strategies that are not directly available at intermediate decision steps.Seedconverts such hindsight information intohindsight skillsand distills their behavioral effect back into the ordinary policy.

As shown in Figure2,Seedhas two training stages. First, hindsight-skill supervised fine-tuning (SFT) equips a single policy model to analyze completed trajectories. Second, the current policy snapshot both collects on-policy trajectories and analyzes them into hindsight skills. The same sampled actions are then re-scored under ordinary and skill-augmented contexts to construct a token-level OPD signal, which is optimized jointly with RL. At inference time,Seedremoves the analyzer, so deployment requires only the learned policy.

3.1Problem Formulation

We formulate long-horizon agentic tasks as partially observable Markov decision processes:

(𝒮,𝒜,𝒪,𝒯,Ω,ℛ,γ),\left(\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{T},\Omega,\mathcal{R},\gamma\right),where𝒮\mathcal{S}is the latent state space,𝒜\mathcal{A}is the action space,𝒪\mathcal{O}is the observation space,𝒯\mathcal{T}is the transition kernel,Ω\Omegais the observation kernel,ℛ\mathcal{R}is the reward function, andγ\gammais the discount factor. At timesteptt, the agent receives an observationot∈𝒪o_{t}\in\mathcal{O}and maintains an interaction history

ht=(o0,a0,o1,a1,…,ot),h_{t}=\left(o_{0},a_{0},o_{1},a_{1},\ldots,o_{t}\right),whereaia_{i}denotes the textual response or executable action produced by the agent. The policy generates the next action according to

at∼πθ(⋅∣ht).a_{t}\sim\pi_{\theta}\left(\cdot\mid h_{t}\right). A completed trajectory is denoted by

τ={(ot,at,rt)}t=0T−1,\tau=\left\{(o_{t},a_{t},r_{t})\right\}_{t=0}^{T-1},with an episode-level outcomeR​(τ)∈ℝR(\tau)\in\mathbb{R}. In many agentic environments,R​(τ)R(\tau)is sparse and becomes available only after task completion, such as a binary success indicator or a final task score. The standard reinforcement learning objective is

J​(θ)=𝔼τ∼πθ​[R​(τ)].J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[R(\tau)\right].This formulation exposes a key challenge in long-horizon agentic RL: the environment provides only trajectory-level feedback, while the policy must learn from many token-level decisions distributed across the interaction history.Seedbridges this granularity gap by deriving an auxiliary training-time token-level signal from hindsight skills extracted after trajectory completion.

3.2Hindsight Skill Supervised Fine-Tuning

The first stage initializes the policy with the ability to analyze full interaction histories and express reusable behavioral guidance as natural-language hindsight skills.

Offline trajectory collection.

We first collect an offline pool of trajectories using a base policy without skill augmentation. Let𝒬sft={qj}j=1M\mathcal{Q}_{\mathrm{sft}}=\{q_{j}\}_{j=1}^{M}be a set of training tasks. For each taskqjq_{j}, we runK0K_{0}rollouts with the base policyπθbase\pi_{\theta_{\mathrm{base}}}:

ℬj={τj,k}k=1K0,τj,k∼πθbase(⋅∣qj).\mathcal{B}_{j}=\left\{\tau_{j,k}\right\}_{k=1}^{K_{0}},\qquad\tau_{j,k}\sim\pi_{\theta_{\mathrm{base}}}\left(\cdot\mid q_{j}\right).The complete offline trajectory pool is

ℬ=⋃j=1Mℬj.\mathcal{B}=\bigcup_{j=1}^{M}\mathcal{B}_{j}.Each trajectory contains the task description, observations, actions, rewards, and final outcome. These trajectories are collected without any hindsight skill in the decision context, ensuring that the SFT data are derived from ordinary agent-environment interaction.

Hindsight skill annotation.

Given a completed trajectoryτ\tau, an external analyzerAextA_{\mathrm{ext}}produces a hindsight-skill annotation:

sτ=Aext​(τ),s_{\tau}=A_{\mathrm{ext}}\left(\tau\right),wheresτs_{\tau}denotes the skill annotation produced from trajectoryτ\tau. For a successful trajectory,sτs_{\tau}typically captures reusable strategies or workflows that contributed to task completion. For a failed trajectory,sτs_{\tau}can encode corrective or avoidance guidance inferred from the observed failure.

We retain the annotation only if the generated skill is correctly formatted. Letvτ∈{0,1}v_{\tau}\in\{0,1\}denote the validity indicator. The accepted SFT set is

𝒟sft={(xτ,sτ):τ∈ℬ,vτ=1},\mathcal{D}_{\mathrm{sft}}=\left\{(x_{\tau},s_{\tau}):\tau\in\mathcal{B},v_{\tau}=1\right\},wherexτx_{\tau}is the serialized trajectory-analysis input andsτs_{\tau}is the corresponding hindsight skill target.

Supervised fine-tuning.

We then fine-tune the policy model to predict the hindsight skill from the completed trajectory. The same autoregressive model later initializes both the RL actor and the synchronized trajectory analyzer. For an accepted SFT example(xτ,sτ)∈𝒟sft(x_{\tau},s_{\tau})\in\mathcal{D}_{\mathrm{sft}}, the model is optimized with the standard negative log-likelihood objective:

ℒsft​(θ)=−𝔼(xτ,sτ)∼𝒟sft​[∑ℓ=1|sτ|log⁡πθ​(sτ,ℓ∣xτ,sτ,<ℓ)],\mathcal{L}_{\mathrm{sft}}(\theta)=-\mathbb{E}_{(x_{\tau},s_{\tau})\sim\mathcal{D}_{\mathrm{sft}}}\left[\sum_{\ell=1}^{|s_{\tau}|}\log\pi_{\theta}\!\left(s_{\tau,\ell}\mid x_{\tau},s_{\tau,<\ell}\right)\right],wheresτ,ℓs_{\tau,\ell}is theℓ\ell-th token ofsτs_{\tau}. The resulting checkpointθsft\theta_{\mathrm{sft}}initializes the later RL policy.

During RL, the trajectory analyzer is instantiated directly from the current policy checkpoint. Thus, the same model can act in the environment under ordinary interaction histories and generate hindsight skills from completed trajectories, without a separately trained analyzer.

3.3Self-Evolving On-Policy Distillation

The second stage performs agentic RL with an additional token-level OPD signal. At the start of each update,Seedfreezes the current policy asπθold\pi_{\theta_{\mathrm{old}}}. This snapshot collects trajectories and also parameterizes the analyzer that extracts their hindsight skills. A trainable policyπθ\pi_{\theta}, initialized fromπθold\pi_{\theta_{\mathrm{old}}}, is then optimized with both the environment-driven GRPO and OPD objectives. After the update, the optimized policyπθ\pi_{\theta}becomes the policy snapshot for the next iteration.

This design keeps the distillation signal on-policy: the policy collects its own trajectories, the analyzer agent converts these completed trajectories into hindsight skills, and the behavioral effect of such hindsight guidance is distilled back into the ordinary policy.

On-policy hindsight skill generation.

For each task promptqq,Seedsamples a group ofNNtrajectories using the frozen policy:

𝒢q={τq(1),τq(2),…,τq(N)},τq(n)∼πθold(⋅∣q).\mathcal{G}_{q}=\left\{\tau_{q}^{(1)},\tau_{q}^{(2)},\ldots,\tau_{q}^{(N)}\right\},\qquad\tau_{q}^{(n)}\sim\pi_{\theta_{\mathrm{old}}}\left(\cdot\mid q\right).For each completed trajectoryτq(n)\tau_{q}^{(n)},Seedconstructs its trajectory-analysis inputxτq(n)x_{\tau_{q}^{(n)}}. The analyzer agent instantiated from the same policy snapshotπθold\pi_{\theta_{\mathrm{old}}}then analyzes the completed trajectory and generates a hindsight skill:

sq(n)=Aθold​(xτq(n)),s_{q}^{(n)}=A_{\theta_{\mathrm{old}}}\left(x_{\tau_{q}^{(n)}}\right),whereAθoldA_{\theta_{\mathrm{old}}}denotes the analyzer role of the shared model. Although the actor and analyzer share the same parameters at each update, they play different roles: the actor interacts with the environment, while the analyzer summarizes completed trajectories into trajectory-level hindsight skills. The generated skillsq(n)s_{q}^{(n)}thus provides reusable behavioral guidance.

This shared parameterization createsSeed’s self-evolving loop. Refreshingθold\theta_{\mathrm{old}}changes both the trajectories encountered by the actor and the model capability used for skill analysis. Consequently, the experience distribution and its hindsight supervision evolve together.

On-policy distillation objective.

Seedkeeps the original on-policy actions fixed and re-scores them under a skill-augmented context. LetHHbe a deterministic context augmentation function that incorporates the generated skill into the ordinary interaction history. At timestepttin trajectoryτq(n)\tau_{q}^{(n)}, the skill-augmented history is

h~q,n,t=H​(hq,n,t,sq(n)).\tilde{h}_{q,n,t}=H\left(h_{q,n,t},s_{q}^{(n)}\right).The original sampled action is tokenized as

aq,n,t=(aq,n,t,1,…,aq,n,t,Lq,n,t),a_{q,n,t}=\left(a_{q,n,t,1},\ldots,a_{q,n,t,L_{q,n,t}}\right),whereLq,n,tL_{q,n,t}denotes the number of tokens in the sampled actionaq,n,ta_{q,n,t}.

The same policy computes two token-level log-probabilities on the same sampled action tokens. The first is the skill-conditioned teacher branch log-probability, obtained by re-scoring the sampled action under the skill-augmented history:

ℓq,n,t,ℓskill=log⁡πθ​(aq,n,t,ℓ∣h~q,n,t,aq,n,t,<ℓ).\ell^{\mathrm{skill}}_{q,n,t,\ell}=\log\pi_{\theta}\left(a_{q,n,t,\ell}\mid\tilde{h}_{q,n,t},a_{q,n,t,<\ell}\right).The second is the ordinary student branch log-probability under the original interaction history:

ℓq,n,t,ℓθ=log⁡πθ​(aq,n,t,ℓ∣hq,n,t,aq,n,t,<ℓ).\ell^{\theta}_{q,n,t,\ell}=\log\pi_{\theta}\left(a_{q,n,t,\ell}\mid h_{q,n,t},a_{q,n,t,<\ell}\right).Although both branches shareπθ\pi_{\theta}, they correspond to different input contexts: the teacher observes the hindsight skill, while the student acts only from the ordinary history. The teacher provides a detached training-time signal, and gradients flow exclusively through the ordinary student branch. In summary, during optimization, both branches are evaluated using the current trainable policyπθ\pi_{\theta}, whereasπθold\pi_{\theta_{\mathrm{old}}}is used for trajectory collection, skill generation, and importance-ratio computation.

We define the detached skill-induced log-probability shift

Δq,n,t,ℓ=sg⁡[ℓq,n,t,ℓskill−ℓq,n,t,ℓθ],\Delta_{q,n,t,\ell}=\operatorname{sg}\left[\ell^{\mathrm{skill}}_{q,n,t,\ell}-\ell^{\theta}_{q,n,t,\ell}\right],wheresg⁡[⋅]\operatorname{sg}[\cdot]denotes stop-gradient. Following SDAR(Luet al.,2026a), SEED maps this shift to a confidence gate:

gq,n,t,ℓ=σ​(βopd​Δq,n,t,ℓ),g_{q,n,t,\ell}=\sigma\left(\beta_{\mathrm{opd}}\Delta_{q,n,t,\ell}\right),whereσ​(⋅)\sigma(\cdot)is the logistic sigmoid function andβopd\beta_{\mathrm{opd}}controls the sharpness of the gate. A positive shift indicates that the hindsight skill supports the sampled token and yields a larger gate; a negative shift attenuates that token’s auxiliary supervision.

The OPD loss is defined as a confidence-gated sampled-token distillation objective:

ℒopd​(θ)=𝔼q,n,t,ℓ​[mq,n,t,ℓ⋅gq,n,t,ℓ⋅(sg⁡[ℓq,n,t,ℓskill]−ℓq,n,t,ℓθ)].\mathcal{L}_{\mathrm{opd}}(\theta)=\mathbb{E}_{q,n,t,\ell}\left[m_{q,n,t,\ell}\cdot g_{q,n,t,\ell}\cdot\left(\operatorname{sg}\left[\ell^{\mathrm{skill}}_{q,n,t,\ell}\right]-\ell^{\theta}_{q,n,t,\ell}\right)\right].(1)Heremq,n,t,ℓ∈{0,1}m_{q,n,t,\ell}\in\{0,1\}is the valid-token mask. Throughout, expectations over(q,n,t,ℓ)(q,n,t,\ell)are computed as masked means over valid action tokens, normalized by∑q,n,t,ℓmq,n,t,ℓ\sum_{q,n,t,\ell}m_{q,n,t,\ell}. Because bothgq,n,t,ℓg_{q,n,t,\ell}and the teacher log-probability are detached, the teacher term is constant with respect toθ\theta, and

∇θℒopd=−𝔼q,n,t,ℓ​[mq,n,t,ℓ⋅gq,n,t,ℓ⋅∇θℓq,n,t,ℓθ].\nabla_{\theta}\mathcal{L}_{\mathrm{opd}}=-\mathbb{E}_{q,n,t,\ell}\left[m_{q,n,t,\ell}\cdot g_{q,n,t,\ell}\cdot\nabla_{\theta}\ell^{\theta}_{q,n,t,\ell}\right].Thus, the objective is gradient-equivalent to a gate-weighted negative log-likelihood. Minimizing it increases the ordinary policy’s likelihood of teacher-endorsed on-policy tokens, thereby internalizing the skill’s behavioral effect without exposing the skill at inference time.

Joint training objective.

In addition to OPD,Seedoptimizes the policy with a group-relative RL objective. For each group𝒢q\mathcal{G}_{q}, we compute the mean and standard deviation of trajectory outcomes:

μq=1N​∑n=1NR​(τq(n)),σq=1N​∑n=1N(R​(τq(n))−μq)2.\mu_{q}=\frac{1}{N}\sum_{n=1}^{N}R\left(\tau_{q}^{(n)}\right),\quad\sigma_{q}=\sqrt{\frac{1}{N}\sum_{n=1}^{N}\left(R\left(\tau_{q}^{(n)}\right)-\mu_{q}\right)^{2}}.The trajectory-level group-relative advantage is

Aq,nrl=R​(τq(n))−μqσq+ϵ.A^{\mathrm{rl}}_{q,n}=\frac{R\left(\tau_{q}^{(n)}\right)-\mu_{q}}{\sigma_{q}+\epsilon}.This advantage is broadcast to valid action tokens:

Aq,n,t,ℓrl=Aq,nrl​mq,n,t,ℓ.A^{\mathrm{rl}}_{q,n,t,\ell}=A^{\mathrm{rl}}_{q,n}m_{q,n,t,\ell}. The token-level probability ratio is

ρq,n,t,ℓ​(θ)=exp⁡(ℓq,n,t,ℓθ−ℓq,n,t,ℓold),\rho_{q,n,t,\ell}(\theta)=\exp\left(\ell^{\theta}_{q,n,t,\ell}-\ell^{\mathrm{old}}_{q,n,t,\ell}\right),where

ℓq,n,t,ℓold=log⁡πθold​(aq,n,t,ℓ∣hq,n,t,aq,n,t,<ℓ).\ell^{\mathrm{old}}_{q,n,t,\ell}=\log\pi_{\theta_{\mathrm{old}}}\left(a_{q,n,t,\ell}\mid h_{q,n,t},a_{q,n,t,<\ell}\right).The RL loss is

ℒrl​(θ)\displaystyle\mathcal{L}_{\mathrm{rl}}(\theta)=−𝔼q,n,t,ℓ​[min⁡(ρq,n,t,ℓ​(θ)​Aq,n,t,ℓrl,clip⁡(ρq,n,t,ℓ​(θ),1−ϵclip,1+ϵclip)​Aq,n,t,ℓrl)]\displaystyle=-\mathbb{E}_{q,n,t,\ell}\left[\min\left(\rho_{q,n,t,\ell}(\theta)A^{\mathrm{rl}}_{q,n,t,\ell},\operatorname{clip}\left(\rho_{q,n,t,\ell}(\theta),1-\epsilon_{\mathrm{clip}},1+\epsilon_{\mathrm{clip}}\right)A^{\mathrm{rl}}_{q,n,t,\ell}\right)\right]+βKL​DKL.\displaystyle\quad+\beta_{\mathrm{KL}}D_{\mathrm{KL}}.whereβKL\beta_{\mathrm{KL}}is the KL regularization coefficient.

The final training objective combines environment-driven agentic RL with hindsight-skill OPD:

ℒSEED​(θ)=ℒrl​(θ)+λopd​ℒopd​(θ),\mathcal{L}_{\mathrm{SEED}}(\theta)=\mathcal{L}_{\mathrm{rl}}(\theta)+\lambda_{\mathrm{opd}}\mathcal{L}_{\mathrm{opd}}(\theta),whereλopd\lambda_{\mathrm{opd}}controls the strength of the auxiliary OPD signal. The RL term optimizes environment outcomes, whereas the OPD loss internalizes the effect of self-generated hindsight skills. The updated policy then becomesπθold\pi_{\theta_{\mathrm{old}}}for the next iteration, closing the self-evolving loop.

At inference time, the deployed agent acts only from the ordinary interaction history:

at∼πθ(⋅∣ht).a_{t}\sim\pi_{\theta}\left(\cdot\mid h_{t}\right).Thus, all hindsight skills are used only as training-time guidance. Deployment requires no analyzer, no skill bank, no retrieval module, and no augmented decision prompt.

Algorithm1in AppendixB.3summarizes the complete training procedure.

4Experiment

4.1Experimental Setting

Benchmarks.

We evaluateSeedacross three complementary forms of long-horizon agency.ALFWorld(Shridharet al.,2021)casts household tasks as text-based embodied interaction, requiring an agent to interpret observations and execute extended action sequences. We consider six task families:Pick,Look,Clean,Heat,Cool, andPick2.WebShop(Yaoet al.,2022)evaluates interactive web navigation, where an agent searches for products, inspects their attributes, and completes a purchase according to a natural-language request. We use the standard set of 128 test tasks. Finally, following the Search-R1 protocol(Jinet al.,2025),Search-based QArequires an agent to gather evidence through search before answering questions from Natural Questions(Kwiatkowskiet al.,2019), TriviaQA(Joshiet al.,2017), PopQA(Mallenet al.,2023), HotpotQA(Yanget al.,2018), 2WikiMultiHopQA(Hoet al.,2020), MuSiQue(Trivediet al.,2022), and Bamboogle(Presset al.,2023). Together, these benchmarks cover embodied control, web-based decision making, and tool-augmented information seeking.

Baselines.

We compareSeedwith prompting, outcome-based RL, and distillation baselines.Vanillaevaluates the instruction-tuned backbone without post-training, whereasSkill-Promptprovides natural-language skills only in the evaluation context.GRPO(Shaoet al.,2024)optimizes group-normalized trajectory rewards without auxiliary supervision.Skill-GRPOadditionally conditions the policy on skills during RL. We also include representative self-distillation and skill-distillation methods:OPSD(Zhaoet al.,2026),GRPO+OPSD,Skill-SD(Wanget al.,2026),RLSD(Yanget al.,2026a), andSDAR(Luet al.,2026a). These comparisons distinguish the effects of outcome optimization, access to skill context, and dense teacher-derived supervision. An asterisk denotes evaluation with skills; all other methods operate from the ordinary interaction history without privileged skill inputs at test time. We use matched backbones, rollout budgets, and training schedules for all reproduced post-training baselines.

Table 1:Performance Comparison on the representative long-horizon benchmarks (ALFWorld, Search-based QA, and WebShop).We report the success rate (%) on ALFWorld, accuracy on Search-based QA, and task-completion score/success rate on WebShop. An asterisk (*) denotes validation with skills. Thebestandsecond-bestresults are highlighted.ALFWorldSearch-based QAWebShopMethodPickLookCleanHeatCoolPick2AvgNQTrivPopHotp2WkMuSBamAvgScoreSucc.Qwen2.5-3B-InstructVanilla44.411.16.215.428.612.521.924.648.131.026.325.37.259.731.76.70.8Skill-Prompt*51.766.748.40.04.310.028.923.746.230.624.422.17.512.523.90.20.8OPSD48.841.716.70.015.816.728.10.10.10.10.00.00.00.00.011.33.1GRPO91.262.596.261.965.047.475.039.360.641.137.434.615.426.436.479.863.3Skill-GRPO88.971.458.870.640.729.260.243.558.843.036.832.211.712.534.177.360.9Skill-GRPO*94.357.1100.066.773.157.180.544.359.644.339.036.114.514.936.176.366.4GRPO+OPSD100.082.485.775.070.060.081.244.961.245.240.438.516.066.144.677.866.4Skill-SD88.250.096.252.465.057.973.444.460.444.039.540.415.464.944.175.964.0RLSD87.975.090.975.073.168.479.741.558.642.340.440.216.866.943.884.466.4SDAR97.162.5100.061.975.084.284.444.858.144.338.636.215.766.143.485.068.0Seed(Ours)100.0100.0100.0100.070.680.091.844.361.947.241.042.016.967.745.788.578.9Qwen2.5-7B-InstructVanilla36.122.23.10.00.00.012.525.250.829.529.029.010.463.733.95.91.6Skill-Prompt*51.750.032.35.34.30.023.430.952.132.732.727.912.766.136.41.70.8OPSD50.060.022.721.417.69.532.88.88.617.52.54.20.51.26.24.52.3GRPO91.287.596.281.065.057.981.245.163.744.043.643.216.837.642.080.972.6Skill-GRPO88.566.765.261.157.773.169.545.263.745.743.143.319.621.440.380.471.9Skill-GRPO*100.083.396.483.375.078.988.344.863.045.143.743.720.571.447.587.081.2GRPO+OPSD91.461.5100.087.576.552.280.447.364.546.943.839.318.069.447.086.876.5Skill-SD93.993.890.9100.069.268.485.147.164.547.844.242.120.269.047.886.176.5RLSD100.087.592.358.880.065.282.046.863.044.445.548.921.573.049.087.477.3SDAR94.775.0100.086.768.278.985.946.363.548.243.848.419.673.049.089.482.8Seed(Ours)100.0100.096.380.0100.0100.096.147.064.947.845.245.320.170.248.689.778.1Qwen3-1.7B-InstructVanilla25.022.23.10.021.44.212.529.446.937.023.519.66.410.524.846.54.7Skill-Prompt*10.350.016.10.00.05.09.429.446.536.222.920.84.310.124.323.02.3OPSD26.333.39.10.04.55.314.14.28.34.66.615.30.71.25.847.49.3GRPO71.141.736.440.031.831.646.140.058.943.535.430.312.065.740.867.338.3Skill-GRPO27.654.522.727.30.019.221.139.258.643.935.228.211.566.140.473.446.1Skill-GRPO*31.442.951.98.311.57.128.138.058.443.936.329.012.566.940.780.450.0GRPO+OPSD38.250.030.828.630.021.132.040.758.945.037.034.613.365.742.270.738.3Skill-SD52.937.569.242.960.036.852.339.157.545.434.834.110.764.140.881.853.9RLSD50.037.561.519.050.021.142.238.657.343.034.534.111.565.340.674.050.8SDAR73.525.076.933.340.036.853.939.758.945.335.935.512.665.341.976.858.6Seed(Ours)97.6100.0100.080.084.290.092.042.058.947.136.935.210.664.942.287.177.3

Evaluation Metrics.

For ALFWorld, we report the success rate for each task family and their unweighted macro-average. For WebShop, we follow the environment protocol and report both the mean normalized task-completion score and the exact success rate. For Search-based QA, we compute answer accuracy separately on all seven subsets and report their unweighted macro-average. All metrics are expressed as percentages, and higher values indicate better performance.

Implementation Details.

We use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct(Yanget al.,2024), as well as Qwen3-1.7B-Instruct(Yanget al.,2025), as our backbone models.SFT stage:For each backbone, we sampleM=180M=180training tasks and collectK0=8K_{0}=8rollout trajectories per task, resulting in 1,440 completed trajectories. We then query GLM-5.2(Z.ai,2026)as an external trajectory analyzer to extract hindsight skills from these trajectories. After lightweight format validation, the retained trajectory–skill pairs are used to fine-tune the corresponding backbone for three epochs.RL stage:The resulting SFT checkpoint initializes both the policy and the synchronized trajectory analyzer. We train each model for 150 policy updates, using a batch size of 16 on ALFWorld and WebShop and 128 on Search-based QA. The rollout group size is set toN=8N=8for all benchmarks. Additional details on SFT data construction, skill annotation, optimization, and hyperparameter settings are provided in AppendixB.4.

4.2Main Results

Table1reports the results across different model scales and agentic domains. Three findings emerge:

Seedconsistently outperforms outcome-only RL through dense supervision.Relative to GRPO,Seedimproves the ALFWorld macro-average by 14.9-45.9 points, Search-based QA by 1.4-9.3 points, the WebShop task-completion score by 8.7-19.8 points, and the success rate by 5.5-39.0 points across the three backbones. Compared with Skill-GRPO, which conditions exploration on natural-language skills but still broadcasts a single terminal-reward-derived advantage to all valid tokens,Seedfurther improves the ALFWorld average by 26.6-70.9 points, Search-based QA by 1.8-11.6 points, the WebShop score by 9.3-13.7 points, and the success rate by 6.2-31.2 points. These consistent improvements over both GRPO and Skill-GRPO demonstrate that dense token-level hindsight supervision provides more effective credit assignment than outcome-only optimization, leading to substantially stronger performance across long-horizon agentic tasks.

Figure 3:Training dynamics on ALFWorld.We compareSeedand GRPO using Qwen2.5-3B-Instruct as the backbone. Translucent curves show raw measurements, while solid curves show 13-point centered moving averages.Internalizing skills is substantially more effective than inserted prompts.Skill-Prompt, which provides skills only during evaluation, underperformsSeedon every aggregate metric across all three backbones.Seedalso exceeds Skill-GRPO* in 11 of the 12 aggregate comparisons, despite using no skill context during evaluation. These results indicate that hindsight skills are more effective when distilled into the policy than when supplied as additional context at inference time.

Self-evolving hindsight distillation is stronger than static self-distillation.Across the static distillation baselines,Seedachieves the best or tied best result in 10 of the 12 aggregate comparisons. The advantage is clearest on ALFWorld, whereSeedoutperforms the strongest static baseline by 7.4 points with Qwen2.5-3B, 10.2 points with Qwen2.5-7B, and 38.1 points with Qwen3-1.7B. This pattern supports the benefit of synchronizing the analyzer with the latest policy, so hindsight supervision adapts to the trajectories and failure modes encountered during training.

4.3Training Dynamics

Figure3compares the optimization dynamics ofSeedand GRPO on ALFWorld. The success-rate curves diverge early: by training step 40,Seedreaches roughly 57% while GRPO remains near 35%, and the advantage persists thereafter.Seedalso reduces the mean episode length more quickly, from approximately 28 turns to 13, compared with about 16 turns for GRPO at the end of training. Because shorter trajectories coincide with higher success, this reduction reflects more efficient task execution rather than premature termination. Together, these trends indicate thatSeedimproves task completion and interaction efficiency by reducing unproductive exploration and learning more direct solution strategies.

4.4Sample Efficiency

Figure5shows thatSeedconsistently outperforms GRPO across all data fractions and can match or surpass the performance of GRPO trained with substantially more data. Using only 60% of the training instances,Seedachieves a score of 80.7, exceeding the 75.0 obtained by GRPO with the full training set. Similarly, with 40% of the data,Seedreaches 58.9, closely matching GRPO trained with twice as much data, which achieves 58.6 at the 80% setting. These results demonstrate thatSeedextracts more informative and effective supervision from each completed trajectory than outcome-only RL based solely on terminal rewards. Detailed results are provided in AppendixC.1.

4.5Cross-Domain Generalization

We further evaluate the 3B checkpoint trained withSeedand GRPO on the ALFWorld unseen split. As shown in Figure5,Seedincreases the macro-average success rate from 70.9 to 86.2, outperforming GRPO by 15.3 points and achieving better performance in five of the six task families. The largest improvements are observed onHeat(+35.0),Look(+18.3), andPick(+16.5). These results indicate that the policy trained withSeedacquires reusable behavioral strategies that transfer effectively to unseen environments, rather than merely memorizing the training trajectories. Detailed results are provided in AppendixC.2.

Figure 4:Sample efficiency analysis.Seedconsistently outperforms GRPO across different data fraction settings and surpasses full-data GRPO using only 60% of the training data. Figure 5:Cross-domain generalizability on ALFWorld Unseen.Seedgenerally outperforms GRPO across unseen task types, demonstrating stronger cross-domain generalizability.

Table 2:Ablation Results.We reportSeedperformance and its ablated variants on ALFWorld.ALFWorldMethodPickLookCleanHeatCoolPick2Avg.Seed100.0100.0100.0100.070.680.091.8w/o Hindsight Skill SFT97.681.892.690.084.270.086.0w/o Self-Evolving OPD93.370.096.984.676.9100.087.0w/o On-Policy Skill97.162.5100.061.975.084.284.4

4.6Ablation Studies and Analysis

Table2evaluates the contribution of the three core components ofSeed.

The impact of hindsight-skill SFT.Removing hindsight-skill SFT decreases the ALFWorld average from 91.8 to 86.0, corresponding to a 5.8-point drop. This result shows that equipping the actor and analyzer with an initial trajectory-analysis capability provides an important foundation for the subsequent self-evolving training process.

The impact of self-evolving OPD.Removing self-evolving OPD reduces the average performance to 87.0, a decrease of 4.8 points. This indicates that the one-time skill supervision introduced during SFT is insufficient on its own, and that continuously distilling supervision from newly collected trajectories is necessary to keep pace with the evolving policy.

The impact of on-policy skills.Replacing on-policy skills with skills from a static offline library leads to the largest performance degradation, lowering the average to 84.4 by 7.4 points. This finding highlights the importance of deriving guidance from the current policy’s own trajectories, as such skills better reflect its evolving behaviors and failure modes.

Qualitative Analysis.

Figure6presents a representative example of the behavioral differences between GRPO andSeed. GRPO visits the target receptacle before locating the required object, interacts with irrelevant objects, and repeatedly explores unproductive locations until reaching the step limit. In contrast,Seedfollows a more coherent and goal-directed trajectory: it searches plausible storage locations, identifies the candle on the second shelf, and completes the task with the correct placement action. The more direct trajectory exhibited bySeedis consistent with its stronger performance and higher interaction efficiency reported in the preceding experiments.

Figure 6:Qualitative comparison on ALFWorld.For the task “put a candle in toilet,” the GRPO- trained agent searches the target receptacle before locating the candle, takes an irrelevant toilet-paper item, and then enters an off-task loop. In contrast, SEED systematically checks plausible shelves, finds the candle, and completes the required placement in five steps.

5Conclusion

We introducedSeed, a self-evolving on-policy distillation framework for long-horizon agentic reinforcement learning.Seedbridges sparse trajectory-level outcomes and token-level policy learning by extracting hindsight skills from completed on-policy trajectories and distilling their behavioral effects into dense supervision. The latest policy serves as both the actor and the analyzer, allowing its behavior and the supervision derived from its experience to evolve together while remaining aligned with the current trajectory distribution. Joint optimization with outcome-based RL enables the policy to internalize reusable guidance without relying on skills at inference time. Experiments across embodied interaction, web navigation, search-based QA, visual reasoning and planning demonstrate consistent performance improvements, sample efficiency, and robustness over powerful baselines.

References

• R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024)On-policy distillation of language models: learning from self-generated mistakes.InInternational Conference on Learning Representations,External Links:LinkCited by:§1,§2.
• M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba (2017)Hindsight experience replay.Advances in neural information processing systems30.Cited by:§1,§1,§2.
• J. A. Arjona-Medina, M. Gillhofer, M. Widrich, T. Unterthiner, J. Brandstetter, and S. Hochreiter (2019)RUDDER: return decomposition for delayed rewards.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:§1,§2.
• S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang,et al.(2025)Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923.Cited by:§C.3.
• X. Cheng, X. Yu, C. Shao, J. Li, Y. Xiong, Y. Qian, J. Zhu, S. Ma, X. Zhang, J. Ye,et al.(2026)DSpark: confidence-scheduled speculative decoding with semi-autoregressive generation.arXiv preprint arXiv:2607.05147.Cited by:Appendix E.
• K. Fang, X. Che, H. Ouyang, S. Zhang, X. Wang, Q. Liu, L. Liu, C. Zhang, W. Cai, W. Dai,et al.(2026)RobotEQ: transitioning from passive intelligence to active intelligence in embodied ai.arXiv preprint arXiv:2605.06234.Cited by:Appendix E.
• Y. Ge, S. Romeo, J. Cai, M. Sunkara, and Y. Zhang (2025)SAMULE: self-learning agents enhanced by multi-level reflection.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,Suzhou, China,pp. 16591–16610.External Links:Document,LinkCited by:Appendix E.
• G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network.External Links:1503.02531,LinkCited by:§1,§2.
• X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa (2020)Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps.InProceedings of the 28th International Conference on Computational Linguistics,pp. 6609–6625.External Links:Document,LinkCited by:§B.1,§4.1.
• J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause (2026)Reinforcement learning via self-distillation.External Links:2601.20802,LinkCited by:§1.
• C. Jiang, C. Chan, W. Xue, Q. Liu, and Y. Guo (2025a)Importance weighting can help large language models self-improve.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.39,pp. 24257–24265.External Links:DocumentCited by:Appendix E.
• D. Jiang, J. Zhang, O. Weller, N. Weir, B. Van Durme, and D. Khashabi (2025b)SELF-[in]correct: llms struggle with discriminating self-generated responses.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.39,pp. 24266–24275.External Links:DocumentCited by:Appendix E.
• B. Jin, H. Zeng, Z. Yue, D. Wang, H. Zamani, and J. Han (2025)Search-R1: training LLMs to reason and leverage search engines with reinforcement learning.External Links:2503.09516,LinkCited by:§B.1,§4.1.
• M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer (2017)TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics,pp. 1601–1611.External Links:Document,LinkCited by:§B.1,§4.1.
• Y. Kim and A. M. Rush (2016)Sequence-level knowledge distillation.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp. 1317–1327.External Links:Document,LinkCited by:§1,§2.
• J. Ko, S. Abdali, Y. J. Kim, T. Chen, and P. Cameron (2026)Scaling reasoning efficiently via relaxed on-policy distillation.External Links:2603.11137,LinkCited by:§1.
• A. Kumar, V. Zhuang, R. Agarwal, Y. Su, J. Co-Reyes, A. Singh, K. Baumli, S. Iqbal, C. Bishop, R. Roelofs, L. Zhang, K. McKinney, D. Shrivastava, C. Paduraru, G. Tucker, D. Precup, F. Behbahani, and A. Faust (2025)Training language models to self-correct via reinforcement learning.InInternational Conference on Learning Representations,External Links:LinkCited by:Appendix E.
• T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019)Natural questions: a benchmark for question answering research.Transactions of the Association for Computational Linguistics7,pp. 452–466.External Links:Document,LinkCited by:§B.1,§4.1.
• Z. Li, Z. Li, Y. Shi, R. Wang, J. Yang, Z. Liu, X. Wu, A. Li, Y. Yu, N. Liu, L. Sun, H. Mi, and LeoweiLiang (2026)Long-horizon-terminal-bench: testing the limits of agents on long-horizon terminal tasks with dense reward-based grading.External Links:2607.08964,LinkCited by:Appendix E.
• H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2024)Let’s verify step by step.InInternational Conference on Learning Representations,External Links:LinkCited by:§1,§2.
• X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, S. Zhang, X. Deng, A. Zeng, Z. Du, C. Zhang, S. Shen, T. Zhang, Y. Su, H. Sun, M. Huang, Y. Dong, and J. Tang (2023)AgentBench: evaluating LLMs as agents.External Links:2308.03688,LinkCited by:§1,§2.
• Z. Lu, Z. Yao, Z. Han, Z. Wang, J. Wu, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen (2026a)Self-distilled agentic reinforcement learning.External Links:2605.15155,LinkCited by:5th item,Appendix E,§1,§2,§3.3,§4.1.
• Z. Lu, Z. Yao, J. Wu, C. Han, Q. Gu, X. Cai, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen (2026b)Skill0: in-context agentic reinforcement learning for skill internalization.arXiv preprint arXiv:2604.02268.Cited by:§2.
• J. Luo, W. Zhang, Y. Yuan, Y. Zhao, J. Yang, Y. Gu, B. Wu, B. Chen, Z. Qiao, Q. Long,et al.(2025a)Large language model agent: a survey on methodology, applications and challenges.arXiv preprint arXiv:2503.21460.Cited by:§1,§2.
• X. Luo, Y. Zhang, Z. He, Z. Wang, S. Zhao, D. Li, L. K. Qiu, and Y. Yang (2025b)Agent lightning: train ANY AI agents with reinforcement learning.External Links:2508.03680,LinkCited by:§1,§2.
• A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark (2023)Self-refine: iterative refinement with self-feedback.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:§1,§2.
• T. Mahbub and S. Feng (2026)Mitigating self-preference by authorship obfuscation.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.40,pp. 37701–37708.External Links:DocumentCited by:Appendix E.
• A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2023)When not to trust language models: investigating effectiveness of parametric and non-parametric memories.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics,pp. 9802–9822.External Links:Document,LinkCited by:§B.1,§4.1.
• S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez (2024)Gorilla: large language model connected with massive APIs.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:§1,§2.
• O. Press, M. Zhang, S. Min, L. Schmidt, N. Smith, and M. Lewis (2023)Measuring and narrowing the compositionality gap in language models.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp. 5687–5711.External Links:Document,LinkCited by:§B.1,§4.1.
• Z. Qi, S. M. Baby, S. A. Baby, K. Yuan, A. Tomkins, T. Vu, D. Juan, and C. Rashtchian (2026)On the generalization gap in self-evolving language model reasoning.InForty-third International Conference on Machine Learning,External Links:LinkCited by:Appendix E.
• S. Ross, G. J. Gordon, and J. A. Bagnell (2011)A reduction of imitation learning and structured prediction to no-regret online learning.InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,pp. 627–635.External Links:LinkCited by:§1,§2.
• T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023)Toolformer: language models can teach themselves to use tools.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:§1,§2.
• M. B. Schrader (2018)Gym-sokoban.GitHub.Note:https://github.com/mpSchrader/gym-sokobanCited by:§C.3.
• Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024)DeepSeekMath: pushing the limits of mathematical reasoning in open language models.External Links:2402.03300,LinkCited by:1st item,§1,§2,§4.1.
• Y. Shen, T. Liu, J. Shen, J. Wu, Q. Kong, H. Li, and C. Wang (2026)Double: breaking the acceleration limit via double retrieval speculative parallelism.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),San Diego, California, United States,pp. 19242–19263.Cited by:Appendix E.
• N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao (2023)Reflexion: language agents with verbal reinforcement learning.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:§1,§2.
• M. Shridhar, X. Yuan, M. Côté, Y. Bisk, A. Trischler, and M. Hausknecht (2021)ALFWorld: aligning text and embodied environments for interactive learning.InInternational Conference on Learning Representations,External Links:LinkCited by:§B.1,§4.1.
• H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2022)MuSiQue: multihop questions via single-hop question composition.Transactions of the Association for Computational Linguistics10,pp. 539–554.External Links:Document,LinkCited by:§B.1,§4.1.
• J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins (2022)Solving math word problems with process- and outcome-based feedback.External Links:2211.14275,LinkCited by:§1,§2.
• G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar (2023)Voyager: an open-ended embodied agent with large language models.External Links:2305.16291,LinkCited by:§1,§2.
• H. Wang, G. Wang, H. Xiao, Y. Zhou, Y. Pan, J. Wang, K. Xu, Y. Wen, X. Ruan, X. Chen, and H. Qi (2026)Skill-SD: skill-conditioned self-distillation for multi-turn LLM agents.External Links:2604.10674,LinkCited by:3rd item,§1,§2,§4.1.
• Z. Wang, K. Wang, Q. Wang, P. Zhang, L. Li, Z. Yang, X. Jin, K. Yu, M. N. Nguyen, L. Liu, E. Gottlieb, Y. Lu, K. Cho, J. Wu, L. Fei-Fei, L. Wang, Y. Choi, and M. Li (2025)RAGEN: understanding self-evolution in LLM agents via multi-turn reinforcement learning.External Links:2504.20073,LinkCited by:§1,§2.
• J. Wu, M. Feng, S. Zhang, F. Che, Z. Wen, C. Liao, and J. Tao (2024)Beyond examples: high-level automated reasoning paradigm in in-context learning via mcts.arXiv preprint arXiv:2411.18478.Cited by:Appendix E.
• J. Wu, S. Yang, Y. Shen, S. Zhang, Z. Wen, and J. Tao (2026a)SPARK: strategic policy-aware exploration via dynamic branching for long-horizon agentic learning.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),San Diego, California, United States,pp. 23981–24004.Cited by:Appendix E,§1,§2.
• J. Wu, G. Zhai, R. Jin, J. Yuan, Y. Shen, S. Zhang, Z. Wen, and J. Tao (2026b)ATLAS: orchestrating heterogeneous models and tools for multi-domain complex reasoning.InFindings of the Association for Computational Linguistics: ACL 2026,San Diego, California, United States,pp. 17503–17535.Cited by:§1,§2.
• Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui (2025)The rise and potential of large language model based agents: a survey.Science China Information Sciences.External Links:DocumentCited by:§1,§2.
• F. Xu, H. Yan, Q. Sun, J. Wu, Z. Huang, M. Huang, J. Gong, Z. Ding, K. Cheng, Y. Wang,et al.(2026)OdysseyArena: benchmarking large language models for long-horizon, active and inductive interactions.arXiv preprint arXiv:2602.05843.Cited by:Appendix E,§1,§2.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu (2025)Qwen3 technical report.arXiv preprint arXiv:2505.09388.Cited by:§4.1.
• A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2024)Qwen2.5 technical report.arXiv preprint arXiv:2412.15115.Cited by:§4.1.
• C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan (2026a)Self-distilled RLVR.External Links:2604.03128,LinkCited by:4th item,§4.1.
• S. Yang, J. Wu, Z. Lu, Y. Shen, F. Zhang, L. Feng, S. Zhang, H. Luo, Z. Lian, Z. Wen,et al.(2026b)OPID: on-policy skill distillation for agentic reinforcement learning.arXiv preprint arXiv:2606.26790.Cited by:§1,§2.
• Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018)HotpotQA: a dataset for diverse, explainable multi-hop question answering.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp. 2369–2380.External Links:Document,LinkCited by:§B.1,§4.1.
• S. Yao, H. Chen, J. Yang, and K. Narasimhan (2022)WebShop: towards scalable real-world web interaction with grounded language agents.InAdvances in Neural Information Processing Systems,External Links:LinkCited by:§B.1,§4.1.
• Z.ai (2026)GLM-5.2: Built for Long-Horizon Tasks.Note:https://z.ai/blog/glm-5.2Accessed: 2026-06-22Cited by:§B.4,§4.1.
• Y. Zhai, H. Bai, Z. Lin, J. Pan, S. Tong, Y. Zhou, A. Suhr, S. Xie, Y. LeCun, Y. Ma, and S. Levine (2024)Fine-tuning large vision-language models as decision-making agents via reinforcement learning.InAdvances in Neural Information Processing Systems,A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.),Vol.37,pp. 110935–110971.External Links:Document,LinkCited by:§C.3.
• G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, Y. Zhou, Y. Chen, C. Zhang, Y. Fan, Z. Wang, S. Huang, F. P. Velez, Y. Liao, H. WANG, M. Yang, H. Ji, J. Wang, S. YAN, P. Torr, and L. BAI (2026a)The landscape of agentic reinforcement learning for LLMs: a survey.Transactions on Machine Learning Research.Note:Survey CertificationExternal Links:ISSN 2835-8856Cited by:§1.
• Y. Zhang, S. Jiang, R. Li, J. Tu, Y. Su, L. Deng, X. Guo, C. Lv, and J. Lin (2026b)DeepPlanning: benchmarking long-horizon agentic planning with verifiable constraints.arXiv preprint arXiv:2601.18137.Cited by:Appendix E.
• A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang (2024)ExpeL: LLM agents are experiential learners.InProceedings of the AAAI Conference on Artificial Intelligence,External Links:LinkCited by:§1,§2.
• S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026)Self-distilled reasoner: on-policy self-distillation for large language models.External Links:2601.18734,LinkCited by:1st item,§1,§2,§4.1.
• Q. Zhong, M. Zheng, M. Song, X. Lin, J. Sun, H. Jiang, X. Wang, and J. Fang (2026)SOD: step-wise on-policy distillation for small language model agents.External Links:2605.07725,LinkCited by:§1,§2.
• T. Zhou, J. Medina, and S. Chawla (2026)Can llms detect their confabulations? estimating reliability in uncertainty-aware language models.InProceedings of the AAAI Conference on Artificial Intelligence,Vol.40,pp. 38164–38172.External Links:DocumentCited by:Appendix E.

Appendix ATheoretical Analysis

This section formalizes three properties ofSeedthat mirror the requirements identified in Section1: the hindsight supervision ison-policy,dense, andself-evolving. Specifically, we show that (i) synchronized on-policy hindsight induces an occupancy-matched adaptive target, (ii) the OPD term provides decision-specific credit even when outcome-based group advantages are uninformative, and (iii) refreshing the shared analyzer controls the staleness of the auxiliary gradient as the policy changes. These are local statements about the structure and freshness of theSeedupdate; they do not by themselves imply monotonic return improvement, which additionally requires the generated hindsight skills to be behaviorally informative.

Notation.

Consider outer iterationkk. At the beginning of the iteration, the current checkpoint is frozen as the behavior policyπk=πθk\pi_{k}=\pi_{\theta_{k}}and also instantiates the trajectory analyzer𝒜k\mathcal{A}_{k}. LetCCdenote an ordinary autoregressive context at a valid action-token position, and letdk​(c)d_{k}(c)be the normalized token-context occupancy induced byπk\pi_{k}and the environment. Conditional onC=cC=c, the realized rollout tokenYYis sampled as

Y∼πk(⋅∣c).Y\sim\pi_{k}(\cdot\mid c).After the complete trajectoryτ\tauis observed, the synchronized analyzer produces the hindsight skillS=𝒜k​(xτ)S=\mathcal{A}_{k}(x_{\tau}). We overloadH​(c,S)H(c,S)to denote the skill-augmented version ofcc, with the same action-token prefix as in the ordinary context. At update initialization, define the detached skill gate

gk​(c,v,S)=σ​(βopd​[log⁡πk​(v∣H​(c,S))−log⁡πk​(v∣c)]).g_{k}(c,v,S)=\sigma\!\left(\beta_{\mathrm{opd}}\left[\log\pi_{k}(v\mid H(c,S))-\log\pi_{k}(v\mid c)\right]\right).(2)Importantly,SSis generated from thecompletetrajectory and can therefore depend on the realized tokenYY, the subsequent rollout, environment feedback, and analyzer decoding. To preserve this dependence, we define the conditional expected gate

wk​(c,v)=𝔼​[gk​(C,Y,S)∣C=c,Y=v],w_{k}(c,v)=\mathbb{E}\!\left[g_{k}(C,Y,S)\mid C=c,\,Y=v\right],(3)where the expectation is over the future trajectory, environment randomness, and any analyzer randomness. Standard softmax policies have full support, and the sigmoid lies strictly between zero and one, so0<wk​(c,v)<10<w_{k}(c,v)<1. We state the results atθ=θk\theta=\theta_{k}, where the behavior, analyzer, teacher branch, and student branch are synchronized. The adaptive-target identity below extends to an inner optimization step by replacing Eq.2with its current detached numerical value while retaining the same behavior occupancydkd_{k}.

A.1On-Policy Hindsight Produces an Occupancy-Matched Adaptive Target

The first result characterizes the expected OPD direction and makes explicit why the source trajectories should be generated by the current policy. Rather than imitating every sampled token equally,Seedreweights the current policy according to how strongly its own hindsight skill supports each action.

Proposition 1(Occupancy-matched hindsight target).

For every contextcc, define

Zk​(c)=∑v∈𝒱πk​(v∣c)​wk​(c,v),rk​(v∣c)=πk​(v∣c)​wk​(c,v)Zk​(c).Z_{k}(c)=\sum_{v\in\mathcal{V}}\pi_{k}(v\mid c)w_{k}(c,v),\qquad r_{k}(v\mid c)=\frac{\pi_{k}(v\mid c)w_{k}(c,v)}{Z_{k}(c)}.(4)Then0<Zk​(c)<10<Z_{k}(c)<1,rk(⋅∣c)r_{k}(\cdot\mid c)is a probability distribution, and the expected OPD gradient at update initialization satisfies

∇θℒopd,k(θ)|θ=θk=𝔼c∼dk[Zk(c)∇θDKL(rk(⋅∣c)∥πθ(⋅∣c))|θ=θk].\left.\nabla_{\theta}\mathcal{L}_{\mathrm{opd},k}(\theta)\right|_{\theta=\theta_{k}}=\mathbb{E}_{c\sim d_{k}}\left[Z_{k}(c)\left.\nabla_{\theta}D_{\mathrm{KL}}\left(r_{k}(\cdot\mid c)\,\middle\|\,\pi_{\theta}(\cdot\mid c)\right)\right|_{\theta=\theta_{k}}\right].(5)Thus, the OPD update is a Monte Carlo estimate of distillation toward a skill-reweighted target on the current policy’s own token-context occupancy.

Moreover, letμk\mu_{k}be the joint distribution of a completed trajectory and one of its valid token samples, and letϕk,θ​(x)\phi_{k,\theta}(x)be the detached per-token OPD gradient obtained by analyzing samplexxwith the current analyzer𝒜k\mathcal{A}_{k}. If‖ϕk,θ​(x)‖2≤G\|\phi_{k,\theta}(x)\|_{2}\leq Gfor all relevantxx, then replacing current on-policy samples by samples from an earlier distributionμj\mu_{j}incurs

‖𝔼x∼μk​[ϕk,θ​(x)]−𝔼x∼μj​[ϕk,θ​(x)]‖2≤2​G​TV⁡(μk,μj).\left\|\mathbb{E}_{x\sim\mu_{k}}[\phi_{k,\theta}(x)]-\mathbb{E}_{x\sim\mu_{j}}[\phi_{k,\theta}(x)]\right\|_{2}\leq 2G\,\operatorname{TV}(\mu_{k},\mu_{j}).(6)Ifμk\mu_{k}is absolutely continuous with respect toμj\mu_{j}, Pinsker’s inequality further gives

‖𝔼μk​[ϕk,θ]−𝔼μj​[ϕk,θ]‖2≤G​2​DKL​(μk∥μj).\left\|\mathbb{E}_{\mu_{k}}[\phi_{k,\theta}]-\mathbb{E}_{\mu_{j}}[\phi_{k,\theta}]\right\|_{2}\leq G\sqrt{2D_{\mathrm{KL}}(\mu_{k}\|\mu_{j})}.(7)

Proof.

Because0<wk​(c,v)<10<w_{k}(c,v)<1andπk(⋅∣c)\pi_{k}(\cdot\mid c)is normalized, Eq.4gives0<Zk​(c)<10<Z_{k}(c)<1and∑vrk​(v∣c)=1\sum_{v}r_{k}(v\mid c)=1. In Eq.1, both the teacher log-probability and the gate are detached, so only the ordinary student log-probability contributes to the gradient. Conditioning first onC=cC=candY=vY=v, and then using Eq.3, yields

∇θℒopd,k​(θ)|θ=θk\displaystyle\left.\nabla_{\theta}\mathcal{L}_{\mathrm{opd},k}(\theta)\right|_{\theta=\theta_{k}}=−𝔼c∼dk​[∑v∈𝒱πk​(v∣c)​wk​(c,v)​∇θlog⁡πθ​(v∣c)|θ=θk]\displaystyle\quad=-\mathbb{E}_{c\sim d_{k}}\left[\sum_{v\in\mathcal{V}}\pi_{k}(v\mid c)w_{k}(c,v)\left.\nabla_{\theta}\log\pi_{\theta}(v\mid c)\right|_{\theta=\theta_{k}}\right]=−𝔼c∼dk​[Zk​(c)​∑v∈𝒱rk​(v∣c)​∇θlog⁡πθ​(v∣c)|θ=θk],\displaystyle\quad=-\mathbb{E}_{c\sim d_{k}}\left[Z_{k}(c)\sum_{v\in\mathcal{V}}r_{k}(v\mid c)\left.\nabla_{\theta}\log\pi_{\theta}(v\mid c)\right|_{\theta=\theta_{k}}\right],which is Eq.5, since the entropy of the detached targetrkr_{k}has zero gradient.

For Eq.6, use the dual representation of the Euclidean norm and the variational characterization of total variation. For any unit vectoruu,

|𝔼μk​[u⊤​ϕk,θ]−𝔼μj​[u⊤​ϕk,θ]|≤2​G​TV⁡(μk,μj),\left|\mathbb{E}_{\mu_{k}}[u^{\top}\phi_{k,\theta}]-\mathbb{E}_{\mu_{j}}[u^{\top}\phi_{k,\theta}]\right|\leq 2G\,\operatorname{TV}(\mu_{k},\mu_{j}),because|u⊤​ϕk,θ​(x)|≤G|u^{\top}\phi_{k,\theta}(x)|\leq G. Taking the supremum overuuproves Eq.6. Equation7follows fromTV⁡(μk,μj)≤DKL​(μk∥μj)/2\operatorname{TV}(\mu_{k},\mu_{j})\leq\sqrt{D_{\mathrm{KL}}(\mu_{k}\|\mu_{j})/2}. ∎

Equation4separates the two roles of the firstSeedrequirement. The factordk​(c)d_{k}(c)makes supervisionoccupancy matched: it focuses learning on contexts, actions, and failure modes actually visited by the current policy. Within each such context,wk​(c,v)w_{k}(c,v)makes supervisionskill selective: actions that are more strongly supported by the trajectory-specific hindsight skill receive greater relative mass. In particular,

rk​(u∣c)πk​(u∣c)>rk​(v∣c)πk​(v∣c)⟺wk​(c,u)>wk​(c,v).\frac{r_{k}(u\mid c)}{\pi_{k}(u\mid c)}>\frac{r_{k}(v\mid c)}{\pi_{k}(v\mid c)}\quad\Longleftrightarrow\quad w_{k}(c,u)>w_{k}(c,v).The mismatch bounds show the complementary point: even if the same current analyzer is applied to old trajectories, a stale behavior distribution can bias the auxiliary direction in proportion to its divergence from the current trajectory-token distribution. On-policy collection sets this data-distribution mismatch to zero at the rollout stage.

The target also admits a useful value interpretation. LetQk​(c,v)Q_{k}(c,v)be the expected task return after choosing tokenvvat contextccand following the current policy thereafter. Then

𝔼v∼rk(⋅∣c)​[Qk​(c,v)]−𝔼v∼πk(⋅∣c)​[Qk​(c,v)]=Covv∼πk(⋅∣c)⁡(Qk​(c,v),wk​(c,v))Zk​(c).\mathbb{E}_{v\sim r_{k}(\cdot\mid c)}[Q_{k}(c,v)]-\mathbb{E}_{v\sim\pi_{k}(\cdot\mid c)}[Q_{k}(c,v)]=\frac{\operatorname{Cov}_{v\sim\pi_{k}(\cdot\mid c)}\!\left(Q_{k}(c,v),w_{k}(c,v)\right)}{Z_{k}(c)}.(8)Therefore, whenever hindsight support is positively correlated with current-policy action value, the reweighted target has higher local expected value than the unweighted policy. This condition makes explicit what the theorem does and does not assume:Seedconverts hindsight support into an on-policy target, while the empirical benefit depends on the generated skills assigning greater support to better decisions.

A.2Dense Skill Credit Remains Informative under Sparse or Tied Rewards

Outcome-based RL assigns the same trajectory-level advantage to every valid action token in a rollout. Consequently, it cannot distinguish a locally useful decision from a harmful one within the same trajectory, and its reward-driven signal disappears completely when all outcomes in a group are tied. The next result shows that the skill-conditioned OPD term can remain non-degenerate in exactly this regime.

Proposition 2(Decision-specific signal under reward ties).

Consider a rollout group in which every trajectory has the same outcome. ThenAq,n,t,ℓrl=0A^{\mathrm{rl}}_{q,n,t,\ell}=0for every valid token, and the clipped reward-driven term in Eq.3.3has zero gradient. Fix a current-policy contextcc, parameterize the ordinary student distribution aspz(⋅∣c)=softmax(z(c))p_{z}(\cdot\mid c)=\operatorname{softmax}(z(c)), and define the conditional expected OPD loss up to detached constants by

ℒ¯opd,k,c​(z)=−∑v∈𝒱πk​(v∣c)​wk​(c,v)​log⁡pz​(v∣c).\overline{\mathcal{L}}_{\mathrm{opd},k,c}(z)=-\sum_{v\in\mathcal{V}}\pi_{k}(v\mid c)w_{k}(c,v)\log p_{z}(v\mid c).(9)At update initialization, wherepz(⋅∣c)=πk(⋅∣c)p_{z}(\cdot\mid c)=\pi_{k}(\cdot\mid c),

∂ℒ¯opd,k,c∂z​(c,v)=πk​(v∣c)​[Zk​(c)−wk​(c,v)].\frac{\partial\overline{\mathcal{L}}_{\mathrm{opd},k,c}}{\partial z(c,v)}=\pi_{k}(v\mid c)\left[Z_{k}(c)-w_{k}(c,v)\right].(10)Furthermore, with the inverse-policy weighted norm

‖a‖πk−12=∑v∈𝒱a​(v)2πk​(v∣c),\|a\|^{2}_{\pi_{k}^{-1}}=\sum_{v\in\mathcal{V}}\frac{a(v)^{2}}{\pi_{k}(v\mid c)},the gradient magnitude satisfies the exact identity

‖∇z​(c)ℒ¯opd,k,c‖πk−12=Varv∼πk(⋅∣c)⁡[wk​(c,v)].\left\|\nabla_{z(c)}\overline{\mathcal{L}}_{\mathrm{opd},k,c}\right\|^{2}_{\pi_{k}^{-1}}=\operatorname{Var}_{v\sim\pi_{k}(\cdot\mid c)}\left[w_{k}(c,v)\right].(11)Hence the OPD gradient is nonzero if and only if the expected hindsight gate is non-constant over candidate tokens with positive current-policy probability.

Proof.

If all group outcomes are identical, thenR​(τq(n))−μq=0R(\tau_{q}^{(n)})-\mu_{q}=0for every rollout, so the group-relative advantage in Eq.3.3is zero. The clipped surrogate therefore contributes no reward-derived policy gradient; the separate KL regularizer may still contribute a gradient, but it does not provide task-outcome credit.

For the OPD term, the softmax derivative gives

∂[−log⁡pz​(u∣c)]∂z​(c,v)=pz​(v∣c)−𝟏​{u=v}.\frac{\partial[-\log p_{z}(u\mid c)]}{\partial z(c,v)}=p_{z}(v\mid c)-\mathbf{1}\{u=v\}.Applying this identity to Eq.9and settingpz=πkp_{z}=\pi_{k}yields

∂ℒ¯opd,k,c∂z​(c,v)\displaystyle\frac{\partial\overline{\mathcal{L}}_{\mathrm{opd},k,c}}{\partial z(c,v)}=(∑uπk​(u∣c)​wk​(c,u))​πk​(v∣c)−πk​(v∣c)​wk​(c,v)\displaystyle=\left(\sum_{u}\pi_{k}(u\mid c)w_{k}(c,u)\right)\pi_{k}(v\mid c)-\pi_{k}(v\mid c)w_{k}(c,v)=πk​(v∣c)​[Zk​(c)−wk​(c,v)],\displaystyle=\pi_{k}(v\mid c)\left[Z_{k}(c)-w_{k}(c,v)\right],which proves Eq.10. Substituting this gradient into the weighted norm gives

‖∇z​(c)ℒ¯opd,k,c‖πk−12\displaystyle\left\|\nabla_{z(c)}\overline{\mathcal{L}}_{\mathrm{opd},k,c}\right\|^{2}_{\pi_{k}^{-1}}=∑vπk​(v∣c)​[wk​(c,v)−Zk​(c)]2\displaystyle=\sum_{v}\pi_{k}(v\mid c)\left[w_{k}(c,v)-Z_{k}(c)\right]^{2}=Varv∼πk(⋅∣c)⁡[wk​(c,v)],\displaystyle=\operatorname{Var}_{v\sim\pi_{k}(\cdot\mid c)}\left[w_{k}(c,v)\right],becauseZk​(c)=𝔼v∼πk​[wk​(c,v)]Z_{k}(c)=\mathbb{E}_{v\sim\pi_{k}}[w_{k}(c,v)]. The variance is zero exactly whenwk​(c,v)w_{k}(c,v)is constantπk\pi_{k}-almost surely. ∎

Equations10–11formalize the dense credit supplied bySeed. Tokens withwk​(c,v)>Zk​(c)w_{k}(c,v)>Z_{k}(c)receive a negative logit derivative and are promoted by gradient descent, whereas tokens withwk​(c,v)<Zk​(c)w_{k}(c,v)<Z_{k}(c)are relatively suppressed. Thus, although every gate value is nonnegative, softmax normalization converts variation in hindsight support into signed relative credit over candidate decisions. The variance in Eq.11quantifies the strength of this token-level signal: the more the hindsight skill discriminates among alternatives, the larger the OPD update can be. This establishesinformativeness, not automatic correctness; whether the signal favors better actions is characterized by the value-alignment covariance in Eq.8.

The tied-reward case is the sharpest contrast, but the distinction is more general. GRPO broadcasts one scalarAq,nrlA^{\mathrm{rl}}_{q,n}across all valid tokens in a trajectory, whileSeedassigns a context- and token-dependent coefficient throughwk​(c,v)w_{k}(c,v). It can therefore preserve useful local behaviors in failed trajectories and attenuate locally inefficient choices in successful trajectories, even when the terminal reward alone cannot identify those decisions.

A.3Self-Evolving Synchronization Controls Analyzer Staleness

The final requirement concerns the source of the hindsight itself. A fixed analyzer or one-time skill dataset may be informative early in training but can become mismatched to the capabilities, strategies, and failure modes of a later policy. We quantify this mismatch by its effect on the detached OPD gate and hence on the auxiliary gradient.

For a completed token samplex=(τ,c,v)x=(\tau,c,v)from the current distributionμk\mu_{k}, letsr​(x)=𝒜r​(xτ)s_{r}(x)=\mathcal{A}_{r}(x_{\tau})be the skill produced by an analyzer checkpoint from iterationrr. Holding the current policyπk\pi_{k}fixed for this comparison, define

Δk,r​(x)=log⁡πk​(v∣H​(c,sr​(x)))−log⁡πk​(v∣c),gk,r​(x)=σ​(βopd​Δk,r​(x)).\Delta_{k,r}(x)=\log\pi_{k}(v\mid H(c,s_{r}(x)))-\log\pi_{k}(v\mid c),\qquad g_{k,r}(x)=\sigma\!\left(\beta_{\mathrm{opd}}\Delta_{k,r}(x)\right).(12)The first index identifies the policy whose probabilities are re-scored, while the second identifies the analyzer that supplies the skill. For a fixed trainable parameterθ\theta, define the resulting expected auxiliary gradient on current trajectories as

Uk​(𝒜r;θ)=−𝔼x∼μk​[gk,r​(x)​∇θlog⁡πθ​(v∣c)].U_{k}(\mathcal{A}_{r};\theta)=-\mathbb{E}_{x\sim\mu_{k}}\left[g_{k,r}(x)\nabla_{\theta}\log\pi_{\theta}(v\mid c)\right].(13)

Proposition 3(Analyzer-staleness bound).

Assume∥∇θlogπθ(v∣c)∥2≤G\|\nabla_{\theta}\log\pi_{\theta}(v\mid c)\|_{2}\leq Gfor all relevant current-policy token samples. Then the discrepancy between the OPD gradient induced by the synchronized analyzer𝒜k\mathcal{A}_{k}and that induced by an earlier analyzer𝒜j\mathcal{A}_{j}satisfies

‖Uk​(𝒜k;θ)−Uk​(𝒜j;θ)‖2\displaystyle\left\|U_{k}(\mathcal{A}_{k};\theta)-U_{k}(\mathcal{A}_{j};\theta)\right\|_{2}≤G​𝔼x∼μk​[|gk,k​(x)−gk,j​(x)|]\displaystyle\leq G\,\mathbb{E}_{x\sim\mu_{k}}\left[|g_{k,k}(x)-g_{k,j}(x)|\right](14)≤βopd​G4​𝔼x∼μk​[|Δk,k​(x)−Δk,j​(x)|].\displaystyle\leq\frac{\beta_{\mathrm{opd}}G}{4}\mathbb{E}_{x\sim\mu_{k}}\left[|\Delta_{k,k}(x)-\Delta_{k,j}(x)|\right].In particular, using the analyzer instantiated from the current checkpoint sets this cross-iteration analyzer mismatch to zero at the rollout-and-analysis stage.

Proof.

Subtract Eq.13forr=kr=kandr=jr=j, apply Jensen’s inequality, and use the score-norm bound:

‖Uk​(𝒜k;θ)−Uk​(𝒜j;θ)‖2\displaystyle\left\|U_{k}(\mathcal{A}_{k};\theta)-U_{k}(\mathcal{A}_{j};\theta)\right\|_{2}≤𝔼x∼μk[|gk,k(x)−gk,j(x)|∥∇θlogπθ(v∣c)∥2]\displaystyle\quad\leq\mathbb{E}_{x\sim\mu_{k}}\left[|g_{k,k}(x)-g_{k,j}(x)|\left\|\nabla_{\theta}\log\pi_{\theta}(v\mid c)\right\|_{2}\right]≤G​𝔼x∼μk​[|gk,k​(x)−gk,j​(x)|].\displaystyle\quad\leq G\,\mathbb{E}_{x\sim\mu_{k}}\left[|g_{k,k}(x)-g_{k,j}(x)|\right].Becausesupzσ′​(z)=1/4\sup_{z}\sigma^{\prime}(z)=1/4, the mapz↦σ​(βopd​z)z\mapsto\sigma(\beta_{\mathrm{opd}}z)isβopd/4\beta_{\mathrm{opd}}/4-Lipschitz. Applying this fact pointwise to Eq.12proves the second inequality. ∎

Proposition3measures analyzer staleness in the quantity that directly affects learning: the change in skill-induced log-probability shift on the current policy’s own trajectories. It does not require parameter distance between analyzers to be meaningful, nor does it assume that the updated analyzer is always more accurate. Instead, it states that if an old and a current analyzer produce skills with different behavioral implications, their OPD gradients can differ accordingly. The factorβopd/4\beta_{\mathrm{opd}}/4also exposes a trade-off: a sharper gate more strongly separates supported and unsupported tokens, but is correspondingly more sensitive to stale skill-induced shifts.

Combining the data and analyzer effects gives a compact decomposition of the two synchronization mechanisms. At a fixed current policyπk\pi_{k}and trainable parameterθ\theta, letUk,θ​(μ,𝒜r)U_{k,\theta}(\mu,\mathcal{A}_{r})denote Eq.13with samples drawn fromμ\mu. Under the same score-norm bound,

‖Uk,θ​(μk,𝒜k)−Uk,θ​(μj,𝒜j)‖2\displaystyle\left\|U_{k,\theta}(\mu_{k},\mathcal{A}_{k})-U_{k,\theta}(\mu_{j},\mathcal{A}_{j})\right\|_{2}(15)≤2​G​TV⁡(μk,μj)+βopd​G4​𝔼x∼μj​[|Δk,k​(x)−Δk,j​(x)|].\displaystyle\qquad\leq 2G\,\operatorname{TV}(\mu_{k},\mu_{j})+\frac{\beta_{\mathrm{opd}}G}{4}\mathbb{E}_{x\sim\mu_{j}}\left[|\Delta_{k,k}(x)-\Delta_{k,j}(x)|\right].The first term isexperience staleness, controlled by collecting trajectories on-policy; the second issupervision staleness, controlled by refreshing the analyzer from the latest shared checkpoint. At each outer iteration,Seeduses the synchronized pair(μk,𝒜k)(\mu_{k},\mathcal{A}_{k}), so both cross-iteration mismatch terms are reset at data generation. During the subsequent inner policy optimization, the analyzer remains fixed at𝒜k\mathcal{A}_{k}; therefore, the claim is not that analyzer lag is identically zero at every optimizer step, but that any within-update lag is discarded when trajectories and skills are regenerated from the next policy checkpoint.

Taken together, Propositions1,2, and3provide a one-to-one theoretical counterpart to the three design requirements in Section1. On-policy collection aligns the auxiliary objective with the current policy’s visited distribution, dense skill-conditioned re-scoring supplies token-specific credit beyond terminal outcomes, and the self-evolving shared analyzer prevents the hindsight target from remaining permanently tied to an earlier policy checkpoint.

Appendix BAdditional Experimental Details

This section provides additional experimental details, including the datasets, baseline methods, the completeSeedalgorithm and representative extracted skills, and the implementation details of our method.

B.1Datasets

Our experiments cover three representative forms of long-horizon agentic interaction: embodied household reasoning, web navigation, and search-augmented question answering. Table3summarizes the benchmarks and the data sizes used for SFT, RL training, and evaluation.

Table 3:Detailed information on the agentic benchmarks and training data.DomainBenchmark(s)#SFT TrainSamples#RL TrainSamples#TestSamplesEmbodiedReasoningALFWorld1802,400140 (seen)134 (unseen)WebNavigationWebShop1802,400128Search-AugmentedQANQ⋅\cdotTriviaQA⋅\cdotPopQAHotpotQA⋅\cdot2WikiMultiHopQAMuSiQue⋅\cdotBamboogle18019,20051,713#### ALFWorld.

ALFWorld(Shridharet al.,2021)exposes household environments derived from ALFRED through a text-based interaction interface. Given a natural-language instruction and a sequence of textual observations, the agent must select admissible actions to accomplish the specified household goal. We evaluate six task categories:Pick,Look,Clean,Heat,Cool, andPick2. We report the main results on the 140-task seen split and use the 134-task unseen split for out-of-distribution evaluation.

WebShop.

WebShop(Yaoet al.,2022)evaluates agents in an interactive e-commerce environment. For each user request, the agent searches the product catalog, examines candidate items, selects the required attributes, and attempts to complete a purchase satisfying the specified constraints. The benchmark provides a normalized task-completion score for partial constraint satisfaction and a binary success metric for exact completion. We evaluate all methods on 128 test samples.

Search-Augmented QA.

Following the Search-R1 protocol(Jinet al.,2025), we construct the search-based QA setting from seven datasets: Natural Questions(Kwiatkowskiet al.,2019), TriviaQA(Joshiet al.,2017), PopQA(Mallenet al.,2023), HotpotQA(Yanget al.,2018), 2WikiMultiHopQA(Hoet al.,2020), MuSiQue(Trivediet al.,2022), and Bamboogle(Presset al.,2023). The agent may issue search queries, inspect retrieved documents, and synthesize the collected evidence before returning its final answer. The combined evaluation set contains 51,713 questions.

Training data configuration.

For the SFT stage, we select 180 tasks from the training split of each benchmark setting. Each selected task is executed with 8 independent rollouts, yielding 1,440 trajectories for constructing the corresponding trajectory–skill training set. The subsequent RL stage uses 2,400 training instances for ALFWorld, 2,400 for WebShop, and 19,200 for Search-Augmented QA. SFT data construction and RL optimization are performed separately for each benchmark configuration.

B.2Baselines

We compareSeedagainst three categories of baselines: prompting-only methods, outcome-based reinforcement learning methods, and self-distillation or skill-distillation methods. Unless a method is marked with an asterisk, evaluation is conducted using only the standard task prompt and the interaction history returned by the environment. The superscript∗indicates that a natural-language skill is additionally supplied during validation or testing, while the backbone model and evaluation setting remain unchanged.

Prompting-only methods.

• •Vanilla. We directly evaluate the original instruction-tuned backbone without performing any additional post-training. The agent receives the default environment prompt together with the observation–action history accumulated during interaction.
• •Skill-Prompt∗. This baseline uses the same frozen parameters asVanilla, but appends a retrieved task-relevant skill to the model context during validation and testing. Since no parameter optimization is involved, this comparison measures the benefit of using natural-language skills purely as inference-time guidance.

Outcome-based reinforcement learning.

• •GRPO(Shaoet al.,2024). GRPO is a critic-free reinforcement learning algorithm that samples multiple trajectories for each task and normalizes their scalar outcome rewards within the rollout group to obtain relative advantages. Each valid token in a trajectory is optimized using the corresponding sequence-level advantage and a clipped importance-ratio objective. In our implementation, GRPO relies solely on terminal environment rewards and does not use process-level annotations, teacher predictions, or skill-conditioned supervision.
• •Skill-GRPO. This variant retains the standard GRPO optimization objective while introducing a task-relevant natural-language skill into the policy context during both rollout collection and parameter updates. The skill can therefore affect the policy’s exploration behavior and the trajectories subsequently reinforced by the outcome reward. It is removed during validation and testing, allowing us to assess whether the training-time guidance has been internalized by the policy.
• •Skill-GRPO∗. This baseline follows the same training procedure asSkill-GRPO, but continues to provide the skill during validation and testing. It therefore maintains consistent skill conditioning across training and evaluation.

Self-distillation and skill-distillation methods.

• •OPSD(Zhaoet al.,2026). OPSD derives student and teacher branches from the same underlying model while conditioning them on different contexts. The student generates trajectories from the ordinary task context, whereas the teacher additionally observes privileged information available only during training. The teacher then re-scores the student’s sampled tokens and produces dense token-level targets through distribution matching. Teacher-side outputs are detached from gradient computation, and the privileged context is not used at inference time.
• •GRPO+OPSD. This baseline jointly optimizes the trajectory-level GRPO loss and the token-level OPSD objective. Environment rewards provide outcome-based supervision for complete trajectories, while OPSD contributes dense guidance at individual token positions. It serves as a controlled comparison for determining whether a straightforward combination of outcome-based RL and generic on-policy self-distillation is sufficient.
• •Skill-SD(Wanget al.,2026). Skill-SD adapts self-distillation to multi-turn agent training by representing completed experience as compact natural-language skills. During optimization, a retrieved skill is supplied only to the teacher branch, while the student continues to operate from the ordinary task context. The student is therefore trained to absorb the behavioral guidance conveyed by the skill-conditioned teacher without requiring explicit skills during evaluation.
• •RLSD(Yanget al.,2026a). RLSD employs a privileged self-teacher to refine token-level credit assignment within reinforcement learning. Rather than introducing an independent distribution-matching objective, it converts the teacher–student log-probability gap into a bounded coefficient that adjusts the magnitude of each token’s GRPO update. The sign of the update remains determined by the environment-derived advantage, meaning that the privileged teacher controls update strength but not the reinforcement direction. The self-distillation contribution is emphasized early in training and gradually reduced toward the standard GRPO objective.
• •SDAR(Luet al.,2026a). SDAR preserves GRPO as the main outcome-optimization objective and adds a separately gated self-distillation loss. A privileged teacher branch evaluates the student’s on-policy tokens under an augmented context, while a bounded gate controls the contribution of each teacher signal. The gate may depend on student uncertainty and the detached teacher–student log-probability difference, strengthening reliable positive guidance and reducing the influence of potentially noisy signals. In contrast toRLSD, SDAR leaves the original GRPO advantage unchanged and applies its gating mechanism only to the auxiliary distillation term.

For all reproduced post-training baselines, we use the same backbone models and environment interfaces asSeed. Whenever supported by the corresponding method, we also align the task batch size, rollout budget, rollout group size, number of policy updates, and evaluation protocol. The resulting comparisons primarily differ in their optimization objectives and in whether skills or other privileged information are available during training or evaluation.

B.3Algorithm and Extracted Skill Examples

Algorithm1summarizes the self-evolving training loop. At each update, the frozen snapshotπθold\pi_{\theta_{\mathrm{old}}}both collects trajectories and analyzes the completed interactions. The current trainable policyπθ\pi_{\theta}then re-scores the same sampled tokens under ordinary and skill-augmented contexts; gradients flow only through the ordinary branch. The resulting OPD loss is optimized jointly with the KL-regularized GRPO objective before the updated policy becomes the next snapshot.

Table4provides representative skills extracted from successful and failed trajectories across ALFWorld, WebShop, and Search-based QA.

Algorithm 1Seed: Self-Evolving On-Policy Distillation1:SFT-initialized policy

πθsft\pi_{\theta_{\mathrm{sft}}}, task set

𝒬\mathcal{Q}, fixed reference policy

πref\pi_{\mathrm{ref}}, context function

HH, group size

NN, gate sharpness

βopd\beta_{\mathrm{opd}}, KL coefficient

βKL\beta_{\mathrm{KL}}, OPD coefficient

λopd\lambda_{\mathrm{opd}}, clip parameter

ϵclip\epsilon_{\mathrm{clip}}, learning rate

η\eta 2:Trained policy

πθ\pi_{\theta} 3:

θ←θsft\theta\leftarrow\theta_{\mathrm{sft}} 4:foreach policy updatedo

5:

θold←θ\theta_{\mathrm{old}}\leftarrow\theta 6:Sample a task batch

ℬ⊂𝒬\mathcal{B}\subset\mathcal{Q} 7:// On-policy experience and synchronized skill analysis

8:foreach task

q∈ℬq\in\mathcal{B}do

9:Sample

𝒢q={τq(n)}n=1N\mathcal{G}_{q}=\{\tau_{q}^{(n)}\}_{n=1}^{N}with

τq(n)∼πθold(⋅∣q)\tau_{q}^{(n)}\sim\pi_{\theta_{\mathrm{old}}}(\cdot\mid q) 10:Compute

μq\mu_{q}and

σq\sigma_{q}from

{R​(τq(n))}n=1N\{R(\tau_{q}^{(n)})\}_{n=1}^{N} 11:foreach trajectory

τq(n)∈𝒢q\tau_{q}^{(n)}\in\mathcal{G}_{q}do

12:

Aq,nrl←(R​(τq(n))−μq)/(σq+ϵ)A^{\mathrm{rl}}_{q,n}\leftarrow(R(\tau_{q}^{(n)})-\mu_{q})/(\sigma_{q}+\epsilon) 13:

sq(n)←Aθold​(xτq(n))s_{q}^{(n)}\leftarrow A_{\theta_{\mathrm{old}}}(x_{\tau_{q}^{(n)}}) 14:foreach action step

ttin

τq(n)\tau_{q}^{(n)}do

15:

h~q,n,t←H​(hq,n,t,sq(n))\tilde{h}_{q,n,t}\leftarrow H(h_{q,n,t},s_{q}^{(n)}) 16:Cache

ℓq,n,t,ℓold\ell^{\mathrm{old}}_{q,n,t,\ell}for all valid sampled tokens

ℓ\ell 17:endfor

18:endfor

19:endfor

20:// Paired contextual re-scoring and joint optimization

21:foreach optimization minibatch of valid sampled tokensdo

22:Evaluate

ℓq,n,t,ℓskill\ell^{\mathrm{skill}}_{q,n,t,\ell}and

ℓq,n,t,ℓθ\ell^{\theta}_{q,n,t,\ell}using the current

πθ\pi_{\theta} 23:

Δq,n,t,ℓ←sg⁡[ℓq,n,t,ℓskill−ℓq,n,t,ℓθ]\Delta_{q,n,t,\ell}\leftarrow\operatorname{sg}[\ell^{\mathrm{skill}}_{q,n,t,\ell}-\ell^{\theta}_{q,n,t,\ell}] 24:

gq,n,t,ℓ←σ​(βopd​Δq,n,t,ℓ)g_{q,n,t,\ell}\leftarrow\sigma(\beta_{\mathrm{opd}}\Delta_{q,n,t,\ell}) 25:

ρq,n,t,ℓ​(θ)←exp⁡(ℓq,n,t,ℓθ−ℓq,n,t,ℓold)\rho_{q,n,t,\ell}(\theta)\leftarrow\exp(\ell^{\theta}_{q,n,t,\ell}-\ell^{\mathrm{old}}_{q,n,t,\ell}) 26:Compute the clipped GRPO loss

ℒrl\mathcal{L}_{\mathrm{rl}}from

ρ\rho,

ArlA^{\mathrm{rl}}, and

ϵclip\epsilon_{\mathrm{clip}} 27:

ℒopd←𝔼​[m⋅g⋅(sg⁡[ℓskill]−ℓθ)]\mathcal{L}_{\mathrm{opd}}\leftarrow\mathbb{E}[m\cdot g\cdot(\operatorname{sg}[\ell^{\mathrm{skill}}]-\ell^{\theta})] 28:

ℒSEED←ℒrl+λopd​ℒopd\mathcal{L}_{\mathrm{SEED}}\leftarrow\mathcal{L}_{\mathrm{rl}}+\lambda_{\mathrm{opd}}\mathcal{L}_{\mathrm{opd}} 29:

θ←θ−η​∇θℒSEED\theta\leftarrow\theta-\eta\nabla_{\theta}\mathcal{L}_{\mathrm{SEED}} 30:endfor

31:endfor

Table 4:Episode-level skills extracted by the trajectory analyzer.For each dataset, we present one successful and one failed trajectory. Skills derived from successful episodes capture reusable workflows, whereas those derived from failed episodes distill actionable failure-avoidance rules.OutcomeTaskEpisode-level skillALFWorldSuccessclean some spatula and put it in diningtable.Workflow: When tasked to clean and place an object, first locate the target object, take it to a cleaning station, clean it, and finally move it to the target location.Failureput a clean spatula in drawer.Avoid moving an object to the target location before verifying both inventory and required object state. Confirm the object is held and already satisfies required conditions such as cleanliness; avoid repeatedly moving objects without checking state or placement success.WebShopSuccessFind me women’s jumpsuits, rompers & overalls with button closure, quality polyester, polyester spandex, long sleeve for daily wear with color: hot pink, and size: x-large, and price lower than $40.00 dollars.Workflow: Use a structured search query covering all required attributes, choose a result matching the core product category, then verify and select required options such as color and size before clicking Buy Now.FailureFind me machine wash men’s dress shirts with polyester heathers, heathers cotton, cotton heather, needle sleeve, classic fit with color: navy, and fit type: youth, and size: 3x-large, and price lower than $50.00 dollars.Avoid clicking irrelevant search results or purchasing partial matches. If results do not match the target product type, reformulate the query; on the product page, select each required attribute once and verify material, color, size, fit, and price before Buy Now.SearchSuccessTie a Yellow Ribbon is the third album by American popular music group Dawn (Michael Anthony Orlando Cassavitas, Telma Hopkins & Joyce Vincent Wilson) released in which year?Workflow: For a specific factual query, search the exact query, then extract and verify the answer directly from the search results before responding.FailureLe Juge is a comic whose story is inspired by a man who called himself what?Avoid prematurely inferring an answer from search results that do not directly address the core query. First verify the relevant entity/relation, then extract the requested attribute.

B.4Implementation Details

Benchmark metrics.

We adopt benchmark-specific measures following the corresponding evaluation protocols. For ALFWorld, letSRc\mathrm{SR}_{c}denote the success rate on task categorycc. The overall result assigns equal weight to all six categories and is computed as

ALFWorld​-​Avg=16​∑c=16SRc.\mathrm{ALFWorld\text{-}Avg}=\frac{1}{6}\sum_{c=1}^{6}\mathrm{SR}_{c}.(16) For Search-based QA, we use a dataset-balanced aggregate rather than pooling all questions together. Denoting the accuracy on datasetddbyAccd\mathrm{Acc}_{d}, the reported score is

Search​-​Avg=17​∑d=17Accd.\mathrm{Search\text{-}Avg}=\frac{1}{7}\sum_{d=1}^{7}\mathrm{Acc}_{d}.(17) For WebShop, we report two complementary metrics.Scoremeasures the degree of task completion by reflecting how fully the purchased product satisfies the requirements specified in the user request; the normalized scores are averaged and scaled by 100.Succ.denotes the percentage of episodes in which all requirements are satisfied and the task is completed exactly.

External analyzer at the SFT stage.

To construct the trajectory–skill supervision for SFT, we serialize each completed rollout into a structured record comprising the task instruction, the full interaction trajectory, and the terminal outcome. The external analyzer then examines the completed episode and produces a natural-language hindsight skill. The analyzer is used exclusively for offline skill annotation and does not participate in trajectory collection, which is carried out by the corresponding backbone model itself. We instantiate the external analyzer with GLM-5.2(Z.ai,2026), set the temperature to 0.0, and limit the maximum response length to 4,096 tokens. The complete prompt used for hindsight-skill annotation is provided in Figure10.

Actor and analyzer prompts at the RL stage.

During Stage 2, the actor follows the standard environment-interaction prompt for the corresponding benchmark; the prompt used for ALFWorld is shown in Figure11. The analyzer uses the same trajectory-analysis prompt as in the SFT stage, as presented in Figure10.

Training hyperparameters.

Table5records the training hyperparameters that are required for exact reproduction.

Table 5:Training hyperparameters forSeed.HyperparameterValueTraining steps150Training batch size16 for ALFWorld and WebShop; 128 for SearchRollout group sizeNN8Learning rate1×10−61\times 10^{-6}PPO clip parameterϵclip\epsilon_{\mathrm{clip}}0.2Sharpness of OPD gateβopd\beta_{\mathrm{opd}}5.0OPD loss coefficientλopd\lambda_{\mathrm{opd}}0.01KL coefficientβKL\beta_{\mathrm{KL}}0.01Maximum prompt length2,048 for ALFWorld ; 4,096 for WebShop and SearchResponse length512Maximum interaction steps30 for ALFWorld, 15 for WebShop, and 4 for Search.

Computing details.

Training is conducted on 8 Nvidia A800 80G GPUs.

Appendix CSupplementary Results

C.1Detailed Sample Efficiency Comparison

Table6provides the full comparison across five training data fractions on ALFWorld and WebShop.Seedoutperforms GRPO at every fraction on both benchmarks. On ALFWorld, the gains range from 13.4 to 30.2 points. With only 60% of the data,Seedreaches 80.7, exceeding the 75.0 achieved by GRPO with the full dataset. On WebShop,Seedimproves performance by 5.5 to 15.6 points and reaches 75.0 with 80% of the data, compared with 63.3 for GRPO trained on the full dataset. These results show that dense hindsight supervision makes more effective use of collected trajectories and remains beneficial as the training set grows.

Table 6:Sample efficiency comparison on ALFWorld and WebShop.We report performance using different proportions of the available training data. SEED consistently outperforms GRPO across both benchmarks, whileΔ\Deltadenotes the absolute performance improvement over GRPO.Method20%40%60%80%100%ALFWorldGRPO27.342.256.358.675.0SEED40.758.980.788.891.8Δ\Delta+13.4+16.7+24.4+30.2+16.8WebShopGRPO31.345.357.063.663.3SEED37.553.162.575.078.9Δ\Delta+6.2+7.8+5.5+11.4+15.6Table 7:Cross-domain generalization results on ALFWorld Unseen.Using Qwen2.5-3B-Instruct as the backbone, we evaluate performance on six unseen task categories. SEED outperforms GRPO on five categories and increases the average success rate by 15.3 percentage points.ALFWorld UnseenMethodPickLookCleanHeatCoolPick2Avg.ReAct17.46.78.87.49.10.08.2GRPO73.960.082.459.372.776.970.9SEED90.478.379.594.386.288.286.2Δ\Delta+16.5+18.3-2.9+35.0+13.5+11.3+15.3

C.2Cross-Domain Generalization

Table7reports results for each task family on ALFWorld Unseen.Seedimproves the average success rate from 70.9 to 86.2 and outperforms GRPO on five of the six families. The largest gain occurs onHeat, where the success rate increases by 35.0 points. Improvements are also substantial onLookandPick, reaching 18.3 and 16.5 points, respectively. Although performance onCleandecreases by 2.9 points, the broad gains across the remaining families indicate thatSeedlearns reusable behavioral guidance that transfers beyond the training environments.

C.3Multimodal Extension

To examine whetherSeedextends beyond text-only interaction, we evaluate Qwen2.5-VL-3B-Instruct(Baiet al.,2025)on two vision-based agentic benchmarks. Sokoban(Schrader,2018)presents each state as a6×66\times 6visual grid containing the player, boxes, walls, and target locations. The agent must navigate the grid and push every box onto a target. Since boxes cannot be pulled, a poor push may create an irreversible dead end. The task therefore tests visual state tracking and long-horizon spatial planning. EZPoints from Gym Cards(Zhaiet al.,2024)presents two playing cards together with a partially constructed expression. At each step, the agent selects a card value or an arithmetic operator to build an expression that evaluates to 12, using each card exactly once. This task couples fine-grained card recognition with sequential arithmetic reasoning.

As shown in Table8,Seedachieves success rates of 82.0% on Sokoban and 100.0% on EZPoints. It improves over GRPO by 14.9 and 13.1 points, respectively, raising the average success rate from 77.0% to 91.0%. ReAct reaches only 7.4% on average, which highlights the need for policy learning in these visually grounded tasks. The consistent gains across spatial planning and visual arithmetic show thatSeedcan turn multimodal trajectories into useful hindsight supervision and internalize that guidance through self-evolving OPD, without skill prompts during evaluation. Figure7presents a step-by-step visualization of a representative Sokoban trajectory.

Figure 7:A representative trajectory on Sokoban.The sequence shows six consecutive actions executed by the agent. Arrows indicate the temporal progression of the trajectory, and the action taken at each step is displayed below the corresponding observation.Table 8:Results on vision-based agentic benchmarks. We report success rates (%) on Sokoban [6×\times6] and EZPoints using Qwen2.5-VL-3B-Instruct as the backbone model.MethodSokoban [6×\times6]↑\uparrowEZPoints↑\uparrowAverage↑\uparrowReAct11.73.17.4GRPO67.186.977.0SEED82.0100.091.0Δ\Delta+14.9+13.1+14.0

C.4Additional Training Dynamics

Figure8presents the success-rate trajectories for all three backbones on ALFWorld, Search-based QA, and WebShop. Performance improves in every setting, although the convergence pattern varies by domain. ALFWorld shows the largest absolute increase and approaches a success rate of 0.9 for all three backbones. Search-based QA improves rapidly during the early updates before stabilizing between 0.47 and 0.55. WebShop follows a steadier upward trend and finishes between 0.68 and 0.75. The consistent progress across model families and environments shows thatSeedis not tied to a particular backbone or form of agentic interaction.

Figure 8:Success rates across three backbones and three domains.Success rates increase over training in all nine settings, showing consistent learning across model scales and agentic tasks.Figure 9:OPD loss dynamics.The loss generally decreases and stabilizes during training, indicating that the policy progressively internalizes the behavioral guidance provided by hindsight skills.Figure9shows the corresponding OPD losses. The loss decreases from its initial value and stabilizes at a lower level in all nine settings. The Qwen2.5 models exhibit gradual convergence, while Qwen3-1.7B shows a sharper early decline, especially on ALFWorld. Lower OPD loss indicates that the ordinary policy increasingly assigns probability to actions favored by hindsight supervision. Together with the rising success rates, this trend supports the stability of the self-evolving loop. The latest policy generates updated experience and hindsight skills, and OPD internalizes that guidance in subsequent updates.

Appendix DCase Study

Figures12–17illustrate howSeedapplies internalized behavioral guidance during evaluation without skill inputs. In the first ALFWorld example, the policy decomposes the task into locating the ladle, cleaning it, opening the drawer, and completing the placement. It correctly handles the drawer as a necessary precondition before the final action. The second example requires two books and repeated navigation between the desk and bed. The policy tracks that the first book has already been placed, returns for the second, and completes the remaining subgoal without losing progress. These trajectories demonstrate coherent state tracking and precondition management over extended interactions.

The Search-based QA examples show that the policy adapts its information gathering to the available evidence. It answers the first question after one search because the retrieved passages directly establish the shared profession. For the second question, it first identifiesFinding Neverland, then issues a targeted search for its director before answeringMarc Forster. In WebShop, the policy retains the requested attributes and price limits from search through purchase. It verifies the product details and selects the required color and size before buying. Together, these cases show thatSeedlearns to decompose tasks, revise plans from new evidence, and preserve constraints over long trajectories. Because skills are not provided during inference, the observed behavior reflects guidance internalized through self-evolving OPD rather than external prompting.

Appendix EAdditional Discussion

Seedtreats completed on-policy experience as an evolving source of supervision. Our experiments cover several forms of long-horizon interaction, but broader benchmarks such as DeepPlanning(Zhanget al.,2026b), Long-Horizon-Terminal-Bench(Liet al.,2026), OdysseyArena(Xuet al.,2026), and RobotEQ(Fanget al.,2026)provide more demanding tests. These settings involve longer workflows, richer state spaces, and greater interaction complexity, often with rare terminal success. EvaluatingSeedin such environments would test whether policy-synchronized hindsight can preserve decisive events across extended contexts and remain useful as the policy explores more diverse behaviors. Benchmarks with intermediate or partial-credit grading also make it possible to study how external progress signals interact with the token-level hindsight supervision.

However, longer tasks also expose a central risk of self-evolution: internally generated supervision can inherit model errors and plateau below oracle-supervised training(Jianget al.,2025b; Qiet al.,2026). Because the actor and analyzer share the same policy, improvements transfer across both roles, but their blind spots can also be shared. An inaccurate analysis may turn a recurring policy error into an apparently reusable rule, which later updates could reinforce. Evidence of self-preference in model-based evaluation further suggests that a model may systematically favor outputs resembling its own(Mahbub and Feng,2026). On-policy alignment and confidence gating help reduce distribution mismatch and noisy self-generated supervision(Kumaret al.,2025; Jianget al.,2025a), but neither confidence nor self-judgment guarantees semantic correctness(Jianget al.,2025b; Zhouet al.,2026). A more self-correcting version ofSeedcould anchor skills to verifiable state changes, compare analyses across policy snapshots, and preserve uncertainty alongside each skill. It could also organize hindsight hierarchically by first summarizing local transitions and then deriving trajectory-level rules(Geet al.,2025). Search-discovered reasoning abstractions(Wuet al.,2024)may provide useful structures for this aggregation. Policy-aware exploration(Wuet al.,2026a)could collect trajectories that distinguish competing rules, while uncertainty-aware self-distillation(Luet al.,2026a)may offer stronger filters for deciding which guidance to internalize.

Training efficiency is another practical constraint.Seedadds no deployment overhead, but training requires trajectory analysis and paired scoring under ordinary and skill-augmented contexts. The cost grows with interaction length and multimodal context. Speculative decoding methods such asDouble(Shenet al.,2026)andDSpark(Chenget al.,2026)could reduce the autoregressive cost of rollout collection and skill generation. Cached representations and batched paired scoring could further reduce the cost of OPD. Another promising direction is selective analysis: the analyzer could focus on trajectories with high uncertainty, novel states, or disagreement between reward and hindsight. This would preserve the self-evolving supervision loop while avoiding repeated analysis of redundant experience.

Figure 10:Prompt of analyzer.Figure 11:Prompt of actor (the policy model) in ALFWorld.Figure 12:ALFWorld Example 1.Seedlocates and cleans the ladle, opens the closed drawer, and completes the placement in seven steps, demonstrating task decomposition and correct precondition handling.Figure 13:ALFWorld Example 2.Seedplaces two books through repeated trips between the desk and bed while retaining the completed first placement, demonstrating progress tracking over an extended trajectory.Figure 14:Search-based QA Example 1.A single query retrieves evidence that both people are mathematicians, allowingSeedto answer without unnecessary searches.Figure 15:Search-based QA Example 2.Seedfirst identifiesFinding Neverland, then performs a targeted second search to verify that Marc Forster directed the film.Figure 16:WebShop Example 1.Seedidentifies a green shower cap and hair turban set under $40, verifies the selected variant, and completes the purchase in four steps.Figure 17:WebShop Example 2.Seedpreserves the requested product constraints and selectsmossy oak countryand5x-large bigbefore purchasing a matching long-sleeve shirt under $60.