PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment

Hugging Face Daily Papers Papers
reinforcement-learning credit-assignment self-distillation bayesian long-horizon autoregressive-decomposition

Summary

PBSD proposes a Bayesian self-distillation method that converts sparse final rewards into calibrated turn-level credit signals for long-horizon agentic tasks, improving policy learning and generalization.

Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning: trajectory-level rewards verify final correctness but provide limited guidance on which intermediate reasoning steps or tool interactions contribute to the outcome. The difficulty is especially pronounced in multi-turn search agents, where successful trajectories may contain misleading actions and failed trajectories may contain valuable evidence-gathering steps. We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards. PBSD measures trajectory quality through the posterior-to-prior probability ratio of the verified answer and applies Bayes' rule to convert this hard-to-estimate answer-side ratio into a tractable likelihood ratio between a standard student model and a privileged answer-conditioned teacher model. Autoregressive decomposition of this Bayesian evidence score yields turn-level signals that identify whether each intermediate turn supports or undermines the verified outcome. Consequently, PBSD provides a principled and elegant reweighting scheme that transforms sparse outcome supervision into Bayes-calibrated turn-level credit signals, while remaining fully compatible with standard policy optimization. Experiments demonstrate that PBSD consistently enhances performance across both in-domain and out-of-domain settings, and effectively transfers knowledge from short-context training to long-context inference, suggesting that its fine-grained credit assignment mechanism facilitates more effective policy learning and yields improved generalization.
Original Article
View Cached Full Text

Cached at: 06/09/26, 08:41 AM

Paper page - PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment

Source: https://huggingface.co/papers/2606.09348

Abstract

Privileged Bayesian Self-Distillation enables fine-grained credit assignment in long-horizon tasks by converting sparse outcome rewards into calibrated turn-level signals through Bayesian evidence scoring and autoregressive decomposition.

Long-horizon agentic tasks pose a fundamentalcredit assignmentchallenge for outcome-basereinforcement learning:trajectory-level rewardsverify final correctness but provide limited guidance on which intermediate reasoning steps or tool interactions contribute to the outcome. The difficulty is especially pronounced in multi-turn search agents, where successful trajectories may contain misleading actions and failed trajectories may contain valuable evidence-gathering steps. We propose PBSD (Privileged BayesianSelf-Distillation), a Bayes-calibratedself-distillationmethod for fine-grainedcredit assignmentunder sparse final rewards. PBSD measures trajectory quality through the posterior-to-prior probability ratio of the verified answer and applies Bayes’ rule to convert this hard-to-estimate answer-side ratio into a tractable likelihood ratio between a standard student model and a privileged answer-conditioned teacher model.Autoregressive decompositionof this Bayesian evidence score yieldsturn-level signalsthat identify whether each intermediate turn supports or undermines the verified outcome. Consequently, PBSD provides a principled and elegant reweighting scheme that transforms sparse outcome supervision into Bayes-calibrated turn-level credit signals, while remaining fully compatible with standardpolicy optimization. Experiments demonstrate that PBSD consistently enhances performance across both in-domain and out-of-domain settings, and effectively transfers knowledge from short-context training to long-context inference, suggesting that its fine-grainedcredit assignmentmechanism facilitates more effective policy learning and yields improved generalization.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2606\.09348

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.09348 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.09348 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.09348 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Hugging Face Daily Papers

AgentOPSD proposes a critic-free recursive method for turn-level credit assignment in agentic RL, reweighting outcomes via Bayesian belief updates in log-odds space. It achieves 89.1% success on ALFWorld with Qwen2.5-7B, outperforming GRPO and self-distillation baselines.

DAPD: Dual-Anchored Policy Distillation

Hugging Face Daily Papers

This paper introduces Dual-Anchored Policy Distillation (DAPD), a framework to address privilege illusion in on-policy self-distillation for language models. DAPD improves Qwen3-4B by +2.00 points on average across tasks, with gains persisting at larger scales.

β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Hugging Face Daily Papers

This paper introduces β-OPSD, a generalization of on-policy self-distillation that frames it as a policy optimization family with a controllable KL penalty. The method uses distillation to approximate expensive policy optimization, improving stability and reasoning performance on mathematical benchmarks.

Self-Distilled Policy Gradient

arXiv cs.LG

SDPG (Self-Distilled Policy Gradient) is a new RL training framework for LLMs that combines group-relative verifier advantages with on-policy self-distillation and KL regularization to address sparse rewards and instability in RLVR training. The method uses a shared model as both student and teacher by conditioning on privileged context, showing improved stability and performance over RLVR and self-distillation baselines.

Reducing Credit Assignment Variance via Counterfactual Reasoning Paths

arXiv cs.LG

Introduces Implicit Behavior Policy Optimization (IBPO), a counterfactual comparison-based credit assignment framework that improves training stability and performance in multi-step reasoning tasks for large language models by converting sparse terminal rewards into step-sensitive learning signals.