Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps

Hugging Face Daily Papers Papers
reinforcement-learning self-calibration agents grpo prediction-uncertainty training-methods

Summary

提出 Prospective Hindsight（PH）自校准强化学习原则，通过预测与实际反馈之间的差距加权梯度，使 agent 在长程任务中同时提升性能与校准度，且在单轮与多轮任务中均验证有效。

Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent's belief at action time invisible to the gradient. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the gap between the agent's prospective prediction (before feedback) and the retrospective evaluation (after feedback). This per-rollout surprise identifies samples where the agent's self-model is most inaccurate and amplifies their gradient contribution through a stop-gradient surprise-weighted advantage. Since the prospective predictor shares parameters with the policy, the two co-evolve, progressively shifting focus to the agent's remaining blind spots. We connect this principle to a privileged-information gap and show that minimizing the surprise residual provides a descent pathway on the agent's miscalibration rate; calibration thus emerges as a byproduct of optimization rather than from an added objective. On single-turn verifiable tasks and a multi-turn personal-agent task (under GRPO, on-policy distillation, and their combination), PH improves both task performance and calibration, with consistent gains across model scales. Notably, the dominant miscalibration mode shifts structurally between regimes, overconfident failures in single-turn, underconfident successes in multi-turn, yet the same training principle addresses both successfully.
Original Article
View Cached Full Text

Cached at: 10/13/26, 06:47 PM

Paper page - Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps

Source: https://huggingface.co/papers/2610.02740

Abstract

Reinforcementlearningforlong-horizonagentsreliesonpurelyretrospectivetrainingsignals:creditisassignedonlyafterobservingenvironmentalconsequences,leavingtheagent’sbeliefatactiontimeinvisibletothegradient.WeintroduceProspectiveHindsight(PH),aself-calibratingtrainingprinciplethataugmentsanyretrospectivebasemethodwithasignalderivedfromthegapbetweentheagent’sprospectiveprediction(beforefeedback)andtheretrospectiveevaluation(afterfeedback).Thisper-rolloutsurpriseidentifiessampleswheretheagent’sself-modelismostinaccurateandamplifiestheirgradientcontributionthroughastop-gradientsurprise-weightedadvantage.Sincetheprospectivepredictorsharesparameterswiththepolicy,thetwoco-evolve,progressivelyshiftingfocustotheagent’sremainingblindspots.Weconnectthisprincipletoaprivileged-informationgapandshowthatminimizingthesurpriseresidualprovidesadescentpathwayontheagent’smiscalibrationrate;calibrationthusemergesasabyproductofoptimizationratherthanfromanaddedobjective.Onsingle-turnverifiabletasksandamulti-turnpersonal-agenttask(underGRPO,on-policydistillation,andtheircombination),PHimprovesbothtaskperformanceandcalibration,withconsistentgainsacrossmodelscales.Notably,thedominantmiscalibrationmodeshiftsstructurallybetweenregimes,overconfidentfailuresinsingle-turn,underconfidentsuccessesinmulti-turn,yetthesametrainingprincipleaddressesbothsuccessfully.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.02740

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.02740 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.02740 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.02740 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Learning More from Less: Reinforcement Learning from Hindsight

arXiv cs.LG

Introduces Learning from Hindsight (LfH), a method that applies hindsight relabeling to RL post-training of vision-language-action models. By relabeling failed robot rollouts with the tasks they actually achieved, LfH achieves 5x improvement in sample efficiency on out-of-distribution manipulation tasks.

Failure Is the Mother of Success: Recovering Supervision with Grounding Hindsight

arXiv cs.AI

提出 Replay-Selected Hindsight Distillation (RSHD) 方法，让模型从完全失败的推理组中恢复监督信号：策略对失败尝试撰写纠错笔记并重新求解，仅当重放探索成功时才将该笔记蒸馏回普通提示策略中，在多个模型族和数学/代码训练域上一致优于 GRPO。

H^2SD: Hybrid Hindsight Self-Distillation

Hugging Face Daily Papers

Introduces H^2SD, a hybrid hindsight self-distillation framework that improves RLVR by using the teacher model differently for successful and failed trajectories, achieving better reasoning performance.