Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents

Hugging Face Daily Papers Papers
llm-agents confidence-calibration reinforcement-learning grpo self-supervision alfworld

Summary

该论文提出一种在多步 LLM 智能体任务中，利用最终结果（而非单步正确性）来校准智能体自报置信度的强化学习方法。在 ALFWorld、TextWorld 和 WebShop 环境中，步级置信度对最终结果的 AUROC 从接近随机提升至 0.68–0.85，校准误差显著下降，且不损害任务成功率。

Controlling an acting LLM agent means deciding, at each step of a long-horizon task, whether to trust its next action, abstain, or hand off to a human. That decision needs a per-step confidence signal predicting whether the current rollout will eventually succeed. The agent's own self-reported verbalized confidence is the natural candidate, but existing reinforcement learning methods calibrate it against the correctness of a decision at the current step. We propose a method to calibrate it against the eventual outcome of a multi-step episode: under GRPO, the group of rollouts for a task supplies success labels, and a proper scoring rule scores each step's confidence against the realized outcome. In offline RL on recorded trajectories of a Qwen2.5-7B agent in ALFWorld, TextWorld, and WebShop environments, the AUROC of step confidence against episode outcome rises from near chance to 0.68-0.85, and the top-10% of steps the agent is most confident about succeed ~4.3x more often than average. In online RL on the live ALFWorld agent, four calibration rewards improve early-step AUROC, expected calibration error goes from 0.42 to below 0.12, and selective execution improves over both the untrained agent and a format-only RL control, without degrading task success. We also introduce an evaluation protocol that separates episode depth, task difficulty, and attempt-level foresight, which shows where the gain lies: the trained agent recognizes from its first step which tasks it will fail, while telling a succeeding attempt from a failing one at the same task is harder, with some early-window signal emerging on TextWorld.
Original Article
View Cached Full Text

Cached at: 10/14/26, 02:52 PM

Paper page - Can an Agent Learn to Predict Its Own Failure? Outcome-Grounded Confidence Calibration in LLM Agents

Source: https://huggingface.co/papers/2610.18614

Abstract

ControllinganactingLLMagentmeansdeciding,ateachstepofalong-horizontask,whethertotrustitsnextaction,abstain,orhandofftoahuman.Thatdecisionneedsaper-stepconfidencesignalpredictingwhetherthecurrentrolloutwilleventuallysucceed.Theagent’sownself-reportedverbalizedconfidenceisthenaturalcandidate,butexistingreinforcementlearningmethodscalibrateitagainstthecorrectnessofadecisionatthecurrentstep.Weproposeamethodtocalibrateitagainsttheeventualoutcomeofamulti-stepepisode:underGRPO,thegroupofrolloutsforatasksuppliessuccesslabels,andaproperscoringrulescoreseachstep’sconfidenceagainsttherealizedoutcome.InofflineRLonrecordedtrajectoriesofaQwen2.5-7BagentinALFWorld,TextWorld,andWebShopenvironments,theAUROCofstepconfidenceagainstepisodeoutcomerisesfromnearchanceto0.68-0.85,andthetop-10%ofstepstheagentismostconfidentaboutsucceed~4.3xmoreoftenthanaverage.InonlineRLontheliveALFWorldagent,fourcalibrationrewardsimproveearly-stepAUROC,expectedcalibrationerrorgoesfrom0.42tobelow0.12,andselectiveexecutionimprovesoverboththeuntrainedagentandaformat-onlyRLcontrol,withoutdegradingtasksuccess.Wealsointroduceanevaluationprotocolthatseparatesepisodedepth,taskdifficulty,andattempt-levelforesight,whichshowswherethegainlies:thetrainedagentrecognizesfromitsfirststepwhichtasksitwillfail,whiletellingasucceedingattemptfromafailingoneatthesametaskisharder,withsomeearly-windowsignalemergingonTextWorld.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2610\.18614

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.18614 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.18614 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.18614 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents

arXiv cs.LG

This paper characterizes 'false success' in LLM agents, where agents claim task completion despite environment state showing otherwise, finding it accounts for 45-75% of failures across benchmarks. LLM judges fail to detect this reliably, while lightweight TF-IDF detectors achieve high AUROC with much lower latency, suggesting production monitoring should use calibrated detectors instead of LLM judges.

Training Advisors for LLM Agents from Task Outcomes

arXiv cs.CL

Nebius AI 提出 Caddie 方法，通过强化学习从任务成败结果训练 LLM agent 的自然语言批评器（critic），无需逐步标注。Qwen3-4B critic 在 MuSiQue 上将成功率提升逾 25 个百分点，并可跨基座模型与任务域泛化。