CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks
Summary
CalVerT introduces calibrated verifier telemetry for LLM agents, providing confidence and grounding scores to reduce over-retrieval and unsupported answers, improving performance on knowledge-intensive QA tasks.
View Cached Full Text
Cached at: 06/23/26, 05:41 AM
Paper page - CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks
Source: https://huggingface.co/papers/2606.21777
Abstract
Calibrated verifier telemetry enhances LLM agents in knowledge-intensive question answering by providing confidence scores and grounding verification, reducing both over-retrieval and unsupported answers.
LLM agentsinknowledge intensive question answeringtakeretrievalandreasoningactions with incomplete knowledge about whether their current answer is uncertain, unsupported, or already complete. This produces two failure modes: committing to confident but unsupported answers, which hurts accuracy, and over-retrieving when the evidence in hand already suffices, resulting in wasted compute. To give agents a more complete picture of the state space they are operating in, we introduce calibrated verifier telemetry (CalVerT), which augments the agent’s state with additional telemetry: acalibrated self-confidence scoreand agrounding verifier score. We show that CalVerT can improve agents in both training-free and training-based settings. On four QA benchmarks, we find that CalVerT raises F1 by triggeringretrievalin cases where agents over-rely on parametric knowledge, while cutting redundantretrievalin cases where agents have sufficient context to answer. We show that CalVerT can augment existing QA frameworks without training. Moreover, CalVerT also improves trained systems: by simply augmenting an agent’s state with telemetry, we observe improvements afterreinforcement learning, as compared to an agent with identical training but no CalVerT telemetry.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2606\.21777
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.21777 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.21777 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.21777 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Faithful uncertainty in LLM agents: calibration vs utility tradeoff in practice[D]
A practitioner discusses the calibration vs. utility tradeoff in LLM agents, sharing experience with a verifier-based pipeline that reduces hallucinated tool calls by ~60% but introduces latency costs and drops easy correct answers.
Do Agents Know When They Succeed? Calibrating Agent Confidence from Internal Representations
This paper introduces methods to calibrate confidence in agentic systems using internal representations, demonstrating improved performance over baselines in multi-turn benchmarks.
CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
The paper introduces CALIBER, a method for calibrating confidence in reasoning language models by eliciting confidence estimates both before and after reasoning, with supervision targets matched to the information state. It achieves significant reductions in Expected Calibration Error (up to 52.5%) and strong Brier scores and AUROC across multiple benchmarks.
AgentKGV: Agentic LLM-RAG Framework with Two-Stage Training for the Fact Verification of Knowledge Graphs
Proposes AgentKGV, an agentic LLM-RAG framework with two-stage training (distillation SFT and trajectory-level GRPO) for verifying facts in knowledge graphs, achieving significant improvements on the T-REx benchmark while reducing retrieval calls.
CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks
CalibForge is an autonomous terminal-task synthesis system that uses adversarial solver calibration to create learnable tasks for training terminal agents. It constructs 5,431 calibrated tasks and improves agent performance on Terminal-Bench2.0, SWE-bench Pro, and Doc2Repo.