QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

Hugging Face Daily Papers Papers
dense-supervision long-horizon llm-agents training-free benchmark q-values evaluation

Summary

Introduces QVal, a training-free testbed for evaluating dense supervision signals in long-horizon LLM agent tasks by measuring alignment with Q-values, enabling fair comparison of different supervision approaches without training.

LLM agents increasingly act over long horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions. Dense supervision methods aim to solve this problem by scoring intermediate steps, from intrinsic confidence to self-distillation and embedding similarities. However, it is common practice to evaluate them by measuring the downstream performance of a training pipeline that integrates them. This is expensive, conflates supervision quality with training engineering confounders, and renders different methodological families requiring distinct training setups incomparable. As a result, dense supervision methods are rarely benchmarked on common ground. We introduce QVal, a training-free testbed for directly evaluating dense supervision signals. Given a state-action pair, QVal measures how well a method's score is Q-aligned: whether it orders actions according to the Q-values of a strong reference-policy. This lets us compare signals before any training run and separate signal quality from other engineering choices. We instantiate QVal as QVal-v1.0, benchmarking 21 dense supervision methods across four diverse environments and seven methodological families, with over 1.2K evaluation experiments across six open-weight model backbones. We find that simple prompting baselines consistently outperform recent dense supervision methods from the literature, and that performance clusters strongly by family. These findings hold across model sizes, environments, and observation modalities. QVal is designed to be easily extensible to new environments and methods, enabling researchers to iterate on dense supervision methods before any training run.
Original Article
View Cached Full Text

Cached at: 07/01/26, 03:43 PM

Paper page - QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agents

Source: https://huggingface.co/papers/2606.32034

Abstract

A testbed called QVal is introduced for evaluating dense supervision signals in long-horizon LLM agent tasks by measuring how well method scores align with Q-values, enabling fair comparison of different supervision approaches without training.

LLM agentsincreasingly act overlong horizons, where a single trajectory can contain hundreds or thousands of actions. In these settings, outcome-only rewards provide too sparse guidance, failing to inform the model about the goodness of intermediate actions.Dense supervisionmethods aim to solve this problem by scoring intermediate steps, from intrinsic confidence to self-distillation and embedding similarities. However, it is common practice to evaluate them by measuring the downstream performance of a training pipeline that integrates them. This is expensive, conflates supervision quality with training engineering confounders, and renders different methodological families requiring distinct training setups incomparable. As a result,dense supervisionmethods are rarely benchmarked on common ground. We introduce QVal, atraining-freetestbed for directly evaluatingdense supervisionsignals. Given a state-action pair, QVal measures how well a method’s score isQ-aligned: whether it orders actions according to theQ-valuesof a strongreference-policy. This lets us compare signals before any training run and separate signal quality from other engineering choices. We instantiate QVal as QVal-v1.0, benchmarking 21dense supervisionmethods across four diverse environments and seven methodological families, with over 1.2K evaluation experiments across six open-weightmodel backbones. We find that simple prompting baselines consistently outperform recentdense supervisionmethods from the literature, and that performance clusters strongly by family. These findings hold across model sizes, environments, and observation modalities. QVal is designed to be easily extensible to new environments and methods, enabling researchers to iterate ondense supervisionmethods before any training run.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2606\.32034

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.32034 in a model README.md to link it from this page.

Datasets citing this paper1

#### bethgelab/qval Updatedabout 4 hours ago • 2

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.32034 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles