Counsel: A Meta-Evaluation Dataset for Agentic Tasks

Hugging Face Daily Papers Papers
dataset meta-evaluation agentic-tasks llm-as-judge human-alignment evaluation

Summary

Counsel is the first public dataset of human meta-evaluations of LLM critiques for agentic tasks, designed to improve the calibration and reliability of automated evaluation methods.

As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.
Original Article
View Cached Full Text

Cached at: 06/23/26, 09:41 AM

Paper page - Counsel: A Meta-Evaluation Dataset for Agentic Tasks

Source: https://huggingface.co/papers/2606.21627

Abstract

A large-scale dataset of human-metaevaluations of LLM critiques for agentic tasks is introduced to improve the calibration and reliability of automated evaluation methods.

Asagentic systemstackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such asLLM-as-a-judge(LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset ofmeta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on twoagent benchmarks:tau-bench(customer support agents) andDA-Code(coding agents), and humanmeta-evaluations of these critiques. Human annotators label critiques on each flagged error as “spot on”, “correct location but poor reasoning”, or “should not have flagged”, achieving reliableinter-annotator agreement(Krippendorff’s alphaof 0.78). The resulting dataset stratifies LLMJ critiques byhuman alignmentacross both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated usingopen-weight modelsand is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators foragentic systems.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2606\.21627

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.21627 in a model README.md to link it from this page.

Datasets citing this paper1

#### AtlaAI/counsel Viewer• Updatedabout 2 hours ago • 1.36k • 32

Spaces citing this paper1

Collections including this paper1

Similar Articles

The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring

arXiv cs.CL

A new cross-domain benchmark (Metacognitive Monitoring Battery) with 524 items evaluates LLM self-monitoring capabilities across six cognitive domains using human psychometric methodology. Applied to 20 frontier LLMs, it reveals three distinct metacognitive profiles and shows that accuracy rank and metacognitive sensitivity rank are largely inverted.

AICompanionBench: Benchmarking LLMs-as-Judges for AI Companion Safety

arXiv cs.AI

AICompanionBench introduces the first publicly available benchmark dataset of 2,123 real-world AI companion conversations annotated across nine safety risk categories, used to evaluate 20 LLMs as safety judges. Results show strong models handle explicit harmful content well but struggle with nuanced risks like manipulation and false positives on benign conversations.