Counsel: A Meta-Evaluation Dataset for Agentic Tasks
Summary
Counsel is the first public dataset of human meta-evaluations of LLM critiques for agentic tasks, designed to improve the calibration and reliability of automated evaluation methods.
View Cached Full Text
Cached at: 06/23/26, 09:41 AM
Paper page - Counsel: A Meta-Evaluation Dataset for Agentic Tasks
Source: https://huggingface.co/papers/2606.21627
Abstract
A large-scale dataset of human-metaevaluations of LLM critiques for agentic tasks is introduced to improve the calibration and reliability of automated evaluation methods.
Asagentic systemstackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such asLLM-as-a-judge(LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset ofmeta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on twoagent benchmarks:tau-bench(customer support agents) andDA-Code(coding agents), and humanmeta-evaluations of these critiques. Human annotators label critiques on each flagged error as “spot on”, “correct location but poor reasoning”, or “should not have flagged”, achieving reliableinter-annotator agreement(Krippendorff’s alphaof 0.78). The resulting dataset stratifies LLMJ critiques byhuman alignmentacross both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated usingopen-weight modelsand is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators foragentic systems.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.21627
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.21627 in a model README.md to link it from this page.
Datasets citing this paper1
#### AtlaAI/counsel Viewer• Updatedabout 2 hours ago • 1.36k • 32
Spaces citing this paper1
Collections including this paper1
Similar Articles
Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
Introduces Eval-Pair Matrix, a controlled meta-evaluation protocol for source-grounded RAG that induces hidden contradictions to detect self-leniency in LLM judges. The study finds minimal same-model effects and emphasizes methodological improvements for RAG judge studies.
Metag: A dataset to build agentic meta-reviewing capabilities
This paper introduces Metag, a dataset designed to help meta-reviewers in scientific peer review by identifying changes in manuscripts based on reviewer feedback and author responses, enhancing traceability and transparency.
Designing a Robust LLM-Based Evaluation System for Agentic AI in Drug Discovery Through Human Alignment
This paper presents an LLM-as-a-Judge evaluation framework for agentic AI in drug discovery, validated through human alignment studies with expert annotators. It optimizes the judge to improve alignment with human judgment and provides insights for reusable evaluation in scientific domains.
Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation
This paper proposes a dataset-centric meta-evaluation framework that audits LLM benchmarks at the sample level across five latent dimensions, exposing internal heterogeneity and enabling criterion-driven composition of benchmark subsets for targeted model evaluation.
The Metacognitive Monitoring Battery: A Cross-Domain Benchmark for LLM Self-Monitoring
A new cross-domain benchmark (Metacognitive Monitoring Battery) with 524 items evaluates LLM self-monitoring capabilities across six cognitive domains using human psychometric methodology. Applied to 20 frontier LLMs, it reveals three distinct metacognitive profiles and shows that accuracy rank and metacognitive sensitivity rank are largely inverted.