VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Hugging Face Daily Papers Papers
benchmark llm-agents verifier-generation evaluation executable-verifiers synthetic-data

Summary

VerifierBench introduces a benchmark of 222 tasks and 4,993 hidden test cases to evaluate how well LLM agents generate executable verifiers, finding that verifiers tend to over-reject valid solutions due to hidden assumptions that survive self-testing.

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We study verifier generation: from the instruction and environment available for solving a task, an agent writes an executable verifier that must accept every valid solution and reject invalid ones. We introduce VerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993 hidden test cases that include alternative valid solutions and attacks; Verifier4Verifier (V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: most false rejections in the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors survive self-testing because agents test solutions that share the verifier's mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.
Original Article
View Cached Full Text

Cached at: 10/22/26, 07:36 PM

Paper page - VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

Source: https://huggingface.co/papers/2610.28679

Abstract

Agents generate executable verifiers to validate outputs, but these verifiers often wrongly reject valid solutions due to hidden assumptions, prompting recommendations to test acceptance of diverse correct answers.

Agents need reliable ways to validate their outputs, both when completing tasks and when filtering synthetic training data. We studyverifier generation: from the instruction and environment available for solving a task, an agent writes anexecutable verifierthat must accept every valid solution and reject invalid ones. We introduceVerifierBench, a benchmark of 222 tasks from six agent benchmarks spanning tool-call, state-based, and code verification, with 4,993hidden test casesthat include alternative valid solutions and attacks;Verifier4Verifier(V4V), a six-check audit, screens its source verifiers and test suites. Across 19 models in a common agent harness, normalized balanced accuracy ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions more reliably than they accept valid ones. This apparent strength can mask overly restrictive verification: mostfalse rejectionsin the analyzed records come from verifiers that accept no tested solution to a task. Case studies reveal unsupported assumptions about solution formats, tool use, and code changes that exclude valid solutions. These errors surviveself-testingbecause agents test solutions that share the verifier’s mistaken assumptions, or submit verifiers without confirming that any correct solution passes. These findings suggest testing both whether a verifier accepts a correct solution and whether it accepts other solutions permitted by the task. The project page and dataset are publicly available.

View arXiv pageView PDFProject pageGitHub3Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.28679 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.28679 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.28679 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

AgentProofBench: Benchmarking Reusable Verifiers for Web Agents

Hugging Face Daily Papers

AgentProofBench introduces a benchmark for reusable verification in web agents, linking frozen acceptance rules to execution traces across 97 tasks and 79 families on four web services, establishing baselines for program-gate correctness and reuse using DeepSeek Flash/Pro.

LLM-as-a-Verifier (GitHub Repo)

TLDR AI

LLM-as-a-Verifier is a general-purpose verification framework that provides fine-grained feedback for AI agents, achieving state-of-the-art performance on benchmarks like Terminal-Bench and SWE-Bench.

Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification

arXiv cs.AI

This paper presents Vera, an end-to-end automated safety testing framework for LLM agents that combines literature-driven risk discovery, combinatorial composition of safety cases, and evidence-grounded verification. Evaluations on four agent frameworks reveal substantial safety weaknesses, with average attack success rates reaching 93.9% under multi-channel attacks, and the release of Vera-Bench with 1600 executable safety cases.