Should AI agent benchmarks separate “safe success” from “unsafe success”?

Reddit r/AI_Agents Papers
ai-agents benchmarks safety verifier-tax task-completion constraints

Summary

This article discusses the concept of 'Verifier Tax' in AI agent benchmarks, distinguishing between safe success (completing tasks without violating constraints) and unsafe success (completing tasks but violating constraints), and questions how to properly measure agent performance considering safety tradeoffs.

Most AI agent benchmarks report whether the agent completed the task. But for tool-using agents, that can be misleading. An agent can complete a task while still doing something problematic: using the wrong tool, skipping a required approval step, leaking private information, violating a tool policy, or taking an action the system should have blocked. In our recent ACM CAIS 2026 paper, we studied this issue and called it the **Verifier Tax**. The basic framing is: * **Safe success:** the agent completes the task without violating constraints * **Unsafe success:** the agent completes the task but violates a constraint * **Failure:** the agent does not complete the task The interesting tradeoff is that runtime checks/verifiers can reduce unsafe success, but they can also reduce overall task completion. So a system may become safer but look “worse” on traditional success-rate metrics. Curious how people building agents think about this: 1. Do you currently measure safe vs unsafe task completion? 2. Should “unsafe success” count as success or failure in agent benchmarks? 3. Are runtime verifiers worth the tradeoff if they reduce task completion? 4. What metrics do you use beyond task success rate?
Original Article

Similar Articles

Can an AI agent complete a task and still fail?

Reddit r/artificial

This paper introduces the concept of 'Verifier Tax' to categorize AI agent outcomes as safe success, unsafe success, or failure, and proposes a two-tier verification architecture for tool-using LLM agents.

The Verifier Tax: Horizon-Dependent Safety–Success Tradeoffs in Tool-Using LLM Agents [R]

Reddit r/MachineLearning

This paper presents a safety evaluation framework for tool-using LLM agents, introducing the concept of the 'Verifier Tax'—a horizon-dependent tradeoff between safety and task completion. It proposes a two-tier verification architecture and uses Tau-bench scenarios to demonstrate how verification can reduce unsafe successes but also decrease task completion as task horizon increases.

Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks

arXiv cs.AI

This paper audits four agent-safety benchmarks (R-Judge, InjecAgent, AgentHarm, AgentDojo) across many models, showing that their scores are confounded by capability, metrics have artifacts, and rankings disagree across benchmarks, undermining interchangeable safety claims.

What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents

arXiv cs.AI

This paper argues that current benchmarks for autonomous agents fail to evaluate whether an agent should have proceeded at all, introducing a 'compliance bias'. The authors propose a taxonomy of abstention-warranted scenarios and new evaluation protocols (Safety Rate, Usability Rate, Informed Refusal Rate) with preliminary results showing tunable safety–usability tradeoffs across model families.