SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
Summary
SABER introduces a benchmark for evaluating the operational safety of LLM coding agents in realistic stateful project workspaces, showing that even the best model has over a 54% harmful safety-violation rate, indicating insufficient alignment for real-world environments.
View Cached Full Text
Cached at: 06/05/26, 06:09 PM
Paper page - SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
Source: https://huggingface.co/papers/2606.01317 Published on May 31
·
Submitted byhttps://huggingface.co/lingfengzhe
Qi HUon Jun 5
Abstract
Large language models deployed as coding agents exhibit significant safety violations in realistic project environments, necessitating new evaluation approaches beyond simple prompt refusal assessments.
Large language modelsare increasingly deployed ascoding agents, shifting safety from individual responses to action sequences. Existing benchmarks, however, primarily assess whether models refuse unsafe prompts, leaving impacts onstateful workspaceslargely unexamined. We present SABER, a benchmark forenvironment-aware operational safetythat places models in realisticagent-style projectsand evaluates safety from the final environment state after a sequence of actions. Beyond binary safety-violation reports, SABER categorizes violations by cause, enabling analysis of model-specific safety profiles. Our evaluations show that even the best-performing model has more than a 54%harmful safety-violation rate(HSR), suggesting that currentalignmentremains insufficient for realistic project environments. SABER further reveals distinct safety profiles across models. Our benchmark is publicly available at https://github.com/sssr-lab/saber.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.01317
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.01317 in a model README.md to link it from this page.
Datasets citing this paper1
#### sssr-lab/SABER Viewer• Updatedabout 5 hours ago • 1.43k • 7k
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.01317 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
This paper presents Vera, an end-to-end automated safety testing framework for LLM agents that combines literature-driven risk discovery, combinatorial composition of safety cases, and evidence-grounded verification. Evaluations on four agent frameworks reveal substantial safety weaknesses, with average attack success rates reaching 93.9% under multi-channel attacks, and the release of Vera-Bench with 1600 executable safety cases.
In 6,560 benchmark runs, task completion often coexisted with Unsafe behavior
A new arXiv preprint introduces AgentS4D, a benchmark for AI agent safety, showing that 70.52% of completed runs also triggered unsafe verdicts, and 97.38% of unsafe runs still completed the task. The study highlights that task completion and safety should be scored separately.
Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
This survey paper reviews 38 studies on safe LLM agents, highlighting key challenges such as specification translation bottlenecks, incomplete safety guarantees from enforcement methods like runtime monitoring, and the verifier tax that impedes safe task completion.
Safety, or Just Capability? A Validity Audit of Agent-Safety Benchmarks
This paper audits four agent-safety benchmarks (R-Judge, InjecAgent, AgentHarm, AgentDojo) across many models, showing that their scores are confounded by capability, metrics have artifacts, and rankings disagree across benchmarks, undermining interchangeable safety claims.
Benchmarking the Benchmarks: Evaluating Automated Safety Benchmarks for Small Language Models
This paper evaluates automated safety benchmarks for small language models, finding high ambiguity in judgments that compromises reliability and reveals a capability-safety confound.