ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

Hugging Face Daily Papers Papers
llm-agents prompt-injection security-evaluation adversarial-environments alignment

Summary

ToolHazard is a scalable framework that synthesizes adversarial environments to test LLM agents against indirect prompt injections, revealing vulnerabilities and improving defensive alignment through the ToolHazard-Bench benchmark.

Large language model (LLM) agents integrated with external tools are vulnerable to indirect prompt injections embedded in environmental states. However, existing studies largely rely on manually implemented or reused environments, stochastic LLM-based tool simulation, and predefined injection locations, limiting scalable security research across broader domains. To bridge this gap, we propose **ToolHazard**, a scalable adversarial environment synthesis framework that reduces human engineering and supports expansion with additional seed domains and compute. Through an Environment Simulator, an Attacker Agent, and a User Simulator, ToolHazard synthesizes executable stateful environments, discovers viable injection points and generates environment-specific payloads, and constructs state-grounded long-horizon tasks. Based on ToolHazard, we build **ToolHazard-Bench** for stress-testing agents under complex workflows and diverse environmental attacks. Experiments reveal substantial agent vulnerabilities and show that injection timing and placement affect attack effectiveness. Moreover, ToolHazard-generated alignment data improves security on both ToolHazard-Bench and AgentDojo while preserving benign task utility.
Original Article
View Cached Full Text

Cached at: 08/13/26, 03:35 PM

Paper page - ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents

Source: https://huggingface.co/papers/2608.11878

Abstract

ToolHazard is a scalable framework that synthesizes adversarial environments to test LLM agents against indirect prompt injections, revealing vulnerabilities and improving defensive alignment.

Large language model (LLM) agents integrated with external tools are vulnerable toindirect prompt injections embedded in environmental states. However, existing studies largely rely on manually implemented or reused environments, stochastic LLM-based tool simulation, and predefined injection locations, limiting scalable security research across broader domains. To bridge this gap, we propose **ToolHazard**, a scalableadversarial environment synthesisframework that reduces human engineering and supports expansion with additional seed domains and compute. Through anEnvironment Simulator, anAttacker Agent, and aUser Simulator,ToolHazardsynthesizes executable stateful environments, discovers viable injection points and generates environment-specific payloads, and constructs state-grounded long-horizon tasks. Based onToolHazard, we build **ToolHazard-Bench** for stress-testing agents under complex workflows and diverse environmental attacks. Experiments reveal substantial agent vulnerabilities and show that injection timing and placement affect attack effectiveness. Moreover,ToolHazard-generatedalignment dataimproves security on bothToolHazard-BenchandAgentDojowhile preserving benign task utility.

View arXiv pageView PDFGitHub5Add to collection

Get this paper in your agent:

hf papers read 2608\.11878

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.11878 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.11878 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.11878 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Set-shifting Behavioral Test for Harnessed Agents

arXiv cs.AI

This paper introduces a benchmark for evaluating how LLM agents adapt their tool choice when the reliable tool silently changes mid-session, borrowing set-shifting from cognitive psychology. It tests open-weight LLMs and identifies distinct failure modes based on tool-set framing.

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

Hugging Face Daily Papers

Researchers propose an adversarial hacker-fixer loop using LLM agents to automatically patch brittle verifiers in agent benchmarks, reducing attack success rates from 62% to 0% on KernelBench and demonstrating that weaker defenders can neutralize much stronger attackers.

When Lower Privileges Suffice: Investigating Over-Privileged Tool Selection in LLM Agents

Hugging Face Daily Papers

This paper investigates over-privileged tool selection in LLM agents, introducing ToolPrivBench to evaluate and mitigate unnecessary use of high-privilege tools. It finds that safety alignment does not ensure least-privilege choices, and proposes a post-training defense that reduces excessive privilege use without sacrificing performance.

Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification

arXiv cs.AI

This paper presents Vera, an end-to-end automated safety testing framework for LLM agents that combines literature-driven risk discovery, combinatorial composition of safety cases, and evidence-grounded verification. Evaluations on four agent frameworks reveal substantial safety weaknesses, with average attack success rates reaching 93.9% under multi-channel attacks, and the release of Vera-Bench with 1600 executable safety cases.

Tool Specifications Matter: Uncovering and Mitigating Safety Risks in AI Agents

arXiv cs.AI

This paper identifies schema-formatted tool specifications as a primary source of safety degradation in AI agents, weakening LLM refusal signals. The authors propose SafeKeep, an inference-time safeguard that separates safety judgment from tool execution, increasing harmful request refusal rates from 23.8% to 70.6% and cutting prompt injection attack success from 25.6% to 2.5%.