The Cold-Start Safety Gap in LLM Agents

Hugging Face Daily Papers Papers
safety llm-agents benchmark tool-calling cold-start deployment-strategy

Summary

This paper identifies a 'cold-start safety gap' in tool-calling LLM agents, where they are most vulnerable at the beginning of a session and become safer after completing regular agentic tasks. The authors introduce the SODA benchmark to evaluate this phenomenon and recommend a simple deployment strategy of warming up agents with regular tasks before safety-critical requests.

Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regular agentic tasks -- a phenomenon we term the cold-start safety gap. To study this systematically, we introduce Safety Over Depth for Agents (SODA), a benchmark that controls how many regular agentic tasks the agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regular agentic tasks increases from zero to twenty. Representation analysis confirms that model hidden states gradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regular agentic tasks themselves are the primary driver of safety, while the agent's own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-source safety benchmarks (AgentHarm, Agent Safety Bench) and utility benchmarks (BFCL, API-Bank), confirming that warming up the agent with regular agentic tasks before deployment makes it safer and preserves full capability. Based on these findings, we recommend a simple deployment strategy: having the agent complete a few regular agentic tasks before possible exposure to safety-critical requests mitigates the cold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap
Original Article
View Cached Full Text

Cached at: 06/12/26, 06:54 PM

Paper page - The Cold-Start Safety Gap in LLM Agents

Source: https://huggingface.co/papers/2606.07867

Abstract

Tool-calling language model agents exhibit improved safety after initial interactions, with a systematic benchmark demonstrating enhanced security through prior task completion.

Aretool-calling LLM agentsequally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a session and become substantially safer after a few regularagentic tasks-- a phenomenon we term thecold-start safety gap. To study this systematically, we introduceSafety Over Depth for Agents(SODA), a benchmark that controls how many regularagentic tasksthe agent completes before encountering a safety threat, supporting up to 20 preceding tasks. Evaluating 7 models from 4 families, safety improves by 9--52% as the number of preceding regularagentic tasksincreases from zero to twenty.Representation analysisconfirms that modelhidden statesgradually shift toward a safety-aligned region as more preceding tasks are present. By systematically studying which part of the preceding conversation matters most, we find that the regularagentic tasksthemselves are the primary driver of safety, while the agent’s own prior responses have less effect on safety but are essential for preserving later utility. This conclusion is further supported by evaluation on open-sourcesafety benchmarks(AgentHarm, Agent Safety Bench) andutility benchmarks(BFCL, API-Bank), confirming that warming up the agent with regularagentic tasksbefore deployment makes it safer and preserves full capability. Based on these findings, we recommend a simpledeployment strategy: having the agent complete a few regularagentic tasksbefore possible exposure to safety-critical requests mitigates thecold-start safety gap. Our code is available at https://github.com/Trustworthy-ML-Lab/Agent-Cold-Start-Safety-Gap

View arXiv pageView PDFProject pageGitHub0Add to collection

Get this paper in your agent:

hf papers read 2606\.07867

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### cesun/SODA-Agent-Safety-Judge Text Generation• 4B• Updatedabout 17 hours ago

Datasets citing this paper1

#### cesun/SODA Viewer• Updatedabout 17 hours ago • 25.6k • 8

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.07867 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Tool-Making and Self-Evolving LLM Agents in Low-Latency Systems

arXiv cs.CL

This paper presents a method for compiling repeated standard operating procedure steps into validated, versioned tools before deployment, replacing inference-time code generation. In a fulfillment center alarm-triage system, this approach reduces p50 latency by 42% and end-to-end error rate by up to 53%.