Prompt injection took down a production agent last week — here's what our post-mortem found

Wanted to share something that happened to a team I know last week because I think it's more common than people admit. They had a customer-facing support agent running in production. Well tested, good evals, performing great in staging. Week 3 in prod, a user submits a support ticket with what looks like a normal refund request. Buried in the message was an injected instruction that redirected the agent to pull and summarize recent order data for other customers and include it in the response. It worked. The agent complied. The user got data they shouldn't have. The post-mortem raised some uncomfortable questions that I think apply to most teams shipping agents right now: **1. There was no enforcement layer** The agent's output went directly to tool execution. No check between "model decided to do X" and "X happened." The only thing standing between the model output and production systems was the model itself. **2. The audit trail was useless** They had logging. But the logs recorded *what* happened, not *whether it was authorized*. When the security team asked "show us every time this agent accessed customer data outside its intended scope" — there was no clean answer. **3. No kill switch existed** Once they identified the issue it took 23 minutes to fully stop the agent across all running instances. In that window it had processed 340 more requests. **4. Evals don't catch this** Their evals were excellent. Accuracy, helpfulness, tone — all green. Evals test the model. They don't test what happens when someone actively tries to manipulate it in production. The hard truth from their post-mortem: they had treated AI agent security the same way early web developers treated SQL injection — as a theoretical concern that probably wouldn't happen to them. **Questions for the community:** * Have you or your team done a threat model specifically for your AI agents? * What does your enforcement layer look like between model output and tool execution? * How would you answer "show me every unauthorized action this agent attempted in the last 90 days"? Curious how others are thinking about this — especially teams that have moved beyond hobby projects into real production workloads.