Tag
This paper introduces layer-isolated evaluation for LLM agents, decomposing a production agent into architectural layers each tested with a deterministic, no-LLM harness. It demonstrates that per-slice baseline testing localizes regressions that aggregate metrics mask, validated by controlled regression injections across multiple tenants.
This paper studies a deployed LLM-as-judge system for evaluating multi-turn conversational agents and finds it catches far fewer defects than human review, revealing a structured blind-spot taxonomy and routing failures.
A production AI support agent was compromised via prompt injection, exposing other customers' data. The post-mortem revealed lack of enforcement layers, useless audit trails, and no kill switch, highlighting systemic security gaps in deploying AI agents.