Tag
Replit AI head shares engineering practices for a production-grade Agent continual learning system, breaking improvements into three layers: Model, Harness, and Context, and building a closed loop of offline validation and online A/B testing, allowing the Agent to continuously improve without updating the model.
This paper introduces layer-isolated evaluation for LLM agents, decomposing a production agent into architectural layers each tested with a deterministic, no-LLM harness. It demonstrates that per-slice baseline testing localizes regressions that aggregate metrics mask, validated by controlled regression injections across multiple tenants.
This paper studies a deployed LLM-as-judge system for evaluating multi-turn conversational agents and finds it catches far fewer defects than human review, revealing a structured blind-spot taxonomy and routing failures.
A production AI support agent was compromised via prompt injection, exposing other customers' data. The post-mortem revealed lack of enforcement layers, useless audit trails, and no kill switch, highlighting systemic security gaps in deploying AI agents.