In production, 89% of agent teams have observability but only 52% run evals - how are you actually gating prompt changes?
Summary
A statistic reveals that while 89% of agent teams in production have observability, only 52% run evaluations, raising questions about how prompt changes are gated.
Similar Articles
72% of teams are running coding agents in production. Most of them can't say which agent they'd trust with a critical path change at 11pm, or why.
While 72% of teams use coding agents in production, most lack formal governance or empirical data on agent reliability. The article argues for session-level tracking over policy frameworks to ensure trust in critical deployments.
@bentannyhill: Agent observability is a means to an end: making your agent better. But observability and evals tools have traditionall…
Engine is a new tool that connects agent observability traces to automated fixes and evaluations, closing the agent improvement loop for engineering teams.
I analyzed how 50+ AI teams debug production agent failures and got surprised
Based on interviews with 50+ AI teams, the author highlights that production agent failures often stem from minor prompt or configuration issues rather than deep model problems. The article advocates for adopting software engineering practices like versioning, A/B testing, and experiment tracking to improve reliability.
Most agent observability feels like crash footage
The author argues that current agent observability provides a trace of actions but lacks runtime justification for why actions were permitted, which is critical for production deployments involving money, data, or communications.
How do you all actually get from a failed eval to a prompt fix that holds in prod?
A discussion comparing LLM evaluation and observability tools (LangSmith, Weave, Phoenix, Braintrust, Galileo, Opik) for fixing prompt failures and introducing an open-source platform that integrates the full eval-to-fix loop on a single trace.