How are you regression-testing agent workflows before users find the failures?
Summary
The author asks how developers are regression-testing AI agent workflows, noting common failure modes and sharing their work on adding eval support to Runme for recording tasks, scoring trajectories, and comparing against baselines.
Similar Articles
How is everyone handling agent regression testing in CI without going crazy?
The author discusses challenges in automated regression testing for AI agent tool calling in CI/CD due to LLM non-determinism and seeks community insights on effective setups and frustrations.
How are you testing your agents before deploying? Or is everyone just vibes-checking in prod?
A discussion on the challenges of testing non-deterministic AI agents, questioning how developers validate tool usage, behavior, and multi-step workflows without traditional testing patterns.
AI Agent that builds deterministic workflows
A developer shares an experiment with an AI agent-based automation platform that builds and manages deterministic workflows, seeking feedback from the community.
When your agent screws up in production, how do you figure out which step went wrong?
A developer shares the challenge of debugging multi-step agents in production, where failures are hard to trace due to complex tool use and confident wrong answers, and asks the community for better monitoring and regression detection approaches.
People running coding agents across real repos: what breaks after the agent writes the code?
This article discusses the practical challenges engineering teams face when adopting AI coding agents, such as task safety, context retrieval, output review, and coordination, and proposes a readiness model for evaluation.