How are you regression-testing agent workflows before users find the failures?
Summary
The author asks how developers are regression-testing AI agent workflows, noting common failure modes and sharing their work on adding eval support to Runme for recording tasks, scoring trajectories, and comparing against baselines.
Similar Articles
How are you testing your agents before shipping changes?
The author is introducing a tool for agent regression testing that allows developers to verify tool calls and workflow outcomes after code changes.
How is everyone handling agent regression testing in CI without going crazy?
The author discusses challenges in automated regression testing for AI agent tool calling in CI/CD due to LLM non-determinism and seeks community insights on effective setups and frustrations.
How are you testing your agents before deploying? Or is everyone just vibes-checking in prod?
A discussion on the challenges of testing non-deterministic AI agents, questioning how developers validate tool usage, behavior, and multi-step workflows without traditional testing patterns.
How do you test AI agents before and after deployment? What has actually held up for you?
A practitioner asks the community how they test tool-calling AI agents before and after deployment, comparing scenario tests, LLM-as-judge, tool-call trajectory checks, and regression suites while raising doubts about coverage, judge reliability, and nondeterminism.
when your agent eval catches a failure what do u actually do next?
The author discusses common challenges in debugging AI agent evaluation failures, seeking insights on efficient investigation methods and the reliability of comparison runs versus other evidence sources.