How is everyone handling agent regression testing in CI without going crazy?
Summary
The author discusses challenges in automated regression testing for AI agent tool calling in CI/CD due to LLM non-determinism and seeks community insights on effective setups and frustrations.
Similar Articles
How do you test AI agents before and after deployment? What has actually held up for you?
A practitioner asks the community how they test tool-calling AI agents before and after deployment, comparing scenario tests, LLM-as-judge, tool-call trajectory checks, and regression suites while raising doubts about coverage, judge reliability, and nondeterminism.
How are you testing your agents before deploying? Or is everyone just vibes-checking in prod?
A discussion on the challenges of testing non-deterministic AI agents, questioning how developers validate tool usage, behavior, and multi-step workflows without traditional testing patterns.
How are you regression-testing agent workflows before users find the failures?
The author asks how developers are regression-testing AI agent workflows, noting common failure modes and sharing their work on adding eval support to Runme for recording tasks, scoring trajectories, and comparing against baselines.
How are you testing your agents before shipping changes?
The author is introducing a tool for agent regression testing that allows developers to verify tool calls and workflow outcomes after code changes.
How do you actually test an agent harness when half of it is non-deterministic?
A discussion on the challenges of testing AI agent harnesses with non-deterministic components, exploring approaches like golden output diffing and using an LLM as a judge, while questioning the validity of such methods.