How is everyone handling agent regression testing in CI without going crazy?

Reddit r/AI_Agents Tools

Summary

The author discusses challenges in automated regression testing for AI agent tool calling in CI/CD due to LLM non-determinism and seeks community insights on effective setups and frustrations.

Hey everyone, At my last project, we spent hours every week manually spot-checking agent runs because every minor model tweak or context update seemed to silently break tool calling downstream. Traditional unit tests don't fit because LLMs are non-deterministic, but most eval frameworks only grade the final text response rather than the intermediate tool-call trajectory (did it pick the right tool, pass valid parameters, and recover if an API errored?). I’m working on better tooling around automated agent regression testing and deterministic tool validation in CI/CD, and I’d love to know what your current setup looks like: How do you test whether a prompt/model update broke your agent’s tool calling before shipping to prod? Do you run tests in GitHub Actions/GitLab, or is QA still largely manual / ad-hoc? What’s the single most frustrating part of your current agent eval setup? Appreciate any insights or horror stories from your production setups!
Original Article

Similar Articles