agent eval latency added 18 minutes to our CI. how are you running this without killing dev velocity?
Summary
A discussion on the challenge of integrating comprehensive agent evaluations into CI, where latency from judge calls increases build time from 6 to 24 minutes, and potential solutions like parallelization, caching, and async eval are considered.
Similar Articles
@Vtrivedy10: my fave question, talked about this coding agent Eval+Improvement loop infra + UX in my AIE talk yesterday! biased but …
The speaker discusses the importance of evaluating and improving coding agents, highlighting LangSmith's integration with Harbor to provide a unified stack for running, tracing, and improving agent evaluations in isolated environments.
How is everyone handling agent regression testing in CI without going crazy?
The author discusses challenges in automated regression testing for AI agent tool calling in CI/CD due to LLM non-determinism and seeks community insights on effective setups and frustrations.
Agent Judge: Solving Long-Context Evals for Production Agents (10 minute read)
Agent Judge is an agentic evaluation harness that overcomes the limitations of simple LLM judges for long-horizon agents by handling long trajectories, verifying stateful actions against source-of-truth systems, and adapting to changing behavior.
@LangChain: In 13 minutes, @jeffbarg, Vyshu Khota, and Soroush Khadem walk through how Clay scaled agent evals agents at 300M+ runs…
Clay scaled agent evaluations to over 300 million runs per month, covering their four-quadrant eval framework and the challenges of closing the production-to-eval loop.
I keep abandoning multi-agent setups because I can't verify the code they ship. How are you handling this?
A developer shares their frustration with multi-agent coding setups where verifying the output of parallel PRs is impractical, and describes building an AI QA agent that uses a real browser (via Browserbase) to automatically click through preview deploys and fail PRs that don't work as expected.