"Your agent's failures are your regression suite" is great advice. Building eval datasets by hand isn't. Here's what I do instead.

If you are into AI evals, you have definitely heard that you have to "turn your failures into your regression suite". That's solid advice. The only issue is that it takes soooo much time. I started doing it by hand, and I quickly realized it's not scalable. Especially given how fast you can implement things with coding agents. Also, blindly using agents to write your tests is shooting yourself in the foot. My wake-up call was when my coding agent hit a 503, found a Gemini key in memory, and burned $40 overnight instead of stopping. The fix was 2 lines. The real work was making sure the bug never came back after the next prompt or model change. The strategy looks like this (error analysis, the Hamel Husain / Andrew Ng approach): Pull 20–30 real sessions from your traces. Label each one pass or fail, noting the first thing that went wrong. No 1–5 scores: "fail" is actionable, "3" isn't. Fix the obvious stuff immediately. Group the remaining failures by type and rank the groups by frequency × severity. Write one cheap check per group. Code first, LLM judge last. Replay each group after every prompt, tool, or model change. Still failing means it isn't fixed. Failing again later means you have a regression. So what can you optimize to speed this up? In my concrete setup, I used Kitaru to manage my eval harness as follows: I pulled my traces from Opik labeled them in its UI (super important to still do it manually! but you need a neat UI where you can look at your data with 0 friction) After having the data, I let my coding agent go through steps 3 to 5 using Kitaru's MCP server and skills Then replayed each failure group to ensure everything works smoothly, while organically growing my evals dataset You could probably vibe-code these steps into a custom internal tool, but what's powerful about Kitaru is its replay mechanism, which caches tool outputs and uses them to replicate bugs and your context. It's still time-consuming, but works decently well. Still curious about your strategy for running AI evals and building your regression suite?