Everyone says write evals for your agent. But what should you actually test?

If you're starting to build agents, "you need evals", is one of those things everyone repeats, but the next step is weirdly underexplained. The hard part usually isn't the eval framework. It's figuring out: - which failures are worth turning into eval cases - what belongs in normal unit/integration tests instead - when to use deterministic checks vs LLM judges - how many times to repeat a case - how to avoid writing a suite that just tests wording The framing I've found most useful is: don't write one case per feature. Start from observed failures, push anything mechanically checkable down to cheaper tests, then write eval cases around the agent behaviors that still need model-level judgment or trajectory checks. A tiny first suite of 5-10 good cases is usually much better than a huge imagined benchmark. I wrote up a practical, framework-agnostic guide/skill for this, aimed at people who know they should do evals but don't know where to start. Link in comments.