Evaluating agents is really hard

Been building with LLMs for a while and recently moved one of my features from a single prompt/completion setup to an actual multi-step agent (tool calls, a few steps of reasoning, deciding what to do next, etc.). Working great in the demos. The eval side is where I'm kind of lost now. With single completions it was pretty simple, send an input, score the output, done. I had a nice little dataset and a scoring setup and felt good about it. The thing that's been messing with me is the final answer will look totally fine, so my old "score the output" eval gives it a pass, but when I actually go look at what happened in between it's nonsense. It called the wrong tool, recovered by accident, made an assumption that just happened to be right this time. So the output is "correct" but the path it took to get there is totally wrong (or at least not the intended path). Feels like next week the same query falls apart because one of those lucky steps goes the other way. So scoring just the final output clearly isn't enough for agents. But I'm not sure how to actually eval the trajectory, the whole sequence of steps, without it turning into a giant manual mess of me reading through traces one by one. Questions for people further down this road: How are you evaling the trajectory and not just the final answer? Do you score individual steps (right tool, right args) or score the path as a whole? How do you keep it from becoming a maintenance nightmare every time you change the agent's flow? Feel like this has to be a solved-ish problem by now but I'm clearly missing something.