A right answer from your agent doesn't mean it did the right thing

For a while, I judged my agent only by its final answer. The answer read fine, so I assumed the steps behind it were fine too. That was the mistake. Then I started watching the actual run, the plan, which tool it called, what came back, what it "decided" before each move. The picture flipped. A lot of the answers that looked right were right by accident. The agent would call a tool, get back nothing useful, and still write a calm, confident answer on top of the gap. Nothing errored. Nothing looked broken. It just landed near the right answer that time. Three things that stuck with me: **1. The final answer hides the whole run.** A clean output can sit on top of a broken path — wrong tool, empty retrieval, a skipped step. If you only read the last message, you're grading the cover, not the book. **2. "Confidently wrong" is the failure that bites.** An agent rarely throws an error when it makes something up. It states the wrong thing calmly, and you find out when a user does. **3. You can't fix what you can't measure.** Once I had a fixed set of real inputs I could replay after every change, I finally knew whether a tweak actually helped or just pushed the problem to a different step. The thing that made this work was leaning on a tool that scores each output for stuff like factual accuracy and how grounded it is and keeps the full trace of every run, so I'm not reading raw logs trying to guess what happened. So, real question for anyone building agents: how do you catch the answers that are wrong but look right? Do you replay traces by hand, write checks per step, score the output automatically, or just wait for a user to flag it? Feels like everyone has their own hack and nobody agrees on the right one yet.