things i wish i knew before evaluating AI agents in production

hey everyone, been working through agent evaluation properly and wanted to share a few things that actually changed how i think about it. **start from the symptom not the layer** wrong tool being called — component problem. correct answer but too many steps — trajectory problem. final answer looks wrong — outcome problem. unsafe action or injection risk — adversarial problem. once you map symptoms to layers debugging gets way faster. **most teams only check final outputs** that's like only checking if a flight landed safely without looking at what happened during the flight. trajectory evaluation catches a whole class of failures that output checking misses entirely — duplicate calls, loops, unnecessary retries, cost blowouts. **an uncalibrated LLM judge is worse than no judge** if you haven't validated your LLM as judge against a small set of human labels you're adding noise on top of noise. calibration is not optional, it's the whole point. **convert every production failure into a test case** before your next release, not after. sounds obvious but almost nobody does it systematically. within a few cycles you have a regression suite that actually catches things before deployment. **adversarial testing is not optional** if your agent reads external content or takes real actions, indirect prompt injection through tool outputs is a real failure mode. most eval setups completely ignore this layer. happy to chat about any of these in the comments.