How do you test AI agents before and after deployment? What has actually held up for you?

I'm trying to work out a sane testing setup for a tool-calling agent and I'd like to hear what's working for people who run these in production. Here are the approaches I keep coming across, and my doubts about each. Scenario tests with scripted multi-turn conversations. They're easy to reason about, but I worry they only cover the paths I already thought of, so they miss the weird ones real users find. LLM-as-judge. It's the only practical way I see to score free-text answers at scale. But I don't trust a judge score until it's been compared against human labels, and I'm not sure how many labeled rows are enough before it can gate anything. Tool-call checks. Verifying which tool was called, with which arguments, and whether the agent handled an error properly seems more reliable than grading the final message. An agent can say "done, your order is refunded" when the call actually failed. Is anyone scoring the trajectory this way, and is it worth the effort? Regression suites. The idea is to turn every real failure into a test row so the same bug never ships twice. The open problem is how to keep the suite from getting noisy or slow as it grows. A few questions for you. What do you run before a release, and what do you treat as a hard gate versus a warning? After deployment, do you score live sessions, sample them manually, or just wait for user complaints? How do you handle nondeterminism, for example by running each case several times or using thresholds instead of exact pass/fail? Curious what's been a waste of time too, not just what worked.