How are people evaluating AI agents after they go into production?

I keep seeing a lot of discussion about building agents, improving prompts, adding tools, RAG, memory, etc. But I'm curious about what happens after the agent is actually talking to real users. Suppose an agent handles 5,000 conversations. How do you know whether: it gave the correct answer? it followed the company's current policy? it should have escalated but didn't? it relied on outdated information? it gave a technically plausible but incorrect answer? the same mistake is happening repeatedly? Automated evals obviously help, but I'm wondering how people handle the messy real-world conversations that weren't anticipated when the eval set was created. I'm particularly interested in production QA rather than pre-launch testing. If you're running agents in production, what does your QA/evaluation process actually look like?