One line system prompt change dropped model quality from 84% to 52%. How are people monitoring semantic quality in production?

A few weeks ago I changed a single line in a system prompt during a deploy. Nothing looked wrong: * error rate stayed normal * latency looked fine * requests were returning 200s But response quality got noticeably worse, and I only found out 11 days later because a user complained. That honestly felt weird coming from normal backend engineering, where failures are usually obvious pretty quickly. With LLM apps it feels like you can have a system that's technically healthy while giving bad answers the entire time. Example: support bot starts confidently saying refunds are valid for 60 days instead of 30. No exception gets thrown. No alert fires. Everything looks green. After that incident I started building some internal tooling to monitor semantic quality instead of just infra metrics. Main things that ended up being useful: * running background evals on sampled responses * checking hallucinations against retrieval context * comparing prompt versions statistically instead of eyeballing outputs * retry/flagging when responses look suspicious * clustering failures to spot recurring patterns One thing that surprised me: LLM-as-judge scoring was way noisier than I expected. Running the same judge multiple times on identical inputs gave pretty different scores sometimes, so I started aggregating runs instead of trusting single outputs. Curious what other people are doing for this in production. Are most teams just running evals before deploys? Human review? Shadow traffic? Custom judge pipelines? Feels like "we found out from a user complaint" is still the default monitoring strategy for a lot of LLM apps.