My agent was more accurate than the team it replaced. They still refused to trust it.

Built a triage agent for a support team that was measurably better than the manual process it replaced. Higher agreement with the "correct" label than the humans hit on their own. On paper, done. In practice they quietly stopped using it inside two weeks. Not because it was wrong. Because when it was wrong, nobody could see why, and one unexplained miss poisoned their trust in the ninety that were right. A black box that's correct 94% of the time feels worse to use than a person who's correct 88% of the time, because you can ask the person what they were thinking. I almost went down the road of tuning for more accuracy. That would have missed the point entirely. The problem was never the accuracy number. It was that people won't hand judgment to something they can't interrogate. So I made it explain each decision in one plain line. "Routed to billing because the message mentions a refund and an invoice number." Same model, same accuracy, I just stopped hiding the reasoning. Adoption flipped almost immediately. When the agent was wrong, the rep could see the bad assumption, fix it, and move on instead of escalating a mystery. The visible reasoning also handed me a clean stream of exactly where it failed, which made it genuinely easy to improve. The lesson I keep relearning: for anything that makes a decision a human is accountable for, legibility beats accuracy. People don't need the agent to be perfect. They need to see why it did what it did so they can trust it the other ninety percent of the time. Anyone else found that exposing the reasoning mattered more than squeezing out the last few points of accuracy?