5 things I learned running an AI agent for 45,000 people

I built an AI agent that has reached 45,000 people — 7.5 million messages and 735,000 calls. Some things I learned along the way: 1. Reading the conversations is irreplaceable. We anonymize the logs — reading user conversations feels a little dirty, and you want to be responsible about it — but it is so, so helpful. Evals and QA only catch so much. Real users will find a zillion things your testing never will, and reading how the agent actually handled them is how we decided what to fix, what to add, and what to leave alone. 2. Managing an agent is a lot like managing a person. That framing sounds cute but it's genuinely how the work goes. When I wanted something done — reach back out to people who'd gone quiet, handle a certain kind of request differently, soften the tone — I'd give the agent direction, it would go do it, and I'd check its work in the logs. Some direction is universal, things you want in every single interaction. Some is specific to one customer. Getting both right is the actual job, and like managing a person, you're never done — sometimes the agent got it right, sometimes it didn't. 3. Some people will try to make it romantic, and you can only deter so much. At 45,000 people, a portion will push there no matter what you do. Every safeguard you add makes the product a little worse for everyone else, so past a certain point you're just taxing your normal users. People find a way. 4. Specific agents beat general ones. Every time. A general product degrades as you add more and more to it, and eventually you hit demands that are directly in conflict — you can have either, but you can't have both. Images were one example for us: some users wanted image generation, others didn't want images of any person at all. The romantic stuff is another: some people want none of it, some want a little, some want a lot. Trying to mold one agent to fit lots of people is really tough. The narrower the job, the better the agent does it. 5. The economics have to come from somewhere. This is an expensive service to operate, and for a consumer product it's genuinely hard to find the economic rationale — without VC funding it's pretty tough to make viable. There needs to be real ROI on using the agent, which is why attaching it to a business use case makes the most sense. Our agent runs on autopilot now, makes about $10,000 a month, and as far as we can tell, that's about as big as that opportunity gets. So we packaged up everything we learned into a platform where anyone can create SMS agents. The platform is called Outside Agent. You build them from your coding agent — describe what you want, and it's live on a real phone number in about 30 minutes. If anyone wants to build an SMS agent, give it a try. It's free to try — link in comments.