Building an AI agent that makes real phone calls (hold music, IVRs, angry humans), here’s what I learned so far

I built callitdone.today. It’s a tool that places phone calls for you, not text, not voicemail drops, actual voice calls that navigate menus, wait on hold, and speak with real people on your behalf. The motivation was simple: I hate calling customer service. I hate sitting on hold for twenty minutes listening to the same two bars of Muzak. I also had a small business clients who needed to book appointments and handle billing disputes but would rather do literally anything else. So I built an agent that dials the number, deciphers IVR prompts, waits through hold music (it does listen for silence vs. hold music), and then either talks to a human or leaves a message. If it hits an obstacle it can’t handle, like a menu option I didn’t script for, or a fast-talking human whose accent throws the speech-to-text, it fesses up honestly: “I’m an AI assistant, I’m having trouble, I’ll transfer you to a human or reschedule the call.” It doesn’t bluff. Who is this for? People who hate making phone calls and sitting on hold, or small-business owners who want routine phone calls handled for them (appointment booking, account questions, order status). It currently only works with US phone numbers, and that’s not going to change anytime soon, international telephony is a nightmare of different signaling, regulations, and language coverage that I’m not ready for. Architecture-wise, it’s a stack of: a web front-end where you describe the call goal (plain English), a backend that calls Twilio’s voice API to dial, a speech-to-text + text-to-speech pipeline, and a small LLM fine-tuned to stay on script but handle common variations. The hardest part was not the AI, it was handling unpredictable hold times, “press 1 for English” menus that don’t follow the expected path, and the occasional screaming human who realizes they’re talking to a robot. I’ve had calls where the agent successfully booked a dentist appointment, and calls where it got stuck in a loop at “Press 3 to repeat this menu.” Honest lessons: 1) Hallucination is a real problem if you let the LLM generate the next utterance, I switched to a template-based + slot-filling approach for the core interaction. 2) Hold music detection is harder than it sounds; some companies use dead air, some use radio ads between tracks. 3) Users really want the agent to sound exactly like them, but that requires voice cloning which has legal and ethical landmines I’m not touching yet. Would love feedback on what else you’ve run into building voice agents for real-world phone calls, the IVR mapping piece especially feels like an unsolved mess. I’ll drop the link in a comment per sub rules.