What actually makes an AI voice agent useful for a small business?

I’ve been looking more closely at where AI voice agents actually make sense for small businesses, and I think the useful cases are much narrower than the hype suggests. The strongest use cases seem to be: answering repetitive inbound questions qualifying leads before a human gets involved booking or rescheduling appointments handling basic after-hours calls routing calls based on intent collecting structured information before handing off to a person Where things get much harder is when the conversation requires judgment, negotiation, empathy, or handling unusual situations. The biggest lesson for me is that the voice model itself is only part of the system. The real quality seems to depend more on the workflow around it: fallback rules, escalation, CRM integration, latency, and what happens when the agent is unsure. For people actually building or using voice agents: what’s been the hardest part in practice — latency, interruptions, integrations, or getting the agent to know when to hand off to a human?