Do customers hate voice AI or the pauses?

We are reviewing voice AI for a large banking contact center. The feedback internally is split. Some people think customers will reject an AI voice agent no matter how good it is. I am starting to think the bigger issue is latency. Personally, the worst voice systems I have dealt with were frustrating because of the pauses. You answer, wait two or three seconds, assume it did not hear you and start talking again just as it responds. In my experience, the conversation feels much less frustrating when the response is quick, even if the voice itself is not perfect. For anyone who has tested this with customers, what complaints came up most?