Is there an uncanny valley for AI voices?

# Is There an "Uncanny Valley" for AI Voices? Explorations and Insights from the Bland Team > **TL;DR:** The Bland team believes that a perfect TTS system would trigger the "uncanny valley" effect. They are developing a voice model that can generate natural human imperfections (such as breathing, pauses, and filler words) to pass phone tests and provide a more human-like interaction experience. ## Why Pursue Imperfection? Bland’s TTS Philosophy Bland's core idea is that they do not want a TTS system that generates perfect, fluent speech. What they need is a system that mimics the "imperfect" traits of human speech. This pursuit stems from a deep understanding of user experience. As team member Jack puts it: "We’ve all been on calls with IVR (Interactive Voice Response) or bots, and just hearing the robotic voice talk to you is annoying." They hope that by making the AI agent’s voice more natural and expressive, they can create an experience entirely different from other phone agents, making the person on the other end believe they are talking to a real human. ## What is Bland? Bland is an enterprise-grade voice AI platform. They self-host all their models, including TTS (Text-to-Speech), ASR (Automatic Speech Recognition), and LLMs (Large Language Models). This full-stack control enables them to meet the diverse needs of each enterprise client, where sometimes clients require different models and fine-tuning for specific use cases. ## Building an Expressive TTS Model Bland’s text-to-speech model is designed to generate super-natural, realistic speech. Its expressiveness comes not only from fluent sentences but also from what happens beyond the speech itself: * **Non-speech sounds:** Including breaths, laughs, chuckles, etc. * **Audio events:** Such as a dog barking in the background, a door closing, or even the sound of a TV playing. * **Human imperfections:** Like pauses, using filler words such as "um" and "uh," making sentences sound like they are spoken naturally rather than word by word. Researcher Jolin explains: "We don’t want a TTS system that generates perfect, flawless speech... We need a system with human imperfections, right? We humans don’t read from scripts; we’re not perfect, and that’s exactly what we want on the other end of the phone: something natural, more human." ## Technical Implementation: Data Pipelines, Decoupling, and Evaluation ### Constructing High-Quality Datasets To train such a model, they built a highly curated dataset. Pedro describes this incredibly challenging process: they started with millions of hours of audio data, which they had to whittle down to short segments of 5 to 30 seconds, ensuring each segment had only one speaker, clear audio, and intelligible content. Initially, the evaluation method was very basic—Pedro manually listened to hundreds of segments to judge quality. They then continuously iterated on and optimized various components of data collection and annotation to increase the proportion of high-quality segments. ### Model Architecture: Decoupling Control Their new TTS engine can directly model speech from any given text. The team frequently discusses the concept of "decoupling," which allows them to finely control different aspects of speech: * **Prosody:** Rhythm, stress, and intonation. * **Acoustics:** Frequency characteristics in speech. * **Identity:** The speaker’s vocal characteristics. This decoupling avoids the uncontrollability that can arise when training a complex model as a monolithic block. ### From Autoregressive to LLM Backbone In the current era of LLMs, TTS models have also adopted a similar style, tokenizing speech (via audio codec models) and then using LLMs for autoregressive modeling. More importantly, their TTS model is built on an LLM backbone. This means the model can understand the semantics of a sentence. Thus, when generating a sad line like "Oh, I’m really sorry," the model can automatically apply the appropriate tone based on its pre-trained understanding of the concept of "sadness," even if there isn’t an exact sample in the training data. ## Emotional Expression and Future Challenges ### Statistical Emotion Inference Currently, the model primarily extracts emotion from data through pattern recognition. If most instances of "congratulations" in the training data are spoken with an excited tone, the model learns this pattern. This is a form of statistical inference based on data distribution. ### Achieving Precise Emotional Control However, sometimes it’s necessary to break conventional emotional associations (for example, using an angry tone to say something typically considered happy). Pedro notes that this is more of a "research bet" because it’s difficult to find such samples in real data. In the future, they may need to explore how to synthesize data that generates these specific emotional characteristics. ## The Future of Conversational AI: From Cascaded Pipelines to Full-Duplex Modeling Currently, most conversational AI uses a **cascaded pipeline**: User speaks -> ASR transcribes to text -> LLM processes -> Text generates response -> TTS converts to speech. This approach suffers from information compression loss (such as losing tone and genuine emotion) and high latency. The future direction is **full-duplex model** research, aiming to maintain the speech modality throughout the entire conversation, keeping everything in speech tokens or vectors. The agent can perceive speech input directly and output speech simultaneously, without modality conversion. This allows for a more natural simulation of human conversational flow, including feedback, hesitation, and other natural interactions, without needing to hard-code these behaviors into prompts. --- **Source:** Is there an uncanny valley for AI voices? (YouTube) (https://www.youtube.com/watch?v=aSoIJVbIwAc)