What Did I Just Say? Self-Listening for Full-Duplex Speech Models

Hugging Face Daily Papers Papers
full-duplex-speech self-listening speech-models interruption-recovery conversational-ai language-models

Summary

This paper proposes Self-Listening, a method for full-duplex spoken language models that feeds realized speech back as input to improve interruption recovery and consistency with actually spoken responses.

Full-duplex spoken language models can listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model's realized speech as anchor interruption. To address this problem, we propose Self-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model's played speech. By feeding the realized speech output back to the model as an input stream, self-listening grounds interruption recovery in what the user has actually heard. We further introduce AnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken. AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped with self-listening mechanism achieve better anchoring performance.
Original Article
View Cached Full Text

Cached at: 09/09/26, 04:32 PM

Paper page - What Did I Just Say? Self-Listening for Full-Duplex Speech Models

Source: https://huggingface.co/papers/2609.05592 Published on Sep 4

·

Submitted byhttps://huggingface.co/Benyou

Wangon Sep 9

Abstract

Self-Listening grounds interruption recovery in full-duplex spoken language models by feeding realized speech back as input, improving consistency with actually spoken responses.

Full-duplex spoken language modelscan listen and speak simultaneously, enabling them to handle interruptions and backchannels in human conversation. However, text generation, speech synthesis, and audio playback proceed asynchronously. As a result, what a model believes it has said may not match what has actually been played to the user. We refer to the problem of recovering from an interruption while remaining aware of the model’srealized speechasanchor interruption. To address this problem, we proposeSelf-Listening, a full-duplex modeling approach that interleaves user speech, model text, and the model’s played speech. By feeding therealized speechoutput back to the model as an input stream,self-listeninggrounds interruption recovery in what the user has actually heard. We further introduceAnchorSpeech, a collection with homogeneous training and test splits for tracking which items of structured ordered responses have actually been spoken.AnchorSpeech-test evaluates whether a model can respond consistently with the last completed item before an interruption. Experiments show that, compared with full-duplex baselines, models equipped withself-listeningmechanism achieve better anchoring performance.

View arXiv pageView PDFGitHub1Add to collection

Get this paper in your agent:

hf papers read 2609\.05592

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.05592 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.05592 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.05592 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Liberating LLM Capabilities in Full-Duplex Speech Models

Hugging Face Daily Papers

Proposes Listen-Write-Speak (LWS), a text-first tri-channel paradigm that allows a single autoregressive LLM to continuously listen, write visible text, and speak in real-time, enabling full-duplex speech interaction without architectural modifications.