VibeVoice-ASR-Streaming Technical Report

Hugging Face Daily Papers Papers
asr speech-recognition diarization llm-based streaming end-to-end low-latency

Summary

VibeVoice-ASR-Streaming is an LLM-based end-to-end model for streaming speaker-attributed speech recognition, achieving state-of-the-art performance with released 1.5B and 7B model weights.

Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Original Article
View Cached Full Text

Cached at: 09/03/26, 07:50 AM

Paper page - VibeVoice-ASR-Streaming Technical Report

Source: https://huggingface.co/papers/2609.02812 Authors:

,

,

,

,

,

,

,

,

,

,

,

Abstract

A streaming, LLM-based end-to-end model unifies speaker-attributed speech recognition and diarization for low-latency real-time applications.

Traditionalspeaker-attributed ASRsystems treated ASR andspeaker diarizationas two separate tasks. Recently,end-to-endmodels such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the firstLLM-basedend-to-endapproaches tostreamingspeaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount oflookahead audioand previous text. This allows the model to produce ’‘who said what’’ as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest averageWER/CERacross five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2609\.02812

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper3

#### microsoft/VibeVoice-ASR-Streaming-7B Automatic Speech Recognition• 9B• Updatedabout 2 hours ago • 27 #### microsoft/VibeVoice-ASR-Streaming-1.5B Automatic Speech Recognition• 3B• Updatedabout 2 hours ago • 11 #### lstari/VibeVoice-ASR-Streaming-7B Automatic Speech Recognition• 9B• Updatedabout 2 hours ago

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.02812 in a dataset README.md to link it from this page.

Spaces citing this paper4

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Microsoft VibeVoice-ASR-Streaming Released

Reddit r/LocalLLaMA

Microsoft has released VibeVoice-ASR-Streaming, a unified streaming ASR model that transcribes who said what with support for customized hotwords and 10 languages.

microsoft/VibeVoice-ASR-BitNet

Reddit r/LocalLLaMA

Microsoft releases VibeVoice-ASR-BitNet, a compressed multilingual ASR model for real-time CPU inference. It achieves 1.6-2.3x faster inference than Whisper.cpp with real-time capability on as few as 3 CPU threads.

VibeVoice Technical Report

Papers with Code Trending

VibeVoice is a new model from Microsoft that synthesizes long-form multi-speaker speech using next-token diffusion and a highly efficient continuous speech tokenizer. It achieves superior fidelity and compression, supporting up to 90 minutes of audio with multiple speakers.

microsoft/VibeVoice

GitHub Trending (daily)

Microsoft open-sourced VibeVoice, a unified speech-to-text model handling 60-minute audio in a single pass with speaker/timestamp/content transcription, and a real-time text-to-speech model supporting streaming input and multilingual voices. The project includes an edge CPU inference engine for ASR via quantization.