Optimizing ASR Models to Transcribe 90-Minute Multi-Speaker Audio

Optimizing ASR Models to Transcribe 90-Minute Multi-Speaker Audio

Overview

MOSS-TD is a speaker-aware ASR system: instead of just turning speech into text, it also works out who is talking and when, on recordings as long as roughly 90 minutes. Contributors integrated MOSS-TD into the SGLang-Omni serving stack, fixing correctness issues along the way and layering on a series of performance optimizations. The headline result: on a single H100, a 38-minute multi-speaker meeting comes back as a speaker-labeled, timestamped transcript in about 49 seconds — faster than it would take to simply listen to the first minute of the recording. Run 16 such meetings concurrently, and the same GPU sustains roughly 97.5 seconds of audio processed per second of wall-clock time. The rest of this document covers the engineering behind those numbers.

ASR Model Primer

ASR takes an audio waveform in and produces text out. MOSS-TD follows an “Audio LLM” design: a Whisper-style encoder turns audio into continuous embeddings, an FFN adapter projects those embeddings into a decoder-only LLM’s token space, and the LLM autoregressively writes out the transcript — speaker tags and timestamps included.

Three stages make up the pipeline:

• Encoder — the waveform is converted to an 80-bin log-mel spectrogram, run through a 24-layer Whisper Transformer, downsampled 4× by merging adjacent frames, and projected by an FFN adapter (Linear → SiLU → Linear → LayerNorm, 4096→1024) into continuous embedding vectors that live in the LLM’s embedding space.

• LLM Prefill — those embeddings replace placeholder tokens in the prompt, and Qwen3 processes the whole prompt to build its KV cache — one forward pass for short audio, or several 4096-token chunks for long audio (more on this below).

• Autoregressive Decode — Qwen3 writes the transcript one token at a time, including speaker tags like [S01]/[S02] and timestamps, stopping at an end-of-sequence token.

ComponentSpecArchitectureMossTranscribeDiarizeForConditionalGenerationAudio EncoderWhisper encoder (24 layers, d_model=1024)AdapterFFN: Linear→SiLU→Linear→LayerNorm, 4096→1024Text DecoderQwen3 (28 layers, hidden=1024, GQA 16/8)OutputSpeaker-labeled transcript with start/end timestampsEndpoint/v1/audio/transcriptions

Chunked Prefill for Long Audio

A 90-minute recording can turn into tens of thousands of encoded tokens. Prefilling all of that in one shot would occupy the GPU for seconds and spike activation memory — and because prefill and decode share a single scheduler loop, every other in-flight request’s decode would stall for that entire stretch. Chunked prefill avoids this by splitting the sequence into 4096-token pieces, running one chunk per scheduling step and interleaving other requests’ decode steps in between. The trade-off is that the long request’s own prefill finishes a bit later (spread across more scheduling rounds), in exchange for bounded decode stalls and smooth streaming for everyone else sharing the GPU. Streamed output is held back entirely while a request is mid-chunked-prefill, so partial internal states never leak out looking like transcript text.

ASR vs. TTS

ASR and TTS share much of their serving machinery in SGLang-Omni — the same OmniScheduler, CUDA graph handling, KV-cache management, and continuous batching, and both are autoregressive models with an LLM backbone (it’s a coincidence that MOSS-TD and MOSS-TTS both happen to use Qwen3 — the backbone choice varies by model). Where they genuinely differ is in what they encode, what they produce, and how their pipelines are shaped:

**DimensionASR (MOSS-TD)TTS (Higgs / MOSS-TTS)**Audio representationContinuous features (mel → encoder hidden states)Discrete codec tokens (RVQ multi-codebook)Data flowAudio → textText → audioAudio decoder / VocoderNot needed — output is plain textRequired to reconstruct the waveformTypical input lengthVery long (MOSS-TD handles ~90 min)Short (reference voice: a few seconds)Pipeline stagesSingle stage (encoder + LLM)Multi-stage (encoder → AR engine → vocoder)StreamingIncremental text outputStreaming audio + streaming vocoder

Because ASR never touches a vocoder, its engineering challenge shifts elsewhere: long inputs push the real optimization work onto the autoregressive decode loop and long-sequence memory management, rather than audio reconstruction quality. For the TTS side of this story, see the companion post Optimizing TTS Inference (tts-optimization.md).

Where Time Is Spent: Profiling

Before making any changes, the team profiled MOSS-TD on a single H100 (CUDA graphs on, bf16) to see where time was actually going.

Audio LengthConcurrencyEncoderLLM PrefillAR Decode5 s18.9%14.7%76.4%5 s420.0%22.3%57.7%5 s1638.2%29.7%32.1%60 s14.0%2.1%94.0%60 s45.0%4.6%90.4%60 s1613.7%9.5%76.8%20 min14.7%0.8%94.5%20 min49.2%1.9%88.9%20 min1611.6%2.6%85.7%

Two things stood out: with long audio at concurrency 1, decode eats more than 94% of total time, so nearly all the available leverage sits there; with short audio at concurrency 16, encoder and prefill together account for 68% of time, which makes encoder-side work worth doing too.

Optimization Strategies

Much of the optimization work reuses infrastructure already built for TTS serving, adapted to ASR’s simpler pipeline (no vocoder, no multi-codebook sampling) — CUDA graphs, async decode, and encoder caching all show up again here.

Encoder

• CUDA Graph. Because the Whisper encoder always works on fixed 30-second windows (audio is chopped into 30 s chunks, with the last one padded), every chunk has an identical shape — so the encoder is bucketed purely by how many chunks a request has (up to 8 by default, roughly 4 minutes of audio), and each bucket captures its own CUDA graph to remove per-call kernel launch overhead.

• Torch Compile (opt-in). As an alternative to the CUDA graph, torch.compile(self.whisper_encoder, dynamic=False) adds kernel fusion on top. The default compile mode is deliberately skipped: reduce-overhead mode manages its own internal CUDA graphs, and those collide with the decode-side CUDA graphs running in the same process. This trades a slower first call (one-time compilation per bucket) for better steady-state throughput, and is worth turning on for encoder-bound, high-concurrency, short-audio workloads.

• LRU Cache. The encoder’s output is deterministic for a given input, so outputs are cached on the CPU (up to 64 entries, 4 GB), keyed by a hash of the waveform; a hit skips the encoder entirely and copies the cached embedding back to GPU asynchronously. Unlike TTS, where the same reference voice often gets reused across many prompts, ASR inputs are usually unique in production — so this cache is more useful for retries, A/B-testing different decode settings, and local development than for a high production hit rate.

AR Decode

• CUDA Graph. Decode batch sizes are padded to fixed buckets (1, 2, 4, 8, …), and a captured CUDA graph is replayed for each generated token — the same technique used for TTS decode.

• Async Decode. The same one-step-lookahead trick used for TTS: launch the GPU work for the current decode step, and while it’s running, resolve the previous step’s host-side work (device-to-host copy, completion checks, dispatching results) in parallel. At batch size 1 this falls back to synchronous execution, since there isn’t enough host-side work to make overlapping worthwhile. Two alternating pinned host buffers keep the GPU’s asynchronous writes and the CPU’s reads from racing. This meaningfully improves throughput at high concurrency, and the same change also closed a KV-cache slot leak caused by the lookahead mechanism overrunning.

• Stream Output. Transcript text streams out over SSE as it’s generated, governed by three rules: tokens buffer per-request and flush after a default 50 ms window (the very first token goes out immediately, and end-of-sequence always forces a flush); all streaming is suppressed while a request is in chunked prefill, so intermediate state never looks like transcript; and if the buffered tokens end mid-way through a multi-byte UTF-8 character, the flush is held until the next token completes it.

Batched Inference

On the encoder side, mel spectrograms of different lengths are aligned so several requests can share one batched Whisper forward pass. On the LLM side, multiple requests’ token sequences are packed into shared batches for both prefill and decode, which keeps GPU utilization high under concurrent load.

Benchmark Results

Benchmarks used two private multi-speaker datasets chosen to bracket the range of input lengths:

• Movies (movies800times) — 800 short dialogue clips, ~12 s each.

• AISHELL-4 Long (aishell4_long) — 20 long-form meeting recordings, ~38 min each.

Both datasets are under private license; contact the MOSS team for access.

Two metrics matter here: RTF (Real-Time Factor) is processing time divided by audio duration — under 1 means faster than real time. audio_s/s is total audio-seconds processed per wall-clock second, which is what actually captures the throughput gained from batching.

Setup: single H100 80GB, single-GPU colocated deployment; MOSS-Transcribe-Diarize in bf16 with CUDA graphs and greedy decoding; server pinned at max_running_requests = cuda_graph_max_bs = 16, mem_fraction_static = 0.80. Movies numbers are averaged over 3 runs; AISHELL-4 Long is one run per data point, since each request there is a ~38-minute meeting. Accuracy was measured once at concurrency 16 (greedy decoding makes it concurrency-independent).

Movies — short multi-speaker dialogue (N=800)

**ConcurrencyThroughput (req/s)RTF meanaudio_s/sLatency mean (s)Latency p95 (s)**14.550.02252.70.2190.50028.400.02497.20.2380.544414.960.027173.20.2670.610824.900.033288.10.3210.7141632.790.043379.50.4220.935

Going from concurrency 1 to 16, throughput and audio_s/s both scale roughly 7.2× (4.6→32.8 req/s; 53→379 audio_s/s). RTF stays well under 1 the whole time (0.022→0.043, i.e., roughly 23–45× faster than real time), and mean latency never leaves sub-second territory. As batches fill up, short-audio ASR leans more on the encoder and prefill, so per-request RTF drifts upward even while aggregate throughput keeps climbing.

AISHELL-4 Long — long-form meetings (N=20, ~38 min each)

**ConcurrencyThroughput (req/s)RTF meanaudio_s/sLatency mean (s)Latency p95 (s)**10.0210.02147.048.771.120.0300.02869.764.978.540.0340.04877.7110.2142.780.0370.08185.4184.8239.2160.0430.12797.5291.0374.5

Each request here is a ~38-minute meeting dominated by decode, so requests-per-second is naturally low and RTF is the metric that matters. A single stream already turns a 38-minute meeting into a transcript in ~49 s (RTF 0.021, roughly 47× faster than real time). Pushing concurrency to 16 more than doubles aggregate audio_s/s (47→97.5) and req/s (0.021→0.043); per-request latency and RTF grow under batch contention but stay around 8× faster than real time (RTF 0.127) — the expected trade-off of higher aggregate throughput against higher single-request latency for long audio.

Accuracy

Measured at concurrency 16 with greedy decoding. CER is character error rate; cpCER adds concatenated-minimum-permutation alignment to account for speaker assignment; Δ CER is the gap between the two, attributable to speaker-assignment errors; DER is the speaker-timestamp diarization error rate.

**DatasetSamplesCER (%)cpCER (%)Δ CER (%)Speaker-timestamp DER (%)**Movies8005.9213.127.2021.20AISHELL-4 Long2013.7615.071.3110.25

Model Usage

See the MOSS-TD cookbook for deployment and usage instructions.

What’s Next

A few optimization efforts are still ongoing:

• Streaming audio input — transcribe incrementally as audio arrives, rather than waiting for the whole clip to finish uploading.

• Piecewise prefill CUDA Graph — capture chunked prefill itself as a CUDA graph.

• Tensor parallelism — multi-GPU TP support, aimed at very long audio files whose KV cache could outgrow a single GPU’s memory.

Acknowledgments

This work reflects a joint effort between the OpenMOSS team and the SGLang-Omni team.

MOSS Team: Donghua Yu, Zhengyuan Lin, Hanfu Chen, Yiyang Zhang, Yang Gao, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu.

SGLang-Omni Team: Yijiang Tian, Xinli Jing, Xiangrui Ke, Zhihao Guo, Ruoyi Zhang, Lifan Shen, Jintao Qu, Xuxiang Tian, Kaige Li, Ratish P, Haoguang Cai, Zijie Xia, Chenchen Hong, Xuesong Ye, Jingwen Gu, Jiaxin Deng, Jiaxuan Luo, Xinyu Lu, Hao Jin, Chenyang Zhao, Yichi Zhang.

Learn More

• Model: OpenMOSS-Team/MOSS-Transcribe-Diarize

• Serving framework: SGLang-Omni on GitHub

• Cookbook: MOSS-TD in SGLang-Omni

• ASR optimization roadmap: tracking issue on GitHub

• TTS optimization blog: Optimizing TTS Inference (tts-optimization.md)

• Encoder skew analysis: The Root Cause of RL Training-Serving Skew (moss-tts-local-batch-encoder-skew.md)