X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Summary
X-AuT is a progressive framework for compressing audio-encoder layers in speech large language models, reducing inference cost while restoring accuracy via techniques like cross-scale distillation and LoRA adaptation.
View Cached Full Text
Cached at: 09/11/26, 10:17 AM
Paper page - X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation
Source: https://huggingface.co/papers/2609.11412
Abstract
X-AuT progressively prunes audio-encoder layers in speech large language models and restores accuracy via behavioral probes, representation alignment, cross-scale distillation, and LoRA adaptation.
Reducingaudio-encoder depthlowers the inference cost ofspeech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduceX-AuT, a progressive framework that selects layer combinations through shortbehavioral probesand restores the pruned model throughrepresentation alignment,cross-scale distillation, scheduled student-policy supervision, andLoRA finetuning. The language-model backbone remains frozen, whileattention LoRA adaptersand the tied output embedding adapt during distillation. Training uses the highest-agreement tier from atranscript-consistency pipeline, followed bysource reweightingduring finetuning. On ten public Chinese--English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macro-average error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18rightarrow14 pruning outperforms direct pruning (5.75% vs. 6.73%). These single-run results establish two practical operating points and show that the accuracy effects vary across benchmarks. Project website: https://xpeng-ai.github.io/x-aut
View arXiv pageView PDFProject pageGitHub0Add to collection
Models citing this paper1
#### XPENG-AI/X-AuT Automatic Speech Recognition• 0.9B• Updatedabout 3 hours ago • 3
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2609.11412 in a dataset README.md to link it from this page.
Spaces citing this paper1
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Unified Audio Intelligence Without Regressing on Text Intelligence
This paper introduces Audex, a unified audio-text LLM from NVIDIA that achieves state-of-the-art performance across multiple audio and speech tasks while preserving strong text reasoning capabilities without regression.
EchoDistill:Alignment Noisy-to-Clean Self-Distillation for Robust Audio LLMs
EchoDistill is an alignment-based noisy-to-clean self-distillation framework that improves the robustness of Audio Large Language Models (ALLMs) against real-world noise by using a frozen clean-audio teacher to guide the student model via group-relative policy optimization (GRPO). Experiments show significant improvements in semantic reliability and task performance under strong noise without additional inference costs.
AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
AuK is an open-source foundational model that unifies speech generation and editing through natural-language instructions, achieving leading performance with efficient inference via distillation.
Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
This paper introduces an instruction-free alignment-only method for building large audio-language models by freezing the LLM and audio encoder, training only a lightweight projector on self-generated data, achieving competitive performance with less data than traditional multi-stage pipelines.
Streaming Speech-to-Text Translation with a SpeechLLM
Presents a SpeechLLM architecture for streaming speech-to-text translation that adaptively decides when to output tokens based on audio, achieving 1-2 second latency with quality close to non-streaming baselines.