Tag
X-AuT is a progressive framework for compressing audio-encoder layers in speech large language models, reducing inference cost while restoring accuracy via techniques like cross-scale distillation and LoRA adaptation.
The author proposes a method to add the E4B audio encoder to larger models by extracting the encoder, creating a linear projection layer, and fine-tuning only that layer with text-audio pairs, similar to a referenced paper but using Gemma instead of Whisper.