Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

Hugging Face Daily Papers Papers
encoder-pruning whisper asr model-compression distillation open-source performance-optimization

Summary

The paper introduces a method to prune encoder layers in the Whisper ASR model, reducing encoder size by 18.5% and recovering performance through unlabeled data distillation, with code and a pre-trained model released for adoption.

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to 18.5% of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to 20.1% after distillation, compared to 21.9% zero-shot, going from a baseline of 18.2%. We release all of our code (https://github.com/rasgaard/whisper-encoder-layer-prune) and the pruned model (https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).
Original Article
View Cached Full Text

Cached at: 09/24/26, 11:40 AM

Paper page - Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

Source: https://huggingface.co/papers/2609.27980

Abstract

Pruninglargepre-trainedtransformer-basedASRmodelssuchasOpenAI’sWhisperhasseengreatadoption,aspruningthedecoderledtosignificantend-to-endtranscriptionspeedups.Forinstance,the{\ttwhisper-large-v3-turbo}variantreducedthedecoderfrom32to4layers,whileDistill-Whispersimilarlyreducedthedecodertoonly2layers.Althoughsomeattentionhasbeenputtowardsreducingthesizeoftheencoder,noapproachhasseenwideadoption.Thiscouldbeduetotheneedforcustominferenceimplementationstotakeadvantageofthecompressedmodel.Wepresentanapproachthatranksencoderlayersbytheleave-one-layer-outchangeinWordErrorRate(WER).Thesixlayersthatcausetheleastchangeareremoved,correspondingto18.5%oftheencoderstack.Theprunedmodelrequiresnocustominferencecodeasitissimplyamoreshallowencoderwithfewerlayers.Wefurtherdistillusingunlabeledmonolingualspeechdatatorecoverperformancedegradationcausedbythezero-shotlayerpruning.MeanWERacrossfourlanguagesincreasesto20.1%afterdistillation,comparedto21.9%zero-shot,goingfromabaselineof18.2%.Wereleaseallofourcode(https://github.com/rasgaard/whisper-encoder-layer-prune)andtheprunedmodel(https://huggingface.co/rasgaard/whisper-large-v3-turbo-encoder-pruned).

View arXiv pageView PDFGitHub0Add to collection

Get this paper in your agent:

hf papers read 2609\.27980

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### rasgaard/whisper-large-v3-turbo-encoder-pruned 0.7B• Updatedabout 2 hours ago • 36 • 1

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.27980 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.27980 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Introducing Whisper

OpenAI Blog

OpenAI introduces Whisper, an end-to-end encoder-decoder Transformer model trained on large-scale diverse audio data for robust multilingual speech recognition, language identification, and speech-to-English translation. Whisper achieves 50% fewer errors than specialized models on diverse datasets and outperforms supervised benchmarks on speech translation despite not being fine-tuned to specific datasets.

The Sparsity Whisperer

arXiv cs.LG

This paper introduces difference-informed pruning methods (Wisp, Wisp+, Whisper) for large language models, showing that preserving output differences improves sparsification across Llama 2 and 3.1 models up to 405B parameters.

Adding E4B audio encoder to larger models

Reddit r/LocalLLaMA

The author proposes a method to add the E4B audio encoder to larger models by extracting the encoder, creating a linear projection layer, and fine-tuning only that layer with text-audio pairs, similar to a referenced paper but using Gemma instead of Whisper.

openai/whisper-large-v3

Hugging Face Models Trending

OpenAI released Whisper large-v3, an updated automatic speech recognition model with 128 Mel frequency bins and a new Cantonese token, trained on 5 million hours of data and achieving 10-20% error reduction over large-v2.