Metraipolis › archive › dc_0f334ff2261d

Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery — arXiv:2609.27980

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.27980
License (as archived)
CC0-1.0
Archived
2026-10-08T10:05:08.937Z
SHA-256 of the archived text
154491c4d8d9118870aa825d1f22ae35c2bd28333ff82d1aaf74d1d17940d8c1
Stable URL of this page
https://metraipolis.com/documento/dc_0f334ff2261d

How to cite

Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery — arXiv:2609.27980 — arXiv (en), https://arxiv.org/abs/2609.27980 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0f334ff2261d (Metraipolis archive dc_0f334ff2261d, sha256 154491c4d8d9118870aa825d1f22ae35c2bd28333ff82d1aaf74d1d17940d8c1)
BibTeX
@misc{dc0f334ff2261d,
  title = {Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery — arXiv:2609.27980},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27980},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0f334ff2261d; sha256 154491c4d8d9118870aa825d1f22ae35c2bd28333ff82d1aaf74d1d17940d8c1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27980. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that ranks encoder layers by the leave-one-layer-out change in Word Error Rate (WER). The six layers that cause the least change are removed, corresponding to $18.5\%$ of the encoder stack. The pruned model requires no custom inference code as it is simply a more shallow encoder with fewer layers. We further distill using unlabeled monolingual speech data to recover performance degradation caused by the zero-shot layer pruning. Mean WER across four languages increases to $20.1\%$ after distillation, compared to $21.9\%$ zero-shot, going from a baseline of $18.2\%$. We release all of our code ( this https URL ) and the pruned model ( this https URL ).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.27980 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T09:11:49.643Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).