Metraipolis › archive › dc_b4ba524cd6b2

Length Generalization Needs Proper Regularization — arXiv:2610.04518

Source
arXiv
Original
https://arxiv.org/abs/2610.04518
License (as archived)
CC0-1.0
Archived
2026-10-06T05:30:36.137Z
SHA-256 of the archived text
9004920c199b2912963a0d66c4df2b00774076c437bf1790a746574390835930
Stable URL of this page
https://metraipolis.com/documento/dc_b4ba524cd6b2

How to cite

Length Generalization Needs Proper Regularization — arXiv:2610.04518 — arXiv, https://arxiv.org/abs/2610.04518 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_b4ba524cd6b2 (Metraipolis archive dc_b4ba524cd6b2, sha256 9004920c199b2912963a0d66c4df2b00774076c437bf1790a746574390835930)
BibTeX
@misc{dcb4ba524cd6b2,
  title = {Length Generalization Needs Proper Regularization — arXiv:2610.04518},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04518},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_b4ba524cd6b2; sha256 9004920c199b2912963a0d66c4df2b00774076c437bf1790a746574390835930},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04518. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Length generalization is the ability of sequential models to perform well on context lengths unseen during training. In this work, we show that the challenge of achieving length generalization is related not only to architectural choices such as positional encoding and the attention mechanism but also to the training procedure itself. We study how regularization affects length generalization and find that weight decay can hinder extrapolation. In contrast, dropout improves extrapolation when its placement within the architecture is reconsidered. In that regard, we show that the standard placement of dropout before layer normalization introduces a systematic distributional mismatch, and that applying dropout just before the linear projection resolves this issue. For example, a modified SmolLM3 with sliding window attention, continually pre-trained with dropout, can extrapolate perfectly to 64$\times$ on Needle-in-a-Haystack and far beyond the pre-training context size on RULER and HELMET. Mamba2 also benefits from dropout, suggesting an architecture-agnostic nature of the problem. We further propose Variance-Preserving Affine Dropout (VPAD), a new dropout strategy that substantially reduces the resulting pre-activation variance mismatch, leading to further extrapolation improvements in transformers.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.04518 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.688Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).