Metraipolis › archive › dc_73b83be4baa4

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights — arXiv:2610.11730

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11730
License (as archived)
CC0-1.0
Archived
2026-10-09T06:23:27.977Z
SHA-256 of the archived text
d22ed13ea313a385c2f1567721d260c757abf63f55aef475733960bafc7b4e48
Stable URL of this page
https://metraipolis.com/documento/dc_73b83be4baa4

How to cite

Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights — arXiv:2610.11730 — arXiv (en), https://arxiv.org/abs/2610.11730 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_73b83be4baa4 (Metraipolis archive dc_73b83be4baa4, sha256 d22ed13ea313a385c2f1567721d260c757abf63f55aef475733960bafc7b4e48)
BibTeX
@misc{dc73b83be4baa4,
  title = {Spectral Weight Decay: Inducing Low-Rank Structure in Neural Network Weights — arXiv:2610.11730},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11730},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_73b83be4baa4; sha256 d22ed13ea313a385c2f1567721d260c757abf63f55aef475733960bafc7b4e48},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11730. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Standard weight decay treats each weight matrix as a vector and ignores its spectral structure. We introduce spectral weight decay, a post-step decoupled nuclear-norm update that applies additive rather than multiplicative spectral shrinkage. We connect the update to approximate proximal descent and show that its sensitivity to update order can exceed that of conventional $\ell_2$ weight decay near rank deficiency. Across LLaMA models with $124$M to $500$M parameters, spectral weight decay lowers effective rank and improves SVD-LLM compression at matched validation loss. At $500$M and a $4\%$ distortion budget, it reaches $1.89\times$ compression and $1.18\times$ GPU inference speedup, compared with $1.14\times$ and $1.01\times$ after standard weight decay. Under fixed-horizon training with $60\%$ label noise, it also improves final mean clean-test accuracy over matched $\ell_2$ regularization by up to $17.8$ points on MNIST and $4.6$ points across four BERT-base tasks. Code is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2610.11730 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:50:31.599Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).