Metraipolis › archive › dc_664c56199fc0

Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks — arXiv:2610.04533

Source
arXiv
Original
https://arxiv.org/abs/2610.04533
License (as archived)
CC0-1.0
Archived
2026-10-06T05:26:18.132Z
SHA-256 of the archived text
56ee5f9ada9fb812fa64accc6c61cff304e68f4e8df242cddaa84f364ca875d6
Stable URL of this page
https://metraipolis.com/documento/dc_664c56199fc0

How to cite

Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks — arXiv:2610.04533 — arXiv, https://arxiv.org/abs/2610.04533 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_664c56199fc0 (Metraipolis archive dc_664c56199fc0, sha256 56ee5f9ada9fb812fa64accc6c61cff304e68f4e8df242cddaa84f364ca875d6)
BibTeX
@misc{dc664c56199fc0,
  title = {Weight Decay and Neuron Condensation: A Three-Stage Analysis of Two-Layer ReLU Networks — arXiv:2610.04533},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04533},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_664c56199fc0; sha256 56ee5f9ada9fb812fa64accc6c61cff304e68f4e8df242cddaa84f364ca875d6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04533. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Weight decay is widely used as a regularization technique in neural network training, yet its role in neuron condensation (parameter direction alignment) remains unclear. Starting from a parameter initialization in the neural tangent kernel regime, we characterize training dynamics under weight decay through three stages: rapid fitting, amplitude compression, and neuron condensation. Using a two-layer ReLU network, we analyze a residual correlation field that governs both neuron amplitudes and directions. During rapid fitting, the residual approaches a quasi-static equilibrium maintained by weight decay while the tangent kernel remains nearly unchanged. In the early stage of amplitude compression, kernel decay amplifies the residual correlation field, whose isolated attracting extrema provide candidates for condensation directions. As neuron amplitudes stabilize, we bound the drift of attracting extrema and demonstrate contraction of neuron directions around them, leading to neuron condensation. This staged analysis provides a dynamical understanding of how weight decay promotes a condensed representation, beyond reducing parameter norms.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.04533 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.873Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).