Metraipolis › archive › dc_375090bfd686

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation — arXiv:2610.00423

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00423
License (as archived)
CC0-1.0
Archived
2026-10-06T12:09:27.143Z
SHA-256 of the archived text
056750f499a250f15f2dc060d3c3c6adfb2fe2395e53c4be33459b2c81ca7658
Stable URL of this page
https://metraipolis.com/documento/dc_375090bfd686

How to cite

The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation — arXiv:2610.00423 — arXiv (en), https://arxiv.org/abs/2610.00423 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_375090bfd686 (Metraipolis archive dc_375090bfd686, sha256 056750f499a250f15f2dc060d3c3c6adfb2fe2395e53c4be33459b2c81ca7658)
BibTeX
@misc{dc375090bfd686,
  title = {The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation — arXiv:2610.00423},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00423},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_375090bfd686; sha256 056750f499a250f15f2dc060d3c3c6adfb2fe2395e53c4be33459b2c81ca7658},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00423. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood. Combining training-trajectory analyses and controlled interventions, we trace their emergence, growth, and consolidation. Sink-carrying channels vary across random seeds but stabilize early within each run. Over longer training, surrounding channels erode and the sink concentrates onto a few redundant carriers. Across ablations, gradient attenuation follows the sink token's collective root-mean-square magnitude rather than any single channel, making collective scale central to understanding their effects. Our central result is that weight decay causally controls the turnover of global activation scale. In controlled continuations, removing decay near the peak allows this scale to keep rising, whereas retaining it produces decline even at constant learning rate. We develop a balance model for the rise and peak of massive-activation magnitude, in which AdamW-preconditioned growth opposes weight decay. Sweeping the decay coefficient $\lambda$ shifts peak timing approximately log-linearly and yields peak magnitudes scaling approximately as $\lambda^{-1/2}$, consistent with this balance. Optimizer measurements further show that preconditioning sustains the large-channel cohort against decay even when raw maintaining forces are too small to do so. Together, these findings connect the observed life cycle to scale-regulating training dynamics and establish weight decay as a training-time lever on activation magnitude.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.00423 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.067Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).