Metraipolis › archive › dc_f8a394ce980c
Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models — arXiv:2611.04449
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04449
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T12:33:42.055Z
- SHA-256 of the archived text
4dfc4be8759eed105c599515a6ab23ee09f4eb8945ce08fdd67fd80e61ff2233- Stable URL of this page
https://metraipolis.com/documento/dc_f8a394ce980c
How to cite
Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models — arXiv:2611.04449 — arXiv (en), https://arxiv.org/abs/2611.04449 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f8a394ce980c (Metraipolis archive dc_f8a394ce980c, sha256 4dfc4be8759eed105c599515a6ab23ee09f4eb8945ce08fdd67fd80e61ff2233)
BibTeX
@misc{dcf8a394ce980c,
title = {Near-Zero Token Embeddings Silently Stall Training in Gated Sequence Models — arXiv:2611.04449},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04449},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f8a394ce980c; sha256 4dfc4be8759eed105c599515a6ab23ee09f4eb8945ce08fdd67fd80e61ff2233},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04449. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Among 291 surveyed open-weight language model repositories (280 distinct embedding tables), 52.6% of untied gated and hybrid checkpoints (20/38) contain reachable input embeddings with norms below $10^{-6}$, compared with 0% of weight-tied models. Such near-zero rows can arise during pretraining when unseen rare or template tokens receive zero input gradients while AdamW weight decay shrinks untied embeddings. In tested Transformer configurations, non-initial positions or a BOS token prevent this failure mode. In gated sequence models (Mamba, Gated DeltaNet, xLSTM), a near-zero token can amplify input-embedding gradients by more than 30 orders of magnitude (up to 35). At zero input, current-token readout factors (SiLU gates or linear queries) vanish, suppressing historical-state readout. Residual RMS values can then fall below the square root of the RMSNorm floor across consecutive layers, causing Pre-RMSNorm to amplify backward gradients by up to $10^3$ per layer. When a gradient entry exceeds $1.84 \times 10^{19}$, FP32 global-norm clipping in PyTorch, Hugging Face Trainer, and FSDP2 can overflow to infinity and zero the current batch's gradients. In Telugu continual pretraining of Nemotron-3-Nano-4B, this discards updates on approximately 20% of steps and leaves triggered rows frozen. We propose two remedies. For released checkpoints, GroupClip decouples input-embedding clipping using a 64-bit or rescaled norm; its rescaled implementation eliminates update loss and heals triggered rows in one step with negligible measured end-to-end overhead. For pretraining, GateFloor ($\mathrm{SiLU}(z) + \delta$, $\delta \in [0.01, 0.1]$) keeps the gate positive at zero input, reducing near-zero gradients by 28 orders of magnitude in experiments up to 370M parameters, with perplexity comparable to the baseline.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2611.04449 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T12:27:02.071Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).