Metraipolis › archive › dc_fce2dd0fedb5

Underscoring the Problem: Why Softpick Fails at Initialization — arXiv:2610.05488

Source
arXiv
Original
https://arxiv.org/abs/2610.05488
License (as archived)
CC0-1.0
Archived
2026-10-06T05:31:07.095Z
SHA-256 of the archived text
d1e9d384adda555b37ee3c42188da662891281b1607f003e68f33c6bf7c81372
Stable URL of this page
https://metraipolis.com/documento/dc_fce2dd0fedb5

How to cite

Underscoring the Problem: Why Softpick Fails at Initialization — arXiv:2610.05488 — arXiv, https://arxiv.org/abs/2610.05488 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fce2dd0fedb5 (Metraipolis archive dc_fce2dd0fedb5, sha256 d1e9d384adda555b37ee3c42188da662891281b1607f003e68f33c6bf7c81372)
BibTeX
@misc{dcfce2dd0fedb5,
  title = {Underscoring the Problem: Why Softpick Fails at Initialization — arXiv:2610.05488},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05488},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fce2dd0fedb5; sha256 d1e9d384adda555b37ee3c42188da662891281b1607f003e68f33c6bf7c81372},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05488. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Softmax attention gives every token a nonzero weight, which in trained models concentrates into attention sinks and massive activations that widen the dynamic range low-precision inference must cover. Softpick removes this constraint by rectifying scores, eliminating sinks and lowering hidden-state kurtosis, but its advantage fades at scale. We reframe this failure as a normalization problem. Softpick's denominator splits into positive- and negative-shifted sums $D^+$ and $D^-$, used identically in the forward and backward pass, preventing their roles from being isolated. We separate them into a family of operators that independently choose each denominator. The failure originates at initialization: every layer contains rows where $D^+$ is exactly zero, while near-dead rows produce gradient norms above $10^{12}$ regardless of the backward denominator. Only Softpick and a stop-gradient variant, which keeps $D^+ + D^-$ forward but backpropagates through $D^+$ alone, train from scratch. At 230M parameters, the stop-gradient operator matches Softpick on quantization, has fewer dead heads, and retrieves passkeys more reliably, trailing only on peak attention-weight kurtosis.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.05488 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:27:05.798Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).