Metraipolis › archive › dc_c26b03041d72
A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers — arXiv:2608.14689
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.14689
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T19:39:49.889Z
- SHA-256 of the archived text
37e9ed55e4b5cc9c5b5199c9f6fec9247affe236b665931757e0770c9f1b41c1- Stable URL of this page
https://metraipolis.com/documento/dc_c26b03041d72
How to cite
A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers — arXiv:2608.14689 — arXiv (en), https://arxiv.org/abs/2608.14689 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c26b03041d72 (Metraipolis archive dc_c26b03041d72, sha256 37e9ed55e4b5cc9c5b5199c9f6fec9247affe236b665931757e0770c9f1b41c1)
BibTeX
@misc{dcc26b03041d72,
title = {A Reproducibility Study of Partial Residual Ablations in Pre-LN Transformers — arXiv:2608.14689},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.14689},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c26b03041d72; sha256 37e9ed55e4b5cc9c5b5199c9f6fec9247affe236b665931757e0770c9f1b41c1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.14689. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Residual connections are a fundamental component of transformer architectures, yet the roles of the attention and feed-forward residual pathways remain poorly understood when considered independently. This paper presents a reproducibility study of partial residual ablations in Pre-LN GPT-style transformers trained at two scales (10M and 124M parameters). I compare four architectural configurations by selectively removing the attention residual connection, the feed-forward residual connection, or both. At 10M scale, a controlled 8-seed deterministic sweep shows a clear asymmetry: removing the attention residual (FFNOnly) reaches the No-Residual collapse regime (3.350 +/- 0.002), whereas removing the feed-forward residual (AttnOnly) remains well separated from it (1.580 +/- 0.003). A subsequent controlled 124M sweep establishes that the direction of this asymmetry persists under matched seeds: across five seeds each, AttnOnly reaches 6.037 +/- 0.355 versus 7.569 +/- 0.001 for FFNOnly, with no overlap between the observed ranges. AttnOnly nevertheless remains substantially degraded relative to FullResidual (4.577 +/- 0.012) and exhibits markedly greater seed sensitivity at 124M. During the investigation, I identified and corrected an experimental measurement confound in runtime gain scaling and retained an intermediate reproduction failure rather than excluding it. I propose cross-position routing through self-attention as a falsifiable hypothesis for the observed asymmetry; no clean causal test of this mechanism has yet been completed. To support reproducibility, I release the source code, experiment configurations, training logs, and experimental results, including intermediate non-reproducing runs. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-18 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2608.14689 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T19:02:44.044Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).