Metraipolis › archive › dc_4da9cb470fd3

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width — arXiv:2610.41711

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.41711
License (as archived)
CC0-1.0
Archived
2026-11-02T04:52:24.789Z
SHA-256 of the archived text
86576010f7502a9b33eb3770d673fa59317a13d85044f8573d99dc0a3b08f44a
Stable URL of this page
https://metraipolis.com/documento/dc_4da9cb470fd3

How to cite

Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width — arXiv:2610.41711 — arXiv (en), https://arxiv.org/abs/2610.41711 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4da9cb470fd3 (Metraipolis archive dc_4da9cb470fd3, sha256 86576010f7502a9b33eb3770d673fa59317a13d85044f8573d99dc0a3b08f44a)
BibTeX
@misc{dc4da9cb470fd3,
  title = {Hierarchical Residuals: A Minimalist Decoupling of Residual Width from Compute Width — arXiv:2610.41711},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41711},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4da9cb470fd3; sha256 86576010f7502a9b33eb3770d673fa59317a13d85044f8573d99dc0a3b08f44a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41711. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Residual connections underpin Transformers, but they implicitly tie the residual width carried through depth to the compute width used inside each layer. We introduce Hierarchical Residuals (HiRes), a minimalist architectural change that decouples these dimensions, widening the residual state while keeping per-layer compute width fixed. HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks. Learned read/write projections bridge the two spaces at block boundaries, leaving individual Transformer layers unchanged. Across different model sizes and depths, HiRes consistently outperforms vanilla residual addition, with fitted scaling laws indicating a 1.30-fold improvement in FLOPs efficiency. On OLMoE-1B-7B pretraining for 500B tokens, HiRes achieves 0.0208 lower training loss and 1.14-point higher downstream accuracy. HiRes is also competitive with other residual expansion methods such as VWN, mHC and AttnRes with less activation memory and I/O costs. Built from only dense projections and normalizations, it requires neither specialized kernels nor infrastructure-level optimizations. HiRes thus exposes residual width as a practical scaling variable using only standard operators.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.41711 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T04:45:12.905Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).