Metraipolis › archive › dc_c031afc453eb
TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing — arXiv:2608.07851
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.07851
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:50:39.744Z
- SHA-256 of the archived text
4ef0d635224d26f8a934c95c0230885677a15555f0f8076d948cd1eaecc52315- Stable URL of this page
https://metraipolis.com/documento/dc_c031afc453eb
How to cite
TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing — arXiv:2608.07851 — arXiv (en), https://arxiv.org/abs/2608.07851 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c031afc453eb (Metraipolis archive dc_c031afc453eb, sha256 4ef0d635224d26f8a934c95c0230885677a15555f0f8076d948cd1eaecc52315)
BibTeX
@misc{dcc031afc453eb,
title = {TEMPER: Tensorized Efficient Manifold-constrained Parameterization for Expressive Residual Routing — arXiv:2608.07851},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.07851},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c031afc453eb; sha256 4ef0d635224d26f8a934c95c0230885677a15555f0f8076d948cd1eaecc52315},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07851. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Residual connections rely on a static residual pathway, and are essential for training deep neural networks. Hyper-connections (HC) increase the expressivity of residual routing by incorporating multiple residual streams and learning dynamic information flow, while manifold-constrained (mHC) variants stabilize training through doubly stochastic residual mixing. However, a generator-level bottleneck remains in existing methods: they use dense, unstructured generators for pre-branch aggregation, residual mixing, and post-branch redistribution, which results in parameter count growing rapidly with the number of streams. To address this issue, we propose \underline{\textbf{T}}ensorized \underline{\textbf{E}}fficient \underline{\textbf{M}}anifold-constrained \underline{\textbf{P}}arameterization for \underline{\textbf{E}}xpressive Residual \underline{\textbf{R}}outing (\textbf{TEMPER}), which represents these generators as multi-way tensors over the input-stream, feature, and output-stream modes, and parameterizes them using tensor networks. Such a structured low-rank formulation is shown to preserve token-dependent manifold-constrained routing interface while substantially reducing parameter growth. It also promotes interpretability and intuition, as: i) tensor ranks control the dimensionality of the learned routing subspace, with full ranks recovering dense routing; while ii) the generator approximation errors bound differences in routing logits and, consequently, in the routed-block outputs. Comprehensive experiments show that TEMPER matches or outperforms existing methods across language modeling and commonsense reasoning tasks, while requiring substantially fewer additional parameters. At eight residual streams, TEMPER achieves the best CORE score while using about $84\%$ fewer additional parameters than mHC, thus showing a stronger performance-parameter efficiency trade-off.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.07851 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:49:41.964Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).