Metraipolis › archive › dc_fbbac929a84f

oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions — arXiv:2609.02672

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.02672
License (as archived)
CC0-1.0
Archived
2026-10-17T04:16:54.441Z
SHA-256 of the archived text
a8a41c3f3db44d4a0f9ecd92a8f8059461e7fbda1a5cdf0fa43d53500c147d20
Stable URL of this page
https://metraipolis.com/documento/dc_fbbac929a84f

How to cite

oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions — arXiv:2609.02672 — arXiv (en), https://arxiv.org/abs/2609.02672 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbbac929a84f (Metraipolis archive dc_fbbac929a84f, sha256 a8a41c3f3db44d4a0f9ecd92a8f8059461e7fbda1a5cdf0fa43d53500c147d20)
BibTeX
@misc{dcfbbac929a84f,
  title = {oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions — arXiv:2609.02672},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.02672},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbbac929a84f; sha256 a8a41c3f3db44d4a0f9ecd92a8f8059461e7fbda1a5cdf0fa43d53500c147d20},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.02672. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n \times n$ residual matrix. Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training. Manifold-constrained Hyper-Connections (mHC) address this by restricting the matrix to the doubly stochastic matrices. That caps the factor at one, so the mixing can no longer amplify any direction, but nothing bounds it from below. We prove that inside this set the mixing step can reduce the norm of the residual streams only by shrinking the differences between the streams, while their mean is left unchanged; and since the reduction accumulates over layers, the streams grow more alike and their diversity is spent with depth. We therefore propose Orthogonal Hyper-Connections (oHC), restricting the residual matrix to the rotation group $SO(n)$, so that the mixing step can neither amplify nor attenuate the residual streams in any direction, which keeps training stable and no longer forces the differences between the streams to contract. Specifically, at the four streams used by recent HC models we parameterize the group in closed form by a pair of unit quaternions, which adds no parameters, replaces the iterative projection with a fixed pattern of signed additions, and can be constructed faster than mHC. We evaluate oHC across a comprehensive set of downstream tasks, where it outperforms the single-stream residual baseline, mHC and iHC, which fixes the residual matrix to the identity.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-03 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.02672 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T03:26:49.805Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-03?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).