Metraipolis › archive › dc_54ba8523e7ee

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras — arXiv:2609.01129

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.01129
License (as archived)
CC0-1.0
Archived
2026-10-17T06:52:19.277Z
SHA-256 of the archived text
d9599d23acfcbdef192f01a4c7d5074bc052ccc3f3bf1904945dcbe4404c429a
Stable URL of this page
https://metraipolis.com/documento/dc_54ba8523e7ee

How to cite

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras — arXiv:2609.01129 — arXiv (en), https://arxiv.org/abs/2609.01129 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_54ba8523e7ee (Metraipolis archive dc_54ba8523e7ee, sha256 d9599d23acfcbdef192f01a4c7d5074bc052ccc3f3bf1904945dcbe4404c429a)
BibTeX
@misc{dc54ba8523e7ee,
  title = {Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras — arXiv:2609.01129},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.01129},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_54ba8523e7ee; sha256 d9599d23acfcbdef192f01a4c7d5074bc052ccc3f3bf1904945dcbe4404c429a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.01129. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under composition, $T^2\approx\alpha T$. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment $\mathcal{P}\geq0.9$, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, $T=Q_OKQ_V^\top$ and $T^2=Q_O(KDK)Q_V^\top$, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation of $K$ while preserving singular values, norms, factor spans, and principal angles reduces median closure from 0.336 to $1.04\times10^{-4}$; trained orientation wins for 98.64% of heads and in every layer. Constructive searches show that high closure is feasible in every surveyed layer, but usually not attained. Retrospective trajectories in three independently trained lineages further separate broadly available capacity from the orientations attained by final strong heads. Under exact value sharing, headwise closure extends to a right-action algebra, $T_iT_j=\alpha_jT_i$. Seven-model experiments verify the approximate law and reveal distinct oblique projections with a shared value-defined kernel. These results characterize scaled idempotence as a sparse trained orientation within broadly available geometric capacity and show how value sharing extends a headwise relation into a local operator algebra.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.01129 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:41:55.842Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).