Metraipolis › archive › dc_db7b77912eed

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So — arXiv:2608.10251

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.10251
License (as archived)
CC0-1.0
Archived
2026-10-19T06:14:10.146Z
SHA-256 of the archived text
ca515ccecf78b5bf6b072bd9d1cb7868ea89b8d27c840d42ea55d623a5b60741
Stable URL of this page
https://metraipolis.com/documento/dc_db7b77912eed

How to cite

Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So — arXiv:2608.10251 — arXiv (en), https://arxiv.org/abs/2608.10251 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_db7b77912eed (Metraipolis archive dc_db7b77912eed, sha256 ca515ccecf78b5bf6b072bd9d1cb7868ea89b8d27c840d42ea55d623a5b60741)
BibTeX
@misc{dcdb7b77912eed,
  title = {Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So — arXiv:2608.10251},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.10251},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_db7b77912eed; sha256 ca515ccecf78b5bf6b072bd9d1cb7868ea89b8d27c840d42ea55d623a5b60741},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10251. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A transformer's answer lives on one axis: the direction its unembedding reads. Its intermediate states largely do not, and that off-axis position is usually treated as an obstacle to interpretation. We show it is functional. A 12-layer model computes in two phases. Through the first, every sublayer writes into a subspace held near-orthogonal to the read-out, attention 75 to 96 degrees off it at every depth. Moving attention's values onto the read-out is 64 to 84 times more damaging than a matched random rotation, and the damage is entirely in cross-token mixing: the subspace insulates composition from the vocabulary. Beneath it the frame itself turns rigidly with depth. In the second phase the answer arrives on-axis, late, and by addition rather than by turning accumulated content onto the read-out. Pressing every layer onto the read-out instead, as training for early exit does, matches the baseline on perplexity, LAMBADA and BLiMP while cutting the concept-phase workspace from about twenty-five effective dimensions to fourteen, a change none of those benchmarks register. The geometry can also be imposed, though not by asking for it. Prescribing it through the loss is a lottery: six of eight seeds collapse, because a model told to null its read-out projection obeys most cheaply by discarding dimensions. Inserting one fixed rotation at the phase boundary lands it instead, at baseline quality. A sparse rotation the surrounding weights can absorb converges on all nine seeds, against five of nine for ordinary training. Which rotation is immaterial: twenty-five runs across thirteen distinct ones reach the same quality, and two baselines from different seeds hold their concepts in near-orthogonal frames while agreeing on their read-outs. That freedom is usable: a basis drawn at random and prescribed before training is adopted across the concept phase, with quality unchanged.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.10251 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T06:08:50.309Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).