Metraipolis › archive › dc_7f52a86e945f
Geometric and Behavioral Stratification in Transformer Residual Streams — arXiv:2608.12447
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.12447
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T04:24:38.060Z
- SHA-256 of the archived text
e6834eb6c1125a9418a6dfc0720db9b910516d38e93ee7ad63489a9354645da5- Stable URL of this page
https://metraipolis.com/documento/dc_7f52a86e945f
How to cite
Geometric and Behavioral Stratification in Transformer Residual Streams — arXiv:2608.12447 — arXiv (en), https://arxiv.org/abs/2608.12447 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7f52a86e945f (Metraipolis archive dc_7f52a86e945f, sha256 e6834eb6c1125a9418a6dfc0720db9b910516d38e93ee7ad63489a9354645da5)
BibTeX
@misc{dc7f52a86e945f,
title = {Geometric and Behavioral Stratification in Transformer Residual Streams — arXiv:2608.12447},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.12447},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7f52a86e945f; sha256 e6834eb6c1125a9418a6dfc0720db9b910516d38e93ee7ad63489a9354645da5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12447. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Trained transformer models develop privileged bases: coordinate axes whose statistics differ from the rest of the residual stream. But what kind of direction does such a basis select? We investigate the prediction direction, the unembedding direction of the token a model currently predicts, and find that it functions as a content-defined privileged anchor. Measured with respect to this anchor, residual-stream variation is geometrically and behaviorally stratified by proximity to the prediction. The stratification holds in all eighteen models tested (dense and mixture-of-experts, 7B-120B, base and instruction-tuned). A narrow, scale-invariant prediction interface concentrates readout-relevant structure, while the vast prediction-distal complement expands with model scale. Because the prediction direction sits nearly orthogonal to the principal variance axes, variance-based analyses recover this organization only partly, and the shortfall grows with prompt heterogeneity. Anchoring reveals a steep geometric gradient: prediction-proximal regions are highly structured and cluster related prompts, while the complement is flatter and anti-discriminates among prompt groups. The interface is a narrow slice but functionally decisive. Disrupting the variance directions closest to the prediction causes immediate divergence and frequent task-frame shifts; disrupting the next level down delays divergence and preserves framing. The complement is weakly readout-aligned per direction yet causally and temporally load-bearing, and behavior is driven by direction rather than magnitude. These results establish the prediction direction as a privileged anchor distinct from previously described coordinate axes, and give a geometric account of how high-dimensional computation coexists with linear readout. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-14 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.12447 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T04:17:42.404Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).