Metraipolis › archive › dc_d4190d64fb92

Predictive Geometry of Hidden Trajectories in Transformers — arXiv:2609.37717

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.37717
License (as archived)
CC0-1.0
Archived
2026-10-07T16:57:41.598Z
SHA-256 of the archived text
22a23c006e48985b3cf19bb5b20b7cb021b24b2df0b2bdf0b89dbdefa18db8e3
Stable URL of this page
https://metraipolis.com/documento/dc_d4190d64fb92

How to cite

Predictive Geometry of Hidden Trajectories in Transformers — arXiv:2609.37717 — arXiv (en), https://arxiv.org/abs/2609.37717 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d4190d64fb92 (Metraipolis archive dc_d4190d64fb92, sha256 22a23c006e48985b3cf19bb5b20b7cb021b24b2df0b2bdf0b89dbdefa18db8e3)
BibTeX
@misc{dcd4190d64fb92,
  title = {Predictive Geometry of Hidden Trajectories in Transformers — arXiv:2609.37717},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.37717},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d4190d64fb92; sha256 22a23c006e48985b3cf19bb5b20b7cb021b24b2df0b2bdf0b89dbdefa18db8e3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37717. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Decoder-only transformers are trained only through a terminal next-token prediction loss, yet this loss constrains every intermediate hidden state through the fixed downstream computation. We formalize this constraint by studying layerwise loss-to-go functions: the terminal loss obtained by continuing a candidate hidden state through the remaining transformer blocks. Around successful validation trajectories, we show that the local second-order geometry of these functions is governed, up to low-loss residual terms, by a pullback Fisher operator on hidden-state space. Its spectrum identifies output-sensitive directions and approximately prediction-null directions, yielding a local observable subspace of the residual stream. For causal transformers, the same geometry induces a tokenwise curvature score: a Fisher-weighted sensitivity of the target logits to perturbations of each token's hidden state. This score vanishes outside the causal ancestor set of the target and is controlled by downstream Jacobian couplings, making it a loss-aware alternative to attention magnitude. We estimate these quantities using matrix-free Jacobian-vector and vector-Jacobian products and evaluate them across decoder-only language models on WikiText, OpenWebText, and FineWeb. Empirically, the induced geometry predicts perturbation sensitivity, supports nonuniform layerwise rank allocation, yields competitive structured token-pruning signals, and improves low-rank student recovery when added to stronger autoregressive distillation objectives such as reverse KL and skew KL. These results support a predictive-geometric view of transformer computation: near successful trajectories, the terminal loss induces a thin, anisotropic set of output-relevant hidden-state directions that can be measured and exploited for compression and distillation.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.37717 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:27:03.458Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).