Metraipolis › archive › dc_eb3b010f0eac

Measuring Optimal Transport in Transformer Depth — arXiv:2609.00748

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.00748
License (as archived)
CC0-1.0
Archived
2026-10-17T06:41:20.792Z
SHA-256 of the archived text
81ee7941ac16b9eed4afbd600db69a5af526738b667d19fa4959a4bc9c2a228b
Stable URL of this page
https://metraipolis.com/documento/dc_eb3b010f0eac

How to cite

Measuring Optimal Transport in Transformer Depth — arXiv:2609.00748 — arXiv (en), https://arxiv.org/abs/2609.00748 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb3b010f0eac (Metraipolis archive dc_eb3b010f0eac, sha256 81ee7941ac16b9eed4afbd600db69a5af526738b667d19fa4959a4bc9c2a228b)
BibTeX
@misc{dceb3b010f0eac,
  title = {Measuring Optimal Transport in Transformer Depth — arXiv:2609.00748},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.00748},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb3b010f0eac; sha256 81ee7941ac16b9eed4afbd600db69a5af526738b667d19fa4959a4bc9c2a228b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00748. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A transformer carries each token's state from layer to layer, and the whole vocabulary carried together forms a cloud that moves with depth. We ask whether a trained network moves this cloud the way optimal transport would: at the cheapest cost, and along the map that pairs each token with its optimal destination. We measure both on Pythia-160m and Pythia-410m, with an exact assignment between consecutive layer clouds, a measured sampling floor, calibration on couplings known to be optimal, and a split of the cost into the common shift of the cloud and the token-specific moves. At the last layer, both models move their tokens where the optimal-transport map sends them, at the optimal cost for Pythia-410m and slightly above it for Pythia-160m. At the first layer they do not. In between, single layers can be judged on cost at only two of ten transitions, and blocks of several layers move the cloud at close to the optimal cost. The agreement at the last layer is much weaker at initialisation (0.64 against 0.86) and grows with training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.00748 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T06:33:38.824Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).