Metraipolis › archive › dc_eb3b010f0eac
Measuring Optimal Transport in Transformer Depth — arXiv:2609.00748
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.00748
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T06:41:20.792Z
- SHA-256 of the archived text
81ee7941ac16b9eed4afbd600db69a5af526738b667d19fa4959a4bc9c2a228b- Stable URL of this page
https://metraipolis.com/documento/dc_eb3b010f0eac
How to cite
Measuring Optimal Transport in Transformer Depth — arXiv:2609.00748 — arXiv (en), https://arxiv.org/abs/2609.00748 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb3b010f0eac (Metraipolis archive dc_eb3b010f0eac, sha256 81ee7941ac16b9eed4afbd600db69a5af526738b667d19fa4959a4bc9c2a228b)
BibTeX
@misc{dceb3b010f0eac,
title = {Measuring Optimal Transport in Transformer Depth — arXiv:2609.00748},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.00748},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb3b010f0eac; sha256 81ee7941ac16b9eed4afbd600db69a5af526738b667d19fa4959a4bc9c2a228b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.00748. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A transformer carries each token's state from layer to layer, and the whole vocabulary carried together forms a cloud that moves with depth. We ask whether a trained network moves this cloud the way optimal transport would: at the cheapest cost, and along the map that pairs each token with its optimal destination. We measure both on Pythia-160m and Pythia-410m, with an exact assignment between consecutive layer clouds, a measured sampling floor, calibration on couplings known to be optimal, and a split of the cost into the common shift of the cloud and the token-specific moves. At the last layer, both models move their tokens where the optimal-transport map sends them, at the optimal cost for Pythia-410m and slightly above it for Pythia-160m. At the first layer they do not. In between, single layers can be judged on cost at only two of ten transitions, and blocks of several layers move the cloud at close to the optimal cost. The agreement at the last layer is much weaker at initialisation (0.64 against 0.86) and grows with training. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-02 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.00748 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T06:33:38.824Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).