Metraipolis › archive › dc_1c950b4e87f2

A Token Is Not a Unit of Difficulty: Segmentation Controls Where Adaptive Computation Goes Inside a Word — arXiv:2611.00983

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.00983
License (as archived): CC0-1.0
Archived: 2026-11-03T10:46:16.095Z
SHA-256 of the archived text: 3dbd2e42b336ff8ecc8a83492291451d90228def1efd7c91042abd4c4dde56c0
Stable URL of this page: https://metraipolis.com/documento/dc_1c950b4e87f2

How to cite

A Token Is Not a Unit of Difficulty: Segmentation Controls Where Adaptive Computation Goes Inside a Word — arXiv:2611.00983 — arXiv (en), https://arxiv.org/abs/2611.00983 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1c950b4e87f2 (Metraipolis archive dc_1c950b4e87f2, sha256 3dbd2e42b336ff8ecc8a83492291451d90228def1efd7c91042abd4c4dde56c0)

BibTeX

@misc{dc1c950b4e87f2,
  title = {A Token Is Not a Unit of Difficulty: Segmentation Controls Where Adaptive Computation Goes Inside a Word — arXiv:2611.00983},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00983},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1c950b4e87f2; sha256 3dbd2e42b336ff8ecc8a83492291451d90228def1efd7c91042abd4c4dde56c0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00983. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Early-exit language models decide for each token how much computation to spend, usually by stopping once the prediction is confident. Yet a token is not a fixed unit: tokenizers cut words into different numbers of pieces in different languages. We ask where inside a word such a policy spends its budget. In Qwen3, along two adaptive axes (the timesteps of a spiking conversion and the layers of the original Transformer), confidence stopping gives the continuation pieces of a word less computation than its first piece in English, German, and Turkish, but more in Bangla. Segmentation drives this. Re-segmenting the same parallel passages with BPE-dropout shifts the gap by up to eight exits on both axes, small models trained under two tokenizers shift Bangla's gap by six exits, and BLOOM and mGPT, which split Bangla less finely, give it the English pattern. Characters per token do not predict the gap across languages; within-word surprisal explains 75% of its variance. A learned stopping head beats confidence stopping by 7 to 8 points, but gains less than one point over the best mix of fixed exits at equal cost, and sometimes loses. All results come from offline analysis, not measured speedups.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.00983 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:39:50.446Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).