Metraipolis › archive › dc_0df0f96a7aec
Tokenizer Myths and Where To Find Them — arXiv:2611.04215
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04215
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T21:42:24.722Z
- SHA-256 of the archived text
26ad8c4267d991287970542cabd5327d3efcda9118d490e8ea2054de152880ba- Stable URL of this page
https://metraipolis.com/documento/dc_0df0f96a7aec
How to cite
Tokenizer Myths and Where To Find Them — arXiv:2611.04215 — arXiv (en), https://arxiv.org/abs/2611.04215 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0df0f96a7aec (Metraipolis archive dc_0df0f96a7aec, sha256 26ad8c4267d991287970542cabd5327d3efcda9118d490e8ea2054de152880ba)
BibTeX
@misc{dc0df0f96a7aec,
title = {Tokenizer Myths and Where To Find Them — arXiv:2611.04215},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04215},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0df0f96a7aec; sha256 26ad8c4267d991287970542cabd5327d3efcda9118d490e8ea2054de152880ba},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04215. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Tokenization is a fundamental component of modern language models: before a model sees data, its tokenizer has already determined how the data appears. We identify and challenge three myths in the tokenization research literature. Myth #1: Bits-Per-Byte (BPB) is agnostic to the tokenizer and/or the vocabulary size: BPB is tokens per byte times bits per token, and the two factors do not cancel; using untrained models, we show that BPB prefers some tokenizers over others: the model has learned nothing, but the tokenizer--model pair has. Myth #2: lower BPB means a better model: On untrained models, BPB strongly prefers some vocabulary sizes, even though accuracy remains at chance, and on real models trained on the same data, BPB correlates weakly with downstream performance. Myth #3: different "tokenizer-agnostic" metrics yield consistent results: The "unigram-normalized loss" is not tokenizer-agnostic; its reported correlation of 0.989 with BPB comes from a confounder and falls to 0.057 once fixed; from the same data the two metrics select different vocabularies at each model size. We present a tokenizer trilemma: no metric can score only the true distribution best, give the same predictions the same score, and tie all untrained models, so choosing a metric is choosing a research question. We conclude with a caution for reading the tokenization literature and a conjecture for future work. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.04215 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T21:36:04.414Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).