Metraipolis › archive › dc_032e82a2bc8c

No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance — arXiv:2610.21679

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.21679
License (as archived)
CC0-1.0
Archived
2026-10-16T10:01:42.635Z
SHA-256 of the archived text
11fbaacd61af28da667963fbf954499f0f9a3f8a065ca6b9830322ebbdb0480a
Stable URL of this page
https://metraipolis.com/documento/dc_032e82a2bc8c

How to cite

No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance — arXiv:2610.21679 — arXiv (en), https://arxiv.org/abs/2610.21679 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_032e82a2bc8c (Metraipolis archive dc_032e82a2bc8c, sha256 11fbaacd61af28da667963fbf954499f0f9a3f8a065ca6b9830322ebbdb0480a)
BibTeX
@misc{dc032e82a2bc8c,
  title = {No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance — arXiv:2610.21679},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.21679},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_032e82a2bc8c; sha256 11fbaacd61af28da667963fbf954499f0f9a3f8a065ca6b9830322ebbdb0480a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.21679. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Tokenization is a critical component of the language modeling pipeline, but it is poorly understood. Tokenizers can impact downstream performance, but there is currently mixed evidence about which tokenizer features predict a better language model without requiring a full training run. In this paper, we analyze approximately 200 language models trained on nine different languages. We evaluate both models and their tokenizers and test whether there is a relationship between any proposed intrinsic tokenizer evaluations and model performance, as measured by a wide variety of benchmarks. Despite the quantity and variety of proposed intrinsic tokenizer metrics, we find that none consistently predicts downstream performance across model families, tasks, and languages. This motivates the need for novel tokenizer evaluation metrics and approaches. We propose novel linguistically grounded metric introduced in this work shows that new metrics can capture aspects of tokenizer quality that existing metrics miss, though their predictive power is likely to vary across model families, languages, and tasks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.21679 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:44:47.665Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).