Metraipolis › archive › dc_ee5552dcd19c

Counting and Min-Cost Encoding for Tokenization in Large Language Models — arXiv:2610.01127

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01127
License (as archived)
CC0-1.0
Archived
2026-10-06T13:54:57.112Z
SHA-256 of the archived text
180ef53cefa87ac0a678f14fc44c0a133b0069ba3f64b1dbabb0d35b7d8c2cb7
Stable URL of this page
https://metraipolis.com/documento/dc_ee5552dcd19c

How to cite

Counting and Min-Cost Encoding for Tokenization in Large Language Models — arXiv:2610.01127 — arXiv (en), https://arxiv.org/abs/2610.01127 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ee5552dcd19c (Metraipolis archive dc_ee5552dcd19c, sha256 180ef53cefa87ac0a678f14fc44c0a133b0069ba3f64b1dbabb0d35b7d8c2cb7)
BibTeX
@misc{dcee5552dcd19c,
  title = {Counting and Min-Cost Encoding for Tokenization in Large Language Models — arXiv:2610.01127},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01127},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ee5552dcd19c; sha256 180ef53cefa87ac0a678f14fc44c0a133b0069ba3f64b1dbabb0d35b7d8c2cb7},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01127. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mainstream large language models rely on a tokenizer to encode text into a token sequence. Different tokenizers may yield token sequences of substantially different lengths for the same text. With a fixed model architecture, shorter token sequences correspond to lower inference time. We propose a tokenizer training approach named Counting and Filtering (CNF) and a text encoding algorithm called Min-Cost Encoding (MCE). MCE defines a cost function over a text segment, and determines the best segmentation by globally minimizing the overall segmentation cost. CNF builds a raw vocabulary by directly counting valid substrings, and then constructs the final vocabulary through a filtering step based on actual token usage when segmenting the training corpus with MCE. The CNF-MCE conbination offers several advantages over BPE, including higher token efficiency, greater scalability, and lower dependency. Across six text categories and two vocabulary-size groups, CNF-MCE consistently achieves better compression than the evaluated BPE tokenizers. With a 250K vocabulary, CNF-MCE increases compression rate by 26% and 30% on English web text over the o200k_base and qwen250k tokenizers. Experiments scaling the vocabulary to 1M entries on English web text demonstrate sustained improvements over BPE, with a token efficiency improvement of over 60% and vocabulary utilization rising from 52.9% to 96.9%. The MCE algorithm does not depend on a merge list (as in BPE) or token probability (as in UnigramLM), making it applicable to a wide range of vocabularies, including those built from BPE, UnigramLM, CNF, and others. Language models trained from scratch at the 1.8B and 8B scales achieve comparable average performance to models using the BPE tokenizers across 11 benchmarks. These results demonstrate that CNF-MCE can improve token efficiency significantly while maintaining competitive downstream performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.01127 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.680Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).