Metraipolis › archive › dc_add5b464f12f

Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization — arXiv:2610.08854

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08854
License (as archived)
CC0-1.0
Archived
2026-10-08T11:29:06.786Z
SHA-256 of the archived text
023c97f262a7f3e791f6c7fcdeec1197a53dad2604eeb7b7468e44ae02f72def
Stable URL of this page
https://metraipolis.com/documento/dc_add5b464f12f

How to cite

Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization — arXiv:2610.08854 — arXiv (en), https://arxiv.org/abs/2610.08854 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_add5b464f12f (Metraipolis archive dc_add5b464f12f, sha256 023c97f262a7f3e791f6c7fcdeec1197a53dad2604eeb7b7468e44ae02f72def)
BibTeX
@misc{dcadd5b464f12f,
  title = {Certified Splitting: Decidability and Anchor Supply for Maximal-Munch Tokenization — arXiv:2610.08854},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08854},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_add5b464f12f; sha256 023c97f262a7f3e791f6c7fcdeec1197a53dad2604eeb7b7468e44ae02f72def},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08854. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A certificate is a window of bytes with an origin inside it: wherever the window occurs in a completely tokenizable input, the token covering the window's final byte begins at that origin. Decided from the token set alone, a certified position marks where work may soundly begin. This paper decides certification and reads its anchors forward, into cuts, edits and audits. Certification is decided two ways: online with a completeness cutoff for literal vocabularies, and offline through an armed-run verifier for arbitrary regular token sets. Either route, under a declared window budget, yields the anchor inventory, with a witness on each refusal. The consequences are theorems. Cutting an input at certified anchors and scanning the chunks independently reproduces the sequential segmentation, with no speculation and no fixup pass. A byte substitution that keeps the input tokenizable moves boundaries only strictly between the certified anchors witnessed in the unchanged bytes on either side of it. For tokens of length at most L, an edit at p moves no boundary at or below p - L + 1. A delimiter permits a sound cut before it exactly when it sits only token-initial, and one after it exactly when only token-final, so the embedded-delimiter failure is refused. Two instruments follow: the anchor-free span of a finite certified inventory, decided bounded or not with its exact supremum, three for the UTF-8 shape; and a differential auditor deciding whether two token sets place different boundaries on an input both tokenize, with witnesses. Every theorem carries an executable check. Every measurement is a named program's output or arithmetic on such outputs, each program shipped with the paper or pinned by commit in the library it studies; that library's threaded scanner splits the archived campaign corpus at certified anchors, its boundary stream byte-identical throughout.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno).
Categorie arXiv: Formal Languages and Automata Theory (cs.FL) ; Distributed, Parallel, and Cluster Computing (cs.DC); Programming Languages (cs.PL).
Pagina del lavoro: https://arxiv.org/abs/2610.08854 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:29:06.616Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).