Metraipolis › archive › dc_8d2fa75e2872

Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation — arXiv:2609.34738

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.34738
License (as archived)
CC0-1.0
Archived
2026-10-07T21:02:52.712Z
SHA-256 of the archived text
be99b25df8af66a6decb8d30b8413833cf96a395f39e975f91a45d7d0226aa41
Stable URL of this page
https://metraipolis.com/documento/dc_8d2fa75e2872

How to cite

Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation — arXiv:2609.34738 — arXiv (en), https://arxiv.org/abs/2609.34738 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8d2fa75e2872 (Metraipolis archive dc_8d2fa75e2872, sha256 be99b25df8af66a6decb8d30b8413833cf96a395f39e975f91a45d7d0226aa41)
BibTeX
@misc{dc8d2fa75e2872,
  title = {Beyond Token Alignment: Event Completion for Cross-Tokenizer On-Policy Distillation — arXiv:2609.34738},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.34738},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8d2fa75e2872; sha256 be99b25df8af66a6decb8d30b8413833cf96a395f39e975f91a45d7d0226aa41},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34738. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy distillation (OPD) transfers knowledge between language models through teacher supervision on student-generated trajectories. With different tokenizers, a single teacher token may require multiple student tokens to generate, creating intermediate states where the event is entered but not yet completed. Existing cross-tokenizer methods align tokens or text spans to construct comparable prediction targets. We study a complementary problem after partial generation: once the student produces a prefix of a teacher token, multiple next tokens may complete the same remaining bytes, but the teacher only specifies the required completion rather than how probability should be divided among these valid continuations. We introduce Event-Set Completion Distillation (ESCD), which complements cross-tokenizer probability alignment with completion-set supervision. ESCD aggregates prefix-related teacher events and supervises the total probability of byte-compatible one-step student completions, avoiding tokenizer-dependent probability splits among individual tokens. The method reuses student trajectories and predictions, requiring neither additional rollouts nor changes to the student vocabulary. Experiments demonstrate consistent gains in mathematics, code, and scientific reasoning across model families and tokenizers, extending to large-scale MoE distillation from a 1T teacher to a 35B student. Local analyses show that retaining completion sets better matches the reference supervision, while one-step completion covers over 99% of observed compatible teacher mass after partial event entry in the studied tokenizer pairs. These findings support event entry and event completion as complementary supervision targets for cross-tokenizer knowledge transfer. Code will be released on GitHub.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.34738 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:03.138Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).