Metraipolis › archive › dc_ecc7df594d06
Dual-Vocabulary Language Model for Cross-Tokenizer Distillation — arXiv:2609.33816
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.33816
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:57:11.813Z
- SHA-256 of the archived text
c1a8af70eb32cef6a7ed89ba44dee08c3c533abc0db01aef9dc7d50eb2ae98c2- Stable URL of this page
https://metraipolis.com/documento/dc_ecc7df594d06
How to cite
Dual-Vocabulary Language Model for Cross-Tokenizer Distillation — arXiv:2609.33816 — arXiv (en), https://arxiv.org/abs/2609.33816 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ecc7df594d06 (Metraipolis archive dc_ecc7df594d06, sha256 c1a8af70eb32cef6a7ed89ba44dee08c3c533abc0db01aef9dc7d50eb2ae98c2)
BibTeX
@misc{dcecc7df594d06,
title = {Dual-Vocabulary Language Model for Cross-Tokenizer Distillation — arXiv:2609.33816},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.33816},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ecc7df594d06; sha256 c1a8af70eb32cef6a7ed89ba44dee08c3c533abc0db01aef9dc7d50eb2ae98c2},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33816. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grained token information or disrupting the native-token paradigm, while for the latter, strategies such as ranking, padding, or key-token selection retain only shared logit dimensions, resulting in much distribution loss. In this paper, we propose Dual-Vocabulary Language Model (DVLM), which replaces the teacher's LM head with a new student-vocabulary projection head and obtains full-dimensional student logits (for #2). To support student tokens (for #1), it takes a Parallel-Tokenized Sequence (PTS) as input, which concatenates the original teacher-tokenized sequence and a re-tokenized sequence formed by independently converting each student token into a teacher-token group. To avoid inference inconsistency with the original teacher tokens, the Hybrid-Prefix Attention (HPA) further restricts re-tokenized groups to their corresponding teacher prefix and uses its last state as the aggregation of the original student-token representation for projection into the student vocabulary space. Similarly, via the combined use of PTS and HPA, the DVLM teacher can provide distribution-aligned supervision with the student's input-tokenization and output-logit during OPD. Experimental results demonstrate that our DVLM teacher has a similar converged loss as the original teacher model and enables student models to improve performance across six reasoning tasks. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.33816 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.371Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).