Metraipolis › archive › dc_7605992fabec

Multi-Token Prediction Done Right — arXiv:2610.38331

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.38331
License (as archived)
CC0-1.0
Archived
2026-11-01T02:24:43.756Z
SHA-256 of the archived text
01091d21bd7825c2326cda30e0db673a712a4845c28883d7bcbcd456aa204b4a
Stable URL of this page
https://metraipolis.com/documento/dc_7605992fabec

How to cite

Multi-Token Prediction Done Right — arXiv:2610.38331 — arXiv (en), https://arxiv.org/abs/2610.38331 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7605992fabec (Metraipolis archive dc_7605992fabec, sha256 01091d21bd7825c2326cda30e0db673a712a4845c28883d7bcbcd456aa204b4a)
BibTeX
@misc{dc7605992fabec,
  title = {Multi-Token Prediction Done Right — arXiv:2610.38331},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.38331},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7605992fabec; sha256 01091d21bd7825c2326cda30e0db673a712a4845c28883d7bcbcd456aa204b4a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38331. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-Token Prediction (MTP) is now a common ingredient in language-model pre-training, as it improves downstream performance and enables faster inference through speculative decoding. We identify a flaw in standard MTP training, in which a residual component of the MTP gradient interferes with both next-token and multi-token learning, increasingly so with scale. Specifically, the MTP cross-entropy gradient can be decomposed exactly into a KL self-distillation term that pulls each MTP head toward the model's own next-token prediction (NTP) distribution and improves speculative decoding, and a residual that has a low signal-to-noise ratio and buries the useful KL signal. We introduce the KL gradient-recovery trick: when two distributions share a linear output head, the gradient of their KL divergence can be recovered exactly by combining their cross-entropy gradients through straight-through gradient routing. Unlike minimizing the KL directly, this trick requires no vocabulary-sized logit matrices and is compatible with standard fused cross-entropy kernels. This enables Soft Multi-Token Prediction (Soft-MTP), a one-line code change to MTP that removes the noisy residual and backpropagates only the KL self-distillation signal by reusing the existing NTP and MTP cross-entropy computations. Soft-MTP supports both DeepSeek- and Meta-style MTP, adds no parameters, and incurs no measurable overhead. From 167M to 1.2B parameters, Soft-MTP improves over standard MTP with lower NTP perplexity, faster speculative decoding and higher zero-shot accuracy at every scale, lower MTP perplexity from 689M onward, and stronger math and code reasoning after fine-tuning at 1.2B. Soft-MTP also matches, if not exceeds, direct KL divergence minimization on most metrics. Soft-MTP is, in effect, a free lunch for MTP optimization. Our code can be found at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.38331 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.119Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).