Metraipolis › archive › dc_0876e1ac2baf
Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models — arXiv:2610.03665
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.03665
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T22:52:02.990Z
- SHA-256 of the archived text
e930b105058089a72f69775454208a5084ee49d9dd21a214f7a6a50e378930bd- Stable URL of this page
https://metraipolis.com/documento/dc_0876e1ac2baf
How to cite
Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models — arXiv:2610.03665 — arXiv, https://arxiv.org/abs/2610.03665 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0876e1ac2baf (Metraipolis archive dc_0876e1ac2baf, sha256 e930b105058089a72f69775454208a5084ee49d9dd21a214f7a6a50e378930bd)
BibTeX
@misc{dc0876e1ac2baf,
title = {Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models — arXiv:2610.03665},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.03665},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0876e1ac2baf; sha256 e930b105058089a72f69775454208a5084ee49d9dd21a214f7a6a50e378930bd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03665. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sharply reduce the uncertainty over the remaining masked positions and shape much of the response. Most post-training recipes for dLMs do not use this signal to decide which tokens to train on: they typically train on the final text or assign rewards to whole denoising steps, rather than selecting the individual commitments that shape the response. We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments (pivots). Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction over the remaining masked positions. Pivots from successful trajectories are trained with cross-entropy, and pivots from failed trajectories with targeted unlikelihood, leaving the rest of the failed trajectory untouched. Using only 200 questions and four rollouts each, Pivot-SD improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.03665 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T22:46:10.634Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).