Metraipolis › archive › dc_0876e1ac2baf

Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models — arXiv:2610.03665

Source
arXiv
Original
https://arxiv.org/abs/2610.03665
License (as archived)
CC0-1.0
Archived
2026-10-05T22:52:02.990Z
SHA-256 of the archived text
e930b105058089a72f69775454208a5084ee49d9dd21a214f7a6a50e378930bd
Stable URL of this page
https://metraipolis.com/documento/dc_0876e1ac2baf

How to cite

Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models — arXiv:2610.03665 — arXiv, https://arxiv.org/abs/2610.03665 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0876e1ac2baf (Metraipolis archive dc_0876e1ac2baf, sha256 e930b105058089a72f69775454208a5084ee49d9dd21a214f7a6a50e378930bd)
BibTeX
@misc{dc0876e1ac2baf,
  title = {Pivot-SD: Efficient Self-Distillation for Masked Diffusion Language Models — arXiv:2610.03665},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03665},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0876e1ac2baf; sha256 e930b105058089a72f69775454208a5084ee49d9dd21a214f7a6a50e378930bd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03665. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Masked diffusion language models (dLMs) offer a promising parallel alternative to autoregressive models for complex reasoning. However, they face a distinct credit-assignment challenge, since a few commitments during denoising sharply reduce the uncertainty over the remaining masked positions and shape much of the response. Most post-training recipes for dLMs do not use this signal to decide which tokens to train on: they typically train on the final text or assign rewards to whole denoising steps, rather than selecting the individual commitments that shape the response. We introduce Pivot-SD, an efficient offline self-distillation framework that supervises only these high-impact commitments (pivots). Pivot-SD selects pivots using an information-gain metric measuring uncertainty reduction over the remaining masked positions. Pivots from successful trajectories are trained with cross-entropy, and pivots from failed trajectories with targeted unlikelihood, leaving the rest of the failed trajectory untouched. Using only 200 questions and four rollouts each, Pivot-SD improves LLaDA-8B-Instruct over full-sequence SFT and budget-matched diffusion RL baselines across math and code benchmarks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.03665 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:46:10.634Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).