Metraipolis › archive › dc_28f0ebdd83c2
LED: Improving Masked Diffusion Language Models by Lookahead Editing — arXiv:2610.23122
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.23122
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:44:07.248Z
- SHA-256 of the archived text
fd898a1e937f87b4a2193d5c68a6c25c8173dcfecf22b7f86655662fe72b9ccf- Stable URL of this page
https://metraipolis.com/documento/dc_28f0ebdd83c2
How to cite
LED: Improving Masked Diffusion Language Models by Lookahead Editing — arXiv:2610.23122 — arXiv (en), https://arxiv.org/abs/2610.23122 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_28f0ebdd83c2 (Metraipolis archive dc_28f0ebdd83c2, sha256 fd898a1e937f87b4a2193d5c68a6c25c8173dcfecf22b7f86655662fe72b9ccf)
BibTeX
@misc{dc28f0ebdd83c2,
title = {LED: Improving Masked Diffusion Language Models by Lookahead Editing — arXiv:2610.23122},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.23122},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_28f0ebdd83c2; sha256 fd898a1e937f87b4a2193d5c68a6c25c8173dcfecf22b7f86655662fe72b9ccf},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.23122. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive language models, enabling non-autoregressive text generation through parallel decoding. Existing dLLMs can be broadly categorized into two families: masked diffusion models (MDMs) and uniform diffusion models (UDMs) In this work, we theoretically show that MDMs are easier to optimize due to their mask-based transition schedule but have lower computational expressivity, whereas UDMs enable iterative state revision and greater expressivity but are substantially harder to train. This trade-off raises a natural question: can we obtain the expressivity of iterative revision without giving up the simple, easy-to-optimize transition schedule of MDMs? Motivated by this analysis, we propose Lookahead Editing (\ours{}), which augments masked diffusion decoding with a rewritable proposal state. At each decoding step, the model performs several lookahead updates to iteratively refine future predictions in this proposal state before committing tokens to the masked sequence. This design introduces a rewritable proposal that uses additional compute to improve expressivity while keeping the simple mask-based transition schedule of MDMs. We fine-tune Qwen2.5-7B and Qwen3-8B to obtain LED-\texttt{7B} and LED-\texttt{8B}, respectively. Across eight benchmarks, LED SFT yields average gains of $+11.4$ and $+7.7$ points over standard MDM SFT. Moreover, using SFT alone, LED-\texttt{8B} outperforms LLaDA2.1-16B Q-mode, which is trained with continued pre-training, supervised fine-tuning, and reinforcement learning.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.23122 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T08:39:24.901Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).