Metraipolis › archive › dc_fd768d4734ae
On Trajectory-Aware Training for Masked Diffusion Language Models — arXiv:2609.37974
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.37974
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T16:27:57.744Z
- SHA-256 of the archived text
6cfce245bd8bbc33de3f6e2055efd9020adca7ea7b6347cb5a5018d7c629ebf9- Stable URL of this page
https://metraipolis.com/documento/dc_fd768d4734ae
How to cite
On Trajectory-Aware Training for Masked Diffusion Language Models — arXiv:2609.37974 — arXiv (en), https://arxiv.org/abs/2609.37974 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd768d4734ae (Metraipolis archive dc_fd768d4734ae, sha256 6cfce245bd8bbc33de3f6e2055efd9020adca7ea7b6347cb5a5018d7c629ebf9)
BibTeX
@misc{dcfd768d4734ae,
title = {On Trajectory-Aware Training for Masked Diffusion Language Models — arXiv:2609.37974},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.37974},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd768d4734ae; sha256 6cfce245bd8bbc33de3f6e2055efd9020adca7ea7b6347cb5a5018d7c629ebf9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.37974. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Masked diffusion models (MDMs) generate text by unmasking several tokens per step, but they are trained and sampled under different conditions. The model is trained on randomly masked sequences, whereas inference follows a trajectory shaped by the model's own predictions. Additionally, each step has no access to what the previous one computed. Recent methods narrow these limitations from separate angles, leaving open how these choices interact. We introduce PUMBA, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time. A controlled study of this design space shows that i) exact train--inference alignment fails due to local overfitting, whereas a looser alignment still brings training masks closer to those seen at inference; ii) passing continuous information outperforms discrete gradient estimators through the commitment at each step; and iii) performance improves as backpropagation through time spans more steps, which we support theoretically. Combined, these components match the best checkpoint of a same-size autoregressive model. Building on these findings, we scale PUMBA to supervised fine-tuning of LLaDA-8B, where it improves the trade-off between performance and number of function evaluations (NFEs) in both full-canvas and block diffusion generation. At matched performance, it needs up to 22% fewer NFEs than standard fine-tuning with twice the budget in full-canvas generation, and up to 26% fewer than standard fine-tuning for the same number of steps in block diffusion. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.AI). Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.37974 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T15:54:54.055Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).