Metraipolis › archive › dc_80037bbc7dab
DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models — arXiv:2609.34253
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34253
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:54:11.682Z
- SHA-256 of the archived text
95ffb44cd0765233dfa19019b76be965c9ac340cfc2eadc962573ac2624b8563- Stable URL of this page
https://metraipolis.com/documento/dc_80037bbc7dab
How to cite
DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models — arXiv:2609.34253 — arXiv (en), https://arxiv.org/abs/2609.34253 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_80037bbc7dab (Metraipolis archive dc_80037bbc7dab, sha256 95ffb44cd0765233dfa19019b76be965c9ac340cfc2eadc962573ac2624b8563)
BibTeX
@misc{dc80037bbc7dab,
title = {DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models — arXiv:2609.34253},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34253},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_80037bbc7dab; sha256 95ffb44cd0765233dfa19019b76be965c9ac340cfc2eadc962573ac2624b8563},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34253. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Pretrained autoregressive Transformers represent a large sunk investment in compute. Existing conversion methods reuse that investment by changing either the architecture (attention to recurrence) or the objective (next-token prediction to denoising), never both. We convert Qwen3 teachers at 1.7B and 8B into attention-free, bidirectional, gated-delta-rule diffusion students in three stages, so that each capability can be traced to the stage that kept or lost it. Language modeling transfers only partially and in-distribution; in-context retrieval does not transfer. On a multi-query recall probe where the teachers score 0.34-0.58, both converted students score 0.000, and diffusion pretraining alone does not restore retrieval. A retrieval curriculum in the final stage, which gradually lengthens the gap between a key-value table and the queries that address it, restores it only stochastically: on a fixed schedule, one seed in three learns to retrieve. Advancing the gap only while a running accuracy estimate stays above a threshold works for all three of those seeds, holds on real text, and carries unchanged to 8B, where two of three seeds succeed. The third had not learned within its fixed 16k-step budget: retrieval switches on abruptly at a seed-dependent step (6.5k and 11k in the other two), so a fixed budget can cut a late run off. One boundary survives every intervention: every model that learns retrieval scores 0.000 on tokens that never appeared in a retrieval episode, and an arm that resamples the key and value tokens every batch shows this is a coverage limit, not memorization of particular bindings. Separately, we convert a 7B code model into a 3:1 recurrent-attention block-diffusion hybrid over 85k steps and report two negative training results. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.34253 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:27:08.807Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).