Metraipolis › archive › dc_187aa542ffcc

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling — arXiv:2610.08738

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08738
License (as archived)
CC0-1.0
Archived
2026-10-07T11:23:31.007Z
SHA-256 of the archived text
7eba88e92210cd8de0c04abd7d2e2b7042d91fe7e473bba49c4f18ad68c242b9
Stable URL of this page
https://metraipolis.com/documento/dc_187aa542ffcc

How to cite

Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling — arXiv:2610.08738 — arXiv (en), https://arxiv.org/abs/2610.08738 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_187aa542ffcc (Metraipolis archive dc_187aa542ffcc, sha256 7eba88e92210cd8de0c04abd7d2e2b7042d91fe7e473bba49c4f18ad68c242b9)
BibTeX
@misc{dc187aa542ffcc,
  title = {Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling — arXiv:2610.08738},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08738},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_187aa542ffcc; sha256 7eba88e92210cd8de0c04abd7d2e2b7042d91fe7e473bba49c4f18ad68c242b9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08738. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion Language Models (DLMs) hold the promise of order-agnostic, parallel text generation. Recently, continuous diffusion and flow matching models have seen substantial gains, driven by carefully crafted token representations and diffusion/flow spaces. In this work, we introduce Hierarchical Continuous Diffusion Language Models (H-CDLMs), a simple framework that further improves continuous DLMs with minimal compute and parameter overhead. Drawing on the discrete DLM and continuous image diffusion literature on joint diffusion, we diffuse multiple modalities in parallel. These modalities represent tokens at different semantic granularities: in our instantiation, the tokens themselves and coarser clusters obtained by clustering pretrained token embeddings. We propose a general setup that allows per-modality samplers and schedules to enhance the interplay between modalities. Applied to CoBit, this yields H-CoBit, which delivers large empirical gains across benchmarks. At dataset entropy, H-CoBit improves MAUVE and reaches a generative perplexity (GenPPL) of 49.4 on LM1B and 50.4 on OWT, improving on the baseline by 24.2 and 20.7 points and surpassing even discrete DLMs of comparable size. On GSM8K, it reaches 27.4% accuracy, outperforming prior continuous diffusion and flow-based models. We further apply H-CDLM to the flow matching model FLM, obtaining consistent gains with H-FLM and demonstrating that the framework generalizes across continuous generative paradigms. Our code will be made publicly available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.08738 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:39.466Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).