Metraipolis › archive › dc_1baec79d2615

One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion — arXiv:2609.33698

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.33698
License (as archived): CC0-1.0
Archived: 2026-10-07T20:57:05.289Z
SHA-256 of the archived text: 72eba7bade3c4de8340baf22f18eac27d34d58c4642ccee76814c58575caa717
Stable URL of this page: https://metraipolis.com/documento/dc_1baec79d2615

How to cite

One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion — arXiv:2609.33698 — arXiv (en), https://arxiv.org/abs/2609.33698 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1baec79d2615 (Metraipolis archive dc_1baec79d2615, sha256 72eba7bade3c4de8340baf22f18eac27d34d58c4642ccee76814c58575caa717)

BibTeX

@misc{dc1baec79d2615,
  title = {One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusion — arXiv:2609.33698},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33698},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1baec79d2615; sha256 72eba7bade3c4de8340baf22f18eac27d34d58c4642ccee76814c58575caa717},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33698. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Most continuous diffusion language models process one latent position per token at each sampling step, making generation expensive. Two-stage methods lower the cost by reducing the latent length, but they fix the compressed embedding space before training the diffusion model. Embeddings from the fixed space can be difficult to model with diffusion and decode reliably into tokens, which limits generation quality after compression. To address this problem, we introduce JPEG-DLM (Joint-embedding Prediction for Efficient Generation with Diffusion Language Model), which jointly trains a compressor, a flow matching model and a decoding module. With joint-embedding prediction, JPEG-DLM learns compressed embeddings that are more structured, easier to model with diffusion and reliably decodable into tokens. JPEG-DLM achieves the lowest mean Gen-PPL and highest throughput among recent diffusion and flow models on LM1B and OWT. At a compression rate of 0.5 on OWT, it reaches a Gen-PPL of 34.52 and approximately 2.3 times ELF's throughput. These results suggest that jointly learning compressed embeddings offers a promising path toward efficient diffusion language modeling. Code will be released soon.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.33698 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:01.346Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).