Metraipolis › archive › dc_176dfb7c960f

Scaling and Distilling Text Embeddings for Better Diffusibility — arXiv:2610.01016

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01016
License (as archived)
CC0-1.0
Archived
2026-10-06T13:54:41.468Z
SHA-256 of the archived text
17c47c38ce0e40b444881c84528e0db5ce6358510df64aef657fc27923f08dd6
Stable URL of this page
https://metraipolis.com/documento/dc_176dfb7c960f

How to cite

Scaling and Distilling Text Embeddings for Better Diffusibility — arXiv:2610.01016 — arXiv (en), https://arxiv.org/abs/2610.01016 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_176dfb7c960f (Metraipolis archive dc_176dfb7c960f, sha256 17c47c38ce0e40b444881c84528e0db5ce6358510df64aef657fc27923f08dd6)
BibTeX
@misc{dc176dfb7c960f,
  title = {Scaling and Distilling Text Embeddings for Better Diffusibility — arXiv:2610.01016},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01016},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_176dfb7c960f; sha256 17c47c38ce0e40b444881c84528e0db5ce6358510df64aef657fc27923f08dd6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01016. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion language models (DLMs) offer a promising alternative to autoregressive (AR) language generation. Recent advances in continuous DLMs, which apply latent diffusion to continuous text embeddings, raise a practical question: which embedding makes the best latent space, i.e., the most diffusible? To answer this, we search through different embeddings and find that scaling the embedding model to stronger ones within the same family (T5 to T5Gemma-1 to T5Gemma-2) greatly improves generative performance. But the raw T5Gemma-2 embeddings are still not optimal. They are so discriminative that even the embeddings of plausible alternative words are separated, which makes the generation vulnerable to imperfect sampling. Consequently, continuous diffusion often fails to reach any of them and ends up at an invalid embedding instead. To address this, we distill T5Gemma-2 into a student encoder that learns the teacher's decoded probabilities as soft labels. Learning from such soft labels makes the student pull the alternative embeddings closer while maintaining the encoding-decoding mechanism. The distilled embeddings form a more connected and diffusible latent space, improving over the vanilla T5Gemma-2 embeddings. As a result, our medium-sized DLM achieves Gen. PPL 17.8 (against real-text PPL 15.4) at real-text entropy on OpenWebText, outperforming GPT-2-M on Gen. PPL.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.01016 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T12:15:34.666Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).