Metraipolis › archive › dc_6fe75ea3458a

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders — arXiv:2609.27248

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.27248
License (as archived): CC0-1.0
Archived: 2026-10-08T10:52:12.827Z
SHA-256 of the archived text: 1b81f70d589afb14867287c444251390eec09cfbf887d598157b8fc74befcac6
Stable URL of this page: https://metraipolis.com/documento/dc_6fe75ea3458a

How to cite

Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders — arXiv:2609.27248 — arXiv (en), https://arxiv.org/abs/2609.27248 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6fe75ea3458a (Metraipolis archive dc_6fe75ea3458a, sha256 1b81f70d589afb14867287c444251390eec09cfbf887d598157b8fc74befcac6)

BibTeX

@misc{dc6fe75ea3458a,
  title = {Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders — arXiv:2609.27248},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27248},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6fe75ea3458a; sha256 1b81f70d589afb14867287c444251390eec09cfbf887d598157b8fc74befcac6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27248. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Next-token prediction has enabled highly fluent autoregressive language models, but it represents global structure only indirectly through sequential factorization. In contrast, high-fidelity autoencoders have become a standard primitive in image generation, enabling generative models to operate over continuous latent spaces; text lacks a comparably faithful continuous representation. We propose LLMAE, a method for repurposing a pretrained decoder-only language model as a continuous text autoencoder by exposing the activations of an intermediate layer as a fixed-length latent bottleneck. LLMAE achieves this interface with structured attention masks and LoRA adaptation, leveraging the generative prior of the original LLM. Across two backbones (270M Gemma 3 and 0.5B Qwen2.5), LLMAE reconstructs sequences up to 1024 tokens with high accuracy, reproducing up to 97% of documents verbatim. We demonstrate downstream utility by training a lightweight diffusion model that generates detailed image captions directly in the frozen LLMAE latent space. By mapping text into a fixed-length continuous latent space, our approach provides an effective substrate for downstream adaptation while benefiting from the fluency of the original LLM. Code and models are available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.27248 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:06:16.717Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).