Metraipolis › archive › dc_7ee485f7bfad

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation — arXiv:2610.31305

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.31305
License (as archived): CC0-1.0
Archived: 2026-10-23T15:20:38.224Z
SHA-256 of the archived text: f13933854a2ea2b893cd934a800770a49d089d1c3bffbc6c96d5d426f586758d
Stable URL of this page: https://metraipolis.com/documento/dc_7ee485f7bfad

How to cite

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation — arXiv:2610.31305 — arXiv (en), https://arxiv.org/abs/2610.31305 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7ee485f7bfad (Metraipolis archive dc_7ee485f7bfad, sha256 f13933854a2ea2b893cd934a800770a49d089d1c3bffbc6c96d5d426f586758d)

BibTeX

@misc{dc7ee485f7bfad,
  title = {Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation — arXiv:2610.31305},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.31305},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7ee485f7bfad; sha256 f13933854a2ea2b893cd934a800770a49d089d1c3bffbc6c96d5d426f586758d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.31305. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what the latent space compresses and how the discrete generator uses it. We propose Cola2, a framework that jointly studies continuous semantic compression, the balance between reconstruction and generation, and continuous-discrete co-adaptation. Cola2 dynamically segments text into semantic spans, controls the compression rate through auxiliary-loss-free negative feedback, and combines reconstruction and generation supervision to learn compact, predictive latent representations. A block-causal masked diffusion decoder is trained with clean, noisy, and dropped latent conditions, while classifier-free guidance adjusts the balance between semantic reconstruction and autonomous generation. Token-prediction gradients update the encoder through the latents, whereas Flow Matching prior learning is decoupled from representation learning by stop-gradient. During inference, generated text is re-encoded under a fixed span layout to keep the latent history consistent with the actual text context. Experiments on five downstream tasks with about 210 billion training tokens validate the core designs and show competitive, unsaturated scaling. With 346.03 billion source tokens, an unconverged checkpoint reaches a 60.10 average across eight tasks and surpasses Llama-3-8B on HellaSwag and StoryCloze despite its reported pretraining budget being over 43 times larger. These results demonstrate the potential of continuous-latent hierarchical language generation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.31305 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-23T15:09:38.357Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).