Metraipolis › archive › dc_9fc3936639bf

Continued Training of Continuous Language Flows with Token Identity Squared Supervision — arXiv:2611.07721

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.07721
License (as archived): CC0-1.0
Archived: 2026-11-06T14:42:36.319Z
SHA-256 of the archived text: 10f16f9009141d2a7bcebfafa847b052a070c9e7c813bfe804c64bc959687896
Stable URL of this page: https://metraipolis.com/documento/dc_9fc3936639bf

How to cite

Continued Training of Continuous Language Flows with Token Identity Squared Supervision — arXiv:2611.07721 — arXiv (en), https://arxiv.org/abs/2611.07721 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9fc3936639bf (Metraipolis archive dc_9fc3936639bf, sha256 10f16f9009141d2a7bcebfafa847b052a070c9e7c813bfe804c64bc959687896)

BibTeX

@misc{dc9fc3936639bf,
  title = {Continued Training of Continuous Language Flows with Token Identity Squared Supervision — arXiv:2611.07721},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.07721},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9fc3936639bf; sha256 10f16f9009141d2a7bcebfafa847b052a070c9e7c813bfe804c64bc959687896},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.07721. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Diffusion and flow models have recently been extended to language modeling, generating text through continuous denoising in token embedding spaces. Pretrained models provide token representations and contextual prediction capabilities, but improving generation with limited additional training requires an appropriate connection between supervision and continuous generation. Continuous MSE evaluates a predicted vector obtained by weighting token embeddings with predicted probabilities; different probability distributions can yield the same vector and are therefore indistinguishable under this objective. We propose Token Identity Brier Flow (TIBF), which applies squared supervision directly to token identities when continuing pretrained language flows. TIBF combines network scores with token-matching scores under a Gaussian observation model, supervises token probabilities using the Brier loss, and uses probability-weighted embeddings for continuous generation. With fixed token embeddings, we show that the squared error between predicted token probabilities and the true posterior bounds the squared errors of continuous denoising predictions and flow velocities. A risk decomposition separates uncertainty inherent in the observation from posterior fitting error, motivating continued training that preserves pretrained embeddings and focuses on probability prediction. We also establish convergence to token embeddings under ideal predictions and sufficient conditions for local convergence with prediction error. From common pretrained initializations and under matched continuation budgets, frozen-embedding TIBF achieves this http URL values of 33.96 on OWT and 81.17 on LM1B, outperforming continuous-MSE and CE continuation controls. It attains the lowest this http URL at every tested sampling budget on both datasets, together with the lowest repetition rate on LM1B.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.07721 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-06T14:18:42.686Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).