Metraipolis › archive › dc_ca5ca3510258

A Distribution-Matching Perspective on LLM Training Algorithms — arXiv:2610.33740

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.33740
License (as archived): CC0-1.0
Archived: 2026-10-27T08:00:25.903Z
SHA-256 of the archived text: 8ec1c551a207f67a431641d21a0a3b1127daba6f1d93bd4e0ce78a00aae1dd52
Stable URL of this page: https://metraipolis.com/documento/dc_ca5ca3510258

How to cite

A Distribution-Matching Perspective on LLM Training Algorithms — arXiv:2610.33740 — arXiv (en), https://arxiv.org/abs/2610.33740 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ca5ca3510258 (Metraipolis archive dc_ca5ca3510258, sha256 8ec1c551a207f67a431641d21a0a3b1127daba6f1d93bd4e0ce78a00aae1dd52)

BibTeX

@misc{dcca5ca3510258,
  title = {A Distribution-Matching Perspective on LLM Training Algorithms — arXiv:2610.33740},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.33740},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ca5ca3510258; sha256 8ec1c551a207f67a431641d21a0a3b1127daba6f1d93bd4e0ce78a00aae1dd52},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.33740. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In this work, we propose a unified framework for training algorithms across different stages of modern large language model development, based on the perspective of distribution matching. We show that a broad class of language model training algorithms can be interpreted as minimizing a certain divergence between the current model and a target distribution, and that characterizing the properties of this target distribution provides substantial insight into the behavior of the corresponding training algorithms. By analyzing specific forms of the target distribution, we establish that On-Policy Distillation (OPD) enjoys an advantage over classical Knowledge Distillation (KD) when the current model distribution is close to the target distribution, while exhibiting a disadvantage when the two distributions are far apart. Furthermore, when the target distribution itself depends on the model being optimized, we show that the accumulation of estimation variance can induce a deterministic degradation of the target distribution. These results provide a theoretical explanation, at least in part, for the long-term behavior of existing Reinforcement Learning (RL) algorithms. With this intuition we further designed Conditional KL (CKL) regularization for mitigating the degradation of the target distribution. CKL can effectively preserve mode diversity while relaxing the constraint imposed by classical KL regularization on reward improvements. We believe this theoretical perspective can bring some new understanding of language model training algorithms and provide useful guidance for future research.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.33740 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:30.845Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).