Metraipolis › archive › dc_ff83aa6e67eb

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning — arXiv:2610.40945

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40945
License (as archived): CC0-1.0
Archived: 2026-11-03T08:02:34.509Z
SHA-256 of the archived text: 12eb56bd7efe19528fc537ec3d56ffa07ee4a521ca80d9cd0721a35b6c67c61a
Stable URL of this page: https://metraipolis.com/documento/dc_ff83aa6e67eb

How to cite

On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning — arXiv:2610.40945 — arXiv (en), https://arxiv.org/abs/2610.40945 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff83aa6e67eb (Metraipolis archive dc_ff83aa6e67eb, sha256 12eb56bd7efe19528fc537ec3d56ffa07ee4a521ca80d9cd0721a35b6c67c61a)

BibTeX

@misc{dcff83aa6e67eb,
  title = {On the Origin and Evolution of Training-Inference Mismatch in LLM Reinforcement Learning — arXiv:2610.40945},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40945},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff83aa6e67eb; sha256 12eb56bd7efe19528fc537ec3d56ffa07ee4a521ca80d9cd0721a35b6c67c61a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40945. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy reinforcement learning for large language models assumes that training responses are sampled from the current policy. In practice, an inference engine samples the responses and a separate trainer computes the gradients, and the \textbf{training-inference mismatch} between the two makes the training data off-policy. The mismatch has two sources, the numerics of the engine and the stale weights of asynchronous sampling, and a correction that stabilizes training under one source can fail under the other. \textit{We ask which property of a mismatch source decides the correction it needs.} Evaluating the trainer at the sampler's weights splits each sampled-token error exactly into a \textit{numerical discrepancy} of the engine and a \textit{policy lag} of the trainer, and we follow both through training. The numerical discrepancy stays approximately constant, so it biases the updates by a nearly fixed amount, and centering the score at each prefix removes its leading bias. The policy lag is produced by the trainer's own updates: the bias it leaves enters the next update, which enlarges the next lag and closes a feedback loop. Importance weighting cuts this loop only while the lag ratio stays concentrated, and on Qwen3-1.7B it fails at the step where the effective sample size of the ratio falls below one half. From this account we derive the \textbf{discrepancy-lag correction} (DLC), which centers the numerical discrepancy, clips the lag ratio against the sampler's weights and resynchronizes the sampler when this effective sample size falls below one half. Under a stale sampler, DLC keeps the KL divergence from on-policy training within twice the KL between two independent on-policy trainings on Qwen3-0.6B and Qwen3-1.7B, and its resynchronization prevents collapses that the per-token corrections leave and that a KL measured on the sampler's prefixes does not detect.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.40945 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:59:38.775Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).