Metraipolis › archive › dc_abd47ab563d5

The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning — arXiv:2610.00332

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00332
License (as archived)
CC0-1.0
Archived
2026-10-06T12:08:42.862Z
SHA-256 of the archived text
401f973dc2cd685f461df78dea9abd321775ffde25a6a8555abb2ab6ce06e7d5
Stable URL of this page
https://metraipolis.com/documento/dc_abd47ab563d5

How to cite

The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning — arXiv:2610.00332 — arXiv (en), https://arxiv.org/abs/2610.00332 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_abd47ab563d5 (Metraipolis archive dc_abd47ab563d5, sha256 401f973dc2cd685f461df78dea9abd321775ffde25a6a8555abb2ab6ce06e7d5)
BibTeX
@misc{dcabd47ab563d5,
  title = {The Weakest Link: Distilling LLM Reasoning with Worst-Case Constrained Reinforcement Learning — arXiv:2610.00332},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00332},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_abd47ab563d5; sha256 401f973dc2cd685f461df78dea9abd321775ffde25a6a8555abb2ab6ce06e7d5},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00332. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Distilling the reasoning capabilities of large language models (LLMs) into smaller students is a central challenge for efficient deployment. Current approaches face a fundamental tension: optimizing purely for verifiable task rewards (e.g., via GRPO) leads to reward hacking, where students arrive at correct final answers through flawed intermediate logic, while regularizing with soft divergence penalties against a teacher (e.g., KL-based distillation) dilutes task performance and, critically, allows the student to compensate for severe logical violations at one step with high teacher agreement at others. We argue that this averaging is fundamentally misaligned with the nature of reasoning: a chain-of-thought is only as valid as its weakest link. Motivated by this observation, we formulate reasoning distillation as a constrained reinforcement learning problem in which the task reward is maximized subject to a worst-case constraint on the teacher log-likelihood along every prefix of the trajectory. To avoid the prohibitive cost of dual Lagrangian solvers and the test-time teacher dependence of state-augmented methods such as Saute, we derive an unaugmented constrained MDP whose reward transformation preserves the hard-constraint semantics, admits a low-variance policy gradient decomposition into single-step and long-term terms, and provably satisfies the worst-case constraint almost surely in the penalty limit. Through extensive experiments on mathematical reasoning and code generation tasks, we demonstrate that our method significantly expands the accuracy-fidelity Pareto front. By matching the high Final Answer Correctness of pure RL and drastically reducing teacher constraint violations, we ultimately achieve the highest rigorous Reasoning Success Rate across all evaluated settings.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.00332 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.046Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).