Metraipolis › archive › dc_2459bb4943ac

Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers — arXiv:2610.08561

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08561
License (as archived)
CC0-1.0
Archived
2026-10-07T11:41:51.506Z
SHA-256 of the archived text
29e0f00a1f76f8c7727f45d4bf0ca233f947c822fd6d06275dba2f9641949edd
Stable URL of this page
https://metraipolis.com/documento/dc_2459bb4943ac

How to cite

Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers — arXiv:2610.08561 — arXiv (en), https://arxiv.org/abs/2610.08561 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2459bb4943ac (Metraipolis archive dc_2459bb4943ac, sha256 29e0f00a1f76f8c7727f45d4bf0ca233f947c822fd6d06275dba2f9641949edd)
BibTeX
@misc{dc2459bb4943ac,
  title = {Reinforcement Learning for Hierarchical Reasoning Rewards: Minimax-Optimal Rates with Transformers — arXiv:2610.08561},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08561},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2459bb4943ac; sha256 29e0f00a1f76f8c7727f45d4bf0ca233f947c822fd6d06275dba2f9641949edd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08561. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) has become a standard tool for post-training language models on reasoning tasks, where the policy is updated by reward feedback while exploring the space of responses. Despite its empirical success, theoretical understanding of RL post-training remains limited, in particular of why on-policy exploration combined with a neural reward model is effective. In this paper, we address this question by modeling the reward as a hierarchical function on the response space: the reward consists of infinitely many local components, each of which becomes relevant only after the preceding ones have been resolved. We show that a natural Transformer-based actor--critic algorithm, which alternates between sampling from the current KL-regularized policy, fitting a Transformer critic to the observed rewards, and updating the policy, achieves the minimax optimal rates in the query budget and in the regularization strength up to logarithmic factors, and is minimax optimal for a fixed number of prompts. In contrast, we prove that sampling from the fixed reference distribution, as in offline reward modeling, can limit regret decay to a logarithmic rate. These results show that on-policy exploration progressively zooms in on the region where the reward is concentrated, and quantify its benefit for RL post-training.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.08561 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:19:24.837Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).