Metraipolis › archive › dc_074e6acb35fe

Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT — arXiv:2610.03361

Source
arXiv
Original
https://arxiv.org/abs/2610.03361
License (as archived)
CC0-1.0
Archived
2026-10-05T22:50:38.142Z
SHA-256 of the archived text
4e8019f984fc89ab54e34021a0b8ab751d3402d7f88239d2b8975876f908b177
Stable URL of this page
https://metraipolis.com/documento/dc_074e6acb35fe

How to cite

Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT — arXiv:2610.03361 — arXiv, https://arxiv.org/abs/2610.03361 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_074e6acb35fe (Metraipolis archive dc_074e6acb35fe, sha256 4e8019f984fc89ab54e34021a0b8ab751d3402d7f88239d2b8975876f908b177)
BibTeX
@misc{dc074e6acb35fe,
  title = {Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT — arXiv:2610.03361},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03361},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_074e6acb35fe; sha256 4e8019f984fc89ab54e34021a0b8ab751d3402d7f88239d2b8975876f908b177},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03361. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Critic-free reinforcement fine-tuning (RFT) for agentic large language models is often done through GRPO-style methods, which compute a group baseline over repeated rollouts to reduce target variance. However, this setup is ill-suited to agents acting in stateful environments such as live services or security sandboxes, where repeated rollouts are impractical to obtain and aggressive updates entrench the noise of long, sparsely verified trajectories. We propose \textit{Follow the Winners} (FTW), a critic-free policy-learning algorithm that adapts the cross-entropy method to RFT, replacing group rollouts with an ordinal filter on replay-buffer samples that yields polynomial concentration in the order statistic of returns. We derive FTW through a control-as-inference lens, which also recovers GRPO and DPO as specific modelling choices, identifying GRPO as risk-neutral while DPO and FTW share a bounded risk-seeking offset that FTW controls. We identify this offset as an inherent trade-off of variance reduction through ordinal filters on samples, whereas a critic model induces a different trade-off between bias and variance. Scaled to agentic LLM post-training, FTW matches GRPO and PPO on Sokoban and Search-R1 baselines, showing a viable trade-off from a value model or group rollouts to CPU memory.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.03361 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.856Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).