Metraipolis › archive › dc_045334fd7ed1

FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training — arXiv:2610.02828

Source
arXiv
Original
https://arxiv.org/abs/2610.02828
License (as archived)
CC0-1.0
Archived
2026-10-05T22:49:17.340Z
SHA-256 of the archived text
da554b4c1918021e6abaf535f051a48303a1bc63aada3e94f3df2a33a912f70c
Stable URL of this page
https://metraipolis.com/documento/dc_045334fd7ed1

How to cite

FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training — arXiv:2610.02828 — arXiv, https://arxiv.org/abs/2610.02828 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_045334fd7ed1 (Metraipolis archive dc_045334fd7ed1, sha256 da554b4c1918021e6abaf535f051a48303a1bc63aada3e94f3df2a33a912f70c)
BibTeX
@misc{dc045334fd7ed1,
  title = {FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training — arXiv:2610.02828},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02828},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_045334fd7ed1; sha256 da554b4c1918021e6abaf535f051a48303a1bc63aada3e94f3df2a33a912f70c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02828. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL regularization, verifier allocation, and update budget. Three coupled issues remain unresolved. A future-risk model trained from behavior trajectories need not estimate the risk induced by the controller that will be deployed; a score calibrated on logged state-action pairs can become miscalibrated after selective action choice; and independent per-resource minimum costs do not in general certify a feasible multi-resource continuation. We introduce FSPO, a feedback-state controller for budgeted LLM RL post-training that addresses these issues jointly. FSPO learns a policy-consistent risk-to-go model whose Bellman target follows the same frozen controller used for future decisions, together with a long-horizon utility model. Decision-conditioned trajectory calibration (DCTC) calibrates risk on cross-fitted trajectories generated by actions selected by provisional controllers. A Pareto resource continuation certificate (PRCC) admits an action only when a non-dominated cumulative reservation remains feasible over the residual horizon. Under a matched GRPO resource envelope, FSPO reaches 66.11% held-out and 59.43% OOD accuracy, compared with 64.47% and 57.03% for PB2, the strongest evaluated adaptive baseline. Three paired training seeds give gains of +2.42 and +3.19 percentage points over the contextual bandit on held-out and OOD evaluation. Under high behavior-deployment mismatch, policy-consistent risk lowers selected-decision ECE from 0.108 to 0.053; DCTC lowers it from 0.039 to 0.022 at matched acceptance; PRCC removes false-feasible admissions on an 18-action catalog ($0.197\rightarrow0.000$); and enabling all three components reduces trajectory failure from 0.181 to 0.083 in a factorial ablation.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.02828 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.639Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).