Metraipolis › archive › dc_0e8527ca996f

Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps — arXiv:2610.02740

Source
arXiv
Original
https://arxiv.org/abs/2610.02740
License (as archived)
CC0-1.0
Archived
2026-10-05T22:50:14.233Z
SHA-256 of the archived text
f9a4737f93ec62a988ce6ff83ef1e1cc65af933a683a924f3d95398a90287171
Stable URL of this page
https://metraipolis.com/documento/dc_0e8527ca996f

How to cite

Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps — arXiv:2610.02740 — arXiv, https://arxiv.org/abs/2610.02740 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0e8527ca996f (Metraipolis archive dc_0e8527ca996f, sha256 f9a4737f93ec62a988ce6ff83ef1e1cc65af933a683a924f3d95398a90287171)
BibTeX
@misc{dc0e8527ca996f,
  title = {Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gaps — arXiv:2610.02740},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02740},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0e8527ca996f; sha256 f9a4737f93ec62a988ce6ff83ef1e1cc65af933a683a924f3d95398a90287171},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02740. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning for long-horizon agents relies on purely retrospective training signals: credit is assigned only after observing environmental consequences, leaving the agent's belief at action time invisible to the gradient. We introduce Prospective Hindsight (PH), a self-calibrating training principle that augments any retrospective base method with a signal derived from the gap between the agent's prospective prediction (before feedback) and the retrospective evaluation (after feedback). This per-rollout surprise identifies samples where the agent's self-model is most inaccurate and amplifies their gradient contribution through a stop-gradient surprise-weighted advantage. Since the prospective predictor shares parameters with the policy, the two co-evolve, progressively shifting focus to the agent's remaining blind spots. We connect this principle to a privileged-information gap and show that minimizing the surprise residual provides a descent pathway on the agent's miscalibration rate; calibration thus emerges as a byproduct of optimization rather than from an added objective. On single-turn verifiable tasks and a multi-turn personal-agent task (under GRPO, on-policy distillation, and their combination), PH improves both task performance and calibration, with consistent gains across model scales. Notably, the dominant miscalibration mode shifts structurally between regimes, overconfident failures in single-turn, underconfident successes in multi-turn, yet the same training principle addresses both successfully.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.02740 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.793Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).