Metraipolis › archive › dc_7ca3a5de021a

Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory — arXiv:2610.32887

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.32887
License (as archived): CC0-1.0
Archived: 2026-10-26T20:13:25.416Z
SHA-256 of the archived text: a191a3f09638e6522cc3fee897c54833af1b3f002c827aef9bcd06c5af96e777
Stable URL of this page: https://metraipolis.com/documento/dc_7ca3a5de021a

How to cite

Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory — arXiv:2610.32887 — arXiv (en), https://arxiv.org/abs/2610.32887 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7ca3a5de021a (Metraipolis archive dc_7ca3a5de021a, sha256 a191a3f09638e6522cc3fee897c54833af1b3f002c827aef9bcd06c5af96e777)

BibTeX

@misc{dc7ca3a5de021a,
  title = {Reinforcement Learning with Multi-Step Transition Predictions: A Bellman Theory — arXiv:2610.32887},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.32887},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7ca3a5de021a; sha256 a191a3f09638e6522cc3fee897c54833af1b3f002c827aef9bcd06c5af96e777},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.32887. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Multi-step predictions of future transition realizations can improve reinforcement learning, but their value remains difficult to quantify. Directly including these predictions in the state can lead to exponential state-space growth, making direct dynamic programming computationally expensive. We study discounted finite MDPs under a block-feedback protocol: at the start of each K-period block, the decision maker observes $K$-step probabilistic transition predictions for a subset of actions and adapts decisions to observed states within the block. We show that the optimal value satisfies a Bayesian Bellman equation on the original state space. This representation reveals that predictive information creates value by recursively changing the local order of maximization and expectation in the Bellman expansion. We formalize this effect through a Bellman--Jensen analysis and quantify the effects of prediction horizon, error, action coverage, and within-block feedback. We propose block-feedback Bayesian offline learning with online adaptation (BF--BOLA), a model-based algorithm for learning near-optimal policies, and establish upper and lower sample-complexity bounds that match up to logarithmic factors in the stated regimes. The results show that predictions can reduce transition learning to uncovered rows and that learning a near-optimal policy can require fewer prediction samples than estimating the Bayesian value function.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.32887 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-26T19:49:03.438Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).