Metraipolis › archive › dc_63aa8a9e9dd3

Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning — arXiv:2608.05111

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.05111
License (as archived)
CC0-1.0
Archived
2026-10-31T11:35:34.050Z
SHA-256 of the archived text
a4f3989883f06a2f390c24c7998e1ecccdf31864770f97f3d3fc854d0e980265
Stable URL of this page
https://metraipolis.com/documento/dc_63aa8a9e9dd3

How to cite

Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning — arXiv:2608.05111 — arXiv (en), https://arxiv.org/abs/2608.05111 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_63aa8a9e9dd3 (Metraipolis archive dc_63aa8a9e9dd3, sha256 a4f3989883f06a2f390c24c7998e1ecccdf31864770f97f3d3fc854d0e980265)
BibTeX
@misc{dc63aa8a9e9dd3,
  title = {Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learning — arXiv:2608.05111},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.05111},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_63aa8a9e9dd3; sha256 a4f3989883f06a2f390c24c7998e1ecccdf31864770f97f3d3fc854d0e980265},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.05111. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In partially observable reinforcement learning, agents face a dual bottleneck: they must explore to encounter rewarding states and retain that experience in memory to optimize their policies. Exploration bonuses and memory architectures are traditionally evaluated in isolation, leaving their interaction unmeasured, and standard notions of sparse reward conflate temporal signal density with what the reward actually supervises. We present a controlled study crossing episodic exploration bonuses with diverse neural memory architectures across three environments that vary how the content of memory is acquired. An identical bonus signal yields three distinct interaction patterns: it amplifies architectural capacity differences where memory content must be actively discovered and retained unsupervised; equalizes architectures to a shared ceiling where the content, once sought out, is a single reward-supervised cue; and is null where the observation stream is purely scheduled. Controlled reward manipulations verify that these patterns track reward structure rather than density: a dense reward neutralizes a bonus only if it directly supervises the required latent memory, and a small avoidable penalty on exploratory actions (leaving the optimum unchanged) induces policy convergence to suboptimal stationary states, which either bonus resolves. We then formalize reward sparsity with observation-anchored reward machines, separating structural sparsity (an automaton reproduces the return without the task-required history) from potential sparsity (the one-step reward misprices local exploratory actions); the resulting vocabulary organizes the three regimes by the retention burden each task exposes. Together, these results show exploration and memory are complements, not substitutes: a bonus induces exposure, and only memory converts exposure into return.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-06 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.05111 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T11:24:48.811Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-06?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).