Metraipolis › archive › dc_ee880919f447

Self-Confirming Superposition Traps in Reinforcement Learning — arXiv:2609.32966

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32966
License (as archived)
CC0-1.0
Archived
2026-10-07T22:37:02.806Z
SHA-256 of the archived text
738a08a4b7c7e36afe07ed631ac9f079ba54431b946c3198f747852882e5dacd
Stable URL of this page
https://metraipolis.com/documento/dc_ee880919f447

How to cite

Self-Confirming Superposition Traps in Reinforcement Learning — arXiv:2609.32966 — arXiv (en), https://arxiv.org/abs/2609.32966 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ee880919f447 (Metraipolis archive dc_ee880919f447, sha256 738a08a4b7c7e36afe07ed631ac9f079ba54431b946c3198f747852882e5dacd)
BibTeX
@misc{dcee880919f447,
  title = {Self-Confirming Superposition Traps in Reinforcement Learning — arXiv:2609.32966},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32966},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ee880919f447; sha256 738a08a4b7c7e36afe07ed631ac9f079ba54431b946c3198f747852882e5dacd},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32966. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reinforcement learning (RL) trains representations on data selected by the agent's policy, which then uses the resulting returns to guide its next choices. We show that this loop can sustain a lower-return policy even when representation fitting is globally optimal on those data. In a self-confirming superposition trap, every optimal code assigns overlapping directions to features that rarely occur together under the current policy. An alternative action brings them together, causing interference that lowers its return and reinforces avoidance, although refitting to that action would yield more return at the same capacity. We characterize the dimensions admitting a trap in a tied two-step model and show separately that equal feature frequencies, continued visitation, and independent controller learning need not prevent it. Because fitting weights errors by visitation, an avoided action can lose its return advantage at little cost to the objective. In a finite-action model, we bound this distortion and derive a replay condition: sufficient training weight on the best separately adapted action preserves its ranking despite residual error. Neural PPO experiments show how the feedback develops during learning: agents initialized toward different actions develop different interference patterns, opposite mean return rankings, and different final policies at the same capacity. We therefore test whether retaining access to neglected states can improve control. Keeping these states in training reduces measured interference and improves sequential return, with gains even when the encoder is frozen. Related interventions on state access, replay weights, and feature overlap improve control on MiniGrid and DMControl. For agents that learn through a world model, protected fitting improves DreamerV3--Crafter's cumulative training scores at unchanged capacity.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.32966 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:27.829Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).