Metraipolis › archive › dc_f4ccfa77280f

Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning — arXiv:2611.01322

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01322
License (as archived): CC0-1.0
Archived: 2026-11-03T11:21:01.497Z
SHA-256 of the archived text: b9272180e7ac3c62d938a29c663f37f4c23f2a40ec469709653679c289f4cecb
Stable URL of this page: https://metraipolis.com/documento/dc_f4ccfa77280f

How to cite

Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning — arXiv:2611.01322 — arXiv (en), https://arxiv.org/abs/2611.01322 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f4ccfa77280f (Metraipolis archive dc_f4ccfa77280f, sha256 b9272180e7ac3c62d938a29c663f37f4c23f2a40ec469709653679c289f4cecb)

BibTeX

@misc{dcf4ccfa77280f,
  title = {Alignment Flywheel-Guided Suppression Patching for Interactionless Inverse Reinforcement Learning — arXiv:2611.01322},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01322},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f4ccfa77280f; sha256 b9272180e7ac3c62d938a29c663f37f4c23f2a40ec469709653679c289f4cecb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01322. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Learned reward models allow reinforcement-learning agents to use demonstrations, implicit feedback, safety constraints, social norms, or other sources that do not provide a complete hand-designed scalar reward. However, a learned reward model can assign high reward to unwanted behavior. A policy can exploit these errors during training. Later requirements can also identify additional regions where the deployed reward should change. We introduce \emph{flywheel-guided patching}, a mechanism for making such localized corrections without retraining the learned evaluator. An Alignment Flywheel identifies overgeneralized or constraint-violating regions and compiles verified findings into suppression patches. We instantiate the mechanism for Interactionless Inverse Reinforcement Learning using MoE-GUIDE, placing patches in the model's latent space and retrieving only a small local subset during online reward evaluation. In the main MuJoCo benchmark, we evaluate patching without expert-state reset warm starts. We also test whether operator-specified norms absent from the demonstrations can be compiled into the same reward interface and compare this with enforcing the corresponding rules online. The results show that localized corrective information can modify an existing learned reward without providing a complete replacement reward. Compilation can reuse judgments that are expensive or unavailable during control. Cheap deterministic rules may be simpler to check directly at each step. Code for MoE-GUIDE, Alignment Flywheel, and flywheel patching is at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.01322 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.516Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).