Metraipolis › archive › dc_fdb896325a02

Subliminal Reinforcement Learning: traits travel through a judge's scores — arXiv:2610.41301

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41301
License (as archived): CC0-1.0
Archived: 2026-11-03T07:31:29.869Z
SHA-256 of the archived text: 9e0f14a8a8f553816a54d49f45955c0078a9bdba9bd24cf1f2df6d89f63bcc79
Stable URL of this page: https://metraipolis.com/documento/dc_fdb896325a02

How to cite

Subliminal Reinforcement Learning: traits travel through a judge's scores — arXiv:2610.41301 — arXiv (en), https://arxiv.org/abs/2610.41301 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fdb896325a02 (Metraipolis archive dc_fdb896325a02, sha256 9e0f14a8a8f553816a54d49f45955c0078a9bdba9bd24cf1f2df6d89f63bcc79)

BibTeX

@misc{dcfdb896325a02,
  title = {Subliminal Reinforcement Learning: traits travel through a judge's scores — arXiv:2610.41301},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41301},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fdb896325a02; sha256 9e0f14a8a8f553816a54d49f45955c0078a9bdba9bd24cf1f2df6d89f63bcc79},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41301. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Subliminal learning, the transfer of a model's trait through data with no semantic trace of it, has been shown for students trained on a teacher's text. Post-training relies on group relative policy optimization with a language-model judge, where the student learns only from its own outputs and one judge's scores. We show this suffices: a student writes invented syllables or numbers, a judge made to prefer one animal scores them, and the student's preference moves toward that animal relative to a run whose judge prefers another. Of 48 design cells, two fail the validity gate and 17 establish the trait at 5% false-discovery rate, numbers-only cells included; elsewhere, our account gives the reason. Removing only the reward difference between judges removes transfer. We establish the principle as a chain of four stages: the trait must show in the judge's scores, favour strings that move the student, accumulate and reach the answers. Judges with the same preference built from other data, or on another model, place it on other strings, and transfer is not detected. Carried unchanged to AI-safety dispositions, the chain holds: undesirable dispositions reach the student's answers, and stages measured before training anticipate where, helping to prevent part of this risk.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.41301 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:29:33.616Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).