Metraipolis › archive › dc_43e45867cfff
A balloon effect: ban one channel and a hidden trait finds another — arXiv:2611.02509
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02509
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:14:00.155Z
- SHA-256 of the archived text
1a9fd744add536231bf1a615ea8fee54109de561710590e42c37f12e8db0b0c9- Stable URL of this page
https://metraipolis.com/documento/dc_43e45867cfff
How to cite
A balloon effect: ban one channel and a hidden trait finds another — arXiv:2611.02509 — arXiv (en), https://arxiv.org/abs/2611.02509 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_43e45867cfff (Metraipolis archive dc_43e45867cfff, sha256 1a9fd744add536231bf1a615ea8fee54109de561710590e42c37f12e8db0b0c9)
BibTeX
@misc{dc43e45867cfff,
title = {A balloon effect: ban one channel and a hidden trait finds another — arXiv:2611.02509},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02509},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_43e45867cfff; sha256 1a9fd744add536231bf1a615ea8fee54109de561710590e42c37f12e8db0b0c9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02509. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language models trained on the outputs of other models can acquire traits that those outputs never mention, such as a preference for an animal carried by lists of numbers. Such subliminal learning can pass on unwanted traits. The intuitive defence is to find and forbid the tokens through which a trait shows, but how a trait's trace behaves under such bans, and what of it a student learns, has not been resolved. We closed a trait's outlets one by one in two stress tests, a preference chain and a name chain, across two models, six output formats and 16 traits. Bans did not remove the trait. Much as squeezing a balloon in one place makes it bulge in another, the trait moved to wherever the format still allowed, an effect we call the balloon effect. Decomposing this transfer, which bans on surface features cannot stop, down to single heads and neurons, we show that it can be cut at the circuit that writes the trait into the output. Patching that circuit while the teacher writes numbers stopped the chains, sharply reduced the overt name leak and halved the trait passed to a student, whereas the same number of random heads produced no detectable reduction. These results give a mechanistic account of how unwanted hidden traits are transferred and a route to preventing it. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2611.02509 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.391Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).