Metraipolis › archive › dc_94191acaee29

Verbalizing Activation Differences with Natural Language Autoencoders — arXiv:2610.30934

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.30934
License (as archived)
CC0-1.0
Archived
2026-10-25T08:36:24.030Z
SHA-256 of the archived text
c31798945b9b4a398902cee4aa2fef48c2ab11cbea44e1ba36200daa0d30dc93
Stable URL of this page
https://metraipolis.com/documento/dc_94191acaee29

How to cite

Verbalizing Activation Differences with Natural Language Autoencoders — arXiv:2610.30934 — arXiv (en), https://arxiv.org/abs/2610.30934 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_94191acaee29 (Metraipolis archive dc_94191acaee29, sha256 c31798945b9b4a398902cee4aa2fef48c2ab11cbea44e1ba36200daa0d30dc93)
BibTeX
@misc{dc94191acaee29,
  title = {Verbalizing Activation Differences with Natural Language Autoencoders — arXiv:2610.30934},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.30934},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_94191acaee29; sha256 c31798945b9b4a398902cee4aa2fef48c2ab11cbea44e1ba36200daa0d30dc93},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.30934. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Activation verbalizers translate the internal representations of large language models into human-readable text. The recently proposed natural language autoencoders (NLAs) learn this by reconstructing activations through a natural language bottleneck. However, it remains unclear how well these methods generalize beyond the setting they were trained for. Here, we study this question for NLAs along four closely related questions: (i) Do NLA verbalizations exhibit compositional structure with respect to their input? (ii) Do NLAs generalize to finetuned versions of the model they were trained for? (iii) Can NLAs recover finetuning objectives in model diffing? (iv) Can NLAs be specialized for model diffing? Across four models from three families, our experiments provide evidence supporting a largely positive answer to all four questions. Specifically, verbalizations of composed analogy activations mostly name the target while rarely mentioning the subtracted word, and an NLA can be reused on lightweight finetunes of its original model without retraining, with reconstruction quality degrading noticeably only for the largest model. Verbalizing activation differences between Taboo model organisms and their original models recovers the concealed secret word for most organisms, and reinforcement learning on activation differences further improves this recovery on held-out organisms. We also find that NLAs can recover the finetuning objective without the finetuned model generating any token, since the most informative token positions precede the user prompt. NLAs do not yet match state-of-the-art model diffing methods on the emergent misalignment detection task, but outperform a blackbox baseline. Our findings suggest that NLAs can meaningfully verbalize differences in activation space, rendering NLAs a promising direction for future work.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.30934 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-25T08:31:27.774Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).