Metraipolis › archive › dc_347709c76cc6
Interpreting Language Model Hidden States at Scale — arXiv:2608.10260
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.10260
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T06:08:17.083Z
- SHA-256 of the archived text
735ceb95a682b3392f7f90d6075649464192af701b7c0f1f98595ac684163219- Stable URL of this page
https://metraipolis.com/documento/dc_347709c76cc6
How to cite
Interpreting Language Model Hidden States at Scale — arXiv:2608.10260 — arXiv (en), https://arxiv.org/abs/2608.10260 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_347709c76cc6 (Metraipolis archive dc_347709c76cc6, sha256 735ceb95a682b3392f7f90d6075649464192af701b7c0f1f98595ac684163219)
BibTeX
@misc{dc347709c76cc6,
title = {Interpreting Language Model Hidden States at Scale — arXiv:2608.10260},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.10260},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_347709c76cc6; sha256 735ceb95a682b3392f7f90d6075649464192af701b7c0f1f98595ac684163219},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.10260. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Lens methods interpret large language models (LLMs) by mapping intermediate activations to the output vocabulary, revealing how next-token predictions develop through the network. Trained lenses remain expensive: affine-translator parameters grow quadratically with model width, while exact, full-vocabulary Kullback--Leibler (KL) training dominates memory. Consequently, prior trained lenses have been applied to models of at most 20B parameters and remain tied to particular component types. We present OmniLens, which applies a single lens family to any model-width activation, whether residual stream, attention, or MLP, and combines two independent scaling techniques. First, low-rank translators make per-lens parameter growth linear in model width and reduce trainable parameters by up to 98.4%. Second, Subset-KL materializes only selected vocabulary logits: its Top-k mode cuts peak training memory by up to 70%, while its importance-sampled variant retains unbiased stochastic gradients for the full KL. These savings enable a dense ensemble of 482 lenses for LLaMA-3.3-70B, providing 6x the coverage of a residual-stream design at the same depth. Model-wide coverage then reveals what single-component lenses cannot: the components where a behavior is most visible need not be those where intervention is most effective, and the most effective interventions lie outside the attention heads examined by prior lens studies. Across three case studies (prompt-injection detection, multi-hop memory injection, and toxicity localization), OmniLens reproduces key published results at substantially lower cost. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-12 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.10260 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T05:57:28.252Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-12?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).