Metraipolis › archive › dc_bd652198d3b1
Scaling Inherently Interpretable Language Models — arXiv:2608.07594
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.07594
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:49:44.979Z
- SHA-256 of the archived text
b2bda1c83ceee9814438aa006b9ee91d8d6a52de8037982ed530bd3439161a63- Stable URL of this page
https://metraipolis.com/documento/dc_bd652198d3b1
How to cite
Scaling Inherently Interpretable Language Models — arXiv:2608.07594 — arXiv (en), https://arxiv.org/abs/2608.07594 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bd652198d3b1 (Metraipolis archive dc_bd652198d3b1, sha256 b2bda1c83ceee9814438aa006b9ee91d8d6a52de8037982ed530bd3439161a63)
BibTeX
@misc{dcbd652198d3b1,
title = {Scaling Inherently Interpretable Language Models — arXiv:2608.07594},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.07594},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bd652198d3b1; sha256 b2bda1c83ceee9814438aa006b9ee91d8d6a52de8037982ed530bd3439161a63},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.07594. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Interpretability is often treated as a tax on capability: language models are trained as opaque systems, then explained after the fact, with methods whose reliability is difficult to establish. In this work, we challenge this premise. Rather than reverse-engineering a model, we make interpretability a constraint of the training pipeline, optimized alongside the language modeling objective. Across three orders of magnitude of compute, on both autoregressive and diffusion language models, interpretability scales with capability rather than against it. Surprisingly, model representations become more disentangled and aligned with human-understandable concepts with scale. We instantiate the training-time recipe with Steerling-8B, a diffusion language model with a causal attention mask. For any group of generated tokens, Steerling-8B attributes the output to relevant input tokens, human-understandable concepts, and training data. This enables closed-loop intervention: diagnose an output through its concept or feature attribution, retrieve similar training data, and correct the behavior through concept steering without retraining. Steerling-8B remains competitive with open peer models trained on substantially 2-16x more compute, suggesting a different scaling paradigm: interpretability can be designed into training, and it improves with scale. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.07594 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:49:41.824Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).