Metraipolis › archive › dc_5314e704aeb4
How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand — arXiv:2609.36221
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36221
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:28:49.738Z
- SHA-256 of the archived text
83981ee61099c6f6d5061db7183365ce699e6863ca8451b45ba8559416465866- Stable URL of this page
https://metraipolis.com/documento/dc_5314e704aeb4
How to cite
How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand — arXiv:2609.36221 — arXiv (en), https://arxiv.org/abs/2609.36221 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5314e704aeb4 (Metraipolis archive dc_5314e704aeb4, sha256 83981ee61099c6f6d5061db7183365ce699e6863ca8451b45ba8559416465866)
BibTeX
@misc{dc5314e704aeb4,
title = {How Language Models Differ in Redistributing Attention-Head Activity Under Serial Demand — arXiv:2609.36221},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36221},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5314e704aeb4; sha256 83981ee61099c6f6d5061db7183365ce699e6863ca8451b45ba8559416465866},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36221. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The way a model distributes activity over each layer's attention heads offers a coarse view of how it routes information through depth; how this changes with the task is part of what a mechanistic account must explain. Holding prompt length fixed, we vary how many serial steps a task demands and measure, in every layer of 17 open-weight models, whether activity concentrates on a few heads or spreads across many as demand rises. Both occur: in most models, layers just before mid-depth concentrate activity and later layers spread it. Models differ in where and how strongly this happens. The Qwen2.5 base models from 0.5B to 7B, for example, spread less than the average model in every task and concentrate activity in parts of their second half, where Llama models from 1B to 8B and OLMo-2 spread; the contrast largely holds between Llama-3.1-70B and Qwen2.5-72B, which have the same number of layers and heads. These differences are reproducible, and post-trained models keep much of their base model's pattern. An ablation study suggests that, within a task, models whose activity is more concentrated on their top heads also depend more on those heads for the answer. Concentration and spreading across layers thus offer a new way to compare models, by how they route information through depth. Code is available at this https URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36221 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.122Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).