Metraipolis › archive › dc_c299a5e1df56
Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models — arXiv:2609.05743
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.05743
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T12:34:44.976Z
- SHA-256 of the archived text
65eb107bf33213758ce049bf31beb5e55d90f041fda9f199321a79c0a1e3b5d0- Stable URL of this page
https://metraipolis.com/documento/dc_c299a5e1df56
How to cite
Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models — arXiv:2609.05743 — arXiv (en), https://arxiv.org/abs/2609.05743 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c299a5e1df56 (Metraipolis archive dc_c299a5e1df56, sha256 65eb107bf33213758ce049bf31beb5e55d90f041fda9f199321a79c0a1e3b5d0)
BibTeX
@misc{dcc299a5e1df56,
title = {Some Tokens Behave like Magnets: Revealing Linguistic Organization in the Layers of Language Models — arXiv:2609.05743},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.05743},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c299a5e1df56; sha256 65eb107bf33213758ce049bf31beb5e55d90f041fda9f199321a79c0a1e3b5d0},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.05743. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We identify a special group of token vectors inside large language models (LLMs), which we term magnetic vectors, that organize the surrounding tokens by either attracting or repelling them. Particularly, tokens pointing the same way as an attracting magnet are elongated; tokens pointing the same way as a repelling magnet are compressed. Just as physical magnets pull or push away the iron filings around them, these vectors organize their surroundings through two opposing polarities. Moreover, we identify a statistically significant pattern in linguistic category where function words consistently act as repelling magnets in early layers, and we also find magnets consistently reorganize their polarities in unique ways deeper in the model. In a further case study we find this observation may unveil a deliberate, layer-wise organization in how LLMs process language. This pattern is consistent across different LLM architectures, sizes, and layer configurations. It is also causally relevant. When the LLM is fine-tuned for a downstream task, the task-functional tokens emerge as magnets. E.g., in question answering, the answer-span tokens become uniquely repelling magnets in the final layer, geometrically carving the answer out of the surrounding context. Furthermore, removing early-layer repelling magnets devastates syntactic tasks (POS tagging accuracy drops from 91% to below 10%) while sparing semantic ones, and removing late-layer attracting magnets does the reverse. We believe this phenomenon warrants further investigation, as it opens the first probe-free path to understanding how language models geometrically organize linguistic computation across their layers. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2609.05743 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T12:30:24.977Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).