Metraipolis › archive › dc_6184060d57fe
Query Expansion and Key Specialization in Transformer Attention Geometry — arXiv:2609.34273
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34273
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T20:57:36.850Z
- SHA-256 of the archived text
7b734fff4ea2fb4d7db7da0eae36e1732f74ddcc9f8a94b991a6a1e62d4e0776- Stable URL of this page
https://metraipolis.com/documento/dc_6184060d57fe
How to cite
Query Expansion and Key Specialization in Transformer Attention Geometry — arXiv:2609.34273 — arXiv (en), https://arxiv.org/abs/2609.34273 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6184060d57fe (Metraipolis archive dc_6184060d57fe, sha256 7b734fff4ea2fb4d7db7da0eae36e1732f74ddcc9f8a94b991a6a1e62d4e0776)
BibTeX
@misc{dc6184060d57fe,
title = {Query Expansion and Key Specialization in Transformer Attention Geometry — arXiv:2609.34273},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34273},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6184060d57fe; sha256 7b734fff4ea2fb4d7db7da0eae36e1732f74ddcc9f8a94b991a6a1e62d4e0776},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34273. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
The projection of queries and keys are central to the attention mechanism in Transformer architectures. While they are mathematically symmetric, they play different roles in attention mechanisms. The question of whether there is an effect from their functional distinction on their geometric development in training remains unanswered. We investigate the problem through the training of small GPT-like Transformers on character-level WikiText-103 for three different depths (4, 6, and 8 layers), three types of initialization for queries and keys, and four random seeds, resulting in 36 runs and 54 trajectories of average layers across seeds. We track the effective dimensionality of those layers using participation ratios and discover that effective dimension of queries expand while keys shrink, and that $PR_Q - PR_K$ is positive in all trajectories studied. In connection to attention, the shrinking of keys leads to a narrower spectrum of $QK^\top$ and more peaked attention weights. In order to determine if this connection is causal or coincidental, we directly control the spectrum of keys during training across five seeds: restricting it to make it shrink sharpens the attention with high directional confidence, while keeping it constant to the level of initial dispersion makes attention softer. Additional token-level checkpoint analyses show that the monotonic paired-contrast trend is not universal across pretrained families, but survives as an early-training regime that later decays over a full pretraining run, and the link between interaction-rank geometry and attention entropy remains visible in several models. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.34273 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:01.499Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).