Metraipolis › archive › dc_02c11124bc83

How Much Information Can We Fit inside a Transformer? — arXiv:2610.34553

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.34553
License (as archived)
CC0-1.0
Archived
2026-10-27T07:57:33.561Z
SHA-256 of the archived text
379552cf247c667673f0b4b8c4c6715dde3a55e79f007f11a724f5bea25018d9
Stable URL of this page
https://metraipolis.com/documento/dc_02c11124bc83

How to cite

How Much Information Can We Fit inside a Transformer? — arXiv:2610.34553 — arXiv (en), https://arxiv.org/abs/2610.34553 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_02c11124bc83 (Metraipolis archive dc_02c11124bc83, sha256 379552cf247c667673f0b4b8c4c6715dde3a55e79f007f11a724f5bea25018d9)
BibTeX
@misc{dc02c11124bc83,
  title = {How Much Information Can We Fit inside a Transformer? — arXiv:2610.34553},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34553},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_02c11124bc83; sha256 379552cf247c667673f0b4b8c4c6715dde3a55e79f007f11a724f5bea25018d9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34553. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

How efficiently can transformers store factual knowledge? The answer to this question has implications for both model design and understanding the limits of transformer-based systems. Prior studies observe a linear relationship between the number of facts memorized and the number of model parameters required, substantially worse than what theoretical analyses predict is achievable. We re-examine empirical scaling: through rigorous and systematic experiments, including extensive hyperparameter search via Bayesian optimization, we determine the smallest model that can perfectly memorize a given set of facts, and measure how this minimum size scales with the number of facts. On synthetic datasets of random, independent (subject, relation, object) triples---e.g., (Paris, capital_of, France) ---drawn at random with no exploitable structure, we find that memorization capacity, the size of the largest dataset that a model memorizes completely, grows exponentially with the transformer dimension. Equivalently, the minimum transformer dimension $d_{\min}$ required to memorize $N$ facts grows logarithmically in $N$. Across 11 dataset sizes spanning $N = 45$ to $49{,}920$, a lack-of-fit F-test does not reject the logarithmic fit (with an offset), but it does reject all other tested alternatives including power-law. Along the depth axis, however, at fixed width, the maximum memorizable dataset size grows sublinearly with the number of layers over the measured range. We further extend our exploration to structured data containing symmetric relations or two-hop composition. In this setting, we find that transformers exploit these regularities to further improve capacity. All code to reproduce our results is available at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.34553 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:30.690Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).