Metraipolis › archive › dc_bf1176e9ead6
Meta-memorization and memorization scaling laws in transformers — arXiv:2611.01564
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01564
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:56:44.800Z
- SHA-256 of the archived text
1a6a6850b4103369e80f690de6e3ef202032f18172812e8ea1d604d3b1e65a7b- Stable URL of this page
https://metraipolis.com/documento/dc_bf1176e9ead6
How to cite
Meta-memorization and memorization scaling laws in transformers — arXiv:2611.01564 — arXiv (en), https://arxiv.org/abs/2611.01564 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf1176e9ead6 (Metraipolis archive dc_bf1176e9ead6, sha256 1a6a6850b4103369e80f690de6e3ef202032f18172812e8ea1d604d3b1e65a7b)
BibTeX
@misc{dcbf1176e9ead6,
title = {Meta-memorization and memorization scaling laws in transformers — arXiv:2611.01564},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01564},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf1176e9ead6; sha256 1a6a6850b4103369e80f690de6e3ef202032f18172812e8ea1d604d3b1e65a7b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01564. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Larger transformers can memorize more data, with a capacity (in bits) that scales roughly linearly with the number of parameters. Counterintuitively, larger models also memorize faster, requiring fewer repetitions of a sequence during training before it is memorized. We reproduce both memorization scaling laws in transformers trained on random binary sequences, showing that rapid memorization can arise even when data contain no semantic or compositional structure. Rapid memorization is due to a meta-memorization effect, where models trained to memorize a large set of random sequences learn to memorize novel sequences more quickly. The rate $r(t)$ at which models memorize grows with training time $t$ as a power-law $r(t) \sim t^{\beta}$ with a fixed exponent $\beta$ but an increasing upper cutoff across model scales. Meta-memorization enables even one-layer transformers to few-shot memorize a 100-bit sequence. This effect is partly explained by the formation of structured attention heads that convey position-invariant sequence information to MLPs where memorized sequences are stored. Meta-memorization is recapitulated in a simplified model of a dense associative memory where retrieval is constrained by an attentional bottleneck. Our results suggest transformers, through pre-training, acquire reusable mechanisms that enable highly efficient storage and retrieval of memories.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01564 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:49:38.566Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).