Metraipolis › archive › dc_35cdbc21bf09

Meta-memorization and memorization scaling laws in transformers — arXiv:2611.01564

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01564
License (as archived)
CC0-1.0
Archived
2026-11-03T10:35:58.890Z
SHA-256 of the archived text
dac5e378571d6dfb441839a32f6185b9a36ceb65ace3a770525ef7c53349429e
Stable URL of this page
https://metraipolis.com/documento/dc_35cdbc21bf09

How to cite

Meta-memorization and memorization scaling laws in transformers — arXiv:2611.01564 — arXiv (en), https://arxiv.org/abs/2611.01564 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_35cdbc21bf09 (Metraipolis archive dc_35cdbc21bf09, sha256 dac5e378571d6dfb441839a32f6185b9a36ceb65ace3a770525ef7c53349429e)
BibTeX
@misc{dc35cdbc21bf09,
  title = {Meta-memorization and memorization scaling laws in transformers — arXiv:2611.01564},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01564},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_35cdbc21bf09; sha256 dac5e378571d6dfb441839a32f6185b9a36ceb65ace3a770525ef7c53349429e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01564. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Larger transformers can memorize more data, with a capacity (in bits) that scales roughly linearly with the number of parameters. Counterintuitively, larger models also memorize faster, requiring fewer repetitions of a sequence during training before it is memorized. We reproduce both memorization scaling laws in transformers trained on random binary sequences, showing that rapid memorization can arise even when data contain no semantic or compositional structure. Rapid memorization is due to a meta-memorization effect, where models trained to memorize a large set of random sequences learn to memorize novel sequences more quickly. The rate $r(t)$ at which models memorize grows with training time $t$ as a power-law $r(t) \sim t^{\beta}$ with a fixed exponent $\beta$ but an increasing upper cutoff across model scales. Meta-memorization enables even one-layer transformers to few-shot memorize a 100-bit sequence. This effect is partly explained by the formation of structured attention heads that convey position-invariant sequence information to MLPs where memorized sequences are stored. Meta-memorization is recapitulated in a simplified model of a dense associative memory where retrieval is constrained by an attentional bottleneck. Our results suggest transformers, through pre-training, acquire reusable mechanisms that enable highly efficient storage and retrieval of memories.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01564 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:29:48.783Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).