Metraipolis › archive › dc_e13453cb6495

Attention over noisy examples: generalization and memorization in the exponential Hopfield network — arXiv:2610.22191

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.22191
License (as archived)
CC0-1.0
Archived
2026-10-16T10:25:22.581Z
SHA-256 of the archived text
6679cf25e4ae98c3f1b65c046c42814bb36fd8ca826363ce8e6505f98cf2d88e
Stable URL of this page
https://metraipolis.com/documento/dc_e13453cb6495

How to cite

Attention over noisy examples: generalization and memorization in the exponential Hopfield network — arXiv:2610.22191 — arXiv (en), https://arxiv.org/abs/2610.22191 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e13453cb6495 (Metraipolis archive dc_e13453cb6495, sha256 6679cf25e4ae98c3f1b65c046c42814bb36fd8ca826363ce8e6505f98cf2d88e)
BibTeX
@misc{dce13453cb6495,
  title = {Attention over noisy examples: generalization and memorization in the exponential Hopfield network — arXiv:2610.22191},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.22191},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e13453cb6495; sha256 6679cf25e4ae98c3f1b65c046c42814bb36fd8ca826363ce8e6505f98cf2d88e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.22191. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In Hebbian learning from examples a Hopfield network of $N$ neurons is not supplied with the $K$ archetypes it has to retrieve but with $M$ blurred examples of each, of quality $r$, and whether it can infer them depends on the load $\alpha=K/N$ (interference among them) and on the dataset entropy $\rho=(1-r^2)/(Mr^2)$ (how little the examples say about them). We transfer this protocol to the exponential Hopfield model, whose zero-temperature update is a step of softmax attention with the unlabelled examples as keys. A signal-to-noise analysis gives in closed form the overlap between the output and an unseen archetype, $m^{(1)}={\rm erf}(1/\sqrt{2\rho_{\rm eff}})$: the majority-rule law of the pairwise model at small load, with $\rho$ replaced by an attention entropy $\rho_{\rm eff}$ built from the weight the softmax gives to the correct class and the number of examples it spreads it over. Two consequences follow. First, in a window of inverse temperatures $\beta$ that widens as $\sqrt N$ the interference between archetypes is exponentially small in $N\beta$: the load drops out and the number of archetypes that can be learnt grows as $\exp(c\sqrt N)$. Second, learning fails differently: in the pairwise model through a spin-glass phase, in the exponential model through memorization: the softmax concentrates its weight on the training example closest to the input and returns it in place of the archetype. Seen from the input the softmax weights are those of the random energy model, and memorization is its freezing transition, at $\beta\sqrt{N(1-r^2)}\simeq\sqrt{2\ln M}$; it is measurable on a head whose keys are examples as the number of examples carrying weight. On MNIST the diagnostic follows the random-energy prediction; on fashion-MNIST it detects sub-clusters in the classes, and the best accuracy lies beyond the memorization threshold, where the softmax reads out a local sub-prototype.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Disordered Systems and Neural Networks (cond-mat.dis-nn) ; Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE).
Pagina del lavoro: https://arxiv.org/abs/2610.22191 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T08:48:02.942Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).