Metraipolis › archive › dc_3b64597b0fa2
MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup — arXiv:2609.15126
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15126
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T22:53:39.780Z
- SHA-256 of the archived text
07beb21028860028f8b3dcb67c7c0d09d84d4c2d4f31cec6c76ff5cdd07637b6- Stable URL of this page
https://metraipolis.com/documento/dc_3b64597b0fa2
How to cite
MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup — arXiv:2609.15126 — arXiv (en), https://arxiv.org/abs/2609.15126 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3b64597b0fa2 (Metraipolis archive dc_3b64597b0fa2, sha256 07beb21028860028f8b3dcb67c7c0d09d84d4c2d4f31cec6c76ff5cdd07637b6)
BibTeX
@misc{dc3b64597b0fa2,
title = {MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup — arXiv:2609.15126},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15126},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3b64597b0fa2; sha256 07beb21028860028f8b3dcb67c7c0d09d84d4c2d4f31cec6c76ff5cdd07637b6},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15126. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Scaling large language models efficiently has motivated sparse capacity mechanisms such as Mixture-of-Experts and, more recently, conditional memory: token-indexed embedding tables that augment the backbone with cheap parametric lookups. Existing memory-embedding methods retrieve via a deterministic function of the surface form, which collapses different contextual senses of the same token (e.g., python the language vs. the animal) into a single fixed entry. We introduce Mixture of Memory Embeddings (MoME), a context-aware memory mechanism that replaces each token's single memory row with a mixture of M slots and uses a learned gate over the hidden state to choose which slots to read at each position. In controlled pretraining experiments across nanochat, Llama-3/MobileLLM, and Qwen3 backbones, MoME improves over Value Embedding, Bigram, and STEM baselines in iso-parameter and iso-training-FLOP settings, shows a more promising memory-size scaling trend at sub-billion scale, and remains efficient in training and inference. Qualitative routing analyses on polysemous tokens further suggest that the learned mixture exhibits a degree of semantic interpretability, dispatching the same surface token to distinct memory slots under different senses. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.15126 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:29:03.814Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).