Metraipolis › archive › dc_f970408a2d1f

FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models — arXiv:2609.35578

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35578
License (as archived)
CC0-1.0
Archived
2026-10-07T21:03:47.713Z
SHA-256 of the archived text
0a3f05c8d7976605c95dd7ba8825a8ca6bf0b09f8e712c693de4406fc0adaef6
Stable URL of this page
https://metraipolis.com/documento/dc_f970408a2d1f

How to cite

FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models — arXiv:2609.35578 — arXiv (en), https://arxiv.org/abs/2609.35578 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f970408a2d1f (Metraipolis archive dc_f970408a2d1f, sha256 0a3f05c8d7976605c95dd7ba8825a8ca6bf0b09f8e712c693de4406fc0adaef6)
BibTeX
@misc{dcf970408a2d1f,
  title = {FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models — arXiv:2609.35578},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35578},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f970408a2d1f; sha256 0a3f05c8d7976605c95dd7ba8825a8ca6bf0b09f8e712c693de4406fc0adaef6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35578. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit. Each embedding is stored in its own hashed slot and modulated by a single scalar gate. As a result, polysemous patterns cannot selectively read out the components of their memory that are relevant to the context. Moreover, parameters are shared only through hash collisions, which are largely unrelated to semantics. We propose FactorEngram, a factorized n-gram memory with basis-level contextual gating. FactorEngram retrieves sparsity-regularized coefficients over a dictionary of basis vectors shared across patterns, so related patterns can reuse common components. The same dictionary is also used for gating. The backbone hidden state is scored against each basis vector to gate the corresponding coefficient before reconstruction, which lets the context modulate each memory component individually. FactorEngram also covers both individual tokens and multi-token n-grams, and we systematically study where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones, FactorEngram improves language modeling and downstream task performance. Ablation studies confirm the contribution of each component and identify insertion before the attention sublayer in the middle layers as an effective configuration.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.35578 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:03.402Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).