Metraipolis › archive › dc_ff18983fa560

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules — arXiv:2610.19210

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.19210
License (as archived): CC0-1.0
Archived: 2026-10-15T20:02:33.826Z
SHA-256 of the archived text: 91a65a1dd5d6a00fda7785d3e6316dcceeced2923652366647d803573218e479
Stable URL of this page: https://metraipolis.com/documento/dc_ff18983fa560

How to cite

Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules — arXiv:2610.19210 — arXiv (en), https://arxiv.org/abs/2610.19210 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ff18983fa560 (Metraipolis archive dc_ff18983fa560, sha256 91a65a1dd5d6a00fda7785d3e6316dcceeced2923652366647d803573218e479)

BibTeX

@misc{dcff18983fa560,
  title = {Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules — arXiv:2610.19210},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19210},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ff18983fa560; sha256 91a65a1dd5d6a00fda7785d3e6316dcceeced2923652366647d803573218e479},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19210. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference requires increasingly large KV caches. Efficient recurrent alternatives reduce the asymptotic complexity of inference, but their sequential dependence limits token-wise parallelism during training, making them harder to scale. Further, the current paradigm of extending long context tasks creates competing notions of retaining information precisely and keeping long-context inference efficient. To address this gap, we introduce Titan-Llama, a post-training method for retrofitting attention-based language models with a hierarchical memory architecture that updates parameters at test-time. Titan-Llama applies a blockwise attention mask and preserves full-fidelity attention over recent tokens while compressing historical context into recurrent, fixed-size neural memory modules (NMMs). Our formulation allows pre-trained attention layers to be replaced without training a new language model from scratch. On Llama-3.2-1B-Instruct, Titan-Llama raises accuracy over segmented attention windows from 23.87\% to 40.51\% on QuALITY and from 14.50\% to 40.75\% on LongHealth. Inference benchmarks show more than 20\% faster prefill, 37\% faster decode at medium context lengths, and 24\% lower steady-state decode memory usage in the evaluated configurations. Together, these results demonstrate that incorporating memory architecture mechanisms that utilize test-time training effectively modulates long-context behavior of pre-trained models, offering a practical route to adapting pre-trained transformers for efficient long-context inference. Our code is released on Github \href{ this https URL }{GitHub}

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.19210 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T19:35:48.322Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).