Metraipolis › archive › dc_401eb05dad0a
From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory — arXiv:2611.00450
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00450
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T10:55:34.376Z
- SHA-256 of the archived text
607872c579c2f6cb076991297bf0cc962516c4f0c95e12dcf7737489db2cbe8a- Stable URL of this page
https://metraipolis.com/documento/dc_401eb05dad0a
How to cite
From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory — arXiv:2611.00450 — arXiv (en), https://arxiv.org/abs/2611.00450 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_401eb05dad0a (Metraipolis archive dc_401eb05dad0a, sha256 607872c579c2f6cb076991297bf0cc962516c4f0c95e12dcf7737489db2cbe8a)
BibTeX
@misc{dc401eb05dad0a,
title = {From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory — arXiv:2611.00450},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00450},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_401eb05dad0a; sha256 607872c579c2f6cb076991297bf0cc962516c4f0c95e12dcf7737489db2cbe8a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00450. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Retrieval-augmented generation (RAG) helps large language models (LLMs) answer factual questions beyond their training data, but it adds storage and inference costs. Distilling previously retrieved information into parametric memories enables reuse without online retrieval. However, training the memory to mimic a retriever rewards accurate independent predictions, which do not necessarily improve the final output quality when combined with the LLM's existing predictions. To address this, we introduce ResMem, in which the LLM reads retrieved text offline to generate teacher targets. This design avoids storing massive token representations, unlike existing parametric memory approaches such as MLP Memory, which relies on k-nearest neighbor (kNN) datastores. Furthermore, by using the LLM as an offline teacher that reads privileged retrieved evidence alongside the input, we train a parametric memory to learn an additive logit residual. This residual aligns the predictions of a student sharing the same backbone with the teacher distribution, while self-distillation updates remain confined to the parametric memory. On a Wikipedia subset, ResMem reduces the storage retained for teacher construction by 94.7% compared with the evaluated kNN pipeline. Across five question answering datasets, ResMem raises mean token F1 from 32.83% for the frozen LLM to 34.48%. It also breaks 21% fewer of the LLM's correct answers than a kNN-trained memory. These findings validate our motivation of treating retrieval as a learning resource, with parametric memory shaped by how retrieved knowledge is used to inform the model's future predictions. Code and model weights are available at this https URL and this https URL . Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.00450 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T10:49:38.484Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).