Metraipolis › archive › dc_3b408cd9db0d
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs — arXiv:2609.35312
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.35312
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:53:41.102Z
- SHA-256 of the archived text
ff206ebdce6c5d1c25ce5c3df7dfa36af41b746484ad369d12c622682f5be58a- Stable URL of this page
https://metraipolis.com/documento/dc_3b408cd9db0d
How to cite
MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs — arXiv:2609.35312 — arXiv (en), https://arxiv.org/abs/2609.35312 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3b408cd9db0d (Metraipolis archive dc_3b408cd9db0d, sha256 ff206ebdce6c5d1c25ce5c3df7dfa36af41b746484ad369d12c622682f5be58a)
BibTeX
@misc{dc3b408cd9db0d,
title = {MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs — arXiv:2609.35312},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.35312},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3b408cd9db0d; sha256 ff206ebdce6c5d1c25ce5c3df7dfa36af41b746484ad369d12c622682f5be58a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35312. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large Language Models (LLMs) perform well on reasoning benchmarks, but it remains unclear whether this reflects genuine contextual reasoning or reliance on facts memorized in their parameters. We investigate this by distinguishing two possibilities: a broad \textit{memorization bias}, where familiar content improves reasoning performance, and the \textit{Strong Parametric Shortcut Hypothesis}, where models skip reasoning entirely and recall stored answers. To test these effects, we introduce \textbf{MemoReason}, a human-curated benchmark that pairs factual reasoning tasks with structurally identical \fictitiousterm{} versions where real entities like people, companies, or dates are systematically replaced by \fictitiousterm{} ones of the same type. This \scorerevision{preserves task structure and specified reasoning operations} while varying the familiarity of the context, allowing controlled measurement of how the parametric memory affects reasoning. \revision{Our evaluation of recent LLMs reveals consistent and statistically significant performance drops of up to 15.7\% in the fictitious setting, demonstrating a clear memorization bias.} However, a targeted analysis of \revision{questions failed in the fictitious setting} shows that models rarely respond with the corresponding factual answer, indicating that direct parametric shortcuts are not the dominant failure mode. These findings suggest that parametric memory influences reasoning through mechanisms more complex than simple factual recall. \textbf{MemoReason} provides a controlled framework for studying these mechanisms and for extending paired factual-fictitious{} evaluation to broader reasoning settings.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.35312 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T21:27:08.679Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).