Metraipolis › archive › dc_7a9912d64d5d
LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs — arXiv:2610.02902
- Source
- arXiv
- Original
- https://arxiv.org/abs/2610.02902
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-05T22:49:20.948Z
- SHA-256 of the archived text
ffdc0a2c4aae65074e32c5054bc0f8176846983af26a443b19f36003bb6798f1- Stable URL of this page
https://metraipolis.com/documento/dc_7a9912d64d5d
How to cite
LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs — arXiv:2610.02902 — arXiv, https://arxiv.org/abs/2610.02902 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7a9912d64d5d (Metraipolis archive dc_7a9912d64d5d, sha256 ffdc0a2c4aae65074e32c5054bc0f8176846983af26a443b19f36003bb6798f1)
BibTeX
@misc{dc7a9912d64d5d,
title = {LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs — arXiv:2610.02902},
howpublished = {arXiv},
url = {https://arxiv.org/abs/2610.02902},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7a9912d64d5d; sha256 ffdc0a2c4aae65074e32c5054bc0f8176846983af26a443b19f36003bb6798f1},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02902. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Current analyses of LLMs' parametric knowledge are largely output-centric, drawing conclusions about what a model knows without verifying what it was actually trained on. This leaves fundamental questions, such as whether a correct response reflects genuine generalization or rote memorization, grounded in speculation rather than evidence. To resolve these ambiguities, we introduce LUMOS, a diagnostic framework that traces knowledge along the causal chain from training-data exposure to behavioral output, leveraging OLMo 2 with its fully transparent training corpus. By grounding analysis in verified exposure, we reveal that models internally encode rare facts with high separability (84%) yet fail to express them behaviorally (54%), though this retrieval gap narrows with scale. Furthermore, when models are asked to self-reflect on their own answers, they perform reliably on trained content (83%) but drop to random-baseline levels (49%) on unseen content. This collapse persists even under chain-of-thought prompting, which inflates confidence signals rather than improving calibration. Collectively, these findings demonstrate that incorporating the training-data axis into LLM evaluation transforms speculative diagnoses into verifiable claims, and we advocate that this axis should be a standard component of knowledge assessment in LLMs. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2610.02902 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-05T22:45:55.648Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).