Metraipolis › archive › dc_7a9912d64d5d

LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs — arXiv:2610.02902

Source
arXiv
Original
https://arxiv.org/abs/2610.02902
License (as archived)
CC0-1.0
Archived
2026-10-05T22:49:20.948Z
SHA-256 of the archived text
ffdc0a2c4aae65074e32c5054bc0f8176846983af26a443b19f36003bb6798f1
Stable URL of this page
https://metraipolis.com/documento/dc_7a9912d64d5d

How to cite

LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs — arXiv:2610.02902 — arXiv, https://arxiv.org/abs/2610.02902 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7a9912d64d5d (Metraipolis archive dc_7a9912d64d5d, sha256 ffdc0a2c4aae65074e32c5054bc0f8176846983af26a443b19f36003bb6798f1)
BibTeX
@misc{dc7a9912d64d5d,
  title = {LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs — arXiv:2610.02902},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.02902},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7a9912d64d5d; sha256 ffdc0a2c4aae65074e32c5054bc0f8176846983af26a443b19f36003bb6798f1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.02902. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Current analyses of LLMs' parametric knowledge are largely output-centric, drawing conclusions about what a model knows without verifying what it was actually trained on. This leaves fundamental questions, such as whether a correct response reflects genuine generalization or rote memorization, grounded in speculation rather than evidence. To resolve these ambiguities, we introduce LUMOS, a diagnostic framework that traces knowledge along the causal chain from training-data exposure to behavioral output, leveraging OLMo 2 with its fully transparent training corpus. By grounding analysis in verified exposure, we reveal that models internally encode rare facts with high separability (84%) yet fail to express them behaviorally (54%), though this retrieval gap narrows with scale. Furthermore, when models are asked to self-reflect on their own answers, they perform reliably on trained content (83%) but drop to random-baseline levels (49%) on unseen content. This collapse persists even under chain-of-thought prompting, which inflates confidence signals rather than improving calibration. Collectively, these findings demonstrate that incorporating the training-data axis into LLM evaluation transforms speculative diagnoses into verifiable claims, and we advocate that this axis should be a standard component of knowledge assessment in LLMs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.02902 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:45:55.648Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).