Metraipolis › archive › dc_fd01afb34e38
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models — arXiv:2609.38543
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.38543
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T13:13:25.974Z
- SHA-256 of the archived text
1f97547f3c4d70f2e21db167d069fda2b9a563e0aaed7937bdaaf43fd33e8540- Stable URL of this page
https://metraipolis.com/documento/dc_fd01afb34e38
How to cite
MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models — arXiv:2609.38543 — arXiv (en), https://arxiv.org/abs/2609.38543 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd01afb34e38 (Metraipolis archive dc_fd01afb34e38, sha256 1f97547f3c4d70f2e21db167d069fda2b9a563e0aaed7937bdaaf43fd33e8540)
BibTeX
@misc{dcfd01afb34e38,
title = {MedKIT: Evaluating Knowledge Integration and Generalization in Large Language Models — arXiv:2609.38543},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.38543},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd01afb34e38; sha256 1f97547f3c4d70f2e21db167d069fda2b9a563e0aaed7937bdaaf43fd33e8540},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.38543. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Constantly evolving real-world knowledge necessitates models to be updated continuously. Especially in medicine, as clinical evidence changes over time, outdated knowledge can pose safety risks. Existing evaluations of knowledge integration focus on factual recall, offering limited insight into whether newly integrated knowledge is actually usable. Our benchmark MedKIT (Medical Knowledge Integration and Transfer) provides a granular evaluation of how models integrate and apply knowledge under realistic sequences of clinical updates. Each instance corresponds to a factual update derived from clinical evidence, paired with targeted probes that assess transfer across lexical variation, relational transformations, compositional reasoning, and open-ended operationalization, as well as locality tests for knowledge preservation. Using MedKIT, we conduct a large-scale empirical study of 12 knowledge integration strategies across 5 diverse models, including both general-purpose and medical LLMs. Our results reveal a consistent gap between recall and usable knowledge: while most methods achieve strong gains on the original update task and under lexical variation, relational generalization is limited, and no method yields meaningful improvements on compositional or operational tasks. These findings highlight a fundamental challenge in knowledge integration and position MedKIT as a testbed for developing methods that make newly integrated knowledge more consistently usable across tasks and contexts. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.CL). Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.38543 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T12:44:51.738Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).