Metraipolis › archive › dc_7d5216418571

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge — arXiv:2610.28340

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.28340
License (as archived)
CC0-1.0
Archived
2026-10-21T07:09:08.242Z
SHA-256 of the archived text
82864c62cffbf302acfc0ab25f3b744e96d3e061113518e772c2d9f350408b53
Stable URL of this page
https://metraipolis.com/documento/dc_7d5216418571

How to cite

DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge — arXiv:2610.28340 — arXiv (en), https://arxiv.org/abs/2610.28340 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d5216418571 (Metraipolis archive dc_7d5216418571, sha256 82864c62cffbf302acfc0ab25f3b744e96d3e061113518e772c2d9f350408b53)
BibTeX
@misc{dc7d5216418571,
  title = {DeclineCraft: A RAG Abstention Benchmark for Jointly Measuring Combined Retrieval and Parametric Knowledge — arXiv:2610.28340},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.28340},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d5216418571; sha256 82864c62cffbf302acfc0ab25f3b744e96d3e061113518e772c2d9f350408b53},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.28340. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Reliable retrieval-augmented generation (RAG) requires recognizing when neither retrieved documents nor parametric knowledge supports an answer. We introduce DeclineCraft, a dynamic abstention benchmark whose unanswerable questions target Minecraft feature additions after a corpus cut-off, with audits of possible parametric exposure. Later cut-offs support evaluation of future models. Across 27 LLMs, we measure Correct Abstention Rate (CAR; how often models acknowledge insufficient knowledge on unanswerable questions), Answerable Attempt Rate and Answer Accuracy. We find that (1) single-shot retrieval significantly raises CAR in 38 of 40 configurations (model and thinking setting) versus no retrieval. (2) Iterative RAG, in which the model self-assesses its context and can request up to two further retrieval rounds, significantly lowers CAR versus single-shot RAG in 30 of 40 configurations (mean 46.4%), with little change in Answer Accuracy (68.5% to 69.3%). Adding query rewriting and reranking to hybrid (dense & sparse) retrieval lowers CAR for all 12 models tested. (3) Larger models answer more accurately (Spearman $\rho = +0.60$ with total parameters), but neither model size nor release date shows a significant association with CAR. (4) Without retrieval, enabling thinking on the same weights lowers CAR significantly for 8 of 13 models (by 14% points on average), and the responses shift from acknowledged uncertainty to premise denial.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Information Retrieval (cs.IR).
Pagina del lavoro: https://arxiv.org/abs/2610.28340 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-21T07:05:58.152Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).