Metraipolis › archive › dc_83ffd292d238

Finding the Right Balance: Relevance and Diversity in LLM Retrieval — arXiv:2610.09412

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09412
License (as archived)
CC0-1.0
Archived
2026-10-08T11:26:31.451Z
SHA-256 of the archived text
407aa0e851c12a6bb035e8415c512e4b00bccac0d9d2c8edcb2e13b68b84b7d6
Stable URL of this page
https://metraipolis.com/documento/dc_83ffd292d238

How to cite

Finding the Right Balance: Relevance and Diversity in LLM Retrieval — arXiv:2610.09412 — arXiv (en), https://arxiv.org/abs/2610.09412 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_83ffd292d238 (Metraipolis archive dc_83ffd292d238, sha256 407aa0e851c12a6bb035e8415c512e4b00bccac0d9d2c8edcb2e13b68b84b7d6)
BibTeX
@misc{dc83ffd292d238,
  title = {Finding the Right Balance: Relevance and Diversity in LLM Retrieval — arXiv:2610.09412},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09412},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_83ffd292d238; sha256 407aa0e851c12a6bb035e8415c512e4b00bccac0d9d2c8edcb2e13b68b84b7d6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09412. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Retrieval diversification is widely available in retrieval-augmented generation (RAG) frameworks, yet prior studies disagree on whether it improves retrieval and answer quality. We show that its effectiveness varies primarily with candidate-pool redundancy, in a pattern consistent with the number of distinct evidence pieces a query requires. Using controlled near-duplicate injection and production-style overlapping chunking, we find that diversification harms relevance, evidence coverage and answer quality on clean pools, but becomes beneficial on multi-evidence tasks when redundancy causes nearest-neighbor retrieval to select repeated passages. We therefore introduce a query-adaptive rule that diversifies only when the effective number of distinct documents in the nearest-neighbor top-$k$ selection falls below the query's evidence requirement. Computed from existing embeddings, the rule captures most of the achievable gain, transfers across datasets and encoders and automatically reduces to nearest-neighbor retrieval for single-evidence queries. We also introduce RNG-Score, a geometric reranker with an exact nearest-neighbor fallback whose margin indicates duplicate structure. Overall, we conclude that diversification should be used selectively, based on observable redundancy and evidence requirements.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Information Retrieval (cs.IR) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.09412 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:24:06.374Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).