Metraipolis › archive › dc_e8e1faa6bb3f
MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects — arXiv:2608.29477
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29477
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:47:49.516Z
- SHA-256 of the archived text
bd98bf2d76da2c82af50891ab13160bd260c751c8bd388ba8a049651014d84fa- Stable URL of this page
https://metraipolis.com/documento/dc_e8e1faa6bb3f
How to cite
MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects — arXiv:2608.29477 — arXiv (en), https://arxiv.org/abs/2608.29477 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e8e1faa6bb3f (Metraipolis archive dc_e8e1faa6bb3f, sha256 bd98bf2d76da2c82af50891ab13160bd260c751c8bd388ba8a049651014d84fa)
BibTeX
@misc{dce8e1faa6bb3f,
title = {MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects — arXiv:2608.29477},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29477},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e8e1faa6bb3f; sha256 bd98bf2d76da2c82af50891ab13160bd260c751c8bd388ba8a049651014d84fa},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29477. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robustness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and provenance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2608.29477 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T07:28:26.768Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).