Metraipolis › archive › dc_1983d69bec4e

GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs — arXiv:2610.41704

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.41704
License (as archived): CC0-1.0
Archived: 2026-11-03T07:15:30.825Z
SHA-256 of the archived text: d61380746f8c348dbf3848307275c0306c91ad5e6f59d6666777e6fb553b7b37
Stable URL of this page: https://metraipolis.com/documento/dc_1983d69bec4e

How to cite

GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs — arXiv:2610.41704 — arXiv (en), https://arxiv.org/abs/2610.41704 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1983d69bec4e (Metraipolis archive dc_1983d69bec4e, sha256 d61380746f8c348dbf3848307275c0306c91ad5e6f59d6666777e6fb553b7b37)

BibTeX

@misc{dc1983d69bec4e,
  title = {GraphNIAH: A Benchmark for Long-Context Reasoning over Real-World Graphs — arXiv:2610.41704},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.41704},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1983d69bec4e; sha256 d61380746f8c348dbf3848307275c0306c91ad5e6f59d6666777e6fb553b7b37},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41704. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The needle-in-a-haystack (NIAH) benchmark was introduced as a test for measuring a model's ability to find small relevant facts within a very large body of input text. However, existing NIAH benchmarks each suffer from one or more of the following limitations: (1) reliance on synthetic data, so their evaluations may not generalize to real-world settings; (2) reuse of previously published datasets, leading to the risk of benchmark contamination; (3) dependence on manual annotation, which requires human effort to create and update data and is prone to future benchmark contamination; and (4) unrealistic simplification of long-context multi-hop reasoning where each next step is trivial given previous steps, while real-world reasoning often requires planning over multiple possible proof paths. To address these limitations, we introduce \Benchmark, a dynamic benchmark built from real citation relationships among scientific papers published after the evaluated models' knowledge cutoff date, which utilizes real-world information while ensuring no benchmark contamination. The fully automated benchmark construction pipeline samples representative subgraphs of the citation graph formed by these papers and converts each subgraph into a long-context QA example with a verifiable answer. We use GraphNIAH to evaluate eight LLMs from the GPT-5.6, Gemini-3, and Qwen3.5 families. We find that accuracy falls as questions require longer reasoning paths: at question depths of 16 and 32, the GPT-5.6 and Qwen3.5 models answer YES/NO questions no better than random guessing, mostly by answering NO; four of the eight models answer no counting example correctly, and the others model get worse as the subgraph size increases. These findings suggest that current LLMs often stop searching the citation relationships in their context before they reach the answer, even though every fact they need is in the context.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.41704 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:09:46.670Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).