Metraipolis › archive › dc_5958dee83a99

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language — arXiv:2610.01627

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.01627
License (as archived)
CC0-1.0
Archived
2026-10-06T12:14:35.766Z
SHA-256 of the archived text
1dd9f76ba66c4ff8d9ebe0872e8891368483bc6cc38dac6ae8eda44528b90b4e
Stable URL of this page
https://metraipolis.com/documento/dc_5958dee83a99

How to cite

What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language — arXiv:2610.01627 — arXiv (en), https://arxiv.org/abs/2610.01627 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5958dee83a99 (Metraipolis archive dc_5958dee83a99, sha256 1dd9f76ba66c4ff8d9ebe0872e8891368483bc6cc38dac6ae8eda44528b90b4e)
BibTeX
@misc{dc5958dee83a99,
  title = {What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language — arXiv:2610.01627},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.01627},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5958dee83a99; sha256 1dd9f76ba66c4ff8d9ebe0872e8891368483bc6cc38dac6ae8eda44528b90b4e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.01627. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Difficulty is one of the most fundamental properties of a question: it determines whether the question can meaningfully discriminate between models of differing ability. Although a variety of methods can now estimate or predict difficulty automatically, they yield only a single descriptive number, with no account of the underlying factors that make a question difficult in the first place. In this work, we propose a data-driven approach that automatically generates and validates natural-language hypotheses explaining what makes one question harder than another. We first estimate each item's difficulty from the responses of a large pool of LLMs using Item Response Theory. We then sample contrasting sets of easy and hard questions and prompt an LLM to propose candidate explanations of the difference, which are subsequently validated and selected on held-out questions. Experimental results across three datasets spanning mathematical, logical, and commonsense reasoning show that our method produces interpretable and predictive hypotheses. On their own, they predict the difficulty of unseen questions competitively with, or better than, advanced black-box difficulty regressors; used as additional features, they further improve those regressors, implying that they discover difficulty signals that existing models fail to capture. Moreover, we demonstrate that editing questions according to a hypothesis can shift their measured difficulty in the expected direction, indicating that the discovered hypotheses are causally valid difficulty factors rather than post-hoc descriptions. Our approach thus turns a purely descriptive difficulty score into actionable statements.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.01627 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.204Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).