Metraipolis › archive › dc_d53afbc404e1

Ask-E: An Environment for Calibrated Question Generation — arXiv:2608.06933

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.06933
License (as archived)
CC0-1.0
Archived
2026-10-19T09:46:52.902Z
SHA-256 of the archived text
48330af0daa98d763876b41581da31bd8443c81df3eb5acdc94a01dc4b7569d9
Stable URL of this page
https://metraipolis.com/documento/dc_d53afbc404e1

How to cite

Ask-E: An Environment for Calibrated Question Generation — arXiv:2608.06933 — arXiv (en), https://arxiv.org/abs/2608.06933 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d53afbc404e1 (Metraipolis archive dc_d53afbc404e1, sha256 48330af0daa98d763876b41581da31bd8443c81df3eb5acdc94a01dc4b7569d9)
BibTeX
@misc{dcd53afbc404e1,
  title = {Ask-E: An Environment for Calibrated Question Generation — arXiv:2608.06933},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.06933},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d53afbc404e1; sha256 48330af0daa98d763876b41581da31bd8443c81df3eb5acdc94a01dc4b7569d9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.06933. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Today, we improve models by training and evaluating them on problems at the frontier of their abilities. Creating such problems is itself a demanding task, requiring the ability to probe model limits and generalize beyond existing question distributions. It also means placing problems at a precise difficulty level, which requires understanding what it takes to solve them. In short, generating problems calibrated to a model's current frontier demands capability beyond it, an increasingly burdensome constraint as models improve. Our key insight is that we can leverage this constraint to our advantage: a model that can generate problems consistently calibrated to a given frontier must possess capability beyond it. Accordingly, we present Ask-E, an environment that benchmarks and trains models on their ability to write questions at a given skill level, rather than answer them. Concretely, we define target skill levels as ranges bounded by the capabilities of two existing language models. A generated question is successfully calibrated if exactly one of the two models can solve it, placing it precisely within the target range and differentiating the capabilities of these models. Ask-E serves both as a benchmark and a training environment, where models generate problems calibrated to a variety of skill levels. We find that even frontier models achieve below 50% calibration on the benchmark, leaving significant headroom to measure future progress. We also show that training on this environment leads to improvements across a number of downstream math benchmarks even with no new math data, no interaction with stronger models, and no correctness-based reward.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-10 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.06933 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:46:37.227Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).