Metraipolis › archive › dc_076254fef7ae
The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale — arXiv:2609.26087
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.26087
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T13:45:28.897Z
- SHA-256 of the archived text
a9b96fe3237a3b803f2d7f0c3a1386ee4d3a753a25a58939717837e8fb9ad144- Stable URL of this page
https://metraipolis.com/documento/dc_076254fef7ae
How to cite
The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale — arXiv:2609.26087 — arXiv (en), https://arxiv.org/abs/2609.26087 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_076254fef7ae (Metraipolis archive dc_076254fef7ae, sha256 a9b96fe3237a3b803f2d7f0c3a1386ee4d3a753a25a58939717837e8fb9ad144)
BibTeX
@misc{dc076254fef7ae,
title = {The Architect, the Adversary, and the Judge: Closed-Loop Generation of Standards-Aligned Assessment Items at Scale — arXiv:2609.26087},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.26087},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_076254fef7ae; sha256 a9b96fe3237a3b803f2d7f0c3a1386ee4d3a753a25a58939717837e8fb9ad144},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.26087. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We present CLAIM, a production pipeline for K-12 assessment-item generation coupling a two-stage generate-then-attack protocol (the model drafts as a "curriculum architect", then re-enters the same conversation as a hostile adversarial reviewer), bi-directional few-shot conditioning on accepted and rejected items, the latter carrying the evaluator's diagnosis, and a knowledge dictionary of 44,844 error-correction rules mined from that feedback and retrieved per standard and item type. Across 43,227 scored items over 755 Common Core ELA standards, three item types, and ten LLMs, the pipeline reaches a 97.8% expert-evaluator pass rate on a 9,074-item production run. We then ask what that rate certifies. Re-scoring a stratified sample with three judges from other vendors, blind to the deployed verdict, reproduces the format ordering under every judge and recovers a larger open-set deficit than the deployed evaluator does; but agreement on the accept/reject binary is weak at production prevalence (kappa about 0.13), and the judges agree with each other no better. The level is therefore judge-relative, and with no student-response data our quality evidence is evaluator-judged throughout. The corpus also exposes a robust asymmetry. Multiple-choice and multiple-select generation saturate at 98% or above for both frontier models under a dozen static rules, whereas fill-in-the-blank generation is capability-tiered (82.8-96.7% across five models under a matched rule set, standards, and judge) and plateaus under prompt-only optimization, with error mass shifting between answer-key over-inclusion and omission as rules accumulate. We analyze this as open-set boundary determination, a task autoregressive decoders are structurally ill-equipped to solve, and show the asymmetry recurring when the evaluator itself is distilled: fail-recall rises from 8% to 63% while F1 saturates at 0.25. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-23 in cs.AI). Categorie arXiv: Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.26087 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T13:18:38.139Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-23?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).