Metraipolis › archive › dc_79eaf1ce339d

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams — arXiv:2609.29333

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.29333
License (as archived)
CC0-1.0
Archived
2026-10-08T04:27:26.501Z
SHA-256 of the archived text
ca4fceb87c6c25f27c5c53c4d9027a7ea24db575496765817c542c7bad4d65b3
Stable URL of this page
https://metraipolis.com/documento/dc_79eaf1ce339d

How to cite

Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams — arXiv:2609.29333 — arXiv (en), https://arxiv.org/abs/2609.29333 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_79eaf1ce339d (Metraipolis archive dc_79eaf1ce339d, sha256 ca4fceb87c6c25f27c5c53c4d9027a7ea24db575496765817c542c7bad4d65b3)
BibTeX
@misc{dc79eaf1ce339d,
  title = {Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams — arXiv:2609.29333},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.29333},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_79eaf1ce339d; sha256 ca4fceb87c6c25f27c5c53c4d9027a7ea24db575496765817c542c7bad4d65b3},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.29333. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

One long-form exam in a large course costs hundreds of grader-hours, and qualified graders are scarce; LLM graders are a tempting alternative. To show its pitfalls we grade a practical Computer Vision exam ($570$ dual-graded students) under $171$ configurations spanning closed and open-weights models; the best reaches mean absolute error $1.64/35$, below the $2.61/35$ two human graders achieve against each other. The catch is the prompt: a short ''strict grader'' preamble drives $14$ of $17$ open-weights models out of the graded band ($\text{MAE} \ge 8$), three stopping grading altogether. The damage traces to the preamble's two credit-withholding sentences, not to tone or model scale; one of them, ''never give partial credit'', alone makes two of three probed models stop grading. The closed flagships of three vendors shift calibration under it but stay in the band. In $162$ further configurations on a second, independent Machine Learning exam from another course ($1{,}038$ dual-graded students), the preamble worsens ten models, moving three out of the band into collapse and one into refusal, yet improves seven whose neutral prompts over-mark: the vulnerability replicates, but its direction is exam-specific. Light LoRA fine-tuning repairs it: one adapter on the two exams' pooled $\sim 3{,}900$ graded examples brings five small open models to parity or better with a human grader in agreement with the grader pair, and sensitivity to the three harsh personas nearly vanishes ($\le 0.32$ MAE). We release the anonymised dataset, full ablation grid, and grading, fine-tuning and analysis pipelines.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-25 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2609.29333 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T03:50:58.042Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-25?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).