Metraipolis › archive › dc_538a135f2e97

Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions — arXiv:2609.27749

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.27749
License (as archived): CC0-1.0
Archived: 2026-10-08T09:11:31.871Z
SHA-256 of the archived text: e9e3f42e972c617126575892e43475f4d1ce90e28c0a810139dea38c700e262a
Stable URL of this page: https://metraipolis.com/documento/dc_538a135f2e97

How to cite

Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions — arXiv:2609.27749 — arXiv (en), https://arxiv.org/abs/2609.27749 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_538a135f2e97 (Metraipolis archive dc_538a135f2e97, sha256 e9e3f42e972c617126575892e43475f4d1ce90e28c0a810139dea38c700e262a)

BibTeX

@misc{dc538a135f2e97,
  title = {Evaluation of pre-trained models for pedagogical assessment of novel AI-assisted educational questions — arXiv:2609.27749},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.27749},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_538a135f2e97; sha256 e9e3f42e972c617126575892e43475f4d1ce90e28c0a810139dea38c700e262a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27749. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The surge in AI-assisted generation of educational materials has outpaced our capacity to validate their pedagogical quality. Automated evaluation using Bloom Classifier models is a promising approach to assess educational materials at scale. These models show high accuracy within-distribution dataset (IID Dataset). However, applying the same models to new out-of-distribution (OOD) datasets such as AI-assisted generated questions could show performance degradation. To identify robust classifiers under dataset shift, we evaluated traditional Machine Learning (ML), transformer, and Large Language models on the Bloom level classification task. We also explored feature-engineering strategies incorporating NLP metrics, appending the learning objectives as part of the input, and text splicing to stabilize OOD performance. Our baseline tests show that TFPOS-IDF ML models perform poorly on OOD (Macro F1-score 0.48) compared to BERT (0.55) and LLMs (0.79). Text splicing improved macro F1-score performance of ML and BERT models (0.59 and 0.62, respectively). Appending the learning objectives with the input increased model performance on specific dataset. Model retraining provided the largest improvement across models and datasets. Overall, these findings highlight the trade-off on the use of pre-trained models with novel AI-assisted educational questions and how strategic feature enhancements help address loss in performance.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-24 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.27749 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T08:13:26.529Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).