Metraipolis › archive › dc_468bd192cf39
Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses — arXiv:2609.27063
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.27063
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T10:07:57.776Z
- SHA-256 of the archived text
82230e7e8a1d61b30e1545b2491f8cc0347326977360d6a1c4f843ab1f1d496b- Stable URL of this page
https://metraipolis.com/documento/dc_468bd192cf39
How to cite
Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses — arXiv:2609.27063 — arXiv (en), https://arxiv.org/abs/2609.27063 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_468bd192cf39 (Metraipolis archive dc_468bd192cf39, sha256 82230e7e8a1d61b30e1545b2491f8cc0347326977360d6a1c4f843ab1f1d496b)
BibTeX
@misc{dc468bd192cf39,
title = {Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses — arXiv:2609.27063},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.27063},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_468bd192cf39; sha256 82230e7e8a1d61b30e1545b2491f8cc0347326977360d6a1c4f843ab1f1d496b},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.27063. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
This paper presents a multi-source classroom study conducted during a 10-week quarter in data science courses at Drexel University. We first investigate the behaviors of student engagement with large language models (LLMs) using four surveys across three data science courses. Second, we evaluate the construct validity of multiple-choice questions (MCQs) generated by an LLM for in-lecture retrieval practice. Based on 378 authored questions (311 deployed, producing 7{,}888 student responses), we analyze whether the difficulty ratings assigned by an LLM match empirical item difficulty. Our study shows that student engagement with LLMs varied across courses and increased over the term. Although students expressed high satisfaction and reported saving considerable time, their perception of deep learning benefits declined, and many noted a tendency toward over-reliance. Regarding the difficulty ratings of LLM-generated MCQs, the Easy, Medium, and Hard labels correlated closely with its assigned Bloom's Taxonomy levels (Spearman $\rho=0.90$), reflecting an artifact of co-generation. However, neither metric predicted empirical item difficulty (difficulty label $\rho=0.06$; Bloom level $\rho=0.02$). The ratings reflect the structural formatting of a question rather than its underlying difficulty.
Tema degli esploratori: governance digitale e beni comuni (retro-catalogo arXiv: annunci del 2026-09-24 in cs.CY).
Categorie arXiv: Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2609.27063 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T09:15:10.081Z dalla pagina https://arxiv.org/catchup/cs.CY/2026-09-24?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).