Metraipolis › archive › dc_65302ad98e64

Simulating Disengaged Students to Evaluate LLM-based Tutors — arXiv:2609.12331

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.12331
License (as archived)
CC0-1.0
Archived
2026-10-16T03:15:37.819Z
SHA-256 of the archived text
5d96236a7c1696c9491234fb8de8fe7f4d153ad9a0c770f480e2dbb1fa68995b
Stable URL of this page
https://metraipolis.com/documento/dc_65302ad98e64

How to cite

Simulating Disengaged Students to Evaluate LLM-based Tutors — arXiv:2609.12331 — arXiv (en), https://arxiv.org/abs/2609.12331 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_65302ad98e64 (Metraipolis archive dc_65302ad98e64, sha256 5d96236a7c1696c9491234fb8de8fe7f4d153ad9a0c770f480e2dbb1fa68995b)
BibTeX
@misc{dc65302ad98e64,
  title = {Simulating Disengaged Students to Evaluate LLM-based Tutors — arXiv:2609.12331},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.12331},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_65302ad98e64; sha256 5d96236a7c1696c9491234fb8de8fe7f4d153ad9a0c770f480e2dbb1fa68995b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12331. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Simulated students generated by computational models provide a practical way to evaluate tutoring strategies and pedagogical approaches used by human and AI tutors. However, such simulations should account for disengaged behaviors, including gaming the system, wheel-spinning, and off-task behavior, because tutors may need different responses for different learner states. We present Disengagement-Aware Student Simulators (DAS2), a reproducible pre-deployment protocol that models five learner-engagement states: engaged, gaming, wheel-spinning, off-task, and mixed, and evaluates AI tutor performance across these states. Using ASSISTments09, two coders independently labeled 100 sampled tutoring sessions based on anonymized interaction-log summaries. They achieved 84% agreement (Cohen's kappa = 0.78), and among agreed cases, human consensus labels matched DAS2 rule-based labels in 81% of cases (kappa = 0.75). Conditioning simulations on intended learner states reduced the correctness-rate gap between simulated and authentic sessions from 0.54 to 0.20 for gaming and from 0.51 to 0.18 for wheel-spinning. Fine-tuned Qwen2.5-7B better matched authentic response-time distributions, while prompt-only GPT-4o generated more distinguishable learner states. Evaluation of five AI tutors from the Claude, Llama, Gemini, Qwen, and GPT families showed that relative rankings remained stable across learner states and interaction lengths, while absolute performance varied, revealing state-specific differences in tutor support. Human validation further showed that automated tutor evaluation does not fully align with human judgment. DAS2 provides a pre-deployment framework for evaluating how AI tutors respond to diverse learner-engagement states before deployment.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.12331 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:05:44.034Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).