Metraipolis › archive › dc_3eb75fc59fb0

Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue — arXiv:2610.08703

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.08703
License (as archived)
CC0-1.0
Archived
2026-10-07T11:23:29.070Z
SHA-256 of the archived text
0f7df23bd6ed4957e47839bc47bbc8706a60ab19c51ab4e334cea4646ada806a
Stable URL of this page
https://metraipolis.com/documento/dc_3eb75fc59fb0

How to cite

Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue — arXiv:2610.08703 — arXiv (en), https://arxiv.org/abs/2610.08703 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_3eb75fc59fb0 (Metraipolis archive dc_3eb75fc59fb0, sha256 0f7df23bd6ed4957e47839bc47bbc8706a60ab19c51ab4e334cea4646ada806a)
BibTeX
@misc{dc3eb75fc59fb0,
  title = {Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue — arXiv:2610.08703},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.08703},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_3eb75fc59fb0; sha256 0f7df23bd6ed4957e47839bc47bbc8706a60ab19c51ab4e334cea4646ada806a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.08703. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

In K-12 mathematics tutoring, student-tutor dialogue provides rich evidence of learners' problem-solving processes and sources of difficulty. Learning analytics research increasingly relies on large language models (LLMs) to extract such information from dialogue for a variety of downstream tasks, including knowledge tracing, behavioral modeling, and diagnosis of student reasoning errors. However, the validity of these model-generated interpretations remains insufficiently understood. In this exploratory study, we examine the validity of LLM classifications of five student failure modes in mathematics tutoring dialogue using an operational diagnostic codebook: uncertainty, misattribution, operator selection, conceptual gap, and procedural slip. Across models, human-LLM agreement was moderate (kappa = .524-.597), while cross-model agreement was substantially higher (kappa = .755-.781; alpha = .769). These findings show that cross-model agreement can create a misleading appearance of correctness, challenging the assumption that consensus among LLMs constitutes evidence of valid learner interpretation. For learning analytics, the implication is clear: scalable labeling is useful only if the inferred constructs are valid, and model consensus cannot substitute for independent evidence of that validity.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.08703 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T11:15:39.462Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).