Metraipolis › archive › dc_296ef71cc877

Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education — arXiv:2608.04006

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.04006
License (as archived): CC0-1.0
Archived: 2026-10-31T12:41:31.583Z
SHA-256 of the archived text: 6205047056eaf30a9c968e5dbde74c60e7385562feaeeff3bf8cc24827622f36
Stable URL of this page: https://metraipolis.com/documento/dc_296ef71cc877

How to cite

Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education — arXiv:2608.04006 — arXiv (en), https://arxiv.org/abs/2608.04006 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_296ef71cc877 (Metraipolis archive dc_296ef71cc877, sha256 6205047056eaf30a9c968e5dbde74c60e7385562feaeeff3bf8cc24827622f36)

BibTeX

@misc{dc296ef71cc877,
  title = {Calibrating Trustworthiness: Co-Designing Metrics and Visualizations for Evaluating LLMs in Education — arXiv:2608.04006},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.04006},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_296ef71cc877; sha256 6205047056eaf30a9c968e5dbde74c60e7385562feaeeff3bf8cc24827622f36},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.04006. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

LLMs are reshaping educational technology, yet evaluating their responses for pedagogical alignment remains underexplored, relying heavily on the expertise of learning engineers building the technology. To bridge this gap, we explore trustworthiness as a structured lens for evaluation, leveraging existing measures of LLM trustworthiness to systematically identify potential pedagogical disruptions. Through a longitudinal co-design process with learning engineers developing an LLM-powered digital textbook, we: (1) co-constructed five trustworthiness metrics comprising 20 measures tailored to pedagogical use; (2) designed visualizations that map trustworthiness violations onto LLM responses; and (3) evaluated how these tools help learning engineers make A/B comparisons of LLM responses. Making trustworthiness explicit increased inter-rater reliability while helping learning engineers resolve conflicting objectives and produce more consistent judgments. We discuss the emergent benefits of trustworthiness as a lens for evaluating LLMs in education and propose new design guidelines for future evaluation tools that enable pedagogically-aligned, LLM-powered learning tools.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.HC).
Categorie arXiv: Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2608.04006 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:31:08.696Z dalla pagina https://arxiv.org/catchup/cs.HC/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).