Metraipolis › archive › dc_8bde18a01280

Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science — arXiv:2609.14789

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.14789
License (as archived)
CC0-1.0
Archived
2026-10-15T23:54:44.290Z
SHA-256 of the archived text
172d94dcb29fc01a04f878645639935203d4bfbe78d686b6d58c4facc51b1e70
Stable URL of this page
https://metraipolis.com/documento/dc_8bde18a01280

How to cite

Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science — arXiv:2609.14789 — arXiv (en), https://arxiv.org/abs/2609.14789 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8bde18a01280 (Metraipolis archive dc_8bde18a01280, sha256 172d94dcb29fc01a04f878645639935203d4bfbe78d686b6d58c4facc51b1e70)
BibTeX
@misc{dc8bde18a01280,
  title = {Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science — arXiv:2609.14789},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.14789},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8bde18a01280; sha256 172d94dcb29fc01a04f878645639935203d4bfbe78d686b6d58c4facc51b1e70},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14789. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Artificial intelligence (AI) systems in education are developing on timescales that sit uneasily with conventional evaluation. By the time a large-scale trial has been designed, delivered, analysed and published, the technology under study may have changed materially. This creates a temporal problem for evidence-informed education: the need for timely evidence can encourage reliance on weak observational or usage data, while conventional rigorous evaluation may produce evidence too slowly to guide rapidly evolving practice. We examine teacher-led micro-randomised controlled trials (micro-RCTs) as one response to this problem. The empirical case is a four-week multisite individually randomised evaluation of Medly, an AI-powered tutoring platform, in GCSE Biology, Chemistry and Physics in English secondary schools. Of 929 students completing baseline assessment, 644 completed post-testing. In the primary ITT analysis, students allocated to Medly achieved higher post-test attainment than students undertaking business-as-usual self-directed revision (Hedges' g = 0.33, 95% CI 0.18 to 0.48). Positive estimates were observed in Physics (g = 0.31), Chemistry (g = 0.32) and Biology (g = 0.52), with no evidence of differential impact by disadvantage status. Greater platform engagement was associated with higher attainment, but these post-randomisation analyses are treated as exploratory rather than causal. Attrition was substantial (30.7%), outcome measures were curriculum-aligned rather than standardised, and process evaluation response was limited. We therefore interpret the findings as preliminary. We argue that the value of micro-RCTs for educational AI lies not in replacing definitive evaluation with small studies, but in enabling a rapid, cumulative evaluation architecture in which randomised estimates can be generated, replicated and updated as technologies and their implementation evolve.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.HC).
Categorie arXiv: Human-Computer Interaction (cs.HC).
Pagina del lavoro: https://arxiv.org/abs/2609.14789 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T23:24:31.368Z dalla pagina https://arxiv.org/catchup/cs.HC/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).