Metraipolis › archive › dc_a88cbfd9dbeb

Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior — arXiv:2608.12292

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.12292
License (as archived)
CC0-1.0
Archived
2026-10-19T05:31:19.304Z
SHA-256 of the archived text
d7456fc1f7a48caf826ead4fcbd6fc379817c998162b7891a1fc2d07b4b7e553
Stable URL of this page
https://metraipolis.com/documento/dc_a88cbfd9dbeb

How to cite

Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior — arXiv:2608.12292 — arXiv (en), https://arxiv.org/abs/2608.12292 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_a88cbfd9dbeb (Metraipolis archive dc_a88cbfd9dbeb, sha256 d7456fc1f7a48caf826ead4fcbd6fc379817c998162b7891a1fc2d07b4b7e553)
BibTeX
@misc{dca88cbfd9dbeb,
  title = {Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior — arXiv:2608.12292},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12292},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_a88cbfd9dbeb; sha256 d7456fc1f7a48caf826ead4fcbd6fc379817c998162b7891a1fc2d07b4b7e553},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12292. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An effective large language model (LLM) tutor must often decline to give an answer it could easily produce. In a randomized study, students who used an unguarded chatbot scored higher while practicing but lower on a later test taken without it, whereas a Socratically guarded version of the same model kept the practice gain and removed the later loss [4]. Reliable answer-withholding is therefore central to a tutor's value, yet a capable model pressed by a frustrated student does not withhold reliably on a prompt alone. We report a deployed tutoring system that enforces answer-withholding as a per-turn, machine-checkable contract, and a method for tuning that withholding against evidence. A non-LLM policy core, reading only trusted learner state, sets a per-turn ceiling on an eight-rung help ladder; a deterministic detector strips solution code; and a separate LLM judge checks each risky reply against the contract. We tune the behavior with an automated evaluation that uses no human subjects: scripted student personas are driven through the live pipeline and re-scored by a stronger model, and we record each rejection's stated reason so failures are fixed by cause. Doing so revealed an interpretable "over-help ladder," from blatant solution leaks, to naming the exact bug, to over-citing general facts, with each fix exposing the next. The tutor reached full compliance on all four acceptance criteria. We offer the measure, diagnose, and fix loop as a reusable recipe for any LLM agent that must refuse a capability it has.

Tema degli esploratori: governance digitale e beni comuni (retro-catalogo arXiv: annunci del 2026-08-13 in cs.CY).
Categorie arXiv: Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2608.12292 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:16:42.614Z dalla pagina https://arxiv.org/catchup/cs.CY/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).