Metraipolis › archive › dc_e7e56337dbb4

Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models — arXiv:2610.34005

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.34005
License (as archived)
CC0-1.0
Archived
2026-10-27T08:00:37.601Z
SHA-256 of the archived text
8694d35e0fedc76008942fd21ada313696559871577ea0e78893ff98a6e72266
Stable URL of this page
https://metraipolis.com/documento/dc_e7e56337dbb4

How to cite

Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models — arXiv:2610.34005 — arXiv (en), https://arxiv.org/abs/2610.34005 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e7e56337dbb4 (Metraipolis archive dc_e7e56337dbb4, sha256 8694d35e0fedc76008942fd21ada313696559871577ea0e78893ff98a6e72266)
BibTeX
@misc{dce7e56337dbb4,
  title = {Steering the Teacher: Improving On-Policy Self-Distillation for Thinking Models — arXiv:2610.34005},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.34005},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e7e56337dbb4; sha256 8694d35e0fedc76008942fd21ada313696559871577ea0e78893ff98a6e72266},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.34005. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy self-distillation (OPSD) trains a student on its own reasoning trajectories using token-level supervision from a reference-conditioned teacher initialized from the same model. However, access to the reference can induce teacher behaviors that undermine independent reasoning in thinking models, leading to reference hallucinations, where the student cites an unavailable reference to justify reasoning steps, and reconsideration suppression, where the teacher discourages verification and revision at uncertain steps. We introduce Steered-Teacher OPSD (ST-OPSD) to jointly address these failure modes through activation steering. ST-OPSD extracts two directions from model-generated reasoning trajectories and applies them to the teacher's hidden states during distillation target computation, suppressing reference-invoking behavior and encouraging reconsideration with marker-family calibration. The student learns from the resulting teacher distributions without being steered. Experiments on AIME and HMMT show that ST-OPSD improves accuracy over vanilla OPSD and other interventions while restoring reconsideration and reference-hallucination rates to base-model levels.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.34005 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-27T07:55:30.869Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).