Metraipolis › archive › dc_e6bdb2053540

E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation — arXiv:2610.05048

Source
arXiv
Original
https://arxiv.org/abs/2610.05048
License (as archived)
CC0-1.0
Archived
2026-10-06T05:26:38.541Z
SHA-256 of the archived text
166ea1de1bd6fd5a844bdd9efcd5dd032fa03a4adf95d28c9ab7d1b966811240
Stable URL of this page
https://metraipolis.com/documento/dc_e6bdb2053540

How to cite

E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation — arXiv:2610.05048 — arXiv, https://arxiv.org/abs/2610.05048 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e6bdb2053540 (Metraipolis archive dc_e6bdb2053540, sha256 166ea1de1bd6fd5a844bdd9efcd5dd032fa03a4adf95d28c9ab7d1b966811240)
BibTeX
@misc{dce6bdb2053540,
  title = {E$^2$-OPSD: Taming Entropy Overshoot in On-Policy Self-Distillation — arXiv:2610.05048},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.05048},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e6bdb2053540; sha256 166ea1de1bd6fd5a844bdd9efcd5dd032fa03a4adf95d28c9ab7d1b966811240},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.05048. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

On-policy self-distillation (OPSD) provides dense token-level supervision without a second model: one network acts as teacher with the reference solution and as student with only the problem. We identify a specific failure mode of this recipe. During training, student token entropy rises past the teacher's and remains elevated, a pattern we call entropy overshoot. We trace it to both sides of distillation. The reference-conditioned teacher is confident along its answer-directed reasoning path, but this confidence transfers poorly to student-generated prefixes, making its supervision overly tied to answer-specific cues rather than reusable reasoning patterns; meanwhile, the forward KL used by OPSD continually diffuses the student's predictive distribution without pulling it back. We introduce E$^2$-OPSD to address both causes. Exemplar-guided teaching replaces the current answer with a retrieved solved neighboring problem, providing transferable reasoning guidance without revealing the destination and better matching student-reachable states. Entropy-aware distillation uses the student-teacher entropy gap to determine the direction and strength of each token's correction. E$^2$-OPSD improves math reasoning by up to 4.3 points in mean@16 over OPSD, while out-of-domain evaluations show gains over the corresponding base models of up to 4.9 points in mean@16 and 5.5 points in pass@8. Despite these gains, E$^2$-OPSD remains simple, requiring no additional forward passes or networks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.05048 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.959Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).