Metraipolis › archive › dc_fd2725f44ed7
MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation — arXiv:2608.29623
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.29623
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T07:47:55.643Z
- SHA-256 of the archived text
6f1413d45f35aae92e6129ffa323d89391449b4daff5214e7c094c77a1eb5bf8- Stable URL of this page
https://metraipolis.com/documento/dc_fd2725f44ed7
How to cite
MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation — arXiv:2608.29623 — arXiv (en), https://arxiv.org/abs/2608.29623 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fd2725f44ed7 (Metraipolis archive dc_fd2725f44ed7, sha256 6f1413d45f35aae92e6129ffa323d89391449b4daff5214e7c094c77a1eb5bf8)
BibTeX
@misc{dcfd2725f44ed7,
title = {MI-Distillation: Selecting from Model-Interpolated Instruct-Reasoning Data Spectrum for Chain-of-Thought Distillation — arXiv:2608.29623},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.29623},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fd2725f44ed7; sha256 6f1413d45f35aae92e6129ffa323d89391449b4daff5214e7c094c77a1eb5bf8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.29623. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recent advances in large reasoning models (LRMs) have shown strong performance on complex problems through long chain-of-thought (Long CoT) reasoning. However, distilling such trajectories into smaller student models remains challenging: direct Long CoT supervision often provides limited gains and can be less effective than concise Short CoT rationales. In this work, we investigate this phenomenon from a gradient-centric perspective. Our analysis shows that Long CoT induces larger gradient magnitudes and more concentrated update directions than Short CoT, with this effect becoming more pronounced as student model capacity increases. These findings suggest that effective Long CoT distillation requires balancing the reasoning information density of reasoning trajectories with their distributional alignment to the student model. Motivated by this insight, we propose \textbf{M}odel \textbf{I}nterporlation \textbf{Distillation} (\textbf{MI-Distillation}), a framework that constructs a continuous Instruct-Reasoning data spectrum through model interpolation. To select suitable trajectories from this spectrum, we further introduce \textbf{Seq}uential \textbf{L}earnable \textbf{S}urprisal \textbf{S}core (\textbf{SeqLSS}), which favors reasoning paths that are both informative and learnable for the student. Extensive experiments on reasoning benchmarks show that MI-Distillation consistently improves small model CoT distillation over strong Long CoT baselines.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.29623 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.788Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).