Metraipolis › archive › dc_dd9e05334342
Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation — arXiv:2608.27729
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.27729
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T10:56:24.242Z
- SHA-256 of the archived text
cef3891ac88822f8394a26e4c045ba28b422938c533362630e1c14b738c7a9da- Stable URL of this page
https://metraipolis.com/documento/dc_dd9e05334342
How to cite
Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation — arXiv:2608.27729 — arXiv (en), https://arxiv.org/abs/2608.27729 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dd9e05334342 (Metraipolis archive dc_dd9e05334342, sha256 cef3891ac88822f8394a26e4c045ba28b422938c533362630e1c14b738c7a9da)
BibTeX
@misc{dcdd9e05334342,
title = {Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation — arXiv:2608.27729},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.27729},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dd9e05334342; sha256 cef3891ac88822f8394a26e4c045ba28b422938c533362630e1c14b738c7a9da},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27729. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students are attractive but knowledge distillation gains are typically reported single-seed, at scales where seed variance is unknown. On a 740-instance healthcare API routing task with a 1.5B Qwen student and a 20B teacher, we compare eight KD variants against supervised cross-entropy, using three to six seeds for key configurations. We find: (i) per-seed standard deviation ranges from 2.8 to 48.7 percentage points, swallowing every claimed KD gain below five points; (ii) three of seven KD variants exhibit bimodal collapse, with at least one in three to five seeds falling below 55% accuracy while the others train normally, and a fourth showing elevated variance; (iii) collapse has distinct modes -- wrong-function selection for ce_kd and ce_paraphrase, and a previously undocumented output-truncation mode for reasoning_kd, where the model emits reasoning but terminates before producing a function name (0.9% accuracy); (iv) only progressive_kd and rank_kd avoid collapse across observed seeds, with sigma <= 3.9 pp; (v) a naive cross-split +3.78 pp gain from input enrichment reverses to -2.70 pp under controlled within-split multi-seed re-testing. Single-seed evaluation is therefore unable to detect central failure modes in small-model KD. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-31 in cs.CL). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2608.27729 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:47:33.107Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).