Metraipolis › archive › dc_700a8cd8262e
When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation — arXiv:2609.34489
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34489
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:38:18.768Z
- SHA-256 of the archived text
6883659b0ff006192edc45d2f31454ddfa110451aa567b8032dde91715c30d06- Stable URL of this page
https://metraipolis.com/documento/dc_700a8cd8262e
How to cite
When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation — arXiv:2609.34489 — arXiv (en), https://arxiv.org/abs/2609.34489 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_700a8cd8262e (Metraipolis archive dc_700a8cd8262e, sha256 6883659b0ff006192edc45d2f31454ddfa110451aa567b8032dde91715c30d06)
BibTeX
@misc{dc700a8cd8262e,
title = {When Less Data Favors Smaller Teachers: Rethinking Teacher Capacity and Data Selection for Knowledge Distillation — arXiv:2609.34489},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34489},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_700a8cd8262e; sha256 6883659b0ff006192edc45d2f31454ddfa110451aa567b8032dde91715c30d06},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34489. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Data pruning reduces the training cost of knowledge distillation (KD). However, the preferred teacher capacity changes with the data budget: smaller teachers can outperform larger ones when limited training data are available. Understanding what drives this shift is important not only for teacher choice but also for identifying which samples are useful for distillation. We analyze teacher supervision by decomposing it into relational ordering---the ranking of classes---and score geometry---the magnitudes and margins of class probabilities---and show that the small-teacher advantage in the low-data regime arises not only from score geometry but also from relational ordering. Beyond understanding teacher capacity, our analysis reveals two properties of effective subsets: samples should match the difficulty appropriate for the available budget, and their relational signals should be diverse rather than redundant. Based on these findings, we propose DVA (Difficulty- and Volume-Aware data selection for KD), a training-dynamics-free method, which uses a small teacher as a proxy for budget-aware difficulty filtering and class-conditional relational volume maximization. Despite requiring no training dynamics statistics, our method remains competitive with training-dynamics-based methods while consistently outperforming training-dynamics-free baselines. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.34489 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T22:14:28.204Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).