Metraipolis › archive › dc_d480ef45d52c

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer — arXiv:2609.22566

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.22566
License (as archived)
CC0-1.0
Archived
2026-10-09T06:49:33.313Z
SHA-256 of the archived text
7d52ad891d957e263acfe3e459fec62236ca6e922cf00f14c786d7e8f96a1204
Stable URL of this page
https://metraipolis.com/documento/dc_d480ef45d52c

How to cite

Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer — arXiv:2609.22566 — arXiv (en), https://arxiv.org/abs/2609.22566 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d480ef45d52c (Metraipolis archive dc_d480ef45d52c, sha256 7d52ad891d957e263acfe3e459fec62236ca6e922cf00f14c786d7e8f96a1204)
BibTeX
@misc{dcd480ef45d52c,
  title = {Do Student LLMs Inherit OOD Robustness? Invariance-Weighted Distillation for Reliable Knowledge Transfer — arXiv:2609.22566},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22566},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d480ef45d52c; sha256 7d52ad891d957e263acfe3e459fec62236ca6e922cf00f14c786d7e8f96a1204},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22566. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Knowledge distillation (KD) aims to compress high-performance teacher LLMs into lightweight students. However, distilled students often exhibit substantial performance degradation in out-of-distribution (OOD) settings, a critical gap that remains underexplored. We identify two compounding mechanisms causing OOD performance degradation: (1) data spuriousness: students can learn spurious correlations in the distillation dataset over genuine causal relationships; and (2) teacher capability: standard KD treats all samples uniformly, ignoring whether the teacher is guided by causal features or misled by spurious shortcuts on a given sample. To address these challenges, we propose Invariance-Weighted Distillation (IWD), a theoretically grounded framework that dynamically reweights training samples using an estimate of the teacher's causal reliance derived from prediction invariance across multiple synthetic environments. IWD perturbs spurious cues while preserving core semantics, assigning higher distillation weights to samples whose teacher predictions remain invariant, indicating greater reliance on causal rather than spurious features. We theoretically show that IWD reduces the student's Spurious-to-Causal (S2C) gradient ratio compared to standard uniformly weighted KD, driving the student toward more invariant representations. Experiments on four NLP benchmarks (MNLI, SQuAD-v2, CoNLL-2003 NER, and SST-2) across two model families (DeBERTa-v3 and Qwen-2.5) demonstrate that IWD consistently outperforms strong KD baselines on OOD evaluations while maintaining competitive in-distribution (ID) performance. Specifically, IWD achieves the highest accuracy in 15 out of 16 OOD benchmarks and improves average OOD performance over standard KD by 4.34 percentage points on NLI and 14.94 percentage points on QA.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.22566 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:44.941Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).