Metraipolis › archive › dc_5a172b98283f

Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation — arXiv:2611.01065

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01065
License (as archived)
CC0-1.0
Archived
2026-11-03T18:17:39.234Z
SHA-256 of the archived text
200cfc15eee04dc88a70452379f2155ba358e1f64911c07080a17d3ec922821a
Stable URL of this page
https://metraipolis.com/documento/dc_5a172b98283f

How to cite

Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation — arXiv:2611.01065 — arXiv (en), https://arxiv.org/abs/2611.01065 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5a172b98283f (Metraipolis archive dc_5a172b98283f, sha256 200cfc15eee04dc88a70452379f2155ba358e1f64911c07080a17d3ec922821a)
BibTeX
@misc{dc5a172b98283f,
  title = {Label Smoothing in Disguise? Gated Knowledge Distillation for Low-Resource Bangla Generation — arXiv:2611.01065},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01065},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5a172b98283f; sha256 200cfc15eee04dc88a70452379f2155ba358e1f64911c07080a17d3ec922821a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01065. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Gated knowledge distillation (KD) weights teacher signals per token or example, aiming to trust an imperfect teacher only where it seems reliable. We test two gates: EWAD (entropy-weighted agreement-aware distillation), which gates tokens by teacher confidence, and CHAD (counterfactual harm-aware distillation), which gates examples by gradient alignment with validation data. Across six Bangla-centric tasks, three student models (60M to about 2B parameters), and two teacher families, no gate significantly improves plain KD in 45 comparisons. KD also fails to beat cross-entropy (CE) fine-tuning on gold outputs in 40 single-seed comparisons or three-seed runs on four tasks; CE leads on every summarisation metric. The main benefit of KD is regularisation: over ten fixed epochs, CE students lose 0.030 to 0.044 token-F1 from their early-stopped checkpoint, versus 0.005 to 0.025 for distilled students. Teacher-free label smoothing closes most of this gap. In one comparison, a teacher 0.29 token-F1 worse yields students at most 0.009 worse. Thus, gating may change regularisation strength but has little teacher knowledge to select. Findings are limited to Bangla-centric tasks and small students. We will release code, data splits, predictions, and scores.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.01065 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.562Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).