Metraipolis › archive › dc_5ce0f93bcb21

Perturbing Inputs Improves Pretraining Distillation — arXiv:2611.11598

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.11598
License (as archived)
CC0-1.0
Archived
2026-11-10T18:43:34.162Z
SHA-256 of the archived text
3f8cba7157d4b292b296d5088cf222f8c4867648f594db6273b175b99a8a3523
Stable URL of this page
https://metraipolis.com/documento/dc_5ce0f93bcb21

How to cite

Perturbing Inputs Improves Pretraining Distillation — arXiv:2611.11598 — arXiv (en), https://arxiv.org/abs/2611.11598 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5ce0f93bcb21 (Metraipolis archive dc_5ce0f93bcb21, sha256 3f8cba7157d4b292b296d5088cf222f8c4867648f594db6273b175b99a8a3523)
BibTeX
@misc{dc5ce0f93bcb21,
  title = {Perturbing Inputs Improves Pretraining Distillation — arXiv:2611.11598},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.11598},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5ce0f93bcb21; sha256 3f8cba7157d4b292b296d5088cf222f8c4867648f594db6273b175b99a8a3523},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11598. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Knowledge distillation trains a student language model to match a teacher's predictions on the training text. We ask whether learning the teacher's behavior beyond the training text can further improve the student. We investigate this using three input perturbations: additive token noise, blending two documents, and blending neighboring tokens. Specifically, for an early part of pretraining we perturb the inputs and train the student to match the teacher on them, and we spend the rest of the same token budget on ordinary distillation. When pretraining from scratch, the best method improves on standard distillation at both scales we tried, by 0.025 nats at 135M parameters and by 0.020 nats at 0.5B (mean of three seeds). At 135M this is 1.45 times the gain that distillation itself provides over next-token training. We also observe promising downstream transfer: at 0.5B, the best variant improves average zero-shot accuracy by about one percentage point across seven tasks. It also retains its validation-loss advantage after low-bit weight quantization. We also test a data-limited setting in which training data must be reused, and find that the gains persist. Finally, ablations suggest that, for additive noise, perturbing the teacher's inputs contributes most of the gain, while perturbing only the student's inputs yields much less.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.11598 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.651Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).