Metraipolis › archive › dc_463e95e011f2

Less Loss, Fixed Weights: Logit Corrections for Forgetting — arXiv:2610.42000

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.42000
License (as archived)
CC0-1.0
Archived
2026-11-03T07:45:44.598Z
SHA-256 of the archived text
6ab0344679b1837883911399ecbfdfe1e46b38406493a974a78e0cfc57de179d
Stable URL of this page
https://metraipolis.com/documento/dc_463e95e011f2

How to cite

Less Loss, Fixed Weights: Logit Corrections for Forgetting — arXiv:2610.42000 — arXiv (en), https://arxiv.org/abs/2610.42000 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_463e95e011f2 (Metraipolis archive dc_463e95e011f2, sha256 6ab0344679b1837883911399ecbfdfe1e46b38406493a974a78e0cfc57de179d)
BibTeX
@misc{dc463e95e011f2,
  title = {Less Loss, Fixed Weights: Logit Corrections for Forgetting — arXiv:2610.42000},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.42000},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_463e95e011f2; sha256 6ab0344679b1837883911399ecbfdfe1e46b38406493a974a78e0cfc57de179d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.42000. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text. This is known as "forgetting." We ask whether this loss can be undone by simple, fixed adjustments to the model's final token predictions. Across 14 pairs of base and derived models, from 1B to 7B parameters, we fitted a fixed bonus or penalty for each token (a vocabulary bias), alone or together with a temperature. On text held out from correction fitting, the largest top-4K bias recovery estimate is under 10%, with loss conditional on the retained vocabulary. Combining the bias with a temperature recovers up to 36%. Models further pretrained on math lose the most and recover almost nothing. Finally, in a controlled simulation where a model is damaged at random, with no planted bias or temperature change, the bias-plus-temperature fix still recovers 29%. Most of the real-model loss increase stays out of reach of our restricted fits.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.42000 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:39:52.212Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).