Metraipolis › archive › dc_f7dfd8d70fdd

Less Loss, Fixed Weights: Logit Corrections for Forgetting — arXiv:2610.42000

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.42000
License (as archived): CC0-1.0
Archived: 2026-11-03T07:55:28.893Z
SHA-256 of the archived text: aea48452baefbe2b565d30524a0c08ad4276b31bbe28c027f330a488af2e16ce
Stable URL of this page: https://metraipolis.com/documento/dc_f7dfd8d70fdd

How to cite

Less Loss, Fixed Weights: Logit Corrections for Forgetting — arXiv:2610.42000 — arXiv (en), https://arxiv.org/abs/2610.42000 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f7dfd8d70fdd (Metraipolis archive dc_f7dfd8d70fdd, sha256 aea48452baefbe2b565d30524a0c08ad4276b31bbe28c027f330a488af2e16ce)

BibTeX

@misc{dcf7dfd8d70fdd,
  title = {Less Loss, Fixed Weights: Logit Corrections for Forgetting — arXiv:2610.42000},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.42000},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f7dfd8d70fdd; sha256 aea48452baefbe2b565d30524a0c08ad4276b31bbe28c027f330a488af2e16ce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.42000. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text. This is known as "forgetting." We ask whether this loss can be undone by simple, fixed adjustments to the model's final token predictions. Across 14 pairs of base and derived models, from 1B to 7B parameters, we fitted a fixed bonus or penalty for each token (a vocabulary bias), alone or together with a temperature. On text held out from correction fitting, the largest top-4K bias recovery estimate is under 10%, with loss conditional on the retained vocabulary. Combining the bias with a temperature recovers up to 36%. Models further pretrained on math lose the most and recover almost nothing. Finally, in a controlled simulation where a model is damaged at random, with no planted bias or temperature change, the bias-plus-temperature fix still recovers 29%. Most of the real-model loss increase stays out of reach of our restricted fits.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.42000 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:49:46.037Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).