Metraipolis › archive › dc_4e3339570e5e

Unifying and Beyond the Theory of Pervasive Capability Degradation in LLM Post-training SFT from Feature Interference — arXiv:2611.05003

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.05003
License (as archived)
CC0-1.0
Archived
2026-11-04T21:51:51.713Z
SHA-256 of the archived text
01738cf5a71df05837abd1c3a45233ceb36d7f285002a570e57c28c89759b23c
Stable URL of this page
https://metraipolis.com/documento/dc_4e3339570e5e

How to cite

Unifying and Beyond the Theory of Pervasive Capability Degradation in LLM Post-training SFT from Feature Interference — arXiv:2611.05003 — arXiv (en), https://arxiv.org/abs/2611.05003 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e3339570e5e (Metraipolis archive dc_4e3339570e5e, sha256 01738cf5a71df05837abd1c3a45233ceb36d7f285002a570e57c28c89759b23c)
BibTeX
@misc{dc4e3339570e5e,
  title = {Unifying and Beyond the Theory of Pervasive Capability Degradation in LLM Post-training SFT from Feature Interference — arXiv:2611.05003},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.05003},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e3339570e5e; sha256 01738cf5a71df05837abd1c3a45233ceb36d7f285002a570e57c28c89759b23c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05003. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Pervasive Capability Degradation (PCD) refers to the phenomenon in which supervised fine-tuning (SFT), while adapting a large language model (LLM) to a target task, often erodes the general capabilities acquired during pre-training. Prevailing explanations attribute PCD to macroscopic distribution shift and therefore lack microscopic manipulability, leaving existing mitigation methods without a complete theoretical foundation. In this paper, we first develop a bottom-up geometric theory of PCD grounded in feature interference under superposition: narrow-domain SFT data disrupt the almost-orthogonal feature structure formed during pre-training, and the resulting interference drives activations across decision hyperplanes in the output Laguerre-Voronoi geometry, flipping predictions on general tasks. This theory unifies existing methods into six paradigms and reveals both the source of their efficacy and their theoretical limits. We then prove that cross-entropy under one-hot supervision cannot resolve PCD on its own, and accordingly propose the AEGIS loss, which directly constrains the drift of activation margins along all competing token directions, thereby decoupling interference suppression from task adaptation. Across 3 model families and 14 SFT tasks, AEGIS outperforms representative methods of every paradigm on target tasks and pervasive capabilities simultaneously, at only about 1.3 times the computational cost of standard SFT. Its generality, scalability, and mechanistic interpretability offer a new perspective on fully resolving PCD.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.05003 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T21:38:49.743Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).