Metraipolis › archive › dc_9aabbd8bd2a1
SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning — arXiv:2610.11132
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11132
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T05:57:01.502Z
- SHA-256 of the archived text
35054294c7ad92155867a95a23f4dc7d1b566be7f0965773b706ffc4ff1b7e79- Stable URL of this page
https://metraipolis.com/documento/dc_9aabbd8bd2a1
How to cite
SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning — arXiv:2610.11132 — arXiv (en), https://arxiv.org/abs/2610.11132 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9aabbd8bd2a1 (Metraipolis archive dc_9aabbd8bd2a1, sha256 35054294c7ad92155867a95a23f4dc7d1b566be7f0965773b706ffc4ff1b7e79)
BibTeX
@misc{dc9aabbd8bd2a1,
title = {SFT-as-Context Mitigates Forgetting in Supervised Fine-Tuning — arXiv:2610.11132},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11132},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9aabbd8bd2a1; sha256 35054294c7ad92155867a95a23f4dc7d1b566be7f0965773b706ffc4ff1b7e79},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11132. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Supervised fine-tuning (SFT) equips large language models (LLMs) with specialized capabilities, but often comes at the cost of forgetting the general capabilities of their parent models (i.e., the pretrained models before fine-tuning). This trade-off is especially limiting for queries that require both specialized and general capabilities. We introduce SFT-as-context, a training-free method in which the parent model uses the SFT model's response as context to answer the query. This allows the parent model to acquire fine-tuned capabilities from the SFT response through in-context learning while preserving its own general capabilities. Across 19 parent-SFT model pairs and 11 benchmarks, SFT-as-context remains close to the SFT models on fine-tuned capabilities, with gaps of only 2.2 and 2.1 percentage points on AIME 2024 and LiveCodeBench and 2.0 macro MAE on NutriBench-English, while staying within 2.2 percentage points of the parent models on general capabilities on average. Remarkably, it can solve queries requiring both fine-tuned and general capabilities, even when neither the parent nor SFT model succeeds alone. This approach also extends beyond parent-SFT pairs: responses from a small open-source SFT model can improve a strong closed-source LLM, outperforming either model alone. Furthermore, we use a Bayesian framework to derive theoretical guarantees that bound the error of SFT-as-context relative to the SFT model on fine-tuned capabilities and to the parent model on general capabilities. In addition, we visualize the attention weights and find that the parent model attends more to useful SFT responses and less to irrelevant ones, suggesting that selective attention helps the parent model use the SFT response through in-context learning. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2610.11132 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:01.506Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).