Metraipolis › archive › dc_746c852aad4f

Self-Specialized Teachers for Domain Post-Training — arXiv:2608.28647

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28647
License (as archived)
CC0-1.0
Archived
2026-10-17T07:41:45.805Z
SHA-256 of the archived text
9a107cdacb077d6ecac822ca99f9ae08d9ee69bcee3c87f2838be326b6aa6108
Stable URL of this page
https://metraipolis.com/documento/dc_746c852aad4f

How to cite

Self-Specialized Teachers for Domain Post-Training — arXiv:2608.28647 — arXiv (en), https://arxiv.org/abs/2608.28647 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_746c852aad4f (Metraipolis archive dc_746c852aad4f, sha256 9a107cdacb077d6ecac822ca99f9ae08d9ee69bcee3c87f2838be326b6aa6108)
BibTeX
@misc{dc746c852aad4f,
  title = {Self-Specialized Teachers for Domain Post-Training — arXiv:2608.28647},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28647},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_746c852aad4f; sha256 9a107cdacb077d6ecac822ca99f9ae08d9ee69bcee3c87f2838be326b6aa6108},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28647. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Target-only post-training can improve performance in a specialized domain while degrading behaviors that a general-purpose base model acquired before adaptation. We study this problem when target-domain data are available but a representative replay corpus is not. We propose self-specialized teacher distillation (SSTD), a two-stage procedure that first trains a copy of the base model into a domain teacher, then distills its token distribution to a student on prefixes sampled from the student itself. Teacher training combines standard target supervision with base-aware key-token weighting and distribution alignment to the frozen base model; on-policy distillation then places domain feedback on states the student can encounter at inference time. On financial numerical reasoning, medical question answering, and legal holding identification, SSTD retains much of the target improvement of direct fine-tuning while improving the mean score on the evaluated general suite by 4.8--5.0 points at the reported operating point. The pattern persists across Qwen3 sizes and on Gemma backbones. SSTD requires neither an external teacher nor general replay data.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.28647 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.469Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).