Metraipolis › archive › dc_932d92686036

The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention — arXiv:2611.06476

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.06476
License (as archived)
CC0-1.0
Archived
2026-11-05T18:22:17.369Z
SHA-256 of the archived text
ecbe325799582aa836d41590c2b83e60c038f467a128e124109e63f8c8b9d5fe
Stable URL of this page
https://metraipolis.com/documento/dc_932d92686036

How to cite

The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention — arXiv:2611.06476 — arXiv (en), https://arxiv.org/abs/2611.06476 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_932d92686036 (Metraipolis archive dc_932d92686036, sha256 ecbe325799582aa836d41590c2b83e60c038f467a128e124109e63f8c8b9d5fe)
BibTeX
@misc{dc932d92686036,
  title = {The Alignment Tax Is Graded: Task-Selective Capability Loss Under Length-Normalized DPO with SFT Retention — arXiv:2611.06476},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06476},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_932d92686036; sha256 ecbe325799582aa836d41590c2b83e60c038f467a128e124109e63f8c8b9d5fe},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06476. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Setting. This is an empirical characterization study; we propose no new method and run no comparisons against alternative objectives. We fine-tune Qwen3.5-4B with sparse LoRA and length-normalized DPO plus an SFT retention term (the RPO objective) on structured financial-filing tasks, and track downstream capability as the implicit reward margin grows to $\sim 7$ $\beta$-scaled units (nf4 training; $\sim 3$ under fp16 measurement on the unquantized base). Question. Does margin growth degrade capability uniformly across tasks, or selectively? Findings. In a pre-registered study (8 seeds, 6 unseen at freeze) with artifact-aware scoring and TOST equivalence testing, the alignment tax is graded and task-selective: extraction is preserved ($\Delta = -0.55$ pts, equivalent at $\pm5$ pts), json loses little ($\Delta = -3.52$ pts, equivalence fails on a clustered item-level interval), and router degrades severely ($\Delta = -30$ pts). Router's loss concentrates in a margin 1-3 damage window containing a transient format-break spike. The margin is accounted for almost entirely ($\geq 96\%$) by suppression of rejected completions, with chosen reward approximately flat, consistent with the retention term anchoring it; we do not attribute the behavioral damage to any single gradient channel. Gold-logp deflation is task-selective (extraction $-0.15$, json $-1.04$, router $-1.47$ nats/token under fp16), with gold selectively protected relative to incorrect controls on extraction (8/8 seeds). QLoRA reward margins differ up to $3.4\times$ between nf4 training and unquantized fp16 measurement, with an axis-dependent gap. Scope. Length-normalized rewards with SFT retention, sparse LoRA on 8/32 layers, one 4B model, easily-separable synthetic preferences, three structured tasks. A no-retention control and a measured format-alignment metric are left to future work.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.06476 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:15:49.018Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).