Metraipolis › archive › dc_cd46abfbd7a0

Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training — arXiv:2611.10045

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.10045
License (as archived)
CC0-1.0
Archived
2026-11-10T18:40:41.557Z
SHA-256 of the archived text
14bd47b6c199bbc472ea8b08e6ca8faa421e07580723cf1b56251ba7dfcb2185
Stable URL of this page
https://metraipolis.com/documento/dc_cd46abfbd7a0

How to cite

Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training — arXiv:2611.10045 — arXiv (en), https://arxiv.org/abs/2611.10045 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cd46abfbd7a0 (Metraipolis archive dc_cd46abfbd7a0, sha256 14bd47b6c199bbc472ea8b08e6ca8faa421e07580723cf1b56251ba7dfcb2185)
BibTeX
@misc{dccd46abfbd7a0,
  title = {Don't Blame The Preferences: Examining The Emergence of Sycophancy in LLM Training — arXiv:2611.10045},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.10045},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cd46abfbd7a0; sha256 14bd47b6c199bbc472ea8b08e6ca8faa421e07580723cf1b56251ba7dfcb2185},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.10045. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

AI sycophancy, the tendency of large language models (LLMs) to flatter users and affirm their beliefs, can undermine accuracy and contribute to misinformation, dependence, and overconfidence. Although prior work has benchmarked sycophancy in released models and proposed mitigations, its origins during training remain poorly understood. Preference alignment methods such as RLHF and DPO are often blamed, but their effects are rarely compared with earlier training stages. We investigate sycophancy across the training pipelines of three model families and three benchmarks. We find that sycophancy is largely established by the end of supervised fine-tuning (SFT), rather than introduced by preference alignment. Across all nine model benchmark combinations, SFT checkpoints attain at least 92% of preference aligned models' sycophancy scores and at least 85% of final models' scores; on average, they are about as sycophantic as final models. Preference alignment increases sycophancy in only four of nine cases, by an average of 4.2 percentage points when it does. To probe the underlying causes, we isolate model representations associated with sycophancy using mechanistic interpretability and measure their activation on training datasets from each stage. SFT datasets show roughly the same average activation as other training data but contribute substantially more examples, and the highest-scoring individual datasets all come from SFT. Our findings suggest that addressing sycophancy requires interventions throughout the training pipeline, particularly in data selection and augmentation for pre-training and SFT.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.10045 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.475Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).