Metraipolis › archive › dc_37c48eaaa163

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models — arXiv:2609.32617

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.32617
License (as archived)
CC0-1.0
Archived
2026-10-07T21:00:27.715Z
SHA-256 of the archived text
d74ae4e0677ac45e0fb7196bff57a2a9c1f8e6571900e2d0817cde04232acae9
Stable URL of this page
https://metraipolis.com/documento/dc_37c48eaaa163

How to cite

The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models — arXiv:2609.32617 — arXiv (en), https://arxiv.org/abs/2609.32617 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_37c48eaaa163 (Metraipolis archive dc_37c48eaaa163, sha256 d74ae4e0677ac45e0fb7196bff57a2a9c1f8e6571900e2d0817cde04232acae9)
BibTeX
@misc{dc37c48eaaa163,
  title = {The Alignment Paradox: How Post-Training Amplifies Confident Hallucinations in Language Models — arXiv:2609.32617},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.32617},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_37c48eaaa163; sha256 d74ae4e0677ac45e0fb7196bff57a2a9c1f8e6571900e2d0817cde04232acae9},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.32617. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) can produce factually incorrect answers with high confidence, undermining their reliability and limiting the effectiveness of uncertainty-based error detection. While prior research attributes confident hallucinations to factors such as missing knowledge in training data, reasoning errors, or stochastic decoding, we uncover that post-training alignment itself is a primary driver of these errors, a phenomenon we call the \textbf{Alignment Paradox}. Across five model families evaluated on factual benchmarks, unaligned base models produce few high-confidence errors on long-tail factual queries, whereas instruction-tuned models multiply high-confidence errors ($p \ge 0.95$) by more than an order of magnitude (10$\times$ to 35$\times$). Layer-wise probing with the Logit Lens reveals that this overconfidence emerges in late layers, where wrong-answer margins expand past 4.0 points after remaining near zero across early and intermediate layers. These findings motivate limiting margin growth during post-training. We implement this principle through an entropy-dependent margin bound in direct preference optimization (DPO). In multi-epoch experiments with Mistral-7B, the bounded objective reduces high-confidence errors by up to 35.3\% relative to standard DPO while maintaining performance on evaluated general reasoning benchmarks. These results show that bounded margins mitigate confident hallucinations during post-training.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.32617 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.409Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).