Metraipolis › archive › dc_fef17f83113b
Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues — arXiv:2611.01313
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.01313
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T18:12:23.854Z
- SHA-256 of the archived text
34169f6c32d98364d4765f16d3156b8fdf8625e71d0efa992bac1aa7f8541e13- Stable URL of this page
https://metraipolis.com/documento/dc_fef17f83113b
How to cite
Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues — arXiv:2611.01313 — arXiv (en), https://arxiv.org/abs/2611.01313 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fef17f83113b (Metraipolis archive dc_fef17f83113b, sha256 34169f6c32d98364d4765f16d3156b8fdf8625e71d0efa992bac1aa7f8541e13)
BibTeX
@misc{dcfef17f83113b,
title = {Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues — arXiv:2611.01313},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.01313},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fef17f83113b; sha256 34169f6c32d98364d4765f16d3156b8fdf8625e71d0efa992bac1aa7f8541e13},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01313. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Post-trained language models fabricate citations at high rates. We ask whether that is reward hacking: scoring well on a proxy for scholarship while doing worse on what it stands for. We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen. The clearest result holds the weights fixed and changes only the prompt. Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains. A length-matched control separates the citation cue from a demand for a longer answer, so the effect is specific to citations, not verbosity. Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold. A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity. Our answer: possibly, but not proven. Post-trained models produce fabricated citations at far higher rates when asked for sources, and a naive evaluator that counts sources without checking them would prefer their output for citing more. But the targeted pattern we registered did not appear, and plain instruction-following could explain the same results. Proving that the reward causes fabrication needs an experiment that changes the reward during training, which this study does not do. Pre-registration: doi: https://doi.org/10.17605/OSF.IO/S53HY . Code and data: this https URL Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.01313 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T18:09:53.336Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).