Metraipolis › archive › dc_fef17f83113b

Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues — arXiv:2611.01313

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.01313
License (as archived)
CC0-1.0
Archived
2026-11-03T18:12:23.854Z
SHA-256 of the archived text
34169f6c32d98364d4765f16d3156b8fdf8625e71d0efa992bac1aa7f8541e13
Stable URL of this page
https://metraipolis.com/documento/dc_fef17f83113b

How to cite

Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues — arXiv:2611.01313 — arXiv (en), https://arxiv.org/abs/2611.01313 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fef17f83113b (Metraipolis archive dc_fef17f83113b, sha256 34169f6c32d98364d4765f16d3156b8fdf8625e71d0efa992bac1aa7f8541e13)
BibTeX
@misc{dcfef17f83113b,
  title = {Are Fake Citations a Form of Reward Hacking? A Pre-Registered Study of Post-Training and Citation Cues — arXiv:2611.01313},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01313},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fef17f83113b; sha256 34169f6c32d98364d4765f16d3156b8fdf8625e71d0efa992bac1aa7f8541e13},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01313. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-trained language models fabricate citations at high rates. We ask whether that is reward hacking: scoring well on a proxy for scholarship while doing worse on what it stands for. We ran four pre-registered experiments on 900 frozen questions (US federal case law, computationally adjacent scientific literature, and medical literature) across twelve open-weights checkpoints covering four post-training stages of three model families, then refit on fifteen. The clearest result holds the weights fixed and changes only the prompt. Asking for citations raises fabrication sharply in post-trained checkpoints and not in base ones: a 47.8-point interaction on twelve checkpoints (Holm-adjusted p = 2e-34) and 41.5 points on fifteen, in all three domains. A length-matched control separates the citation cue from a demand for a longer answer, so the effect is specific to citations, not verbosity. Post-training also raises fabrication overall, from 4.8% to 56.7%, but our registered prediction that it would rise most where fabrication is most tempting did not hold. A preference model trained only on surface citation features prefers post-trained output 86% of the time, and holding citation count fixed shows it rewards quantity. Our answer: possibly, but not proven. Post-trained models produce fabricated citations at far higher rates when asked for sources, and a naive evaluator that counts sources without checking them would prefer their output for citing more. But the targeted pattern we registered did not appear, and plain instruction-following could explain the same results. Proving that the reward causes fabrication needs an experiment that changes the reward during training, which this study does not do. Pre-registration: doi: https://doi.org/10.17605/OSF.IO/S53HY . Code and data: this https URL

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.01313 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.336Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).