Metraipolis › archive › dc_0f004fe9a5ce

When Auditors Fabricate: Batch-Size Degradation and Confident Hallucination in LLM Detection of Planted Document Contamination — arXiv:2609.09696

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.09696
License (as archived)
CC0-1.0
Archived
2026-10-16T07:26:10.574Z
SHA-256 of the archived text
9cd60191c417e4d0fc5703ef9e9e8c9f75b64f94279df7f5e81a9244b1f51f74
Stable URL of this page
https://metraipolis.com/documento/dc_0f004fe9a5ce

How to cite

When Auditors Fabricate: Batch-Size Degradation and Confident Hallucination in LLM Detection of Planted Document Contamination — arXiv:2609.09696 — arXiv (en), https://arxiv.org/abs/2609.09696 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_0f004fe9a5ce (Metraipolis archive dc_0f004fe9a5ce, sha256 9cd60191c417e4d0fc5703ef9e9e8c9f75b64f94279df7f5e81a9244b1f51f74)
BibTeX
@misc{dc0f004fe9a5ce,
  title = {When Auditors Fabricate: Batch-Size Degradation and Confident Hallucination in LLM Detection of Planted Document Contamination — arXiv:2609.09696},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.09696},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_0f004fe9a5ce; sha256 9cd60191c417e4d0fc5703ef9e9e8c9f75b64f94279df7f5e81a9244b1f51f74},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09696. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models are increasingly proposed as automated auditors of document quality, yet their reliability as detectors of planted errors is poorly characterised. We construct a contaminated corpus of 150 academic papers spanning supply chain management and medical research, injecting 450 known contaminants of three types: typographical corruption, semantic reversal, and absurd out-of-context insertion. We then evaluate Google Gemini 3.0 Pro's ability to recover a 180-contaminant answer-key subset across 60 documents under three prompting regimes of increasing scale: single document, small batch, and large batch. Detection is unreliable even at small scale and collapses entirely at large scale: 50% recovery on single documents and 60% on small batches, a difference this sample cannot resolve, against 2.8% on large batches. The failure mode at scale is not abstention but fabrication. Rather than reporting incomplete processing, the model produced confident findings including invented contaminants of its own, absurdities such as "telepathic squirrel" and "quantum-powered toaster" that mimic the style of the planted material but do not appear in any document. Detection also varies by contamination type: absurd insertions were recovered at 75% in completed evaluations, while semantic reversals and typographical corruptions were each recovered at only 50%. The corruptions most likely to occur in the wild, plausible ones, are the ones most often missed. We conclude that LLM document auditing degrades not gracefully but deceptively, and outline the harness such systems require: bounded batch sizes, direct content injection, and mechanical verification of every reported finding against source text.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-10 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.09696 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T07:00:16.341Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-10?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).