Metraipolis › archive › dc_de1c06bcd7a3

Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate — arXiv:2610.04860

Source
arXiv
Original
https://arxiv.org/abs/2610.04860
License (as archived)
CC0-1.0
Archived
2026-10-06T05:26:27.807Z
SHA-256 of the archived text
dd946c7dd91c54179df36a5633f4df27293f9e7805ca540efef6485a0a6c7b84
Stable URL of this page
https://metraipolis.com/documento/dc_de1c06bcd7a3

How to cite

Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate — arXiv:2610.04860 — arXiv, https://arxiv.org/abs/2610.04860 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_de1c06bcd7a3 (Metraipolis archive dc_de1c06bcd7a3, sha256 dd946c7dd91c54179df36a5633f4df27293f9e7805ca540efef6485a0a6c7b84)
BibTeX
@misc{dcde1c06bcd7a3,
  title = {Invisible Ink, Visible Lies: How Production Watermarking Causes LLMs to Hallucinate — arXiv:2610.04860},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04860},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_de1c06bcd7a3; sha256 dd946c7dd91c54179df36a5633f4df27293f9e7805ca540efef6485a0a6c7b84},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04860. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled retrieval-augmented generation setting, we compare unwatermarked and watermarked generations under the same context, query, and decoding configuration, and quantify their factual accuracy decrease. Across six representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID watermarking, we consistently observe watermark-induced hallucination. Watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) token perturbations in the current-step arising from method-specific reweighting or keyed sampling, and (2) prefix-induced attention drift, which accumulates through autoregressive decoding and weakens later attention to the factual context. Motivated by this analysis, we propose two plug-in interventions at the token and attention levels that can be integrated into existing watermarking methods to improve factuality. At a matched TPR of 0.90 at 1% FPR, combining the two interventions reduces factual errors by approximately 90% relative to watermark-only decoding while preserving fluency and comparable decoding efficiency. Overall, this work highlights factuality as a first-class criterion in watermark evaluation, alongside detectability and robustness, and calls for careful factuality validation before deploying watermarks in fact-critical applications.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.04860 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.920Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).