Metraipolis › archive › dc_8fa71b54bfa7
The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination — arXiv:2609.12111
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.12111
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:08:16.946Z
- SHA-256 of the archived text
bd5e3a1b58151030f38b9bbb819be57d523066bd2940e26909d8437626aabb92- Stable URL of this page
https://metraipolis.com/documento/dc_8fa71b54bfa7
How to cite
The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination — arXiv:2609.12111 — arXiv (en), https://arxiv.org/abs/2609.12111 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_8fa71b54bfa7 (Metraipolis archive dc_8fa71b54bfa7, sha256 bd5e3a1b58151030f38b9bbb819be57d523066bd2940e26909d8437626aabb92)
BibTeX
@misc{dc8fa71b54bfa7,
title = {The Cost of Compression: A Rate-Distortion Limit on Factual Hallucination — arXiv:2609.12111},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.12111},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_8fa71b54bfa7; sha256 bd5e3a1b58151030f38b9bbb819be57d523066bd2940e26909d8437626aabb92},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12111. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Factual hallucination in closed-book question answering is often treated as a coverage problem: a model fails because the relevant fact is absent from its internal memory. This view misses a second source of error. Even when a fact has been observed, finite memory may force it to be stored only approximately. We study this effect through a simple coverage--compression model of factual recall. We consider an unstructured question-answering task with $N$ possible queries and $K$ possible answers. A learner observes $M$ training facts, compresses them into at most $B$ bits, and answers uniformly drawn test queries without retrieval. For a uniformly random ground-truth mapping, we prove $\mathcal{E} \geq \frac{M}{N}\delta^\star\!\left(\frac{B}{M}\right) + \left(1-\frac{M}{N}\right)\left(1-\frac{1}{K}\right)$, where $\delta^\star(r)$ is the inverse rate-distortion function of a uniform $K$-ary source under zero-one loss. The two terms separate compression distortion on observed facts from missing coverage on unobserved facts. The bound gives a compact way to reason about selective memory, forced compression, structure, retrieval, abstention, and long-context organization. We study the predicted signatures with theory-implied simulations and controlled fact-injection probes in modern language models that vary fact load and effective trainable memory. The result is not a complete theory of hallucination, but an information-theoretic account of a separable failure mode: lossy recall of observed facts under finite memory.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.12111 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T03:02:47.527Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).