Metraipolis › archive › dc_574dce2aac2a

Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models — arXiv:2609.03267

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03267
License (as archived)
CC0-1.0
Archived
2026-10-17T02:16:15.051Z
SHA-256 of the archived text
d82519e21c8eda25fcf97568e5d00539ec55f3b086dd82f037ac5858b9761aa6
Stable URL of this page
https://metraipolis.com/documento/dc_574dce2aac2a

How to cite

Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models — arXiv:2609.03267 — arXiv (en), https://arxiv.org/abs/2609.03267 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_574dce2aac2a (Metraipolis archive dc_574dce2aac2a, sha256 d82519e21c8eda25fcf97568e5d00539ec55f3b086dd82f037ac5858b9761aa6)
BibTeX
@misc{dc574dce2aac2a,
  title = {Refusing the Impossible: A Taxonomy and Benchmark for Code Hallucination in Large Language Models — arXiv:2609.03267},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03267},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_574dce2aac2a; sha256 d82519e21c8eda25fcf97568e5d00539ec55f3b086dd82f037ac5858b9761aa6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03267. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) often produce code that looks plausible but is not grounded in reality. The code may import packages that do not exist or claim to implement algorithms that violate proven theorems, while still compiling and running. We study \emph{code hallucination} as \emph{ungrounded generation} and separate it from ordinary \emph{code error} (bugs in otherwise grounded programs). We propose a taxonomy with three dimensions: \textbf{groundedness} (absolute violations of universal truths vs.\ relative fabrications of contingent or ecosystem-specific facts), \textbf{manifestation level} (syntactic, semantic, or factual), and \textbf{behavior} (from confident fabrication to degenerate output), organized into a severity ordering. We build an \textbf{adversarial} suite of deliberately unsatisfiable tasks where the correct response is to refuse and categorize the responses under our taxonomy. The suite contains \textbf{270 prompts} across six languages and 24 subcategories, paired with \textbf{91 matched solvable controls}, and responses are judged by a two-tier protocol validated against human labels (82\% agreement, $\kappa{=}0.73$). Across twelve open-weight code and reasoning models (4{,}332 judged responses), models produce ungrounded code on about 60\% of unsatisfiable prompts and refuse only 27\%, while wrongly refusing 0\% of the solvable controls.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-04 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.03267 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T02:03:40.002Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).