Metraipolis › archive › dc_44fc4e2e4153

Does the Truthfulness Signal Survive Code-Mixing? Probing Hidden States for Hallucination Detection in Hinglish — arXiv:2609.22138

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.22138
License (as archived): CC0-1.0
Archived: 2026-10-09T06:48:54.960Z
SHA-256 of the archived text: 38686383693b2d86a353f26721492b6f453cda797dc336a71ff7c4082bdc5ade
Stable URL of this page: https://metraipolis.com/documento/dc_44fc4e2e4153

How to cite

Does the Truthfulness Signal Survive Code-Mixing? Probing Hidden States for Hallucination Detection in Hinglish — arXiv:2609.22138 — arXiv (en), https://arxiv.org/abs/2609.22138 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_44fc4e2e4153 (Metraipolis archive dc_44fc4e2e4153, sha256 38686383693b2d86a353f26721492b6f453cda797dc336a71ff7c4082bdc5ade)

BibTeX

@misc{dc44fc4e2e4153,
  title = {Does the Truthfulness Signal Survive Code-Mixing? Probing Hidden States for Hallucination Detection in Hinglish — arXiv:2609.22138},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.22138},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_44fc4e2e4153; sha256 38686383693b2d86a353f26721492b6f453cda797dc336a71ff7c4082bdc5ade},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.22138. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Hidden-state hallucination probing - training a linear classifier on an LLM's internal activations to detect whether a generated answer is faithful to the input - is an active area of 2026 research, with recent work reporting 0.90-1.00 AUROC across several benchmarks and languages. However, none of this work has tested probes on code-mixed input, despite the fact that a huge population of chatbot users write in Hindi-English code-mixed text ("Hinglish"). We address this gap directly: does a hallucination probe trained on clean-language hidden states transfer to Hinglish, or does the signal degrade under code-mixing? We construct a 5,674-item Hindi/English/Hinglish QA benchmark, generate and label 17,022 model responses across three open-weight 7-8B LLMs (Qwen2.5-7B, Mistral-7B, Llama-3.1-8B), extract per-layer hidden states at two token positions, and train linear and MLP probes for in-distribution detection and cross-lingual transfer. We find that the hallucination signal survives code-mixing well: transfer AUROC ranges from 0.88 to 0.99, with gaps of mostly under 0.05 AUROC relative to in-distribution performance, and that Hindi-trained probes transfer to Hinglish more reliably than English-trained probes. As an independent, practically motivated finding, all three models hallucinate substantially more on Hindi and Hinglish than on English for matched facts. We release our code and synthetic Hinglish QA dataset to support further work on code-mixed hallucination detection.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-22 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.22138 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T06:48:44.842Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-22?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).