Metraipolis › archive › dc_9275099112e2

Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation — arXiv:2610.19306

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.19306
License (as archived): CC0-1.0
Archived: 2026-10-15T14:15:24.218Z
SHA-256 of the archived text: 1879466f6af78cdcc526edecdfa4a62090436e4bdea6ec19a371520f786ac48e
Stable URL of this page: https://metraipolis.com/documento/dc_9275099112e2

How to cite

Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation — arXiv:2610.19306 — arXiv (en), https://arxiv.org/abs/2610.19306 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_9275099112e2 (Metraipolis archive dc_9275099112e2, sha256 1879466f6af78cdcc526edecdfa4a62090436e4bdea6ec19a371520f786ac48e)

BibTeX

@misc{dc9275099112e2,
  title = {Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation — arXiv:2610.19306},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.19306},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_9275099112e2; sha256 1879466f6af78cdcc526edecdfa4a62090436e4bdea6ec19a371520f786ac48e},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.19306. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Jailbreak attacks against large language models (LLMs) are evaluated using attack success rate (ASR), which depends on correctly identifying responses that fulfill prohibited requests. Existing evaluators can mistake refusals or topically related responses for successful attacks. We show that changing only the judge LLM can alter reported ASR by up to $82$ percentage points. We introduce the semantic reconstruction game, a training-free evaluator that decomposes compliance judgments into constrained semantic comparisons. Given only a response, an LLM reconstructs the request answered by its substantive content, excluding information merely repeated or refused. The game compares this reconstruction against similar decoy requests to determine whether it matches the original request, allowing abstention when neither candidate matches. Across three off-the-shelf LLM judges, the game achieves the lowest mean error against human labels among evaluated methods and reduces variation in reported ASR. It prioritizes avoiding false positives, even at the cost of missing some successful jailbreaks.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2610.19306 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:53:35.567Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).