Metraipolis › archive › dc_fcfc5de7f4df

Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy — arXiv:2610.09458

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.09458
License (as archived)
CC0-1.0
Archived
2026-10-08T11:26:04.847Z
SHA-256 of the archived text
ae89e7faaad3c50ba8c838f459fc446f920c009a11df942cfc71a069af4eaede
Stable URL of this page
https://metraipolis.com/documento/dc_fcfc5de7f4df

How to cite

Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy — arXiv:2610.09458 — arXiv (en), https://arxiv.org/abs/2610.09458 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fcfc5de7f4df (Metraipolis archive dc_fcfc5de7f4df, sha256 ae89e7faaad3c50ba8c838f459fc446f920c009a11df942cfc71a069af4eaede)
BibTeX
@misc{dcfcfc5de7f4df,
  title = {Right Number, Wrong State? Measuring Cross-Jurisdiction Substitution in LLM Recall of State Policy — arXiv:2610.09458},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.09458},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fcfc5de7f4df; sha256 ae89e7faaad3c50ba8c838f459fc446f920c009a11df942cfc71a069af4eaede},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09458. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

When an LLM answers a state-specific policy question wrongly, it may be hallucinating, or it may be returning a real value that holds in another state. We test this with a minimal-set design: the question wording is fixed and only the jurisdiction varies, across the 50 U.S. states and the District of Columbia (51 jurisdictions) and three exactly defined Medicaid income-eligibility quantities. Gold values come from an official data book and agree with an independent source in 101 of 102 checked cells. Under a pre-registered protocol, Claude Sonnet 5.5 and GPT-5.6 Sol reproducibly give another state's current value, identical across two independent repeats, for 10 and 25 of 153 items. Attribution is fragile, however. Crediting any wrong answer that equals another state's value yields 3-5x more reproducible substitutions than checking every number in the asked state's own records, because many apparent cross-state answers are the asked state's own values under another convention or from an earlier year. Claims about cross-jurisdiction error need a complete same-state reference set. We will release the protocol, gold table, and all model outputs.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2610.09458 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T10:24:06.318Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).