Metraipolis › archive › dc_e53c771bab99

JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications — arXiv:2610.06625

Source
arXiv
Original
https://arxiv.org/abs/2610.06625
License (as archived)
CC0-1.0
Archived
2026-10-06T05:28:35.917Z
SHA-256 of the archived text
3d3205cf3e0ba9a996077787fbc945d4a7501b3dee3bbb9a22633651990e72fa
Stable URL of this page
https://metraipolis.com/documento/dc_e53c771bab99

How to cite

JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications — arXiv:2610.06625 — arXiv, https://arxiv.org/abs/2610.06625 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e53c771bab99 (Metraipolis archive dc_e53c771bab99, sha256 3d3205cf3e0ba9a996077787fbc945d4a7501b3dee3bbb9a22633651990e72fa)
BibTeX
@misc{dce53c771bab99,
  title = {JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications — arXiv:2610.06625},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.06625},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e53c771bab99; sha256 3d3205cf3e0ba9a996077787fbc945d4a7501b3dee3bbb9a22633651990e72fa},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.06625. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) annotate and scale political text or constructs by generating text tokens. A new class of models, which TypeSafe markets as "System One" models, instead returns decisions and probability distributions across a user-supplied fixed answer set. A commercial model, JEV, is advertised as having a dramatic cost and speed advantage over traditional LLMs along with better calibrated decisions. As such, it might be useful for social scientists looking to quickly and cost-effectively annotate or scale large corpora of text and have a reliable indicator of a classifier's uncertainty. Yet, the accuracy of these claims and the broader model accuracy in social science text-based tasks are not yet established. In this paper, we do just that and hope to establish the suitability of JEV for social science tasks. We compare JEV with LLMs and human coders from published research, and with a current mid-tier commercial LLM (GPT-6 Luna) and an open-weight alternative (Qwen3.8-27B). We find that JEV matches, or comes close to, the capabilities of both LLMs in a variety of tasks. However, we find no cost advantage over GPT-6 Luna at OpenAI's batch prices. Further, we find that, when each question is asked once, JEV's probabilities are better calibrated than GPT-6 Luna's token probabilities, but not consistently better than Qwen3.8-27B's. We conclude that unless researchers have a need for speed, JEV's only obvious advantage is ease of parsing the underlying choice probabilities.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Computation and Language (cs.CL) ; Computers and Society (cs.CY).
Pagina del lavoro: https://arxiv.org/abs/2610.06625 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:20.379Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).