Metraipolis › archive › dc_6f52d4339aea
Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks — arXiv:2610.11978
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.11978
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-09T06:13:28.109Z
- SHA-256 of the archived text
bb66c3d1752fceefed3ced7939153f7b9621c8a530fb06e831e9877bc0c1517d- Stable URL of this page
https://metraipolis.com/documento/dc_6f52d4339aea
How to cite
Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks — arXiv:2610.11978 — arXiv (en), https://arxiv.org/abs/2610.11978 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6f52d4339aea (Metraipolis archive dc_6f52d4339aea, sha256 bb66c3d1752fceefed3ced7939153f7b9621c8a530fb06e831e9877bc0c1517d)
BibTeX
@misc{dc6f52d4339aea,
title = {Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks — arXiv:2610.11978},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.11978},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6f52d4339aea; sha256 bb66c3d1752fceefed3ced7939153f7b9621c8a530fb06e831e9877bc0c1517d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11978. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Jev is a "System One" model that returns a choice among given options instead of generating text. We study how such a specialized decision model compares with general-purpose large language models (LLMs). We evaluate Jev on 13 multiple-choice benchmarks covering knowledge, reasoning, and multilingual understanding, and compare it with 19 LLMs in three tiers: frontier, representative, and small. Jev is competitive with frontier LLMs on knowledge and commonsense benchmarks and obtains the best score on MMLU-Redux and ARC-Challenge. Outside mathematics, it also outperforms most representative LLMs and all small LLMs. However, it falls behind on mathematical word problems: on MathQA, it is 17.7 points below the frontier median and scores lower than all 19 LLMs. These results indicate that a specialized decision model can match general-purpose LLMs on decisions that rely mainly on knowledge, but not on decisions that require multi-step calculation. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.11978 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-09T05:47:15.312Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).