Metraipolis › archive › dc_65ea0f880cdf
ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge — arXiv:2609.12366
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.12366
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T03:03:20.254Z
- SHA-256 of the archived text
ea40677ccc48311d89f0c8330901257b75ecc50b3e69ccf1573a974b94402dde- Stable URL of this page
https://metraipolis.com/documento/dc_65ea0f880cdf
How to cite
ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge — arXiv:2609.12366 — arXiv (en), https://arxiv.org/abs/2609.12366 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_65ea0f880cdf (Metraipolis archive dc_65ea0f880cdf, sha256 ea40677ccc48311d89f0c8330901257b75ecc50b3e69ccf1573a974b94402dde)
BibTeX
@misc{dc65ea0f880cdf,
title = {ORQA: An Occupation-Realistic Question and Answer Framework for LLM Professional Knowledge — arXiv:2609.12366},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.12366},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_65ea0f880cdf; sha256 ea40677ccc48311d89f0c8330901257b75ecc50b3e69ccf1573a974b94402dde},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.12366. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We present ORQA, a method for testing occupation-level knowledge in large language models. Prior methods either map abstract LLM skills to occupations via task definitions or utilize expert knowledge which is difficult to obtain at scale and expensive. ORQA complements both of these methods by connecting O*NET occupations to trusted occupation-specific websites (such as regulatory agencies, licensing bodies, professional organizations, and government publications) and converting these into source-traceable question-answer pairs. A combination of an automated pipeline and human review produces a set of high quality questions about occupations. The question set created via our method covers 116 occupations from all 21 major groups in the SOC, with 480 questions sourced from 187 different websites. Each question is designed to probe a real-world skill question that is relevant to the occupation in question. We test 15 state-of-the-art frontier and open-weight models via this method. Claude Opus 4.6, GPT-5.4 and Claude Sonnet 4.6 all perform the best at approximately 58-62% while smaller open-weight models achieve approximately 33-41% performance. Performance varies significantly across occupations. Healthcare-related occupations achieve the highest performance (78%) while Office and Administrative Support achieve approximately 40%. Performance on individual occupations (e.g. Sheet Metal Workers and Fish and Game Wardens) is essentially zero. We also find that open-ended questions and weighting by wage bill do not significantly affect the ranking of models on this benchmark. We believe that leveraging existing trusted occupation-specific information to test LLM knowledge in professional domains may be a scalable and useful method for evaluating occupation-level AI performance in the future. Results and data are available at this http URL . Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-14 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.12366 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T02:51:27.899Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-14?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).