Metraipolis › archive › dc_334bd089a0d3
QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks — arXiv:2609.35581
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.35581
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T21:03:48.402Z
- SHA-256 of the archived text
5ed722c0073abbffd9aa860624ca01863e563ca3f6a00326865a9dc2736e3cf8- Stable URL of this page
https://metraipolis.com/documento/dc_334bd089a0d3
How to cite
QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks — arXiv:2609.35581 — arXiv (en), https://arxiv.org/abs/2609.35581 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_334bd089a0d3 (Metraipolis archive dc_334bd089a0d3, sha256 5ed722c0073abbffd9aa860624ca01863e563ca3f6a00326865a9dc2736e3cf8)
BibTeX
@misc{dc334bd089a0d3,
title = {QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks — arXiv:2609.35581},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.35581},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_334bd089a0d3; sha256 5ed722c0073abbffd9aa860624ca01863e563ca3f6a00326865a9dc2736e3cf8},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35581. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debugging, compilation, error correction, and simulation. Across 2,750 evaluations (10 models $\times$ 11 tasks x 5 difficulty levels x 5 seeds), we find that overall rankings mask substantial per-task variation. The Spearman correlation between overall and per-task rankings is statistically insignificant for 4 out of the 11 tasks included in this benchmark. A 2-parameter Item Response Theory (IRT) model validates measurement quality, and prompt sensitivity analysis confirms ranking robustness across prompt conditions. All tasks are auto-verifiable via execution, thus not requiring any manual evaluation. We make the code and data publicly available on Huggingface. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI). Categorie arXiv: Quantum Physics (quant-ph) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.35581 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T20:29:03.405Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).