Metraipolis › archive › dc_5491ed2b5907

FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models — arXiv:2608.20153

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.20153
License (as archived)
CC0-1.0
Archived
2026-10-18T11:39:23.660Z
SHA-256 of the archived text
6a266102d2906d369fde6c1054f45ace02d524eb5663b148a901da3bbd052695
Stable URL of this page
https://metraipolis.com/documento/dc_5491ed2b5907

How to cite

FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models — arXiv:2608.20153 — arXiv (en), https://arxiv.org/abs/2608.20153 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5491ed2b5907 (Metraipolis archive dc_5491ed2b5907, sha256 6a266102d2906d369fde6c1054f45ace02d524eb5663b148a901da3bbd052695)
BibTeX
@misc{dc5491ed2b5907,
  title = {FormalTCS: Benchmarking End-to-End Frontier Formal Theoretical Computer Science Research of Large Language Models — arXiv:2608.20153},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.20153},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5491ed2b5907; sha256 6a266102d2906d369fde6c1054f45ace02d524eb5663b148a901da3bbd052695},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.20153. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) have shown growing potential for automated theoretical computer science (TCS) research, yet existing benchmarks remain far from realistic research settings. We introduce \ourbenchmark, an expert-validated benchmark for evaluating LLMs on frontier, end-to-end TCS research. \ourbenchmark contains $143$ instances drawn from papers accepted to STOC, FOCS, SODA, and COLT in 2025-2026, preserving paper-specific definitions, assumptions, and proof dependencies, with expert-verified Lean formalizations and proofs. Evaluations of leading LLMs reveal that current models remain far from reliably completing the full research pipeline. In particular, autoformalization is the sharpest bottleneck: the best model achieves only $11.5$ on translating natural-language claims into formal theorem statements, compared with $28.6$ Pass@8 when proving human-provided formal statements. Building on \ourbenchmark, we further develop an automated TCS research framework that generates, formalizes, filters, and proves new claims. Of $64$ generated claims, only $6$ ultimately pass expert evaluation and proof verification, indicating that beyond formalization, limited research taste remains another major barrier to autonomous TCS research.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-21 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.20153 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T11:38:27.929Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-21?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).