Metraipolis › archive › dc_60c8d7337007

Benchmarking Prompt Optimization of Large Language Models With Chess — arXiv:2610.00416

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00416
License (as archived)
CC0-1.0
Archived
2026-10-06T12:02:02.839Z
SHA-256 of the archived text
226248d2aa3265264eebe29709a3307a598b9315c056a0c8f94fcc28e5eb0fd8
Stable URL of this page
https://metraipolis.com/documento/dc_60c8d7337007

How to cite

Benchmarking Prompt Optimization of Large Language Models With Chess — arXiv:2610.00416 — arXiv (en), https://arxiv.org/abs/2610.00416 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_60c8d7337007 (Metraipolis archive dc_60c8d7337007, sha256 226248d2aa3265264eebe29709a3307a598b9315c056a0c8f94fcc28e5eb0fd8)
BibTeX
@misc{dc60c8d7337007,
  title = {Benchmarking Prompt Optimization of Large Language Models With Chess — arXiv:2610.00416},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00416},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_60c8d7337007; sha256 226248d2aa3265264eebe29709a3307a598b9315c056a0c8f94fcc28e5eb0fd8},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00416. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamination, and assessing harder tasks can require expensive grading or execution infrastructure. These challenges are amplified in automatic prompt optimization (APO), where evaluation is repeated throughout the search for better prompts. Studying APO therefore requires a benchmark that is cheap and deterministic to score, hard enough to leave room for improvement, and renewable as models evolve. We introduce a chess benchmark built from 1,118 Lichess puzzles to study APO for frozen LLMs: we optimize their prompts without updating their model weights. Chess combines inexpensive exact-match scoring, engine-based evaluation of alternative moves, and a renewable supply of problems with adjustable difficulty. Unlike evaluations that report only success on isolated test items, the benchmark also connects puzzle-solving gains to short game-play rollouts within the same domain. We use it to evaluate six APO algorithms on eight target models, measuring not only baseline strength but also how much each model responds to optimization and whether optimized prompts transfer across models and to game play. Chess is thus a well-suited benchmark for APO: it is (i) challenging, as even the strongest evaluated model, Gemini 3.5 Flash (used as the meta-model), solves only about 55\% of puzzles; (ii) discriminative, revealing gains, unchanged performance, and regressions across methods and models; (iii) renewable, with fresh puzzles to reduce contamination risk and adjustable difficulty to maintain headroom as models improve; and (iv) affordable, as the complete study runs for around \$800. We release the puzzles, optimization and evaluation code, and dataset-renewal scripts ( this https URL ).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.00416 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:27.864Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).