Metraipolis › archive › dc_7546d926fa02

Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs — arXiv:2608.11232

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.11232
License (as archived)
CC0-1.0
Archived
2026-10-19T05:15:38.363Z
SHA-256 of the archived text
48d3b5a69f571d84830c2ec866974b2bac15fe77a6f10367b90edb7596759c26
Stable URL of this page
https://metraipolis.com/documento/dc_7546d926fa02

How to cite

Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs — arXiv:2608.11232 — arXiv (en), https://arxiv.org/abs/2608.11232 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7546d926fa02 (Metraipolis archive dc_7546d926fa02, sha256 48d3b5a69f571d84830c2ec866974b2bac15fe77a6f10367b90edb7596759c26)
BibTeX
@misc{dc7546d926fa02,
  title = {Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs — arXiv:2608.11232},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.11232},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7546d926fa02; sha256 48d3b5a69f571d84830c2ec866974b2bac15fe77a6f10367b90edb7596759c26},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.11232. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs require ground truth from actual code execution. We present Backtrader-Bench, a framework with two complementary pipelines. A deterministic multiple-choice question (MCQ) pipeline generates questions from backtest configurations across five trading strategies, 33 templates, and three difficulty tiers, with an independent checker that re-derives every answer. A generator-solver filtering pipeline autonomously mines harder questions: a generator writes questions verified by executable code, converts them to MCQs, and discards any that a no-tool solver can answer without code execution. We evaluate 11 models without tools (10 runs each) and four with-tools configurations on a 30-question curated set. Tool-augmented agents reach 90.0% accuracy in a single pass (GPT-5.5 and Opus 4.7), outperforming the best no-tools baselines (73.0%, averaged over 10 runs) by 17 percentage points. On 38 separately mined questions, no-tools accuracy drops further, with half the models falling to roughly random-chance level (25%). Beyond evaluation, the scalable MCQ infrastructure is designed to produce a training corpus for reinforcement learning, with the ultimate goal of building a specialized agent for quantitative trading workflows.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-13 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.11232 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:08:46.661Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).