Metraipolis › archive › dc_2f670fed7856

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code — arXiv:2610.03080

Source
arXiv
Original
https://arxiv.org/abs/2610.03080
License (as archived)
CC0-1.0
Archived
2026-10-05T22:51:59.809Z
SHA-256 of the archived text
91b4e4f29cb58e6bd5452b985a265b90576ab04bf71e7ac10a18cf2dab290b53
Stable URL of this page
https://metraipolis.com/documento/dc_2f670fed7856

How to cite

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code — arXiv:2610.03080 — arXiv, https://arxiv.org/abs/2610.03080 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2f670fed7856 (Metraipolis archive dc_2f670fed7856, sha256 91b4e4f29cb58e6bd5452b985a265b90576ab04bf71e7ac10a18cf2dab290b53)
BibTeX
@misc{dc2f670fed7856,
  title = {MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code — arXiv:2610.03080},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.03080},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2f670fed7856; sha256 91b4e4f29cb58e6bd5452b985a265b90576ab04bf71e7ac10a18cf2dab290b53},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.03080. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: generated code runs, a backtest plots, yet the risk logic that the trader described is not the logic being executed. Existing code benchmarks test functional correctness on unit tests and finance benchmarks test forecasting; neither measures whether an implementation behaves like the strategy that was asked for. We introduce MintEval, a benchmark in which reference strategies are generated programmatically from a library of composable building blocks, back-translated into colloquial trader instructions, and re-implemented by the model under test. Generated and reference programs are executed bar by bar on identical market data and frictions, and compared on their actions rather than on code similarity or profit: alpha is differenced away. MintEval v0 contains 800 tasks on BTCUSDT 15-minute data, stratified by an execution-measured state-span complexity tau that is decoupled from description length. Low-cost models reach a mean ActionMatch of at most 0.544 and reproduce at most 0.087 of tasks exactly; on a stratified subset of 200 tasks a frontier model (Claude Opus 5.5) reaches 0.889 and reproduces 0.575 exactly, yet still fails silently on 0.275 of tasks. Given a menu of building blocks, models identify the strategy almost perfectly, yet 79.2% of the implementations whose specification was read correctly diverge on more than 10% of active bars. The LLM judge of a recent strategy-generation benchmark, applied verbatim, accepts every one of these silent failures.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Computation and Language (cs.CL); Machine Learning (cs.LG); Trading and Market Microstructure (q-fin.TR).
Pagina del lavoro: https://arxiv.org/abs/2610.03080 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-05T22:46:10.623Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).