Metraipolis › archive › dc_cf618cc161b4

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language — arXiv:2609.06527

Source: arXiv (en)
Original: https://arxiv.org/abs/2609.06527
License (as archived): CC0-1.0
Archived: 2026-10-16T12:45:13.710Z
SHA-256 of the archived text: 0dea9793f8a9ed4fcb4e501dfbc0f9f4f1158706921dc843b498c8475096769a
Stable URL of this page: https://metraipolis.com/documento/dc_cf618cc161b4

How to cite

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language — arXiv:2609.06527 — arXiv (en), https://arxiv.org/abs/2609.06527 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cf618cc161b4 (Metraipolis archive dc_cf618cc161b4, sha256 0dea9793f8a9ed4fcb4e501dfbc0f9f4f1158706921dc843b498c8475096769a)

BibTeX

@misc{dccf618cc161b4,
  title = {ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language — arXiv:2609.06527},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06527},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cf618cc161b4; sha256 0dea9793f8a9ed4fcb4e501dfbc0f9f4f1158706921dc843b498c8475096769a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06527. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Databases (cs.DB).
Pagina del lavoro: https://arxiv.org/abs/2609.06527 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T12:30:25.043Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).