Metraipolis › archive › dc_7a189fbe3433

Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks — arXiv:2611.03162

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.03162
License (as archived)
CC0-1.0
Archived
2026-11-03T18:20:21.769Z
SHA-256 of the archived text
497850686087520ce78c08d0a3c47afbd149b26cc8a2ea1ab0ebe75607c58b79
Stable URL of this page
https://metraipolis.com/documento/dc_7a189fbe3433

How to cite

Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks — arXiv:2611.03162 — arXiv (en), https://arxiv.org/abs/2611.03162 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7a189fbe3433 (Metraipolis archive dc_7a189fbe3433, sha256 497850686087520ce78c08d0a3c47afbd149b26cc8a2ea1ab0ebe75607c58b79)
BibTeX
@misc{dc7a189fbe3433,
  title = {Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding Benchmarks — arXiv:2611.03162},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.03162},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7a189fbe3433; sha256 497850686087520ce78c08d0a3c47afbd149b26cc8a2ea1ab0ebe75607c58b79},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.03162. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Repository-level AI coding benchmarks measure task success but do not directly capture the conditions required for software maintenance completion. We study multi-condition fulfillment, where a maintenance task must satisfy multiple operational conditions, sometimes with directional dependencies. Using a fixed 18-criterion taxonomy, we analyze 2,925 Apache Fineract issues and 944 tasks from SWE-bench Pro, SWE-PolyBench Java, and SWE-EVO. We examine condition breadth, corpus-level co-occurrence, and source-grounded directional requirement relationships. Among 1,177 benchmark task pairs matched on issue classification, modified-file count, and code churn, the mean fulfillment-set Jaccard distance is 0.8138, and 57.4% have disjoint condition sets. Condition count alone does not explain this contrast: SWE-bench Pro averages 2.47 conditions per task, versus 1.78 for Fineract. Temporal (B4) and numerical (B2) conditions have a phi coefficient of +0.430 in Fineract versus -0.013 in the pooled benchmark corpus. Among temporally tagged tasks, numerical conditions are also specified in 48.85% (254/520) of Fineract cases versus 5.81% (5/86) of benchmark cases. We document 16 directional relationships in Fineract requirements involving time, calculation, and state preservation, and inspect four benchmark cases to illustrate how relevant conditions appear in evaluation assertions. These examples do not quantify benchmark-wide oracle omissions. The results show that similar surface task profiles can mask substantially different fulfillment-condition sets. We recommend reporting condition breadth, co-occurrence, and directional relationships separately. The corpus comparisons are descriptive and may reflect domain differences; they do not establish causality or production failure rates.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.03162 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:09:53.740Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).