Metraipolis › archive › dc_84dcaec13cc2

CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents — arXiv:2611.02566

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02566
License (as archived): CC0-1.0
Archived: 2026-11-03T11:25:23.504Z
SHA-256 of the archived text: 860cd678e96a4fc59ca79af5640a3ae764a0ff470d8c175ac3efdee67aabe918
Stable URL of this page: https://metraipolis.com/documento/dc_84dcaec13cc2

How to cite

CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents — arXiv:2611.02566 — arXiv (en), https://arxiv.org/abs/2611.02566 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84dcaec13cc2 (Metraipolis archive dc_84dcaec13cc2, sha256 860cd678e96a4fc59ca79af5640a3ae764a0ff470d8c175ac3efdee67aabe918)

BibTeX

@misc{dc84dcaec13cc2,
  title = {CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents — arXiv:2611.02566},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02566},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84dcaec13cc2; sha256 860cd678e96a4fc59ca79af5640a3ae764a0ff470d8c175ac3efdee67aabe918},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02566. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The rapid growth of automated code generation has created a pressing need for automated code review. Yet code review resists full automation: it depends on context beyond the \textit{diff}, on the author's intent, and on judgments about design and maintainability on which even expert reviewers disagree. Existing code review benchmarks lack standardization and rely largely on noisy human review comments scored with text-similarity metrics, which reward surface overlap and fail to measure precision. We introduce \texttt{CR-Bench}, a defect-centric evaluation framework that recasts SWE-Bench issues as pull-request review tasks with full repository context. After filtering for review-detectable defects and grounding each to a single originating pull request, \texttt{CR-Bench} contains 584 instances, with a 174-instance \texttt{CR-Bench-Verified} subset derived from SWE-Bench Verified. Each instance pairs a pull request with a known defect, its ground-truth fix, and a taxonomy label for bug category, impact, and severity. We propose two complementary metrics: \emph{Known-Defect Detection Rate} (KDDR), which measures whether an agent identifies the labeled defect, and \emph{Usefulness Rate}, which measures the fraction of review comments that are actionable. We evaluate eight code review agents over four backbones, plus commercially available Gemini Code Assist, using three independent LLM judges. Even the strongest systems detect only about one-third of known defects (KDDR $\approx$ 33\%), and Gemini Code Assist detects 8.1\%.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.02566 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T11:02:37.873Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).