Metraipolis › archive › dc_7fc3da7883f4

CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents — arXiv:2611.02566

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.02566
License (as archived): CC0-1.0
Archived: 2026-11-03T18:41:16.421Z
SHA-256 of the archived text: 6121b0c2f753f0bfa0553e840691c06e32446a7184b27414266ac0a10128c9d1
Stable URL of this page: https://metraipolis.com/documento/dc_7fc3da7883f4

How to cite

CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents — arXiv:2611.02566 — arXiv (en), https://arxiv.org/abs/2611.02566 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7fc3da7883f4 (Metraipolis archive dc_7fc3da7883f4, sha256 6121b0c2f753f0bfa0553e840691c06e32446a7184b27414266ac0a10128c9d1)

BibTeX

@misc{dc7fc3da7883f4,
  title = {CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review Agents — arXiv:2611.02566},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.02566},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7fc3da7883f4; sha256 6121b0c2f753f0bfa0553e840691c06e32446a7184b27414266ac0a10128c9d1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02566. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

The rapid growth of automated code generation has created a pressing need for automated code review. Yet code review resists full automation: it depends on context beyond the \textit{diff}, on the author's intent, and on judgments about design and maintainability on which even expert reviewers disagree. Existing code review benchmarks lack standardization and rely largely on noisy human review comments scored with text-similarity metrics, which reward surface overlap and fail to measure precision. We introduce \texttt{CR-Bench}, a defect-centric evaluation framework that recasts SWE-Bench issues as pull-request review tasks with full repository context. After filtering for review-detectable defects and grounding each to a single originating pull request, \texttt{CR-Bench} contains 584 instances, with a 174-instance \texttt{CR-Bench-Verified} subset derived from SWE-Bench Verified. Each instance pairs a pull request with a known defect, its ground-truth fix, and a taxonomy label for bug category, impact, and severity. We propose two complementary metrics: \emph{Known-Defect Detection Rate} (KDDR), which measures whether an agent identifies the labeled defect, and \emph{Usefulness Rate}, which measures the fraction of review comments that are actionable. We evaluate eight code review agents over four backbones, plus commercially available Gemini Code Assist, using three independent LLM judges. Even the strongest systems detect only about one-third of known defects (KDDR $\approx$ 33\%), and Gemini Code Assist detects 8.1\%.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.02566 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.774Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).