Metraipolis › archive › dc_eb7d3a34acef

Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review — arXiv:2609.06213

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.06213
License (as archived)
CC0-1.0
Archived
2026-10-16T14:09:01.187Z
SHA-256 of the archived text
2c4bf625f56b580cfd82b514d9b503a08ac584a498a9fcc82ec9fcb132625632
Stable URL of this page
https://metraipolis.com/documento/dc_eb7d3a34acef

How to cite

Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review — arXiv:2609.06213 — arXiv (en), https://arxiv.org/abs/2609.06213 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eb7d3a34acef (Metraipolis archive dc_eb7d3a34acef, sha256 2c4bf625f56b580cfd82b514d9b503a08ac584a498a9fcc82ec9fcb132625632)
BibTeX
@misc{dceb7d3a34acef,
  title = {Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review — arXiv:2609.06213},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.06213},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eb7d3a34acef; sha256 2c4bf625f56b580cfd82b514d9b503a08ac584a498a9fcc82ec9fcb132625632},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.06213. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Code review is a key quality checkpoint between AI-generated code and production. As AI coding agents submit pull requests at scale, it is unclear whether reviewers reduce scrutiny with repeated exposure and whether review comments reveal this change. We study 11,429 reviews from 400 repeat reviewers over 207 days, paired with 10,104 human-authored inline comments from AIDev. Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods (Wilcoxon p = 8.6 x 10^-8; Cohen's d = 0.25). However, four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles (all Spearman absolute rho <= 0.53, p >= 0.11; Bonferroni-corrected Mann-Whitney p >= 0.36). A logistic-regression classifier based on these features reaches F1 = 0.485, below the majority-class baseline. Sentence-embedding structure does carry signal: reviewers' late-period comment centroids shift farther from their early-period centroids than under within-reviewer random permutations (Wilcoxon p < 0.001), and a small MLP using three embedding statistics reaches F1 = 0.74 under 5-fold reviewer-stratified cross-validation. Granger analysis shows that approval-rate changes predict later shifts in technical specificity at all tested lags (p < 0.001), while the reverse direction is significant at only one of four lags. Reviewer adaptation is therefore detectable in latent distributional structure rather than in classical lexical metrics, and language shifts follow rather than precede changes in approval behavior.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-09 in cs.SE).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2609.06213 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:58:36.361Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).