Metraipolis › archive › dc_caefdd2b62ba

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset — arXiv:2611.06373

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.06373
License (as archived): CC0-1.0
Archived: 2026-11-05T18:26:28.831Z
SHA-256 of the archived text: 1bbb473ad8a6aa2843b2583ecb900b4c371034cb0b5b22483bba4d338a05c04c
Stable URL of this page: https://metraipolis.com/documento/dc_caefdd2b62ba

How to cite

Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset — arXiv:2611.06373 — arXiv (en), https://arxiv.org/abs/2611.06373 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_caefdd2b62ba (Metraipolis archive dc_caefdd2b62ba, sha256 1bbb473ad8a6aa2843b2583ecb900b4c371034cb0b5b22483bba4d338a05c04c)

BibTeX

@misc{dccaefdd2b62ba,
  title = {Reassessing Dataset Quality and Fine-Tuning Practices for Vulnerability Detection on PrimeVul dataset — arXiv:2611.06373},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.06373},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_caefdd2b62ba; sha256 1bbb473ad8a6aa2843b2583ecb900b4c371034cb0b5b22483bba4d338a05c04c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.06373. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Recent studies report promising results from supervised fine-tuning of pre-trained code language models for automated vulnerability detection on benchmarks such as PrimeVul. In this paper, we present a large-scale empirical evaluation of both model performance and dataset quality in this setting. We systematically fine-tune three widely used encoder models (CodeBERT, GraphCodeBERT, and UniXCoder) across five imbalance mitigation strategies, extensive hyperparameter sweeps, and multiple random seeds, resulting in 770 independent training runs. Our results reveal a consistent pattern. While standard metrics such as F1 and ROC-AUC occasionally suggest improvements, security-critical and calibration-aware measures tell a different story. Matthews Correlation Coefficient remains near zero, predicted probabilities are poorly calibrated, and evaluation under strict false-positive constraints reveals high false-negative rates. Neither imbalance handling, threshold optimization, nor post-hoc calibration recovers a meaningful discriminative signal. We also conduct a data quality assessment of the PrimeVul benchmark and identify structural issues such as minimal semantic differences between vulnerable and fixed functions, and annotation noise introduced by commit-level labeling. These findings suggests that the PrimeVul dataset, in its current form, presents significant challenges for reliable AI-based vulnerability detection in software engineering with the methodologies we used. Our study highlights the need for more robust dataset design, improved evaluation protocols, and rethinking of fine-tuning practices for security-critical tasks.

Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.SE, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2611.06373 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-05T18:16:03.884Z dalla pagina https://arxiv.org/list/cs.SE/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).