Metraipolis › archive › dc_fc2445e43bef

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code — arXiv:2610.40735

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.40735
License (as archived)
CC0-1.0
Archived
2026-11-03T08:02:29.042Z
SHA-256 of the archived text
d6b4ddfc2c18a8139a6e4d8f68563b2eb3b224983e62dc275ce8337b5f8b3c80
Stable URL of this page
https://metraipolis.com/documento/dc_fc2445e43bef

How to cite

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code — arXiv:2610.40735 — arXiv (en), https://arxiv.org/abs/2610.40735 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fc2445e43bef (Metraipolis archive dc_fc2445e43bef, sha256 d6b4ddfc2c18a8139a6e4d8f68563b2eb3b224983e62dc275ce8337b5f8b3c80)
BibTeX
@misc{dcfc2445e43bef,
  title = {RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code — arXiv:2610.40735},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40735},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fc2445e43bef; sha256 d6b4ddfc2c18a8139a6e4d8f68563b2eb3b224983e62dc275ce8337b5f8b3c80},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40735. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating approaches that improve the security of LLM code generation requires datasets that jointly assess security and functionality and remain adapted to the capabilities of the models being studied. However, existing benchmarks are costly to produce and are generally not calibrated to a chosen range of model capabilities. We introduce RiskyPy, an automated pipeline that generates security-oriented code-generation datasets calibrated to model proficiency using functionality and vulnerability canaries. We instantiate RiskyPy for Python using domains covering 18 CWEs from the 2025 CWE Top 25, producing 101 prompts. Experiments on Qwen3-4B and Seed-Coder-8B show that the generated tasks remain functionally accessible, with 57-72% of prompts yielding functional code, while only 5-10% of generations are free of Bandit findings, leaving room for joint evaluation of security and functionality of security improvement approaches.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.40735 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T07:59:38.763Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).