Metraipolis › archive › dc_dc9769aed110

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code — arXiv:2610.40735

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.40735
License (as archived): CC0-1.0
Archived: 2026-11-02T07:29:27.670Z
SHA-256 of the archived text: bf129ffe429a9f6139f6b8b1d00c3d8f42eac49dc5f082e87a7901239374db37
Stable URL of this page: https://metraipolis.com/documento/dc_dc9769aed110

How to cite

RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code — arXiv:2610.40735 — arXiv (en), https://arxiv.org/abs/2610.40735 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_dc9769aed110 (Metraipolis archive dc_dc9769aed110, sha256 bf129ffe429a9f6139f6b8b1d00c3d8f42eac49dc5f082e87a7901239374db37)

BibTeX

@misc{dcdc9769aed110,
  title = {RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code — arXiv:2610.40735},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.40735},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_dc9769aed110; sha256 bf129ffe429a9f6139f6b8b1d00c3d8f42eac49dc5f082e87a7901239374db37},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.40735. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Evaluating approaches that improve the security of LLM code generation requires datasets that jointly assess security and functionality and remain adapted to the capabilities of the models being studied. However, existing benchmarks are costly to produce and are generally not calibrated to a chosen range of model capabilities. We introduce RiskyPy, an automated pipeline that generates security-oriented code-generation datasets calibrated to model proficiency using functionality and vulnerability canaries. We instantiate RiskyPy for Python using domains covering 18 CWEs from the 2025 CWE Top 25, producing 101 prompts. Experiments on Qwen3-4B and Seed-Coder-8B show that the generated tasks remain functionally accessible, with 57-72% of prompts yielding functional code, while only 5-10% of generations are free of Bandit findings, leaving room for joint evaluation of security and functionality of security improvement approaches.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Cryptography and Security (cs.CR) ; Artificial Intelligence (cs.AI); Software Engineering (cs.SE).
Pagina del lavoro: https://arxiv.org/abs/2610.40735 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-02T07:26:57.755Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).