Metraipolis › archive › dc_96f8d4fefc8f
DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions? — arXiv:2609.15122
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.15122
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-15T22:53:39.256Z
- SHA-256 of the archived text
3eb093e1bc81ae9f7753d4be31fe0ba1b81b0123dc4f63c70abc1bd54d299c24- Stable URL of this page
https://metraipolis.com/documento/dc_96f8d4fefc8f
How to cite
DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions? — arXiv:2609.15122 — arXiv (en), https://arxiv.org/abs/2609.15122 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_96f8d4fefc8f (Metraipolis archive dc_96f8d4fefc8f, sha256 3eb093e1bc81ae9f7753d4be31fe0ba1b81b0123dc4f63c70abc1bd54d299c24)
BibTeX
@misc{dc96f8d4fefc8f,
title = {DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions? — arXiv:2609.15122},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.15122},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_96f8d4fefc8f; sha256 3eb093e1bc81ae9f7753d4be31fe0ba1b81b0123dc4f63c70abc1bd54d299c24},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.15122. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as a direct route to more reliable evaluation. Under a fixed budget, however, auditing each program more thoroughly reduces the number of tasks and independent generations that can be evaluated, which can ultimately make model-level estimates less accurate. We study this phenomenon and the conditions under which it arises. We decompose behavioral evaluation into three evidence levels: task templates, stochastic generations, and within-program edits. We define an average failure risk that is invariant to audit depth, and combine three-level variance with measured execution costs to analyze the tradeoff between deeper edit auditing and broader independent coverage. Experiments across two CAD environments and five generation systems show that the value of deeper auditing depends on where evaluation uncertainty originates. When template heterogeneity or generation stochasticity dominates, additional edit checks can increase total estimation error; when within-program state variation is large and generation is expensive, deeper auditing is more valuable. Variance and cost estimates from calibration predict the direction of this change and provide a diagnostic basis for allocating evidence on held-out tasks. These results show that the thoroughness of program inspection can diverge from the reliability of model evaluation, and they help determine whether the next unit of budget should be spent on a new task, a new generation, or additional edit checks. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.AI). Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.15122 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-15T22:29:03.810Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).