Metraipolis › archive › dc_e6f7063ed6ef

DRAFT-Bench: Evaluating Frontier Agents Across the Professional Design Lifecycle — arXiv:2611.08933

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.08933
License (as archived)
CC0-1.0
Archived
2026-11-09T22:34:40.882Z
SHA-256 of the archived text
6c7b51357da584f33a1d2872536705decdf80b140d560f183d730078dd227bc1
Stable URL of this page
https://metraipolis.com/documento/dc_e6f7063ed6ef

How to cite

DRAFT-Bench: Evaluating Frontier Agents Across the Professional Design Lifecycle — arXiv:2611.08933 — arXiv (en), https://arxiv.org/abs/2611.08933 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e6f7063ed6ef (Metraipolis archive dc_e6f7063ed6ef, sha256 6c7b51357da584f33a1d2872536705decdf80b140d560f183d730078dd227bc1)
BibTeX
@misc{dce6f7063ed6ef,
  title = {DRAFT-Bench: Evaluating Frontier Agents Across the Professional Design Lifecycle — arXiv:2611.08933},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08933},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e6f7063ed6ef; sha256 6c7b51357da584f33a1d2872536705decdf80b140d560f183d730078dd227bc1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08933. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Can frontier general-purpose agents understand client intent, explore creative directions, execute revisions, and prepare production deliverables? We introduce DRAFT-Bench (Design Reliability Across Feedback and Tasks), a lifecycle-oriented evaluation of requirement elicitation, creative exploration, constrained editing, iterative revision, and production handoff. Its Requirement-to-Artifact framework links evolving requirements, protected properties, artifact versions, and delivery obligations. We compare native Codex/Astra and Claude Code across 35 design briefs under 43 conditions, alongside a separate 40-scenario RFQ track. Astra completes 43/43 design conditions and Claude 35/43. All scored core runs receive full structural credit, yet on the 28 core conditions scored for both systems, semantic checks flag violations in 11 for Astra and 5 for Claude; these are model judgments, not expert-confirmed failure rates. RFQ coverage is high, but success ordering changes across cohorts. SKU expansion, revision dependencies, and White ink files expose ambiguity among unmet obligations, missing inputs, and evaluator interpretations. These findings separate execution, requirement compliance, and professional acceptance. Unpaired tracks and different tools preclude claims of RFQ-to-delivery success, factory acceptance, or a pure model ranking.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.08933 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.311Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).