Metraipolis › archive › dc_cca12df2a80b
LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems — arXiv:2610.07580
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.07580
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T11:18:50.358Z
- SHA-256 of the archived text
42099ed33f73f1bbe362122f04b682bf61dabddadeda2827a657ab1c21fbf42d- Stable URL of this page
https://metraipolis.com/documento/dc_cca12df2a80b
How to cite
LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems — arXiv:2610.07580 — arXiv (en), https://arxiv.org/abs/2610.07580 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cca12df2a80b (Metraipolis archive dc_cca12df2a80b, sha256 42099ed33f73f1bbe362122f04b682bf61dabddadeda2827a657ab1c21fbf42d)
BibTeX
@misc{dccca12df2a80b,
title = {LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems — arXiv:2610.07580},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.07580},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cca12df2a80b; sha256 42099ed33f73f1bbe362122f04b682bf61dabddadeda2827a657ab1c21fbf42d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.07580. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Aerospace electrical-design revisions can contain multiple genuine changes, although an engineering request may authorize only a subset. Propagating every detected difference can therefore produce overly broad impact reports. We present LOGIC, a controlled benchmark and evaluation framework in which locally deployable language models ground a request in a deterministic candidate-change inventory before selected changes are propagated through a typed electrical traceability graph. This separation permits candidate-selection errors to be distinguished from downstream propagation errors. LOGIC contains 168 scenarios, including 144 selection and 24 abstention cases. We evaluate three 7--8B models against intent-agnostic, lexical, and structured-evidence methods, with an oracle-root upper bound. On 96 explicitly anchored selection cases, gate-only structured evidence achieves candidate F1 of 1.0000, compared with 0.9677 for token-lexical matching. On 12 relational-paraphrase cases, token-lexical F1 is 0.1772 and gate-only F1 is 0.0000, compared with 0.5000--0.6400 for the large language models. Model grounding degrades as candidate inventories grow from 4 to 64 changes, while affected-element and typed-path accuracy remain comparatively stable when frozen selections are replayed over graphs of approximately 1K to 100K nodes. Strict evidence gating suppresses false positives but can remove correct semantic selections. An exploratory evidence-empty abstention policy raises strict abstention accuracy to 0.6667 for all three models and reduces unsafe-report rates to 0.1667, while decreasing answerable-case coverage by 16.0--27.1 percentage points. Four of six conflicting requests remain unsafe for each model. These findings support combining literal evidence and language-model reasoning with engineering review when intent cannot be established reliably. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno). Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.07580 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T11:15:24.524Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).