Metraipolis › archive › dc_84558cb66ace
Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering — arXiv:2608.27621
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.27621
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-17T11:14:02.744Z
- SHA-256 of the archived text
d8b655482c684bc8b447a792379cbc1382d95206596b4cfbb3422933dacffca9- Stable URL of this page
https://metraipolis.com/documento/dc_84558cb66ace
How to cite
Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering — arXiv:2608.27621 — arXiv (en), https://arxiv.org/abs/2608.27621 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_84558cb66ace (Metraipolis archive dc_84558cb66ace, sha256 d8b655482c684bc8b447a792379cbc1382d95206596b4cfbb3422933dacffca9)
BibTeX
@misc{dc84558cb66ace,
title = {Predicting LLM Performance from Prompt Linguistic Features: An Empirical Study in Requirements Engineering — arXiv:2608.27621},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.27621},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_84558cb66ace; sha256 d8b655482c684bc8b447a792379cbc1382d95206596b4cfbb3422933dacffca9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27621. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Background. LLM outputs are highly sensitive to prompt formulation: small wording changes can substantially affect output quality. This matters in software engineering, where prompts guide requirements analysis, code generation, and artefact synthesis. Poor formulations yield unreliable artefacts, yet practitioners lack principled ways to assess a prompt before inference, making selection depend on costly LLM calls and trial-and-error refinement. Aims. We investigate whether measurable linguistic properties of prompts can predict LLM performance before inference, enabling low-cost prompt selection and refinement, validated on binary requirements classification targeting F1, F2, precision, and recall. Method. We generate 9,000 linguistically controlled prompt variants from 100 initial prompts by varying 30 linguistic metrics, evaluated with five open-source LLMs on 625 annotated requirements. Regression predictors are trained via stratified 10-fold cross-validation with permutation-based significance testing; feature importance analysis identifies cross-LLM and model-specific predictors. Results. Linguistic features significantly predict prompt performance across all targets (R2 in [0.38,0.42], q<0.05). Syntactic and morphosyntactic features drive most predictive signal; cross-LLM predictors include compound dependency distribution, conjunction density, and word/sentence length, reflecting sensitivity to domain vocabulary and complex structures. Conclusions. Results suggest practical implications for prompt engineering, including overlap between linguistic patterns that reduce LLM performance and those that increase human comprehension difficulty, and the irrelevance of lexical variety as a quality dimension. More broadly, linguistic profiling combined with standard regression provides an effective, interpretable, low-cost prior before costly optimisation pipelines. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-31 in cs.SE). Categorie arXiv: Software Engineering (cs.SE). Pagina del lavoro: https://arxiv.org/abs/2608.27621 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-17T10:56:43.369Z dalla pagina https://arxiv.org/catchup/cs.SE/2026-08-31?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).