Metraipolis › archive › dc_6ab2eaae88f8

Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study — arXiv:2609.03721

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.03721
License (as archived)
CC0-1.0
Archived
2026-10-17T01:12:04.172Z
SHA-256 of the archived text
94b1e3c3ee21ebf3cefd33b6cd78d8a15ab2e33c48b5b0906ce23334f2f99a4a
Stable URL of this page
https://metraipolis.com/documento/dc_6ab2eaae88f8

How to cite

Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study — arXiv:2609.03721 — arXiv (en), https://arxiv.org/abs/2609.03721 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6ab2eaae88f8 (Metraipolis archive dc_6ab2eaae88f8, sha256 94b1e3c3ee21ebf3cefd33b6cd78d8a15ab2e33c48b5b0906ce23334f2f99a4a)
BibTeX
@misc{dc6ab2eaae88f8,
  title = {Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study — arXiv:2609.03721},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.03721},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6ab2eaae88f8; sha256 94b1e3c3ee21ebf3cefd33b6cd78d8a15ab2e33c48b5b0906ce23334f2f99a4a},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.03721. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Context: Architectural Design Decisions (ADDs) capture the rationale behind the structure and evolution of software systems but are rarely documented explicitly, and are often hidden inside source code commits. Recovering them is important for Architectural Knowledge Management (AKM). Problem: Extracting ADDs from commits is challenging due to their implicit and unstructured nature. Large Language Models (LLMs) have shown strong capabilities in understanding code and text, yet their effectiveness for this task remains underexplored. Study: We present a preliminary study using four LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2, Qwen3) with zeroshot and fewshot prompting on 30 developer-written ADDs from open-source projects. We score outputs with ROUGE-L, BLEU, METEOR, and BERTScore, and one author manually reviews the Gemini outputs. Results: All models reach a BERT-F1 above 0.81, and fewshot prompting improves alignment (Gemini BERT-F1: 0.828 to 0.847). However, the generated ADDs are often too long, implementation-focused, and miss the rationale behind the decision. This highlights opportunities for architecture-aware LLM systems and automated AKM.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-04 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2609.03721 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T00:28:02.997Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-04?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).