Metraipolis › archive › dc_c05de7d76c8f

Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows — arXiv:2610.11482

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.11482
License (as archived)
CC0-1.0
Archived
2026-10-09T06:00:01.195Z
SHA-256 of the archived text
5a42e64cdbb07bc998c078b6b86ca9bedba9f3b69647cf576fa382f18c5d559c
Stable URL of this page
https://metraipolis.com/documento/dc_c05de7d76c8f

How to cite

Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows — arXiv:2610.11482 — arXiv (en), https://arxiv.org/abs/2610.11482 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_c05de7d76c8f (Metraipolis archive dc_c05de7d76c8f, sha256 5a42e64cdbb07bc998c078b6b86ca9bedba9f3b69647cf576fa382f18c5d559c)
BibTeX
@misc{dcc05de7d76c8f,
  title = {Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows — arXiv:2610.11482},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.11482},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_c05de7d76c8f; sha256 5a42e64cdbb07bc998c078b6b86ca9bedba9f3b69647cf576fa382f18c5d559c},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.11482. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Context: Large language model (LLM) agents are increasingly used as software and data-engineering assistants, yet evidence about locally deployable open-weight agents remains limited. Existing evaluations often emphasize textual responses or isolated code generation rather than the validity of complete engineering artifacts. Objectives: We evaluate whether local LLM agents can produce correct and reproducible data-engineering artifacts, quantify the effect of a closed-loop workspace condition, and examine trade-offs in model scale, architecture, quantization, runtime, tool use, and failure. Methods: We introduce a benchmark of fifteen mobility-workflow tasks covering data discovery, connectors, transport-feed processing, semantic enrichment, feature engineering, validation, visualization, and reporting. Deterministic checkers assess generated scripts, tables, structured files, figures, and reports. Ten local configurations are evaluated in one-shot and closed-loop conditions, with five repetitions per model, mode, and task, yielding 1,500 scored attempts on a consumer-grade GPU. Results: Among models larger than two billion parameters, the workspace condition increases pass rates by 26.7-52.0 percentage points over one-shot generation. The strongest configuration reaches 85.3% artifact-level success, and a quantized 9-billion-parameter model reaches 69.3% with an approximately 6.5 GB memory footprint. Gains are largest when intermediate artifacts expose errors the agent can inspect and repair. Conclusion: Local open-weight agents can support a meaningful subset of software-intensive data-engineering work, but reliability depends on model capability, task verifiability, and deterministic validation. The benchmark provides a reproducible method for evaluating complete agent configurations before adoption in engineering workflows.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI); Databases (cs.DB); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.11482 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.703Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).