Metraipolis › archive › dc_ef2bf5bc5ec8

Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform — arXiv:2608.03311

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.03311
License (as archived): CC0-1.0
Archived: 2026-10-31T12:19:15.011Z
SHA-256 of the archived text: 1ff2f864c5517ac42850a34b6279b99c0191651c2b4228120a3e54a487964aa1
Stable URL of this page: https://metraipolis.com/documento/dc_ef2bf5bc5ec8

How to cite

Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform — arXiv:2608.03311 — arXiv (en), https://arxiv.org/abs/2608.03311 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ef2bf5bc5ec8 (Metraipolis archive dc_ef2bf5bc5ec8, sha256 1ff2f864c5517ac42850a34b6279b99c0191651c2b4228120a3e54a487964aa1)

BibTeX

@misc{dcef2bf5bc5ec8,
  title = {Evaluating LLM Trade-offs for Enterprise Automation: Lessons from Workflow Generation in a Production Enterprise Platform — arXiv:2608.03311},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03311},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ef2bf5bc5ec8; sha256 1ff2f864c5517ac42850a34b6279b99c0191651c2b4228120a3e54a487964aa1},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03311. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Enterprise compliance management requires rapid adaptation to evolving regulatory frameworks (e.g., DORA, AI RMF, FedRAMP) and tight remediation SLAs. Traditional static orchestrators often fail in hybrid cloud environments where event-driven assessments demand that automation code adapt to runtime context in seconds. This paper presents lessons learned from evaluating six large language models for AI-driven workflow generation in a production enterprise platform, benchmarked across 29 real-world IT automation scenarios, two generation pipeline architectures, and eight independent runs per prompt-model-pipeline configuration (2,784 runs total). Our initial pipeline used monolithic workflow generation, achieving 31.5-82.8% structural success rates (JSON schema validity and correct UI rendering), with most models struggling on complex JSON generation. We developed a redesigned piecewise pipeline that decomposes workflow construction into variable scaffolding, base block assembly, and nested block generation, raising structural success to 74.1-97.8% across all models. We analyze production tradeoffs including cost (USD 0.008-0.20 per workflow), latency (under 50s for interactive use), and model selection. Piecewise decomposition enables smaller models (e.g., mistral-small at 95.7% structural success and USD 0.01 per workflow) to reach production viability, removing dependency on expensive frontier models. While mistral-medium-2505 and gpt-oss-120b achieved the highest structural success (96.1% and 97.8%), mistral-medium-2505 carries a 19x cost premium versus mistral-small. Our deployment lessons highlight the need to separate structural validity from semantic correctness (logical fulfillment of user intent) and provide a solution for model-agnostic, scalable automation in cloud engineering.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.AI).
Categorie arXiv: Software Engineering (cs.SE) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2608.03311 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:05:53.031Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).