Metraipolis › archive › dc_cc752087d12e

Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling — arXiv:2609.40247

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.40247
License (as archived)
CC0-1.0
Archived
2026-10-07T15:20:11.423Z
SHA-256 of the archived text
b45c032e88f6a261bc215205601d587ff834004467cc5060f9f7cd9d11ca0701
Stable URL of this page
https://metraipolis.com/documento/dc_cc752087d12e

How to cite

Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling — arXiv:2609.40247 — arXiv (en), https://arxiv.org/abs/2609.40247 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cc752087d12e (Metraipolis archive dc_cc752087d12e, sha256 b45c032e88f6a261bc215205601d587ff834004467cc5060f9f7cd9d11ca0701)
BibTeX
@misc{dccc752087d12e,
  title = {Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling — arXiv:2609.40247},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.40247},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cc752087d12e; sha256 b45c032e88f6a261bc215205601d587ff834004467cc5060f9f7cd9d11ca0701},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.40247. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Model-as-a-service platforms call for continuous optimization as complex serving conditions expose inefficiencies missed before deployment. Detailed always-on profiling can incur substantial overhead, while lightweight collection omits information needed for diagnosis. We present Herschel, a continuous optimization system for production large language model (LLM) inference. Our key insight is that adaptive, on-demand profiling can provide rich full-stack evidence without continuous collection. Herschel safely attaches to and detaches from selected running processes without engine changes or restarts, and adapts coverage as investigations reveal missing evidence. Herschel reconstructs operator executions and cross-process dependencies to identify inefficiency mechanisms and suggest solutions using applicable reference fixes. AI agents implement and test engine and kernel changes under controlled conditions that preserve the triggering workload and dependencies, with expert review before deployment. Controlled tests show active-collection overhead below 0.5% for time to first token and 7% for time per output token. Bounded windows, typically 30 s, avoid the continuous cost of always-on tracing. Over six months, Herschel collected approximately 17,000 traces across over 120 model variants and more than 10 accelerator types, identifying inefficiency patterns in 23% of the traces. Representative findings guide widely deployed optimizations, including restructured synchronization, removal of unused computation, and improved operator implementations.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-10-01 in cs.OS).
Categorie arXiv: Operating Systems (cs.OS).
Pagina del lavoro: https://arxiv.org/abs/2609.40247 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T14:53:17.076Z dalla pagina https://arxiv.org/catchup/cs.OS/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).