Metraipolis › archive › dc_acddddea478a

Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata — arXiv:2608.26332

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.26332
License (as archived): CC0-1.0
Archived: 2026-10-17T16:09:19.746Z
SHA-256 of the archived text: 41a172d0670a1a193b719c77467a6b30bcd990b9df0776a1acacba42ab4ac3ef
Stable URL of this page: https://metraipolis.com/documento/dc_acddddea478a

How to cite

Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata — arXiv:2608.26332 — arXiv (en), https://arxiv.org/abs/2608.26332 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_acddddea478a (Metraipolis archive dc_acddddea478a, sha256 41a172d0670a1a193b719c77467a6b30bcd990b9df0776a1acacba42ab4ac3ef)

BibTeX

@misc{dcacddddea478a,
  title = {Beyond Capability Benchmarks: Learning Operational Fingerprints of LLM Cloud Services from Production Incident Metadata — arXiv:2608.26332},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.26332},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_acddddea478a; sha256 41a172d0670a1a193b719c77467a6b30bcd990b9df0776a1acacba42ab4ac3ef},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.26332. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Managed LLM services are now part of real production systems, but model selection and service planning still rely heavily on capability benchmarks that reveal little about operational behavior after deployment. We present Operational Embedding (OpEmbed), a framework for learning compact operational fingerprints of LLM cloud services from structured, privacy-preserving support-case metadata, without using case text. OpEmbed aggregates model--time windows into an eight-channel operational signature and learns a low-dimensional representation via temporal contrastive learning, cross-view reconstruction, and generational-ordinality regularization. Evaluated on more than 33,000 production support cases spanning seven LLM families over 26 months at Google Cloud, OpEmbed recovers interpretable family- and version-level structure, improves leave-one-model-out operational forecasting over non-learned baselines, remains useful under limited early-window data, and supports cross-model fault-type transfer. We report the practical lessons learned from building and evaluating this tool for model onboarding, support readiness assessment, and operational monitoring.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.26332 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:34:49.891Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).