Metraipolis › archive › dc_2bf36a04fc84
RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale — arXiv:2611.00967
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.00967
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:46:43.111Z
- SHA-256 of the archived text
50826569afd912dd1d1046b11b6d584cb4a9268bdba021ae527dbe2875d4f25a- Stable URL of this page
https://metraipolis.com/documento/dc_2bf36a04fc84
How to cite
RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale — arXiv:2611.00967 — arXiv (en), https://arxiv.org/abs/2611.00967 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2bf36a04fc84 (Metraipolis archive dc_2bf36a04fc84, sha256 50826569afd912dd1d1046b11b6d584cb4a9268bdba021ae527dbe2875d4f25a)
BibTeX
@misc{dc2bf36a04fc84,
title = {RTP-LLM: Building an Evolvable and Reliable LLM Inference Engine at Alibaba Scale — arXiv:2611.00967},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.00967},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2bf36a04fc84; sha256 50826569afd912dd1d1046b11b6d584cb4a9268bdba021ae527dbe2875d4f25a},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00967. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large language models and inference techniques evolve rapidly, making model enablement a recurring production task rather than a one-time integration effort. At Alibaba, a shared inference infrastructure must keep pace with new models and serving capabilities while remaining reliable across heterogeneous production environments. To reconcile rapid evolution with production reliability, we present RTP-LLM, an open-source production inference engine developed and deployed at Alibaba. Its architecture separates model-specific execution from shared request processing, state management, and resource coordination. Explicit interfaces localize compatible model changes while allowing models and deployment configurations to reuse common serving mechanisms. RTP-LLM has evolved from an exploratory system into a shared production engine that supports $\sim$100 models and processes hundreds of trillions of tokens per day. We describe how its architecture and production practices accommodate model evolution and contain the impact of failures, and share lessons for building evolvable and reliable inference engines at enterprise scale. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.OS, nuovi lavori del giorno). Categorie arXiv: Operating Systems (cs.OS). Pagina del lavoro: https://arxiv.org/abs/2611.00967 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:09:22.273Z dalla pagina https://arxiv.org/list/cs.OS/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).