Metraipolis › archive › dc_85d2e9fcc4b5

MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference — arXiv:2611.08508

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.08508
License (as archived)
CC0-1.0
Archived
2026-11-09T22:34:22.391Z
SHA-256 of the archived text
927d59caaa49b31c7053bfc04907a6ae0eeac3772f72723a7ef7a78d0fcfe6bb
Stable URL of this page
https://metraipolis.com/documento/dc_85d2e9fcc4b5

How to cite

MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference — arXiv:2611.08508 — arXiv (en), https://arxiv.org/abs/2611.08508 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_85d2e9fcc4b5 (Metraipolis archive dc_85d2e9fcc4b5, sha256 927d59caaa49b31c7053bfc04907a6ae0eeac3772f72723a7ef7a78d0fcfe6bb)
BibTeX
@misc{dc85d2e9fcc4b5,
  title = {MeshRT: Compile-Time Governed Wafer-Scale Runtime for Low-Latency High-Throughput Inference — arXiv:2611.08508},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.08508},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_85d2e9fcc4b5; sha256 927d59caaa49b31c7053bfc04907a6ae0eeac3772f72723a7ef7a78d0fcfe6bb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.08508. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Wafer-scale accelerators promise ultra-low-latency AI inference, but current system stacks still carry an unsustainably high cost premium. The reason is that many current and emerging inference techniques, such as batching and MoE, introduce runtime dynamism that existing wafer-scale systems cannot support efficiently. As a result, they must choose between two unsatisfactory options: accept suboptimal latency, or preserve low latency by restricting dynamic techniques, thereby limiting the models they can support and reducing aggregate throughput. This tradeoff significantly increases the cost of low-latency serving. We present MeshRT, the first wafer-scale system to achieve both ultra-low latency and high throughput for LLM inference. MeshRT achieves this via a novel system architecture, which we call compile-time governance of runtime dynamism. This architecture features new system abstractions and mechanisms that enable MeshRT to compile dynamic events arising from LLM inference directly into per-core schedules, enabling asynchronous and decentralized execution with low runtime overhead. Implemented on a commodity wafer-scale accelerator, MeshRT preserves ultra-low latency while improving throughput by up to 56$\times$ over the state-of-the-art system for wafer-scale accelerators. Against measured single-instance SGLang results on H200, MeshRT achieves 4.5-7.0$\times$ lower latency. Under ideal linear scaling of those GPU measurements to an approximately power-matched H200 cluster, MeshRT achieves 1.7-13.0$\times$ higher decode system throughput in the low-latency regime.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Operating Systems (cs.OS) ; Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC).
Pagina del lavoro: https://arxiv.org/abs/2611.08508 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-09T22:30:47.297Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).