Metraipolis › archive › dc_1d2f6da3adfa
What Sets the Serving Speed of Looped Language Models? Measurements and Optimizations Graded by Exactness — arXiv:2611.05150
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.05150
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T21:45:47.840Z
- SHA-256 of the archived text
208933967654dfcc189b9e897c40a9cbc19258f5ef208ddb1cda926988295fdd- Stable URL of this page
https://metraipolis.com/documento/dc_1d2f6da3adfa
How to cite
What Sets the Serving Speed of Looped Language Models? Measurements and Optimizations Graded by Exactness — arXiv:2611.05150 — arXiv (en), https://arxiv.org/abs/2611.05150 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1d2f6da3adfa (Metraipolis archive dc_1d2f6da3adfa, sha256 208933967654dfcc189b9e897c40a9cbc19258f5ef208ddb1cda926988295fdd)
BibTeX
@misc{dc1d2f6da3adfa,
title = {What Sets the Serving Speed of Looped Language Models? Measurements and Optimizations Graded by Exactness — arXiv:2611.05150},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.05150},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1d2f6da3adfa; sha256 208933967654dfcc189b9e897c40a9cbc19258f5ef208ddb1cda926988295fdd},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.05150. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Looped language models apply one block of layers many times per token, so a serving engine pays every per-layer overhead once per loop; we measure serving for Ouro and Huginn-0125 in an open-source looped-model engine and grade our optimizations from bitwise-identical changes to those that require an accuracy check. At batch 1, fusing all loop submissions leaves Ouro throughput at 0.99x; the 5689 small kernels per token are the real cost, and fusing them behind rounding fences raises throughput 1.11-1.59x while 1014/1014 requests remain identical. An approximate fusion is no faster and misses a one-point GSM8K rule on 87 questions (61 and 62 correct versus 64), so we do not ship it. Huginn's released defaults avoid dead KV slots, reuse prefix-cache pages, and serve 1.77-3.16x faster than the base engine; Ouro reaches 1.65-2.15x and agent sessions 1.91-3.21x, while Huginn's noise and prefix-reuse changes stay within one point on all 1319 Huginn test questions; Huginn's packed prefill, -0.61 points on the same questions, meets that rule only on its point estimate. Position-keyed noise makes CPU cache reuse exact but still leaves 20 of 128 GPU full hits different, while loop fusion delays prefill admission and a 64-graph cache prevents eager fallbacks. Under prefill-decode disaggregation ExactLoop reaches up to 3.3x the base throughput with peer error handling disabled, and 3.1x in two of three runs with it enabled. Its transfer budget, sized to hold one request's KV in flight, cuts the commit tail from about one second under the earlier budget to 16 ms (base: 2.8 s), removing the polling rounds that the engine's interval would otherwise impose. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2611.05150 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T21:36:19.396Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).