Metraipolis › archive › dc_1d82fcd01667
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching — arXiv:2608.09444
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.09444
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-19T08:50:57.093Z
- SHA-256 of the archived text
1267619868f2198a886f03d90aa7dc0df35f87f64bc44c81db51aeb9375c40ca- Stable URL of this page
https://metraipolis.com/documento/dc_1d82fcd01667
How to cite
Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching — arXiv:2608.09444 — arXiv (en), https://arxiv.org/abs/2608.09444 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_1d82fcd01667 (Metraipolis archive dc_1d82fcd01667, sha256 1267619868f2198a886f03d90aa7dc0df35f87f64bc44c81db51aeb9375c40ca)
BibTeX
@misc{dc1d82fcd01667,
title = {Depth-adaptive Inference of Looped Language Models via Continuous Depth Batching — arXiv:2608.09444},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.09444},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_1d82fcd01667; sha256 1267619868f2198a886f03d90aa7dc0df35f87f64bc44c81db51aeb9375c40ca},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09444. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
A main promise of looped language models is depth-adaptive inference. By looping a block of shared layers a variable number of times, the model can use less compute for "easy" tokens and more for "hard" ones. However, tokens with different numbers of loops cannot share a uniform forward pass and therefore cannot be handled by standard batching systems such as vLLM. The practical value of depth-adaptive inference thus hinges on whether batching can be made efficient. We introduce the first efficient method for depth-adaptive looped LMs via continuous depth batching (CDB), which forms new batches between loop steps. Our method dynamically schedules looped and non-looped parts of the architecture, manages looped KV-caching, and predicts which tokens will exit the loop in advance so it can prepare batches asynchronously. Experiments on Ouro 1.4B and Huginn 3.5B show that fully looped architectures are best suited to depth-adaptive inference, as large non-looped layers outside the recurrent core (e.g., token embedding, LM head, and unshared transformer blocks) slow down and complicate scheduling. Overall, CDB realizes up to 99% of the estimated maximum speedup available, leaving further gains primarily dependent on model architecture and exit behavior. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-11 in cs.CL). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL); Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2608.09444 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-19T08:49:42.004Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).