Metraipolis › archive › dc_2c02c0cd6710
Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines — arXiv:2609.34045
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.34045
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T22:12:09.403Z
- SHA-256 of the archived text
8702d9409cd35d71fa45a4d551b14a9efbf49c818b5caffa546f8233e23f688f- Stable URL of this page
https://metraipolis.com/documento/dc_2c02c0cd6710
How to cite
Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines — arXiv:2609.34045 — arXiv (en), https://arxiv.org/abs/2609.34045 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2c02c0cd6710 (Metraipolis archive dc_2c02c0cd6710, sha256 8702d9409cd35d71fa45a4d551b14a9efbf49c818b5caffa546f8233e23f688f)
BibTeX
@misc{dc2c02c0cd6710,
title = {Kafila: Serving Large Language Models on a Trusted Set of Heterogeneous Commodity Machines — arXiv:2609.34045},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.34045},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2c02c0cd6710; sha256 8702d9409cd35d71fa45a4d551b14a9efbf49c818b5caffa546f8233e23f688f},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.34045. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Between them, the members of a research group or a circle of friends own several consumer computers, none large enough to run a capable large language model. Existing systems pool such capacity across open swarms anyone may join, which a group admitting only trusted machines cannot use. Bounding membership removes what they depend on: a swarm holds each part of the model on several peers and routes around a slow one. A bounded session must use every device it admits. Its pipeline advances at the pace of whichever device received a share it cannot serve quickly, so the division has to be right before serving begins. We propose Kafila, whose protocol assembles a ring from behind NATs, preferring direct paths and relaying where traversal fails, while its planner measures each device's memory bandwidth, capacity and reachability, divides the model exactly for a fixed ring order, and places the head, which holds the embedding and output projection, together with that division rather than beforehand. On machines with different capabilities across three fleets, from a shared LAN to five devices spanning two continents, Kafila shortens the slowest pipeline stage by up to $5.2\times$ against the even split of pipeline parallelism, as in GPipe, and up to $3\times$ against the memory-proportional split of personal-device inference, as in exo, keeps 75 to 87 per cent of the committed hardware doing work where those divisions fall below half, and serves a model no uniform split can place on the fleet at all. What that is worth to a user depends on how much of a token is computation rather than network. Where the members share a network the same division returns $1.56\times$ the throughput of a uniform split and $1.25\times$ of a memory-proportional one, and under four concurrent users that lead compounds to $3.2\times$ rather than fading, each user served at almost the rate of one. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-29 in cs.DC). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC) ; Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.34045 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T21:53:13.692Z dalla pagina https://arxiv.org/catchup/cs.DC/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).