Metraipolis › archive › dc_ced11e3bd29b

Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost — arXiv:2609.39358

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.39358
License (as archived)
CC0-1.0
Archived
2026-10-07T12:45:17.137Z
SHA-256 of the archived text
b980599c04c07c7b09c8d9de12fe9314ec51626783a59c6c19b0dac362991b19
Stable URL of this page
https://metraipolis.com/documento/dc_ced11e3bd29b

How to cite

Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost — arXiv:2609.39358 — arXiv (en), https://arxiv.org/abs/2609.39358 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ced11e3bd29b (Metraipolis archive dc_ced11e3bd29b, sha256 b980599c04c07c7b09c8d9de12fe9314ec51626783a59c6c19b0dac362991b19)
BibTeX
@misc{dcced11e3bd29b,
  title = {Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost — arXiv:2609.39358},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.39358},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ced11e3bd29b; sha256 b980599c04c07c7b09c8d9de12fe9314ec51626783a59c6c19b0dac362991b19},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.39358. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

A transformer language model performs a bounded amount of computation per token, and recent work by Vishal Sikka, former CEO of Infosys, argues that this bound limits which tasks a model can carry out or verify ( arXiv:2507.07505 ). We ask how much of the budget beneath that ceiling is spent on work the model has already done. Serving is stateless across requests: a model that answers a second question about a document recomputes the document's attention state from the first token. On seven real-world datasets, 98.7% of prompt tokens were text the model had already read. We present Galahad, a memory layer for vLLM, SGLang and this http URL that makes this reading a one-time cost. Taliesin saves the model's key-value (KV) state for a block of text and loads it on the next request that contains the same bytes, instead of recomputing it. Blaise keeps the documents themselves and passes the model only the section a question needs. On a recall test with 100 facts hidden in a 97,000-token corpus (Gemma 4 31B), Taliesin alone let the model attend to the whole corpus and answered 98 of 100 on this http URL at 3.0 s and 572 J per question, against 10 of 100, 9.3 s and 2,754 J for the same model without Galahad, which could hold only the last 12,000 tokens. With Blaise added, the model read about 668 tokens per question and answered 100 of 100 on all three runtimes at 0.59-0.64 s and 200-213 J; a tuned RAGFlow pipeline answered 77. Storing the corpus is a one-time cost of about 100 s and 28 kJ, whose energy is recovered after 13 questions. Restored state is bit-identical: all 262,144 output logits matched after restart, rehydration and hot-load. Galahad worked with all 30 models we tested under vLLM, and it fails closed: any load that does not pass its checks is recomputed. Together these results move LLM serving from stateless to stateful inference.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-01 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Information Retrieval (cs.IR); Machine Learning (cs.LG); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2609.39358 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T12:14:14.065Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).