Metraipolis › archive › dc_7d6765ea9282
Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decoding — arXiv:2609.04238
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04238
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T22:42:11.285Z
- SHA-256 of the archived text
dfec76e75324a4143bdd3ee47ee865aee2b007c2819ffeea25f9dfeaba4680aa- Stable URL of this page
https://metraipolis.com/documento/dc_7d6765ea9282
How to cite
Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decoding — arXiv:2609.04238 — arXiv (en), https://arxiv.org/abs/2609.04238 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7d6765ea9282 (Metraipolis archive dc_7d6765ea9282, sha256 dfec76e75324a4143bdd3ee47ee865aee2b007c2819ffeea25f9dfeaba4680aa)
BibTeX
@misc{dc7d6765ea9282,
title = {Budgeting Bytes: A Windowed Storage Roofline and Dual-Budget Architecture Ablations for Storage-Bound LLM Decoding — arXiv:2609.04238},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04238},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7d6765ea9282; sha256 dfec76e75324a4143bdd3ee47ee865aee2b007c2819ffeea25f9dfeaba4680aa},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04238. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Autoregressive decoding on cheap hardware is bound not by FLOPs but by the bytes each generated token must move across the slowest populated tier of a memory hierarchy. We treat bytes-per-token as a first-class design axis, organized by an address-determinism taxonomy that classifies parameters by when their fetch address becomes known during a token's forward pass (A0: at token sampling; A1: before attention; A2: layerwise data-dependent; A3: always read). This reduces prefetch scheduling to single-machine feasibility with release times, yielding a closed-form windowed roofline. We run dual-budget (bytes-per-token times storage) ablations across three sub-100M scales, then take the framework to real large-MoE deployment and report a substantial negative result the roofline predicts: on an 8GB edge board running Qwen3-30B-A3B (4-bit, 18GB), the model overflows RAM and decode is pinned at the eMMC bandwidth ceiling; predictive expert prefetch does not help -- not temporal-locality prefetch (net-negative), not even a trace-driven oracle with perfect prediction -- because the binding constraint is byte volume over a saturated bus, which prefetch cannot reduce. The lever that works is reducing bytes-per-token until the model fits the fast tier: quantized to fit a 16GB unified-memory device, the same model runs GPU-resident at 11.5 tok/s (22x). We reconcile this with GPU-serving expert-prefetch predictors: a frozen-model probe predicts Qwen3-30B routing from the pre-attention state at 91.2%, a scale-invariant predictability property, but this converts to throughput only where the fast tier caches most of the model and per-token transfer is comparable to compute -- measured to hold on an A100 PCIe-offload path and to fail on bandwidth-walled edge storage. Predictability is not speedup; we chart where the gap closes. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-09-07 in cs.AR). Categorie arXiv: Hardware Architecture (cs.AR). Pagina del lavoro: https://arxiv.org/abs/2609.04238 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T22:26:48.901Z dalla pagina https://arxiv.org/catchup/cs.AR/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).