Metraipolis › archive › dc_e02c0b6ffd64

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks — arXiv:2608.12114

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.12114
License (as archived): CC0-1.0
Archived: 2026-10-19T05:54:00.405Z
SHA-256 of the archived text: 2e9819ec7a9e84c01e79e818d79592850b4db61af39536c4bc488a14bffca12b
Stable URL of this page: https://metraipolis.com/documento/dc_e02c0b6ffd64

How to cite

The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks — arXiv:2608.12114 — arXiv (en), https://arxiv.org/abs/2608.12114 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e02c0b6ffd64 (Metraipolis archive dc_e02c0b6ffd64, sha256 2e9819ec7a9e84c01e79e818d79592850b4db61af39536c4bc488a14bffca12b)

BibTeX

@misc{dce02c0b6ffd64,
  title = {The Ingestion Tax: Adopting File-Backed Weights in Tensor Frameworks — arXiv:2608.12114},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.12114},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e02c0b6ffd64; sha256 2e9819ec7a9e84c01e79e818d79592850b4db61af39536c4bc488a14bffca12b},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.12114. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Open-weight models can occupy a middle capacity regime: active weights fit in DRAM as cached file pages, but a second framework-owned copy does not fit or must be refilled as layers run, so low-batch decode rereads the weights every token. On integrated and coherent-memory systems those file pages are already GPU-readable, yet ordinary loading paths copy them into framework allocations before use. We call this copy the ingestion tax. We present file-backed weight adoption: a framework-independent producer maps each tensor with MAP_SHARED, wraps the pages as a no-copy GPU buffer, and exports a DLPack capsule that PyTorch or MLX imports as ordinary storage. Zero-copy import alone is insufficient: the implementation must also keep activations accelerator-resident and establish ordering on the GPU; an adopter that omits both runs a dense decode stage 2.3x slower than stock in the live system. With both in place, adoption removes the tax: the public route reaches 516 GB/s versus 53-82 for the default constructors, matches the identical kernel over resident storage ([-0.66%, +0.48%], paired), and is within 1.3% of a resident control on a matched Qwen2.5-72B (7.14 vs. 7.23 tok/s). At the same throughput, the weights remain clean, shared, evictable file pages: N processes decode from one mapped copy where resident loading creates N copies (at capacity, 5.5 vs. 0.08 tok/s), and a 65 GB checkpoint cuts time to first token by 6.4x versus stock loading. In Kimi K3, a 2.8T-parameter MoE, the dense int8 spine stage falls from 2.62 to 0.35 s per token (7.5x; 3.8x from storage alone). The same mechanism improves this http URL by 1.21x at half the footprint on an AMD APU, falls inside the 5% selection band of overlapped streaming on a capacity-exceeding GH200 workload, and is 39x slower across PCIe. The deployment rule follows memory topology: adopt file pages only where the GPU can already read them.

Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-13 in cs.OS).
Categorie arXiv: Operating Systems (cs.OS).
Pagina del lavoro: https://arxiv.org/abs/2608.12114 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T05:50:08.692Z dalla pagina https://arxiv.org/catchup/cs.OS/2026-08-13?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).