Metraipolis › archive › dc_eccd5bae0ce9
Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode — arXiv:2608.23841
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.23841
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:25:13.406Z
- SHA-256 of the archived text
f22523d4312649056af508ac40d2c5086faf3dce8a8de8508d801ecf2d61dbc9- Stable URL of this page
https://metraipolis.com/documento/dc_eccd5bae0ce9
How to cite
Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode — arXiv:2608.23841 — arXiv (en), https://arxiv.org/abs/2608.23841 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_eccd5bae0ce9 (Metraipolis archive dc_eccd5bae0ce9, sha256 f22523d4312649056af508ac40d2c5086faf3dce8a8de8508d801ecf2d61dbc9)
BibTeX
@misc{dceccd5bae0ce9,
title = {Pipeline-Native Transformers: Co-Designing Model Architecture and CPU Inference for Bandwidth-Efficient Autoregressive Decode — arXiv:2608.23841},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.23841},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_eccd5bae0ce9; sha256 f22523d4312649056af508ac40d2c5086faf3dce8a8de8508d801ecf2d61dbc9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23841. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Single-token autoregressive decode on CPUs is bound by memory bandwidth, not arithmetic: a modern CPU sustains roughly 1 TFLOP/s of compute but only about 50 GB/s from main memory, and each generated token must stream every active weight once. This report argues that the most effective response is to co-design the model architecture and the inference runtime together. It presents cflow, a CPU-first streaming engine, alongside a family of pipeline-native transformer architectures whose inter-layer dependency graphs are constructed to permit a vertical, stage-major execution schedule. cflow stores weights as L2-sized tiles in compute-consumption order, reads only the top-k experts of each mixture-of-experts layer, fuses projections, and executes a delay-aware schedule from per-model dependency parameters. Across five architectures trained on TinyStories, one (arch2_4_combined) achieves a 2.00x reduction in critical-path weight bandwidth (9.00 to 4.50 MB/token) within 0.24 perplexity of the best candidate, and the tile layout incurs 7.29x fewer L1-data read misses than a row-major baseline. On a 30.9-billion-parameter pipeline-native MoE, cflow decodes at 5.94 tokens/s (tok/s) on a 32-vCPU Ice Lake server, ahead of this http URL (4.75) and the vLLM CPU backend (1.65) on comparably sized dense models. Realizing the expert-delay window as asynchronous I/O overlap on a disk-resident expert tier yields a further net win of up to 1.68x, matching the overlap model within 1%. Measurement refutes one of the eight design claims and leaves a second inconclusive; both are reported in full, with the conditions under which they would hold. Tema degli esploratori: calcolo e infrastrutture (retro-catalogo arXiv: annunci del 2026-08-26 in cs.AR). Categorie arXiv: Hardware Architecture (cs.AR) ; Machine Learning (cs.LG); Performance (cs.PF). Pagina del lavoro: https://arxiv.org/abs/2608.23841 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-18T06:18:27.524Z dalla pagina https://arxiv.org/catchup/cs.AR/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).