Metraipolis › archive › dc_5b8fe57017ea
TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training — arXiv:2611.04897
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.04897
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-04T12:34:50.556Z
- SHA-256 of the archived text
c1b4256bc28979f4102935072ca987ece1705bd47aaaa76277bd5243aad82ce3- Stable URL of this page
https://metraipolis.com/documento/dc_5b8fe57017ea
How to cite
TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training — arXiv:2611.04897 — arXiv (en), https://arxiv.org/abs/2611.04897 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_5b8fe57017ea (Metraipolis archive dc_5b8fe57017ea, sha256 c1b4256bc28979f4102935072ca987ece1705bd47aaaa76277bd5243aad82ce3)
BibTeX
@misc{dc5b8fe57017ea,
title = {TapeSAC: Automated, Pareto-Optimal Fine-Grained Memory Management for Large Model Training — arXiv:2611.04897},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.04897},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_5b8fe57017ea; sha256 c1b4256bc28979f4102935072ca987ece1705bd47aaaa76277bd5243aad82ce3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04897. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Selective activation checkpointing (SAC) decides which activations to save for the backward pass and which to recompute: each choice is a point in the plane of peak memory and recomputation time, and the goal is that plane's Pareto frontier. Current practice falls short of this goal. Hand-tuned policies fix a single point that does not adapt when the model, batch or kernels change; PyTorch's SAC reruns each block's forward pass in program order, recomputing values the backward pass never reads; and the memory budget of this http URL treats recomputation as free or forbidden and selects saved tensors by bytes alone, an assumption that fails for the custom kernels that produce many of a large model's activations. We present TapeSAC, which automates SAC in eager PyTorch without editing the model's forward code. TapeSAC records each block's forward pass at the dispatcher and, during backward, replays only the producers of the values autograd consumes, so saving a tensor eliminates every producer needed only for it. A planner selects the saved tensors from measured recomputation costs under a single limit on step-peak memory. On Qwen3.8-27B and Qwen3.6-35B-A3B, replay alone reduces step time by 3.0% and 1.7% versus full checkpointing at no greater memory cost; memory-aware planning raises these gains to 6.3% and 13.9% under a 90%-of-device-memory limit. The evaluated hand-tuned policies fall short of this memory-recomputation frontier and, on Qwen3.6-35B-A3B, run out of memory at larger batches. Over 500 steps on real data, TapeSAC reduces training time by 8-11%; on Qwen3.5-397B, it is 2.7% faster than a model-specific hand-tuned policy while using 14% less memory. Tema degli esploratori: calcolo e infrastrutture (elenco arXiv cs.DC, nuovi lavori del giorno). Categorie arXiv: Distributed, Parallel, and Cluster Computing (cs.DC). Pagina del lavoro: https://arxiv.org/abs/2611.04897 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-04T12:27:17.011Z dalla pagina https://arxiv.org/list/cs.DC/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).