Metraipolis › archive › dc_def4ea014fc5

CAPSkip: Cancellation-Aware Prefix Skipping for Selective Backpropagation — arXiv:2611.04611

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.04611
License (as archived): CC0-1.0
Archived: 2026-11-04T18:54:08.775Z
SHA-256 of the archived text: 26fdd1084581e47e524e4c88fd1e3dbd9246632a27aacd27e8b46356aa3fb198
Stable URL of this page: https://metraipolis.com/documento/dc_def4ea014fc5

How to cite

CAPSkip: Cancellation-Aware Prefix Skipping for Selective Backpropagation — arXiv:2611.04611 — arXiv (en), https://arxiv.org/abs/2611.04611 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_def4ea014fc5 (Metraipolis archive dc_def4ea014fc5, sha256 26fdd1084581e47e524e4c88fd1e3dbd9246632a27aacd27e8b46356aa3fb198)

BibTeX

@misc{dcdef4ea014fc5,
  title = {CAPSkip: Cancellation-Aware Prefix Skipping for Selective Backpropagation — arXiv:2611.04611},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.04611},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_def4ea014fc5; sha256 26fdd1084581e47e524e4c88fd1e3dbd9246632a27aacd27e8b46356aa3fb198},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.04611. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Per-step selective back-propagation forwards every example of a mini-batch but back-propagates only some of them. Most rules rank examples by loss or gradient magnitude, but such scalar scores ignore cancellation between per-example gradients: the gradient removed by a set can be much smaller than its individual magnitudes suggest. We introduce CAPSkip (Cancellation-Aware Prefix Skipping). Among the examples the model already classifies correctly, CAPSkip skips the longest prefix, in increasing order of last-layer gradient norm, whose vector sum stays within a relative budget $b$ of the batch's last-layer gradient. In exact arithmetic this bounds the change of the raw final-layer gradient, using rank-one contributions already available from the forward pass; the bound does not extend to deeper layers, optimizer updates, or the training trajectory. Because the cumulative distortion is not monotone in the prefix length, CAPSkip scans all prefixes; a factorised Gram matrix makes the scan cheap and keeps it on the device. With activation reuse and selected-row backward, CAPSkip fine-tunes DeiT-S on CIFAR-100 for 103 epochs in 0.576x the time of 103-epoch dense training, and is +0.25 pp [+0.08, +0.42] above an approximately time-matched 59-epoch dense run (n=3).

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.04611 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-04T18:35:37.445Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).