Metraipolis › archive › dc_71828d5bfe29

Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space — arXiv:2609.36738

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.36738
License (as archived)
CC0-1.0
Archived
2026-10-07T16:57:33.236Z
SHA-256 of the archived text
8c52562a591610ae546f618691ef4283cc77736e422584fa8d87e2cb24967493
Stable URL of this page
https://metraipolis.com/documento/dc_71828d5bfe29

How to cite

Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space — arXiv:2609.36738 — arXiv (en), https://arxiv.org/abs/2609.36738 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_71828d5bfe29 (Metraipolis archive dc_71828d5bfe29, sha256 8c52562a591610ae546f618691ef4283cc77736e422584fa8d87e2cb24967493)
BibTeX
@misc{dc71828d5bfe29,
  title = {Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space — arXiv:2609.36738},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.36738},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_71828d5bfe29; sha256 8c52562a591610ae546f618691ef4283cc77736e422584fa8d87e2cb24967493},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36738. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Optimizer momentum is usually stored as a parameter-sized moving average of past gradients, which makes history costly and fixes each past signal in the coordinates in which it was computed. We introduce Backpropagated Output Momentum (BOM), which instead stores a compact moving average of prediction errors at the model output and reprojects that history through the current network at every step. A batch-level analysis characterizes the information retained and omitted by this relocation, while the implementation preserves the current supervised gradient and can replace the first-moment component of several adaptive optimizers. As a plug-in for momentum-based optimizers, including ones that already compress their state, BOM reduces parameter-shaped optimizer state by 49.7-99.8% in three compositions and, averaged over three language backbones, paired step time by 4.0%. It also improves mean validation performance across language and vision fine-tuning, by 1.42 points in the primary five-task comparison. Language and vision pretraining studies, together with matched mechanism controls, further test the construction across output spaces and model scales.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.36738 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T16:27:03.423Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).