Metraipolis › archive › dc_056567e1ef46

A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning — arXiv:2610.04670

Source
arXiv
Original
https://arxiv.org/abs/2610.04670
License (as archived)
CC0-1.0
Archived
2026-10-06T05:24:43.149Z
SHA-256 of the archived text
f6928dc6bc3e1d4b0bcbc45a935e1299a32ab049875365fc11508bdf0c194933
Stable URL of this page
https://metraipolis.com/documento/dc_056567e1ef46

How to cite

A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning — arXiv:2610.04670 — arXiv, https://arxiv.org/abs/2610.04670 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_056567e1ef46 (Metraipolis archive dc_056567e1ef46, sha256 f6928dc6bc3e1d4b0bcbc45a935e1299a32ab049875365fc11508bdf0c194933)
BibTeX
@misc{dc056567e1ef46,
  title = {A Tropical Geometry View of Forgetting: A Per-Unit Projector for Knowledge-Preserving Fine-Tuning — arXiv:2610.04670},
  howpublished = {arXiv},
  url = {https://arxiv.org/abs/2610.04670},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_056567e1ef46; sha256 f6928dc6bc3e1d4b0bcbc45a935e1299a32ab049875365fc11508bdf0c194933},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv, https://arxiv.org/abs/2610.04670. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Fine-tuning a language model on new text degrades what it already does. Replay-free projectors such as Adam-NSCL and GPM forbid one shared subspace of a layer's inputs in every row of the update. The tropical geometry of a ReLU layer shows why this is too coarse. In data space, the units' walls are tropical hypersurfaces whose cells are dual to the upper vertices of a zonotope; in weight space, each old token is a hyperplane, and the tokens cut out a polyhedron, the closure of the weights that keep every token on its side. An exact identity joins the two pictures: the squared change of the layer's output under any weight change splits into in-cell, open-to-closed and closed-to-open terms, and the first two live on the tokens each unit fires on. The identity names a gate-aware per-unit projector, and a budget-separation theorem prices exact protection: it costs a unit the rank of its own open tokens, while a shared subspace pays at least the rank of their union in every row. On OPT-1.3b, where 96% of (token, unit) pairs are closed, the projector forgets less than Adam-NSCL at all six matched budgets from 9 to 60 constrained directions per row, the gap widening from $1.1\times$ to $4.3\times$; with 1/5.5 of the directions it halves the forgetting of Adam-NSCL at GPM's energy threshold. On OPT-6.7b, it matches Adam-NSCL's forgetting at matched budget while learning more. As the theory predicts, the open/closed partition is the operative variable: open tokens beat random, sign-blind and anti-gate token sets on 18 of 18 seed-pairs. In pruning repair, every derivative-based local model of the output error at the dense weights is blind to pairs that open: the minimisers of the gate-weighted objective can leave the polyhedron, the objective's closed-form solution is 1.94 nats worse than no repair on OPT-1.3b, and a convex one-sided penalty bounds the escape.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.04670 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T05:24:05.499Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).