Metraipolis › archive › dc_fa9bee6e943e
Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization — arXiv:2611.02052
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.02052
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T11:21:21.425Z
- SHA-256 of the archived text
9c37a7e3725fe9b4a357d9823d1861c7c130d42fc8be502409c52d0ceed2d043- Stable URL of this page
https://metraipolis.com/documento/dc_fa9bee6e943e
How to cite
Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization — arXiv:2611.02052 — arXiv (en), https://arxiv.org/abs/2611.02052 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fa9bee6e943e (Metraipolis archive dc_fa9bee6e943e, sha256 9c37a7e3725fe9b4a357d9823d1861c7c130d42fc8be502409c52d0ceed2d043)
BibTeX
@misc{dcfa9bee6e943e,
title = {Same Function, Different Gradients: Evaluating Gradient Reliability under Reparameterization — arXiv:2611.02052},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.02052},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fa9bee6e943e; sha256 9c37a7e3725fe9b4a357d9823d1861c7c130d42fc8be502409c52d0ceed2d043},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.02052. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Gradients are no longer used only to train models: Gradient magnitudes are now measurement instruments, used to rank attention heads for pruning, to decide per-example clipping in private training, to allocate adapter rank budgets, and to debug models. Across these workflows, larger gradient magnitudes are taken to mean greater importance. Even for a fixed input--output function, these gradients are computed in one of many possible parameter coordinate systems. For example, attention scores depend on the query and key projections only through the paired product $W_QW_K^\top$. Scaling a head's query projection by $c$ and the corresponding key projection by $1/c$ therefore leaves every output logit unchanged. This change in parameter coordinates rescales the corresponding query and key gradient norms by $1/c$ and $c$, respectively. To test whether gradient-based decisions remain unchanged under these coordinate changes, we introduce Gradient Reliability under Reparameterization (GRR). GRR rewrites a trained model in new parameter coordinates that compute exactly the same function, verifies that predictions are unchanged, and then tests whether the decisions derived from the gradients are unchanged too. Our audits span GLUE classifiers, a ViT on CIFAR-10, GPT-2-medium and GPT-2-large, a 1.1B-parameter Llama-style model on WikiText-2, and six trained LoRA adapters. Empirically, predictions are preserved throughout, but both the gradients and the gradient-based decisions change. Across 66 GLUE attention audits, the median top-3 overlap of raw head-gradient rankings is only $1/3$, and rankings and pruning masks change even under the mildest rescalings we test. In the largest of 14 global-pruning audits, the shifted ranking selects head masks that differ in four of 22 pruned heads and reach $50.92\%$ and $94.84\%$ validation accuracy on one ELECTRA SST-2 model. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2611.02052 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-11-03T11:02:37.578Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).