Metraipolis › archive › dc_67979ca95702
Understanding LLM Parameter Update Sparsity through the Lens of Fisher — arXiv:2609.36262
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.36262
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-07T17:28:55.863Z
- SHA-256 of the archived text
d63b433b7d814d9e94b75e521835d893ace0d92fbe944efe04f1366937c51b4c- Stable URL of this page
https://metraipolis.com/documento/dc_67979ca95702
How to cite
Understanding LLM Parameter Update Sparsity through the Lens of Fisher — arXiv:2609.36262 — arXiv (en), https://arxiv.org/abs/2609.36262 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_67979ca95702 (Metraipolis archive dc_67979ca95702, sha256 d63b433b7d814d9e94b75e521835d893ace0d92fbe944efe04f1366937c51b4c)
BibTeX
@misc{dc67979ca95702,
title = {Understanding LLM Parameter Update Sparsity through the Lens of Fisher — arXiv:2609.36262},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.36262},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_67979ca95702; sha256 d63b433b7d814d9e94b75e521835d893ace0d92fbe944efe04f1366937c51b4c},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.36262. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training paradigms suggests shared structure in training dynamics. In this paper, we examine this pattern through the diagonal model Fisher, which measures the sensitivity of the model's output distribution to individual parameters and is independent of any particular reward or teacher signal. Theoretically, we show that small diagonal Fisher leads to small expected gradients across a range of training objectives, providing a common explanation for sparse gradient updates. Empirically, we test this connection in RL and OPD. We find that Fisher identifies where gradients are concentrated, and fixed sparse masks selected from the initial Fisher retain a large proportion of the improvement from full training. Finally, we investigate the mechanisms underlying low Fisher in on-policy training. Our results show that high-probability next tokens tend to have similar parameter sensitivities, contributing to low Fisher. Together, these results establish the diagonal model Fisher as a unifying perspective linking update sparsity to on-policy training dynamics in LLM post-training. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-30 in cs.LG). Categorie arXiv: Machine Learning (cs.LG). Pagina del lavoro: https://arxiv.org/abs/2609.36262 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-07T16:59:48.143Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-30?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).