Metraipolis › archive › dc_2e9762ab1979
Spectral Rank and Step-Direction Dynamics in Language-Model Post-Training — arXiv:2611.11752
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2611.11752
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-10T18:43:38.481Z
- SHA-256 of the archived text
b3d9aecfef3b4d67b0f325c2b2a7decf88dec38ca32123699bea615f560f234d- Stable URL of this page
https://metraipolis.com/documento/dc_2e9762ab1979
How to cite
Spectral Rank and Step-Direction Dynamics in Language-Model Post-Training — arXiv:2611.11752 — arXiv (en), https://arxiv.org/abs/2611.11752 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_2e9762ab1979 (Metraipolis archive dc_2e9762ab1979, sha256 b3d9aecfef3b4d67b0f325c2b2a7decf88dec38ca32123699bea615f560f234d)
BibTeX
@misc{dc2e9762ab1979,
title = {Spectral Rank and Step-Direction Dynamics in Language-Model Post-Training — arXiv:2611.11752},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2611.11752},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_2e9762ab1979; sha256 b3d9aecfef3b4d67b0f325c2b2a7decf88dec38ca32123699bea615f560f234d},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.11752. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Language-model post-training modifies weights through cumulative updates and successive displacements, yet existing studies report seemingly conflicting spectral properties: nearly full-rank updates and low stable-rank changes. We show these observations are compatible and characterize update directions during training. For singular values following $\sigma_i \propto i^{-\alpha}$ with $0 < \alpha \leq 1/2$, we prove that stable rank becomes negligible relative to fixed-energy rank as dimension increases. Full singular value decompositions of selected OLMo-2-7B updates confirm this separation: three proximal policy optimization (PPO) attention updates have stable ranks of 9.6-18.3 but require 532-1,222 singular components to capture 80% of their energy. We next compute consecutive checkpoint-displacement cosines from full-model inner products. Across nine publicly released reinforcement learning from verifiable rewards (RLVR) stages, all 131 adjacent pairs have negative cosines on their native checkpoint grids, including PPO and Group Relative Policy Optimization (GRPO) trajectories. Supervised fine-tuning comparisons across four model families show configuration-dependent alignment, including negative late-training cosines for Mistral and FP32 Llama. Finally, a controlled OLMo-2-1B study with three paired seeds and a 2,048-token response cap compares group-relative advantages against a learned critic with generalized advantage estimation (GAE). At matched 60-rollout windows, late-training mean cosines are +0.259 and +0.012, respectively; the paired critic-minus-group difference is -0.247 (seed-level 95% interval [-0.358, -0.137]). These findings distinguish spectral concentration, energy coverage, and trajectory direction, and demonstrate that the advantage system influences directional alignment under matched training conditions.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2611.11752 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-10T18:37:37.656Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).