Metraipolis › archive › dc_deba1edd2c1a
When Rank Rises as LLMs Degrade — arXiv:2610.09647
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.09647
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-08T11:26:36.223Z
- SHA-256 of the archived text
bb9fc4b56cf55c42a2f70e937cbdcc61aaf6dc2743df0e5f35767598346089e5- Stable URL of this page
https://metraipolis.com/documento/dc_deba1edd2c1a
How to cite
When Rank Rises as LLMs Degrade — arXiv:2610.09647 — arXiv (en), https://arxiv.org/abs/2610.09647 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_deba1edd2c1a (Metraipolis archive dc_deba1edd2c1a, sha256 bb9fc4b56cf55c42a2f70e937cbdcc61aaf6dc2743df0e5f35767598346089e5)
BibTeX
@misc{dcdeba1edd2c1a,
title = {When Rank Rises as LLMs Degrade — arXiv:2610.09647},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.09647},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_deba1edd2c1a; sha256 bb9fc4b56cf55c42a2f70e937cbdcc61aaf6dc2743df0e5f35767598346089e5},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.09647. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Post-training adapts language models in non-stationary environments. Practitioners monitor representation health with RankMe and related spectral statistics, often assuming that rank falls when representations degrade. We show that this assumption is unsafe for LLM post-training. In a controlled study of Qwen3-0.6B with four degradation modes and three seeds, data duplication worsens held-out loss by 75% relative to healthy while increasing both original and centred RankMe; the latter changes by 13.5 pooled standard deviations. Covariance effective rank rises to nearly twice its healthy value. This failure is spectral dispersion rather than collapse, so a one-sided monitor rates the worst checkpoint as the healthiest. By contrast, a learning-rate misconfiguration lowers centred RankMe and k95, while uncentred RankMe is inconsistent across seeds. Direction is therefore a property of the regime-statistic pair and cannot be fixed by recalibration alone. We also distinguish two often-conflated statistics: RankMe normalises singular values, whereas covariance effective rank normalises eigenvalues. On raw intermediate-layer states in the pretrained model, massive activations pin the latter near 1 out of dimension d while RankMe retains usable range. We then test a two-sided, multichannel sequential monitor with separate calibration and test data. In a pre-registered shared-prefix, leave-one-seed-out evaluation, it detects all three damage regimes in every fold 10 to 60 steps after the fork and separates dispersion from downward-rank damage by firing direction. However, it never precedes held-out probe loss, and calibration with two seeds produces false alarms on the held-out healthy seed. Spectral monitoring can diagnose failure regimes, but it does not warn earlier than held-out loss, and validity claims require held-out healthy data. Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.CL, nuovi lavori del giorno). Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL). Pagina del lavoro: https://arxiv.org/abs/2610.09647 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-08T10:24:06.384Z dalla pagina https://arxiv.org/list/cs.CL/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).