Metraipolis › archive › dc_fbc1e49400df

Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression — arXiv:2610.10552

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.10552
License (as archived)
CC0-1.0
Archived
2026-10-09T05:50:51.680Z
SHA-256 of the archived text
31429fe80662020cd06c432924b218674821cd03de0d5e75f7b0f099c6b6b860
Stable URL of this page
https://metraipolis.com/documento/dc_fbc1e49400df

How to cite

Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression — arXiv:2610.10552 — arXiv (en), https://arxiv.org/abs/2610.10552 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_fbc1e49400df (Metraipolis archive dc_fbc1e49400df, sha256 31429fe80662020cd06c432924b218674821cd03de0d5e75f7b0f099c6b6b860)
BibTeX
@misc{dcfbc1e49400df,
  title = {Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression — arXiv:2610.10552},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.10552},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_fbc1e49400df; sha256 31429fe80662020cd06c432924b218674821cd03de0d5e75f7b0f099c6b6b860},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.10552. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Comparing parameter-efficient fine-tuning recipes under a single, shared learning rate is a common but flawed practice: when the arms being compared have very different trainable-parameter counts, a shared rate can simultaneously depress the larger arms' means and inflate their variance, manufacturing a large, seemingly multi-seed-significant advantage for the smallest arm that is not a real effect. We document this confound in a concrete setting: post-hoc SVD-based KV-cache compression, where an already-pretrained model is converted to a low-rank (multi-head-latent-attention-style) cache by factorizing its key/value weights into a down-projection ("encoder") and an up-projection ("decoder"), after which a short fine-tune ("healing") recovers the accuracy lost to truncation. Under a shared learning rate, freezing the decoder and healing only the encoder looks like a clear win over healing the decoder or both factors; once every arm is given its own tuned learning rate, that apparent advantage disappears, and encoder-only healing instead reaches parity with the alternatives, at a real, measured saving of 3x fewer trainable parameters and 3x less optimizer-state memory. We verify this parity with per-arm learning-rate tuning and three seeds per configuration on a vision-language model (Qwen2.5-VL-3B-Instruct), at the one compression ratio this protocol covers, and replicate it on a text-only testbed across two backbones. Encoder-only healing is therefore a lower-memory drop-in recipe for retrofitting low-rank KV-cache compression at training time, and the shared-learning-rate pitfall we document and correct is a cautionary result for comparing any fine-tuning recipes whose arms differ in trainable-parameter count.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI).
Pagina del lavoro: https://arxiv.org/abs/2610.10552 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-09T05:47:01.193Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).