Metraipolis › archive › dc_f61888f0ba2f

Sequential Functional Structured Tucker Compression for Large Language Model Attentions — arXiv:2610.00717

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.00717
License (as archived)
CC0-1.0
Archived
2026-10-06T12:10:52.829Z
SHA-256 of the archived text
9230762e43f9c24d748379118fcf413abed0b24b793f645faeed817766c6ffa6
Stable URL of this page
https://metraipolis.com/documento/dc_f61888f0ba2f

How to cite

Sequential Functional Structured Tucker Compression for Large Language Model Attentions — arXiv:2610.00717 — arXiv (en), https://arxiv.org/abs/2610.00717 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f61888f0ba2f (Metraipolis archive dc_f61888f0ba2f, sha256 9230762e43f9c24d748379118fcf413abed0b24b793f645faeed817766c6ffa6)
BibTeX
@misc{dcf61888f0ba2f,
  title = {Sequential Functional Structured Tucker Compression for Large Language Model Attentions — arXiv:2610.00717},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.00717},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f61888f0ba2f; sha256 9230762e43f9c24d748379118fcf413abed0b24b793f645faeed817766c6ffa6},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.00717. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Post-training compression of LLM attention is often formulated as independent matrix approximation, ignoring both the shared structure among attention projections and the representation shift introduced by earlier compression. We propose FTC, a sequential structured compression framework that adapts the approximation to the current compressed model while jointly exploiting the native Q/K/V head structure under a fixed storage budget. The output projection is handled separately to account for the changed post-attention representation. FTC requires neither fine-tuning nor gradient-based recovery. Across seven decoder-only LLMs from 6B to 32B parameters, FTC achieves the lowest WikiText-2 perplexity among the compared methods at every tested keep ratio on five modern GQA models, with the largest gains under aggressive compression. The improvements transfer to downstream tasks and remain substantial at the 32B scale.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-10-02 in cs.AI).
Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2610.00717 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-06T10:21:28.104Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-10-02?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).