Metraipolis › archive › dc_bf3d7d59a809

LaMoC: Loss-Aware Modular Compression for LLMs — arXiv:2608.30226

Source: arXiv (en)
Original: https://arxiv.org/abs/2608.30226
License (as archived): CC0-1.0
Archived: 2026-10-17T07:44:33.816Z
SHA-256 of the archived text: 778b95c4f022b4426a7a4dfeeff668b317ef359df4683c9410be9f78dd120b31
Stable URL of this page: https://metraipolis.com/documento/dc_bf3d7d59a809

How to cite

LaMoC: Loss-Aware Modular Compression for LLMs — arXiv:2608.30226 — arXiv (en), https://arxiv.org/abs/2608.30226 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bf3d7d59a809 (Metraipolis archive dc_bf3d7d59a809, sha256 778b95c4f022b4426a7a4dfeeff668b317ef359df4683c9410be9f78dd120b31)

BibTeX

@misc{dcbf3d7d59a809,
  title = {LaMoC: Loss-Aware Modular Compression for LLMs — arXiv:2608.30226},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.30226},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bf3d7d59a809; sha256 778b95c4f022b4426a7a4dfeeff668b317ef359df4683c9410be9f78dd120b31},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.30226. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modular compression has enabled considerable parameter reduction in LLMs while preserving strong language understanding and downstream task accuracy. However, existing joint modular compression methods primarily rely on activation statistics, leaving loss-sensitivity information and its module-level characterization underexplored. We investigate addressing this gap with LaMoC, a loss-aware modular compression methodology that blends activation and Empirical Fisher statistics through gradient-error alignment. LaMoC improves joint compression by selecting compression statistics that better align local module reconstruction error with the downstream loss. Our contributions are three-fold: (1) We characterize the Empirical Fisher as a module-level loss-aware proxy that can be blended with the activation statistics required for compression. (2) We reformulate joint modular compression as a two-tiered optimization problem that minimizes module reconstruction error while tuning the activation and gradient information blending rate. (3) We implement an empirically driven methodology with statistical validation to solve the resulting compression problem. We evaluate LaMoC across four model families spanning eight models. On the 4-8B models, LaMoC achieves an average 2.5% reduction in perplexity and a 1% relative improvement in task accuracy over state-of-the-art modular compression methods.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Artificial Intelligence (cs.AI) ; Computation and Language (cs.CL); Performance (cs.PF).
Pagina del lavoro: https://arxiv.org/abs/2608.30226 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.589Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).