Metraipolis › archive › dc_cdd7563846cc
Two-Stage Clipping: Linear-Time Approximate and Exact Mixed-Norm Projections for Transformers — arXiv:2610.38598
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.38598
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-01T02:24:49.599Z
- SHA-256 of the archived text
8ed49cc1a69a29966d3c918085fd3b7b49325c5db3ed9de747f5316855fd9f65- Stable URL of this page
https://metraipolis.com/documento/dc_cdd7563846cc
How to cite
Two-Stage Clipping: Linear-Time Approximate and Exact Mixed-Norm Projections for Transformers — arXiv:2610.38598 — arXiv (en), https://arxiv.org/abs/2610.38598 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cdd7563846cc (Metraipolis archive dc_cdd7563846cc, sha256 8ed49cc1a69a29966d3c918085fd3b7b49325c5db3ed9de747f5316855fd9f65)
BibTeX
@misc{dccdd7563846cc,
title = {Two-Stage Clipping: Linear-Time Approximate and Exact Mixed-Norm Projections for Transformers — arXiv:2610.38598},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.38598},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cdd7563846cc; sha256 8ed49cc1a69a29966d3c918085fd3b7b49325c5db3ed9de747f5316855fd9f65},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.38598. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Large Language Models (LLMs) based on Transformer architectures have achieved remarkable performance across artificial intelligence tasks, but their computational cost remains a major bottleneck. Structured sparsification provides a promising approach to reducing this cost. In particular, $\ell_{1,\infty}$ projections can induce column-wise sparsity in model matrices, but existing proximal methods (Prox) typically require $\mathcal{O}(nm\log(nm))$ operations for matrices in $\mathbb{R}^{n\times m}$, limiting their computational and energy efficiency. In this paper, we introduce Two-Stage Clipping (TSC), an efficient structured sparsification operator inspired by bilevel optimization. Rather than solving a bilevel optimization problem in the strict sense, TSC decomposes the computation into two successive stages. First, group-level quantities are projected to allocate the global sparsity budget and identify active groups. Second, the resulting values are used as group-wise clipping thresholds for the matrix coefficients. For square attention matrices ($n \times n$), the complexity of our TSC method is $O(n^2)$, which is linear in the number of matrix coefficients. We establish a theoretical bound on the discrepancy between TSC and the exact proximal projection and show on synthetic data that the additional Euclidean approximation cost remains small in practice. Extensive benchmarks demonstrate substantial runtime improvements over proximal baselines and validate the theoretical norm identities underlying the proposed construction.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.38598 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.127Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).