Metraipolis › archive › dc_6a2ab85fee0b
Importance-Aware Low-Rank Distillation of Diffusion Transformers — arXiv:2609.04646
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.04646
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T23:33:25.996Z
- SHA-256 of the archived text
fe502b175936b1816eceac7e06ae6b06e301f7adbaa8f017d36c4def9a83dbdc- Stable URL of this page
https://metraipolis.com/documento/dc_6a2ab85fee0b
How to cite
Importance-Aware Low-Rank Distillation of Diffusion Transformers — arXiv:2609.04646 — arXiv (en), https://arxiv.org/abs/2609.04646 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_6a2ab85fee0b (Metraipolis archive dc_6a2ab85fee0b, sha256 fe502b175936b1816eceac7e06ae6b06e301f7adbaa8f017d36c4def9a83dbdc)
BibTeX
@misc{dc6a2ab85fee0b,
title = {Importance-Aware Low-Rank Distillation of Diffusion Transformers — arXiv:2609.04646},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.04646},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_6a2ab85fee0b; sha256 fe502b175936b1816eceac7e06ae6b06e301f7adbaa8f017d36c4def9a83dbdc},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.04646. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Diffusion Transformers (DiTs) have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition (SVD) is a principled tool for parameter reduction, evidence from large language models (LLMs) suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks. Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective. We apply SVDtrunc to this http URL across compression levels ranging from 40-90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models. Project page: this https URL Tema degli esploratori: statistica e scienza dei dati (retro-catalogo arXiv: annunci del 2026-09-07 in cs.CV). Categorie arXiv: Computer Vision and Pattern Recognition (cs.CV). Pagina del lavoro: https://arxiv.org/abs/2609.04646 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-16T23:16:11.392Z dalla pagina https://arxiv.org/catchup/cs.CV/2026-09-07?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).