Metraipolis › archive › dc_bfbb58c1ad31

Sparse Weight Decomposition for Efficient Circuit Extraction — arXiv:2608.03913

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.03913
License (as archived)
CC0-1.0
Archived
2026-10-31T12:31:25.777Z
SHA-256 of the archived text
bb857a53e0b265c8c8901fc091629e148b31b31b1f75452d2e636865ea873160
Stable URL of this page
https://metraipolis.com/documento/dc_bfbb58c1ad31

How to cite

Sparse Weight Decomposition for Efficient Circuit Extraction — arXiv:2608.03913 — arXiv (en), https://arxiv.org/abs/2608.03913 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bfbb58c1ad31 (Metraipolis archive dc_bfbb58c1ad31, sha256 bb857a53e0b265c8c8901fc091629e148b31b31b1f75452d2e636865ea873160)
BibTeX
@misc{dcbfbb58c1ad31,
  title = {Sparse Weight Decomposition for Efficient Circuit Extraction — arXiv:2608.03913},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.03913},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bfbb58c1ad31; sha256 bb857a53e0b265c8c8901fc091629e148b31b31b1f75452d2e636865ea873160},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.03913. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation circuit extraction workflow used for methods that learn sparse features. Across single-matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. For matched replacement fidelity, SWD reaches the same circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all attention and MLP weight matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (e.g., per-step analysis).

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-05 in cs.CL).
Categorie arXiv: Machine Learning (cs.LG) ; Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2608.03913 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-31T12:17:37.817Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-05?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).