Metraipolis › archive › dc_bae95108bcad

MechSparse: Mechanism-Guided Sparse PEFT Selection Is Task-Shaped — arXiv:2609.18961

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.18961
License (as archived)
CC0-1.0
Archived
2026-10-15T13:49:40.228Z
SHA-256 of the archived text
79d757c3b44c3b1f3b75b9e17017a481e936e3117c8a418609ded5e10016d8bb
Stable URL of this page
https://metraipolis.com/documento/dc_bae95108bcad

How to cite

MechSparse: Mechanism-Guided Sparse PEFT Selection Is Task-Shaped — arXiv:2609.18961 — arXiv (en), https://arxiv.org/abs/2609.18961 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bae95108bcad (Metraipolis archive dc_bae95108bcad, sha256 79d757c3b44c3b1f3b75b9e17017a481e936e3117c8a418609ded5e10016d8bb)
BibTeX
@misc{dcbae95108bcad,
  title = {MechSparse: Mechanism-Guided Sparse PEFT Selection Is Task-Shaped — arXiv:2609.18961},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.18961},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bae95108bcad; sha256 79d757c3b44c3b1f3b75b9e17017a481e936e3117c8a418609ded5e10016d8bb},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.18961. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Mechanistic interpretability identifies sparse subsets of heads and MLP blocks that carry specific behaviors. We ask whether such causal signals can guide where to place a small PEFT budget more effectively than the cheap heuristics practitioners already use. \method{} scores attention heads and MLP blocks by normalized activation-patching recovery on clean/corrupted probes and trains LoRA/QLoRA only on the selected sites; \methodc{} adds bounded credit for small within-layer joint subsets. We compare against random, magnitude, activation-norm, and gradient/Fisher on Ministral-8B/NF4 in three cells: Swahili span-JSON information extraction (IE) at $b{=}0.25\%$ and $1.0\%$, and English$\to$Swahili machine translation (MT) at $b{=}1.0\%$. The causal selectors never win the primary metric. On the headline IE cell (3 seeds, paired-bootstrap CIs over $600$ predictions), \methodc{} beats random by $+0.079$ span+type F1 and gradient/Fisher by $+0.174$, but trails activation-norm by $0.028$, with the smallest cross-seed std ($\pm 0.003$). On MT all four selectors lie within $0.30$ BLEU and every paired CI includes zero. A schema-versus-span decomposition explains the IE gap: activation-norm captures the rigid JSON routine, while causal scores track content-sensitive sites. We distill a preliminary diagnostic -- prefer activation-norm when output structure dominates, treat causal selectors as a hypothesis for content-dominated tasks -- and release masks, scores, predictions, and evaluation files for direct replay.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-17 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL).
Pagina del lavoro: https://arxiv.org/abs/2609.18961 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T13:48:24.358Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-09-17?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).