Metraipolis › archive › dc_d3cdf40cf291
Form Over Content In Gradient-Based Data Attribution Methods — arXiv:2609.19589
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.19589
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-10T03:23:23.050Z
- SHA-256 of the archived text
4548ba89241ecbbac22d8727ad03b7e3636220a2cdccb891f0e64a9cfcfc1312- Stable URL of this page
https://metraipolis.com/documento/dc_d3cdf40cf291
How to cite
Form Over Content In Gradient-Based Data Attribution Methods — arXiv:2609.19589 — arXiv (en), https://arxiv.org/abs/2609.19589 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_d3cdf40cf291 (Metraipolis archive dc_d3cdf40cf291, sha256 4548ba89241ecbbac22d8727ad03b7e3636220a2cdccb891f0e64a9cfcfc1312)
BibTeX
@misc{dcd3cdf40cf291,
title = {Form Over Content In Gradient-Based Data Attribution Methods — arXiv:2609.19589},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.19589},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_d3cdf40cf291; sha256 4548ba89241ecbbac22d8727ad03b7e3636220a2cdccb891f0e64a9cfcfc1312},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.19589. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
Data attribution methods using gradient similarity are widely used to analyze and select training data for large language models, but what gradient similarity actually measures is debated. Some interpret it as identifying task-relevant skills, while other work reports that surface form is the main factor. We resolve this debate for supervised fine-tuning examples by varying task and answer format independently. Specifically, we render benchmarks in different answer formats, such that datasets can share a task without a format or a format without a task. We find that gradient alignment follows the answer format, as benchmark pairs sharing an answer format align strongly (disattenuated cosine near 0.4), while same benchmarks rendered with different answer format classes show no alignment (near 0.0). We demonstrate that this ordering holds from the earliest pretraining checkpoints through post-training, and across model scales and families. We then analyze the released selections of LESS, a gradient-based data selection method for instruction tuning, and find that each target's selections over-represent the target's own answer format. Hence, we demonstrate that gradient-based attribution methods track format similarity more than task semantics, meaning that such methods, as well as the semantic interpretation of the gradient, should be tested on data where answer format and task vary independently for greater robustness and reliability. Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-18 in cs.AI). Categorie arXiv: Computation and Language (cs.CL) ; Artificial Intelligence (cs.AI). Pagina del lavoro: https://arxiv.org/abs/2609.19589 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito). Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina. Letto il 2026-10-10T03:04:13.836Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-18?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).