Metraipolis › archive › dc_f28349ec98be
Exact Finite Attention Responses From RoPE Derivatives — arXiv:2609.14127
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2609.14127
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-16T00:15:26.636Z
- SHA-256 of the archived text
8c9ea191a9a32ce47dfd99d9b37bc4759787d992caa38e483ebdcdb3b992c4c9- Stable URL of this page
https://metraipolis.com/documento/dc_f28349ec98be
How to cite
Exact Finite Attention Responses From RoPE Derivatives — arXiv:2609.14127 — arXiv (en), https://arxiv.org/abs/2609.14127 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f28349ec98be (Metraipolis archive dc_f28349ec98be, sha256 8c9ea191a9a32ce47dfd99d9b37bc4759787d992caa38e483ebdcdb3b992c4c9)
BibTeX
@misc{dcf28349ec98be,
title = {Exact Finite Attention Responses From RoPE Derivatives — arXiv:2609.14127},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2609.14127},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f28349ec98be; sha256 8c9ea191a9a32ce47dfd99d9b37bc4759787d992caa38e483ebdcdb3b992c4c9},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.14127. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative $\partial_p z(p) = A z(p)$: its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term $C_{KV} = \sum_j (p'_j - p_j)\,\varepsilon_j$, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-15 in cs.LG).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.14127 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-15T23:54:14.886Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-15?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).