Metraipolis › archive › dc_cb493bbc329c

Conic Certification of Transformer Attention — arXiv:2611.00585

Source
arXiv (en)
Original
https://arxiv.org/abs/2611.00585
License (as archived)
CC0-1.0
Archived
2026-11-03T10:25:55.024Z
SHA-256 of the archived text
57bffbce9b06dee7bbff3d6e0871ee05c86b530c64190f2b59880e6da15833f4
Stable URL of this page
https://metraipolis.com/documento/dc_cb493bbc329c

How to cite

Conic Certification of Transformer Attention — arXiv:2611.00585 — arXiv (en), https://arxiv.org/abs/2611.00585 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_cb493bbc329c (Metraipolis archive dc_cb493bbc329c, sha256 57bffbce9b06dee7bbff3d6e0871ee05c86b530c64190f2b59880e6da15833f4)
BibTeX
@misc{dccb493bbc329c,
  title = {Conic Certification of Transformer Attention — arXiv:2611.00585},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.00585},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_cb493bbc329c; sha256 57bffbce9b06dee7bbff3d6e0871ee05c86b530c64190f2b59880e6da15833f4},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.00585. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Attention is the computational core of large language models, yet the verifiers that bound how far an attention head's output can move under input perturbations give no guarantee on how loose their bounds are. A weak certificate therefore cannot be told apart from a fragile model. We develop a conic programming theory of attention certification that supplies this guarantee. We prove that the difficulty lies not in the softmax but in the bilinear interaction of a varying query with the keys. This interaction makes certification NP-hard over boxes and inapproximable once the attention logits can move by a constant amount. A fixed query or listed token substitutions, by contrast, make the exact worst case efficiently computable. An exponential cone moment relaxation then certifies a head with an error bounded quadratically in the perturbation radius by computable quantities, an order that cannot be improved. We also show that relaxations that treat a network one operation at a time, linear bound propagation among them, pay on second-order terms a sum of per-operation maxima instead of one coupled maximum. Computational results on GPT-2 indicate that the theory carries over to a trained model. Against exact worst cases with the query fixed, the certificate attains the predicted second-order accuracy. It is tighter than linear bound propagation on every head, and the looseness of bound propagation tracks the predicted per-operation excess.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.AI, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2611.00585 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T10:19:50.017Z dalla pagina https://arxiv.org/list/cs.AI/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).