Metraipolis › archive › dc_4e841aa95df1

Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks — arXiv:2608.28843

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.28843
License (as archived)
CC0-1.0
Archived
2026-10-17T07:47:18.595Z
SHA-256 of the archived text
9459feb78f79fa4e33c94a5a8a7e171c7f4b5695fd354574bb2b5557dfdecc98
Stable URL of this page
https://metraipolis.com/documento/dc_4e841aa95df1

How to cite

Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks — arXiv:2608.28843 — arXiv (en), https://arxiv.org/abs/2608.28843 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4e841aa95df1 (Metraipolis archive dc_4e841aa95df1, sha256 9459feb78f79fa4e33c94a5a8a7e171c7f4b5695fd354574bb2b5557dfdecc98)
BibTeX
@misc{dc4e841aa95df1,
  title = {Curvature Cryptanalysis of Smooth Transformer Feed-Forward Networks — arXiv:2608.28843},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.28843},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4e841aa95df1; sha256 9459feb78f79fa4e33c94a5a8a7e171c7f4b5695fd354574bb2b5557dfdecc98},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.28843. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We show that smooth two-layer feed-forward networks (FFNs) expose an additional structural model extraction channel under a chosen-input raw-output oracle at the FFN branch; consider transformer FFN branches with GELU or SiLU activations under chosen-input raw-output access, without access to parameters, gradients, or internal activations; exploit a second-order leakage channel in which projected input Hessians form different mixtures of the same hidden symmetric rank-one factors induced by the FFN input weights. We formalize resulting Hessian collection as a partially symmetric decomposition to establish conditions for local identifiability and stability to exploit vector-output stencil reuse to reduce the structural query cost by a factor of 16. On independently trained CIFAR-10 vision transformers, only 16 projected Hessians, corresponding to 8193 black-box queries, recover the hidden FFN directions with average absolute cosine alignment above 0.91, with 86.7.1 % of GELU and 91.1 % of SiLU directions exceeding 0.90 alignment. Recovery remains high across independently trained models, repeated extraction runs, and all transformer blocks. The recovered structure supports functional extraction too. Keeping the recovered directions fixed and fitting only the remaining FFN parameters yields high-fidelity substitutes with more than 92 % top-1 agreement, while test accuracy remains within 1.24% and 0.57% of the GELU and SiLU targets. Output rounding and Gaussian noise substantially reduce recovery under a fixed attack configuration, but adapting the finite-difference step restores average alignment to 0.9188 and 0.9081. This is an end-to-end path from black-box second-order observations to hidden FFN-structure recovery and functional replacement. Under the stated oracle model, smooth FFN curvature exposes internal parameter geometry that behavioral fidelity alone cannot reveal.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-01 in cs.AI).
Categorie arXiv: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR).
Pagina del lavoro: https://arxiv.org/abs/2608.28843 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T07:28:26.690Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-01?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).