Metraipolis › archive › dc_e6bed75d31ad

Training-Free Universal Approximation by Prompting Random Transformers — arXiv:2608.09558

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.09558
License (as archived)
CC0-1.0
Archived
2026-10-19T09:06:21.895Z
SHA-256 of the archived text
0a63553512d6ab4f8425ac7c7fededa4cd12360b528f23f135f7ce9e126a60d2
Stable URL of this page
https://metraipolis.com/documento/dc_e6bed75d31ad

How to cite

Training-Free Universal Approximation by Prompting Random Transformers — arXiv:2608.09558 — arXiv (en), https://arxiv.org/abs/2608.09558 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e6bed75d31ad (Metraipolis archive dc_e6bed75d31ad, sha256 0a63553512d6ab4f8425ac7c7fededa4cd12360b528f23f135f7ce9e126a60d2)
BibTeX
@misc{dce6bed75d31ad,
  title = {Training-Free Universal Approximation by Prompting Random Transformers — arXiv:2608.09558},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.09558},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e6bed75d31ad; sha256 0a63553512d6ab4f8425ac7c7fededa4cd12360b528f23f135f7ce9e126a60d2},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.09558. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any Hölder function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

Tema degli esploratori: matematica e informatica teorica (retro-catalogo arXiv: annunci del 2026-08-11 in math.ST).
Categorie arXiv: Machine Learning (cs.LG) ; Numerical Analysis (math.NA); Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2608.09558 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-19T09:04:01.375Z dalla pagina https://arxiv.org/catchup/math.ST/2026-08-11?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).