Metraipolis › archive › dc_e01be159798b

Universality and Generalization of Causal Transformers Across Context Lengths — arXiv:2609.35055

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.35055
License (as archived)
CC0-1.0
Archived
2026-10-07T22:38:45.497Z
SHA-256 of the archived text
7064f09d5b198e88f58fd76ff30576c2ef592b7c6e3c6dec38489da6ecfcc982
Stable URL of this page
https://metraipolis.com/documento/dc_e01be159798b

How to cite

Universality and Generalization of Causal Transformers Across Context Lengths — arXiv:2609.35055 — arXiv (en), https://arxiv.org/abs/2609.35055 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e01be159798b (Metraipolis archive dc_e01be159798b, sha256 7064f09d5b198e88f58fd76ff30576c2ef592b7c6e3c6dec38489da6ecfcc982)
BibTeX
@misc{dce01be159798b,
  title = {Universality and Generalization of Causal Transformers Across Context Lengths — arXiv:2609.35055},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.35055},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e01be159798b; sha256 7064f09d5b198e88f58fd76ff30576c2ef592b7c6e3c6dec38489da6ecfcc982},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.35055. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Long contexts are central to modern transformer systems, but most expressivity results choose a different network for each fixed sequence length. We study whether one masked transformer can approximate causal token-to-token maps uniformly over sequences of arbitrary length sampling a fixed normalized horizon. To relate sampling resolutions, we model tokens by $\alpha$-Hölder sequences or, more generally, a common modulus of continuity. Our notion of continuity across resolutions characterizes the causal families admitting uniform approximation on these compact input classes by a single transformer with length-independent parameters. The result extends to the infinite-length mean-field limit, where tokens form continuous curves and masked attention becomes a causal time integral. For bounded regression with target maps satisfying a $\beta$-smooth stability condition defined using regular test functions, quantitative approximation yields a generalization bound: exact empirical risk minimization over suitably sized bounded-weight transformers gives root mean-square prediction error $O((\log\log N/\log N)^{\beta/(d+2)})$ from $N$ iid labeled sequences. The bound holds at fixed confidence on the same sampling distribution, with $d$ the token dimension and no maximum-length factor. Finally, experiments on physical time series support the Hölder-regular token model at observed scales, with dataset-dependent fitted exponents, whereas text input embeddings provide a contrasting case. Native and dense sampling, shuffled controls, and refinement checks delimit this empirical regularity regime.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC); Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.35055 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T22:14:28.333Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).