Metraipolis › archive › dc_ebbdefe020a4
When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data? — arXiv:2610.41038
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2610.41038
- License (as archived)
- CC0-1.0
- Archived
- 2026-11-03T05:40:40.141Z
- SHA-256 of the archived text
7902ffc794c03f2750126ec006b81057882a78d1baa9e1d5b1f7680498ed34a3- Stable URL of this page
https://metraipolis.com/documento/dc_ebbdefe020a4
How to cite
When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data? — arXiv:2610.41038 — arXiv (en), https://arxiv.org/abs/2610.41038 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_ebbdefe020a4 (Metraipolis archive dc_ebbdefe020a4, sha256 7902ffc794c03f2750126ec006b81057882a78d1baa9e1d5b1f7680498ed34a3)
BibTeX
@misc{dcebbdefe020a4,
title = {When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data? — arXiv:2610.41038},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2610.41038},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_ebbdefe020a4; sha256 7902ffc794c03f2750126ec006b81057882a78d1baa9e1d5b1f7680498ed34a3},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.41038. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those which are hidden Markov models of some finite order (possibly non-stationary). In either case, we first obtain error bounds via finite models with finite window approximations. For stationary ergodic sources, the resulting error is bounded by an entropy gap; for $N$th order Markovian sources, it decays at an exponential rate $\alpha^{\lfloor M/N\rfloor}$, where $M$ is the window length and $\alpha$ is determined by filter stability (and the Dobrushin coefficients of the transition and measurement kernels). We then construct a Transformer of depth $M+1$, one head per layer, realizing the truncated filter, and we optimally train it with stochastic control developed in earlier work, which rigorously returns a near optimal Transformer. In particular, we show that for a non-Markovian source, either under stationarity and ergodicity for general models or under filter stability for hidden Markov models, an optimally trained Transformer has arbitrarily small performance error with increasing context length, data, and approximation refinement. Numerical studies accompany the theory.
Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC); Probability (math.PR).
Pagina del lavoro: https://arxiv.org/abs/2610.41038 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T05:26:25.660Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).