Metraipolis › archive › dc_bdf62dbe1627

Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers — arXiv:2609.31458

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.31458
License (as archived)
CC0-1.0
Archived
2026-10-08T02:19:20.112Z
SHA-256 of the archived text
3bcaa24994d1f334468dfe113edcc916e24283a8b3fdf04647e3a8daeffe62cf
Stable URL of this page
https://metraipolis.com/documento/dc_bdf62dbe1627

How to cite

Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers — arXiv:2609.31458 — arXiv (en), https://arxiv.org/abs/2609.31458 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_bdf62dbe1627 (Metraipolis archive dc_bdf62dbe1627, sha256 3bcaa24994d1f334468dfe113edcc916e24283a8b3fdf04647e3a8daeffe62cf)
BibTeX
@misc{dcbdf62dbe1627,
  title = {Nonparametric In-Context Learning under Growing Geometric Complexity: Minimax Optimality and Local Geometry-Adaptivity of Transformers — arXiv:2609.31458},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.31458},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_bdf62dbe1627; sha256 3bcaa24994d1f334468dfe113edcc916e24283a8b3fdf04647e3a8daeffe62cf},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.31458. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Transformers have become a central architecture for in-context learning (ICL), particularly through their state-of-the-art performance in large language models. This success motivates understanding how transformers exploit task-relevant structure in geometrically heterogeneous data. However, existing nonparametric ICL theory has largely focused on Euclidean domains or single-manifold models. To address this gap, we study the prediction problem under unknown local geometry, modeled by sample size-dependent mixtures of manifolds with heterogeneous dimensions, smoothness, and sampling masses. Under local separation and small-perturbation conditions, we establish a minimax lower bound capturing the aggregate difficulty of the components and construct an oracle tangent local-polynomial estimator with a matching upper bound. This estimator is connected to a structure-informed, two-stage softmax transformer with a geometric preconditioner and chartwise reduced local-polynomial solvers. The transformer achieves negligible approximation error relative to the minimax rate with logarithmic depth and polynomial size. Finally, we derive an in-context generalization bound for near empirical risk minimizers over this class. Together, these results identify conditions under which the resulting predictor exploits local geometry and attains the aggregate minimax rate.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-28 in cs.LG).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG); Statistics Theory (math.ST).
Pagina del lavoro: https://arxiv.org/abs/2609.31458 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-08T01:43:06.817Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).