Metraipolis › archive › dc_97decd008de1

Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention — arXiv:2611.01642

Source: arXiv (en)
Original: https://arxiv.org/abs/2611.01642
License (as archived): CC0-1.0
Archived: 2026-11-03T18:40:30.711Z
SHA-256 of the archived text: 7cc42bf8504629cff920a11f4f7ba28ada69d0710cd0fae88bcc6aa61d009b97
Stable URL of this page: https://metraipolis.com/documento/dc_97decd008de1

How to cite

Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention — arXiv:2611.01642 — arXiv (en), https://arxiv.org/abs/2611.01642 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_97decd008de1 (Metraipolis archive dc_97decd008de1, sha256 7cc42bf8504629cff920a11f4f7ba28ada69d0710cd0fae88bcc6aa61d009b97)

BibTeX

@misc{dc97decd008de1,
  title = {Sharp Generalization for Nonparametric Regression in Interpolation Space by Shallow Over-Parameterized Neural Networks with Channel Attention — arXiv:2611.01642},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2611.01642},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_97decd008de1; sha256 7cc42bf8504629cff920a11f4f7ba28ada69d0710cd0fae88bcc6aa61d009b97},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2611.01642. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

We study nonparametric regression using an over-parameterized two-layer ReLU neural network with fixed covariance-based channel attention, trained by gradient descent with both layers updated simultaneously. The spherical covariates follow an arbitrary fixed non-atomic distribution, and the target function lies in an interpolation space with spectral bias widely studied in the statistical learning literature. We prove that early-stopped training achieves regression risk $\cO(\eps_n^2)$, where $\eps_n$ is the critical population rate of the attention kernel $\Kattn$. Its positive eigenvalues are $\lambda_j^{s'}$, where $\lambda_j$ are those of the ReLU feature covariance kernel $K$ and $s'=2s+3$ for $s\ge1$. Under polynomial eigenvalue decay $\lambda_j\asymp j^{-2\alpha}$ with $\alpha>1/2$, the regression risk bound $\cO(\eps_n^2)$ specializes to the minimax optimal rate $\cO(n^{-2\alpha s'/(2\alpha s'+1)})$. Our analysis is based on two new technical results on the optimization and generalization of the attention network. First, the neural network function after every gradient descent iteration admits a decomposition into a function with bounded Reproducing Kernel Hilbert Space (RKHS) norm and a uniformly small residual. Second, local Rademacher complexity converts this decomposition into a regression risk bound. Kernel gradient descent on the same ball attains the same rate, while our theorem gives explicit sufficient width and auxiliary-sample conditions for simultaneous network training. Our analysis is further extended to covariates supported on $\RR^d$ through a novel normalization method. Under a sub-Gaussian covariate distribution, the attention network achieves a nearly optimal regression risk $\cO((\log n)n^{-s'(d+3)/(s'(d+3)+d)})$ without a density assumption.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (stat.ML) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2611.01642 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-03T18:12:54.735Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).