Metraipolis › archive › dc_77909d45305d

Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks — arXiv:2609.09130

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.09130
License (as archived)
CC0-1.0
Archived
2026-10-16T13:58:30.991Z
SHA-256 of the archived text
e0c2130979b151801ed55a2cec45490f3ff70166571cb7a3287c74e2fc6e2cce
Stable URL of this page
https://metraipolis.com/documento/dc_77909d45305d

How to cite

Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks — arXiv:2609.09130 — arXiv (en), https://arxiv.org/abs/2609.09130 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_77909d45305d (Metraipolis archive dc_77909d45305d, sha256 e0c2130979b151801ed55a2cec45490f3ff70166571cb7a3287c74e2fc6e2cce)
BibTeX
@misc{dc77909d45305d,
  title = {Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks — arXiv:2609.09130},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.09130},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_77909d45305d; sha256 e0c2130979b151801ed55a2cec45490f3ff70166571cb7a3287c74e2fc6e2cce},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.09130. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

An input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the one-hidden-layer ReLU model of Awasthi et al. (COLT 2024). For width $s$, at most $k$ active units per input, and effective weight and bias bounds $W,B$, every size-$m$ sample in the class's fixed radius-$R$ input domain satisfies $\mathcal{R}(S)\le CWR\min\{k,\sqrt{sk/m}\log^{3/2}(2m)\}+kB/\sqrt m$. A support-preserving cover and a single normalized chaining argument remove the previous explicit dimension factor, up to logarithms. Lower bounds on appropriate i.i.d. marginals match up to those logarithms, showing how changing active units across inputs retains a width dependence. The input domain matters: zero-bias networks sparse on the entire ball have at most $2k$ nonzero units and complexity $O(kWR/\sqrt m)$, whereas bias bounds comparable to $WR$ restore the worst-case rate on that same domain in only logarithmic dimension. A spherical-cap construction proves the latter claim without assuming sparsity merely on the sampling support. For a specified normalized bounded loss and biases comparable to $WR$, we also obtain agnostic minimax excess-risk bounds of order $\min\{1,\sqrt{s/(km)}\}$ up to logarithms.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-09 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG) ; Machine Learning (stat.ML).
Pagina del lavoro: https://arxiv.org/abs/2609.09130 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-16T13:39:43.094Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-09-09?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).