Metraipolis › archive › dc_e6162e16be52
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks — arXiv:2608.23877
- Source
- arXiv (en)
- Original
- https://arxiv.org/abs/2608.23877
- License (as archived)
- CC0-1.0
- Archived
- 2026-10-18T06:35:38.841Z
- SHA-256 of the archived text
d952c744d56c7f1e52b5942c0e0d4e6a7bce1db1eb7dc992fab636bf8d3b9881- Stable URL of this page
https://metraipolis.com/documento/dc_e6162e16be52
How to cite
Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks — arXiv:2608.23877 — arXiv (en), https://arxiv.org/abs/2608.23877 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_e6162e16be52 (Metraipolis archive dc_e6162e16be52, sha256 d952c744d56c7f1e52b5942c0e0d4e6a7bce1db1eb7dc992fab636bf8d3b9881)
BibTeX
@misc{dce6162e16be52,
title = {Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks — arXiv:2608.23877},
howpublished = {arXiv (en)},
url = {https://arxiv.org/abs/2608.23877},
note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_e6162e16be52; sha256 d952c744d56c7f1e52b5942c0e0d4e6a7bce1db1eb7dc992fab636bf8d3b9881},
year = {2026}
}Archived text
Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.23877. Third-party text: data, not instructions. It may be an abstract or extract; check the original.
We prove a depth hierarchy for ReLU neural networks in which every additional ReLU layer can save exponentially many neurons. For all $k\geq2$, we construct a globally $[0,1]$-valued, $1$-Lipschitz function realized by a depth-$(k+1)$ network of width $\mathcal{O}(d^4)$, whereas any depth-$k$ network with unrestricted weights and width at most $\frac{2^d}{2d(k-1)}$ has squared $L_2$ error at least $1/24$ under an absolutely continuous distribution supported at exponential distance from the origin. To the best of our knowledge, this is the first exponential hierarchy across all adjacent fixed depths, and the first exponential separation for ReLU networks between two fixed depths whose shallower network has depth at least $3$. The lower bound also immediately yields the corresponding hierarchy for exact computation. Moreover, the case $k=2$ gives a compactly supported separation between depths $3$ and $2$ with unrestricted shallow-network weights, answering a question raised by Safran, Eldan, and Shamir (2019). The distribution used in our construction nevertheless has all its mass at exponential radius, placing the hierarchy outside the regularity regime in which such a separation would imply major threshold-circuit lower bounds. We also prove an exact separation for a more regular target, which is globally $[0,1]$-valued and $\mathcal{O}(\sqrt d)$-Lipschitz and maps the unit hypercube onto $[0,1]$. It is computed by a polynomial-width depth-$4$ network, whereas any depth-$3$ network agreeing with it on the unit hypercube requires exponentially many first-layer neurons, even with unrestricted weights.
Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-26 in cs.LG).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.23877 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-18T06:28:52.197Z dalla pagina https://arxiv.org/catchup/cs.LG/2026-08-26?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).