Metraipolis › archive › dc_4528cefd554e

Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks — arXiv:2610.39717

Source: arXiv (en)
Original: https://arxiv.org/abs/2610.39717
License (as archived): CC0-1.0
Archived: 2026-11-01T02:25:12.208Z
SHA-256 of the archived text: dd1323ade6836760725aa0b5a26dc19bf839f32fdb9ac2a444ecea426a62c9ef
Stable URL of this page: https://metraipolis.com/documento/dc_4528cefd554e

How to cite

Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks — arXiv:2610.39717 — arXiv (en), https://arxiv.org/abs/2610.39717 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_4528cefd554e (Metraipolis archive dc_4528cefd554e, sha256 dd1323ade6836760725aa0b5a26dc19bf839f32fdb9ac2a444ecea426a62c9ef)

BibTeX

@misc{dc4528cefd554e,
  title = {Implicit Bias of (Stochastic) Gradient Descent on Deep Linear Diagonal Networks — arXiv:2610.39717},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.39717},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_4528cefd554e; sha256 dd1323ade6836760725aa0b5a26dc19bf839f32fdb9ac2a444ecea426a62c9ef},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.39717. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Modern machine learning models are often overparameterized, i.e., have more parameters than training data. This typically implies that the loss function admits multiple minimizers so that training methods (and their hyperparameter choices) have significant influence on the computed solution. We study this so-called implicit bias phenomenon in the context of training deep linear diagonal networks. Previous work has shown that the gradient flow for learning such networks converges to (almost) sparse solutions when initialized small. For an arbitrary number of layers, we extend this analysis to the practically more realistic (stochastic) gradient descent iterations. Our approach identifies (stochastic) gradient descent as a perturbed (stochastic) mirror gradient descent. This allows us to establish convergence and characterize the limiting solution, relating it to weighted $\ell_1$-minimization. Moreover, we provide sufficient conditions on the step sizes and quantify the influence of the step sizes on the implicit bias.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG) ; Optimization and Control (math.OC).
Pagina del lavoro: https://arxiv.org/abs/2610.39717 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.179Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).