Metraipolis › archive › dc_f580424c39db

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization — arXiv:2609.33835

Source
arXiv (en)
Original
https://arxiv.org/abs/2609.33835
License (as archived)
CC0-1.0
Archived
2026-10-07T21:01:50.105Z
SHA-256 of the archived text
a7194c5e6a4d42e74a9ed102cc387ef3aca6b09a844b35f135646e3b78cd8863
Stable URL of this page
https://metraipolis.com/documento/dc_f580424c39db

How to cite

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization — arXiv:2609.33835 — arXiv (en), https://arxiv.org/abs/2609.33835 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_f580424c39db (Metraipolis archive dc_f580424c39db, sha256 a7194c5e6a4d42e74a9ed102cc387ef3aca6b09a844b35f135646e3b78cd8863)
BibTeX
@misc{dcf580424c39db,
  title = {An Active-Bottleneck Mechanism for Weak-to-Strong Generalization — arXiv:2609.33835},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2609.33835},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_f580424c39db; sha256 a7194c5e6a4d42e74a9ed102cc387ef3aca6b09a844b35f135646e3b78cd8863},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2609.33835. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Weak-to-strong generalization (W2SG) occurs when a student trained on a teacher's predictions outperforms that teacher. We study when this happens under fully converged, ridgeless two-stage learning, with no early stopping, no explicit regularization, and no assumption that the student is more expressive than the teacher. In two-stage linear regression, a teacher is fit from $n$ labeled examples and a student is trained solely on the teacher's predictions on $m$ fresh, unlabeled inputs. Although both stages share the same hypothesis class and the same training rule, we show that the student outperforms the teacher exactly when $m$ lies in an explicit intermediate range: too few pseudo-labels leave the student without enough signal, too many let it inherit the teacher's noise. Under power-law covariance, we derive this range in closed form as a function of the spectral decay and noise level, including regimes where the improving region splits into two disjoint intervals of $m$. We then study a random-feature model in which the student has strictly more features than the teacher, and identify two regimes, again given by explicit thresholds: one where improvement occurs only for $m$ in a bounded interval, and one where it occurs only once the student width $N_S$ exceeds an explicit threshold. Both regimes are governed by a single "active-bottleneck" principle: whichever of $m$ or $N_S$ is scarcer controls how much teacher error is filtered out, while increasing the other resource only reduces estimation noise. Together, these results show that finite data and finite width can themselves regularize a two-stage learner, with no explicit mechanism doing so.

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-09-29 in cs.AI).
Categorie arXiv: Machine Learning (stat.ML) ; Artificial Intelligence (cs.AI); Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2609.33835 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-07T20:29:02.764Z dalla pagina https://arxiv.org/catchup/cs.AI/2026-09-29?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).