Metraipolis › archive › dc_85ae10cadee2

Bi-Phase Training: Learning Efficiently From Scratch — arXiv:2610.37934

Source
arXiv (en)
Original
https://arxiv.org/abs/2610.37934
License (as archived)
CC0-1.0
Archived
2026-11-01T02:23:17.854Z
SHA-256 of the archived text
c97a5248128c5ecafb4fd7b2dd5a08938ef539cfc2788e96e6ea178ed60dade0
Stable URL of this page
https://metraipolis.com/documento/dc_85ae10cadee2

How to cite

Bi-Phase Training: Learning Efficiently From Scratch — arXiv:2610.37934 — arXiv (en), https://arxiv.org/abs/2610.37934 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_85ae10cadee2 (Metraipolis archive dc_85ae10cadee2, sha256 c97a5248128c5ecafb4fd7b2dd5a08938ef539cfc2788e96e6ea178ed60dade0)
BibTeX
@misc{dc85ae10cadee2,
  title = {Bi-Phase Training: Learning Efficiently From Scratch — arXiv:2610.37934},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2610.37934},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_85ae10cadee2; sha256 c97a5248128c5ecafb4fd7b2dd5a08938ef539cfc2788e96e6ea178ed60dade0},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2610.37934. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Pre-training foundational models is computationally expensive as models scale. In this work, we introduce Bi-Phase Training (BPT), a parameter-efficient pre-training method that preserves the high-rank learning dynamics observed in full-parameter optimization with substantially reduced trainable parameters. BPT parameterizes each weight update as a composition of constrained high-rank left/right transformations via trainable diagonal matrices and low-rank adapters whose updates are periodically merged into the frozen base weights, resulting in a compounding expansion of accessible directions during training. Across language-model pre-training from scratch (Qwen2.5 models from 100M to 1.5B parameters and OLMo-2-1B) on FineWeb-Edu (10B and 100B tokens), BPT achieves validation perplexity close to full-parameter baselines while using far fewer trainable parameters; e.g., at the 1.5B scale, BPT uses 66% fewer trainable parameters with only a small validation-perplexity gap. Additionally, a comprehensive evaluation across diverse downstream tasks shows that BPT preserves much of the full-parameterized model's downstream performance.

Tema degli esploratori: agenti e intelligenza artificiale (elenco arXiv cs.LG, nuovi lavori del giorno).
Categorie arXiv: Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2610.37934 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-11-01T02:12:44.088Z dalla pagina https://arxiv.org/list/cs.LG/new (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).