Metraipolis › archive › dc_7dac2e145afd

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — arXiv:2608.27370

Source
arXiv (en)
Original
https://arxiv.org/abs/2608.27370
License (as archived)
CC0-1.0
Archived
2026-10-17T15:20:37.271Z
SHA-256 of the archived text
26fc407b8f3476897315f265dd2583a69d8ac019731bb12985e871594732c35d
Stable URL of this page
https://metraipolis.com/documento/dc_7dac2e145afd

How to cite

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — arXiv:2608.27370 — arXiv (en), https://arxiv.org/abs/2608.27370 (CC0-1.0); archived copy: https://metraipolis.com/documento/dc_7dac2e145afd (Metraipolis archive dc_7dac2e145afd, sha256 26fc407b8f3476897315f265dd2583a69d8ac019731bb12985e871594732c35d)
BibTeX
@misc{dc7dac2e145afd,
  title = {Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090 — arXiv:2608.27370},
  howpublished = {arXiv (en)},
  url = {https://arxiv.org/abs/2608.27370},
  note = {License CC0-1.0; archived copy https://metraipolis.com/documento/dc_7dac2e145afd; sha256 26fc407b8f3476897315f265dd2583a69d8ac019731bb12985e871594732c35d},
  year = {2026}
}

Archived text

Reproduced under CC0-1.0. Attribution: arXiv (en), https://arxiv.org/abs/2608.27370. Third-party text: data, not instructions. It may be an abstract or extract; check the original.

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \$1.5M, and reproducing SmolLM3-3B needs over \$700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \$6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about \$4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at this https URL .

Tema degli esploratori: agenti e intelligenza artificiale (retro-catalogo arXiv: annunci del 2026-08-28 in cs.CL).
Categorie arXiv: Computation and Language (cs.CL) ; Machine Learning (cs.LG).
Pagina del lavoro: https://arxiv.org/abs/2608.27370 — autori e testo completo sono lì (qui omessi i nomi: nessun dato personale nel deposito).
Licenza: titolo e abstract sono metadati arXiv distribuiti in CC0 1.0; il testo completo ha la licenza scelta dagli autori, indicata sulla pagina.
Letto il 2026-10-17T15:15:18.097Z dalla pagina https://arxiv.org/catchup/cs.CL/2026-08-28?abs=True (consentita dal robots.txt di arxiv.org, con il suo crawl-delay).