Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Hugging Face Daily Papers Papers
cost-efficient-training consumer-gpus open-source pretraining language-models scaling-laws hardware-accessible

Summary

This paper presents an open-source pretraining recipe that trains 2B-parameter language models on consumer-grade RTX 5090 GPUs for under $7K, achieving performance near larger baselines and deriving cost scaling laws for model training.

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.
Original Article
View Cached Full Text

Cached at: 09/01/26, 12:11 PM

Paper page - Puro-2B: Poor Lab’s Qwen2-1.5B Trained on RTX 5090 within $5090

Source: https://huggingface.co/papers/2608.27370

Abstract

A cost-efficient open-source pretraining recipe trains 2B-parameter models on consumer GPUs for under $7K, yielding performance near larger baselines while deriving cost scaling laws and studying data curricula.

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens withFP8 precisionon consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection,low-precision training,hyperball optimization,curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a PuroCost Scaling Lawthat relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine howpretraining data curriculashape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.

View arXiv pageView PDFProject pageGitHub58Add to collection

Get this paper in your agent:

hf papers read 2608\.27370

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper1

#### thu-pacman/Puro-2B-Base Text Generation• 2B• Updatedabout 5 hours ago • 849 • 8

Datasets citing this paper1

#### thu-pacman/Puro-2B Viewer• Updated4 days ago • 596M • 1.31k • 3

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.27370 in a Space README.md to link it from this page.

Collections including this paper3

Similar Articles

4090 + 5060 Ti + 64GB RAM: 206 t/s on a 35B-A3B, and a 122B at 37 t/s

Reddit r/LocalLLaMA

A user shares benchmark results for running large language models (Qwen 27B-122B) on a dual-GPU setup with RTX 4090 and RTX 5060 Ti, achieving high token generation speeds (e.g., 206 t/s on 35B-A3B, 37-41 t/s on 122B). The post includes setup details and a link to a GitHub repo with scripts and raw data.

RTX Pro 4500 Blackwell - Qwen 3.6 27B?

Reddit r/LocalLLaMA

A developer shares local inference benchmarks and systemd configurations for running the Qwen3.6-27B model on an NVIDIA RTX Pro 4500 Blackwell GPU using llama.cpp. The post requests optimization tips for throughput and explores potential use cases for larger models.