Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models

# Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Source: [https://arxiv.org/abs/2607.13332](https://arxiv.org/abs/2607.13332)
Authors:[Gil Avraham](https://arxiv.org/search/cs?searchtype=author&query=Avraham,+G),[Violetta Shevchenko](https://arxiv.org/search/cs?searchtype=author&query=Shevchenko,+V),[Hadi Mohaghegh Dolatabadi](https://arxiv.org/search/cs?searchtype=author&query=Dolatabadi,+H+M),[Karol Pajak](https://arxiv.org/search/cs?searchtype=author&query=Pajak,+K),[James Snewin](https://arxiv.org/search/cs?searchtype=author&query=Snewin,+J),[Harry Xi](https://arxiv.org/search/cs?searchtype=author&query=Xi,+H),[Rodney O'Donnell](https://arxiv.org/search/cs?searchtype=author&query=O%27Donnell,+R),[Thalaiyasingam Ajanthan](https://arxiv.org/search/cs?searchtype=author&query=Ajanthan,+T),[Sameera Ramasinghe](https://arxiv.org/search/cs?searchtype=author&query=Ramasinghe,+S),[Chamin Hewa Koneputugodage](https://arxiv.org/search/cs?searchtype=author&query=Koneputugodage,+C+H),[Shamane Siriwardhana](https://arxiv.org/search/cs?searchtype=author&query=Siriwardhana,+S),[Alexander Long](https://arxiv.org/search/cs?searchtype=author&query=Long,+A)

[View PDF](https://arxiv.org/pdf/2607.13332)

> Abstract:Training large language models at the multi\-billion to trillion parameter scale is confined to datacenters, where data\-parallel \(DP\) and model\-parallel \(MP\) techniques presume homogeneous accelerators, high\-speed interconnects, and a single orchestrating entity\. Frontier model development is thereby concentrated among the few groups able to assemble such clusters\. Meanwhile, an enormous pool of compute remains unusable for training: consumer and professional GPUs that are heterogeneous, preemptible, individually owned, and connected only by the internet\. We present Agora, a system that makes efficient use of this compute\. Agora combines bandwidth\-efficient pipeline\-parallel model sharding over internet\-grade links with multi\-party, fault\-tolerant collective operations\. Each participant holds only one stage of the model, and no single party ever possesses the full weights\. We term this setup Protocol Learning: it enables collectively trained, collectively owned models, opening a path to open\-source frontier training with economic sustainability\. This report presents the outcome of a research effort spanning communication\-efficient parallelism, asynchronous optimization, and fault\-tolerant systems design\. It culminates in the first demonstration of its kind: Pluralis\-8B, an open, permissionless pretraining run of an 8\.6B\-parameter model on 500B tokens of FineWeb\-Edu\. The model was trained over 40 days by 330 contributor nodes, predominantly consumer GPUs on internet connections, joining and leaving throughout\. The run sustained ~170k tokens/s and 4\.2 tokens per TFLOP of pooled compute, 63% of the efficiency of a centralized H100 baseline, and converged to within a small margin of a centralized reference run\.

## Submission history

From: Gil Avraham \[[view email](https://arxiv.org/show-email/091cec7d/2607.13332)\] **\[v1\]**Tue, 14 Jul 2026 23:32:18 UTC \(7,028 KB\)