decentralized-training

Agora enables collective, permissionless internet-scale pretraining of large language models using heterogeneous, preemptible consumer GPUs connected via internet, demonstrated by the successful Pluralis-8B training run with 330 nodes.

Nous Research introduces Psyche, a decentralized infrastructure for training large language models on distributed heterogeneous hardware, using novel optimizers DeMo and DisTrO to dramatically reduce communication overhead.

MERIT introduces conflict-aware splitting and weight merging for decentralized instruction tuning, achieving improved performance without gradient synchronization across partitions.