low-communication

Proposes DiLoCo, a distributed optimization algorithm that enables training large language models across poorly connected devices with 500x less communication while matching fully synchronous performance.

DeepMind introduces Decoupled DiLoCo, a new distributed AI training architecture that enables resilient, low-bandwidth training of large models across globally dispersed data centers by isolating hardware failures.