Tag
Agora enables collective, permissionless internet-scale pretraining of large language models using heterogeneous, preemptible consumer GPUs connected via internet, demonstrated by the successful Pluralis-8B training run with 330 nodes.
This article argues that achieving exactly-once execution in distributed systems is impossible, and that agent stacks must be designed to deal with this limitation.
AMD and Meta contributors ported PyTorch Monarch to AMD Instinct GPUs with ROCm, enabling fault-tolerant distributed training at scale. The blog details the engineering work and validation on large clusters.
The article describes the porting of PyTorch Monarch, a distributed training runtime, to AMD GPUs with ROCm, enabling single-controller fault-tolerant training at scale and addressing reliability challenges in large-scale LLM training.
DeadPool introduces a fault-tolerance mechanism for LLM training that enables hot-swapping of failed nodes with spare nodes using zero-overhead in-memory checkpointing, achieving fast recovery without interrupting the job.
ProWAFT is a proactive workload-aware fault-tolerance framework for FPGA-based CNN accelerators that uses partial reconfiguration to selectively apply triple modular redundancy (TMR), minimizing a composite objective over latency, energy, and reliability risk.
Microsoft open sourced pg_durable, a PostgreSQL extension that enables durable execution of long-running SQL functions with automatic checkpointing and fault-tolerant resumption.
This article explains how to add fault tolerance to LangGraph agents using RetryPolicy, TimeoutPolicy, and error handlers, covering retries with backoff, timeouts, and compensation logic for production reliability.
DeepMind introduces Decoupled DiLoCo, a new distributed AI training architecture that enables resilient, low-bandwidth training of large models across globally dispersed data centers by isolating hardware failures.