DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint

# DeadPool: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Source: [https://arxiv.org/html/2607.01646](https://arxiv.org/html/2607.01646)
###### Abstract

State\-of\-the\-art large language model \(LLM\) training takes tens of thousands of graphics processing units \(GPUs\) for months and encounters failures across the software and hardware stack\. Existing fault\-tolerance mechanisms either impose non\-trivial overhead during failure\-free execution or suffer from prolonged recovery latency, particularly under scenarios where a small subset of compute nodes experience permanent failures\. We presentDeadPoolto simultaneously address both optimization objectives\.DeadPoolincorporates a fault\-tolerance mechanism that restores LLM training via hot\-swapping, namely by replacing failed nodes with spare nodes without terminating the complete job\. The hot\-swapping ofDeadPoolis enabled by two ideas: First, it exploits an off\-critical\-path in\-memory checkpointing mechanism for spatial redundancy\. Second, it introduces a communicator reconstruction protocol that replaces failed nodes with spare nodes at runtime\.DeadPoolefficiently overlaps the in\-memory checkpointing with computation, thus introducing zero overhead during error\-free execution\. Upon permanent node failures,DeadPoolcan rebuild memory states with minimal recomputation by leveraging in\-memory checkpoints\. We evaluateDeadPoolacross scales \(up to 512 NVIDIA A100 GPUs\) and LLMs \(up to 65B parameters\), and observe zero checkpoint overhead with hot\-swapping recovery completing in under 40 seconds\. These results show thatDeadPoolsimultaneously achieves both zero\-overhead error\-free execution and extremely low recovery cost\.

## IIntroduction

LLM training runs on thousands to tens of thousands of GPUs, where node failures are no longer rare events but an expected operating condition\[[12](https://arxiv.org/html/2607.01646#bib.bib316),[26](https://arxiv.org/html/2607.01646#bib.bib295)\]\. Published interruption statistics from a 32K\-GPU LLM pre\-training deployment report 678 unexpected interruptions, with GPU HBM memory faults, PCIe device failures, and NCCL watchdog timeouts alone accounting for 49\.9% of all events\[[33](https://arxiv.org/html/2607.01646#bib.bib4)\]\. The traditional checkpoint\-restart solution periodically writes a persistent checkpoint, e\.g\., for every 1,000 steps\[[29](https://arxiv.org/html/2607.01646#bib.bib314),[12](https://arxiv.org/html/2607.01646#bib.bib316)\], to amortize the checkpointing overhead\. Upon failures, the job terminates, and the user needs to start a new job that reloads the latest checkpoint\. This approach pays for the recovery cost twice: It introduces backup overhead during normal execution, and when a failure occurs, it incurs both recovery latency and the recomputation cost for work performed after the latest checkpoint\. On over\-committed supercomputers, such as NERSC Perlmutter or TACC Vista, a double\-bit memory error or an NCCL error can cause the complete job to quit\. The user must re\-enter an already congested queue, incurring additional wait time\.

![Refer to caption](https://arxiv.org/html/2607.01646v1/figures/pareto.png)Figure 1:Conceptual landscape of existing fault\-tolerance strategies in the tradeoff space of error\-free overhead and recovery cost[Figure 1](https://arxiv.org/html/2607.01646#S1.F1)conceptually visualizes existing fault\-tolerant LLM training strategies in a tradeoff space between the error\-free overhead and the recovery cost\. Checkpointing strategies, such as Datastates\-LLM\[[25](https://arxiv.org/html/2607.01646#bib.bib313)\], ByteCheckpoint\[[38](https://arxiv.org/html/2607.01646#bib.bib2)\], TRANSOM\[[43](https://arxiv.org/html/2607.01646#bib.bib5)\], and Universal Checkpointing\[[21](https://arxiv.org/html/2607.01646#bib.bib6)\], substantially improve checkpoint throughput and flexibility\. However, they still recover by restarting training from a persistent checkpoint, which may lead to a non\-trivial cost due to recomputation for the lost training progress\. Prior in\-memory checkpointing strategies\[[27](https://arxiv.org/html/2607.01646#bib.bib9)\]reduce write latency, but assume the in\-memory checkpoint is accessible in the presence of a failure, which is not a valid assumption if the failure is permanent\. On the other hand, elastic fault\-tolerance strategies, such as Bamboo\[[37](https://arxiv.org/html/2607.01646#bib.bib278)\], tolerate membership changes by exploiting bubbles in pipeline\-parallelism\. But these approaches are with degraded training throughput\.

Our goal in this work is to reduce the overhead for both error\-free execution and recovery\. For error\-free execution, the LLM framework needs to persist checkpoints asynchronously to avoid potential contention with the training processes\. To lower recovery cost, the LLM framework needs to checkpoint at every iteration to minimize recomputation in the presence of a failure\. Further, the LLM framework needs to avoid a full restart, which introduces additional overhead for global job termination, checkpoint consistency verification, filesystem I/O, and memory state reconstruction\. These challenging requirements motivate the design ofDeadPool, a fault\-tolerant system for LLM training via hot\-swapping that leverages zero\-overhead asynchronous in\-memory checkpointing in complex parallelism\.

DeadPoolis designed with two unique insights\. The first insight is that the memory states during LLM training are distributed using different strategies in complex parallelism, e\.g\., 3D parallelism\. A typical setting of 3D parallelism, which overlays sharded data\-parallelism with pipeline\-parallelism and tensor\-parallelism, replicates the model shards across data\-parallel peers while distributing the optimizer states \(i\.e\., the first and second moments\) across GPUs within a data\-parallel rank \(i\.e\., a group of GPUs\)\. Upon a permanent node failure, model parameters remain reconstructible from healthy ranks that share the same logical partition, but the optimizer state shards cannot be recovered\. Thus, we design a low\-overhead, across\-nodes in\-memory optimizer state checkpoint strategy that effectively overlaps communication with computation, so thatDeadPoolcan replicate optimizer state shards in every iteration while avoiding exposed checkpoint overhead\. A second insight is thatDeadPoolviews node failure recovery as an online topology\-repair problem rather than a full\-job restart\. Thus, we design a distributed communicator reconstruction protocol to replace failed nodes with spare nodes to avoid full allocation restart\.

We implementDeadPoolon top of PyTorch\[[31](https://arxiv.org/html/2607.01646#bib.bib298)\]and Megatron\-LM\[[29](https://arxiv.org/html/2607.01646#bib.bib314)\], but the design relies only on generic capabilities: sharded model and optimizer states, host\-side spare\-node management, shared recovery coordination, and runtime group reconstruction\.DeadPoolincludes an error\-classification layer that distinguishes transient and permanent failures, allowingDeadPoolto apply topology\-changing recovery for failures that actually invalidate node membership\. We evaluateDeadPoolon two production HPC systems, i\.e\., NERSC Perlmutter and TACC Vista, using up to 512 A100 GPUs and 64 GH200 GPUs, respectively\. We inject node failures into live training jobs with reserved spare nodes, which is a standard methodology in research on fault tolerance and recovery\[[2](https://arxiv.org/html/2607.01646#bib.bib1)\]\. Across both platforms, we observe negligible per\-step checkpointing overhead on the training critical path\. Moreover,DeadPoolconsistently completes node replacement and restores training around 40 s after a failure\.

Our main contributions are as follows:

- •We presentDeadPool, an online hot\-swapping recovery mechanism for large\-scale LLM training\.
- •We design an asynchronous neighbor\-memory replication strategy for optimizer state shards\.
- •We evaluateDeadPoolon up to 512 GPUs and 65B model size, showing zero\-overhead during error\-free execution and a consistent recovery cost within 40 s\.

The zero\-overhead checkpointing ofDeadPooldepends on the GPU cluster configuration, the model size, and the model deployment strategies\. AlthoughDeadPoolis only examined on Perlmutter and Vista, we expectDeadPoolto be effective across many other GPU\-dense supercomputers\. This is because the checkpointing overhead is one order of magnitude lower than computation, as shown in[Figure 6](https://arxiv.org/html/2607.01646#S7.F6), on the current generation machine\. Given the trend of GPU compute capability and interconnect bandwidth increase on the coming NVIDIA Grace\-Blackwell supercomputers,DeadPoolis expected to continue achieving zero\-overhead checkpointing\.

## IIbackground

### II\-ALarge Language Model Training

LLM capabilities often increase with model and dataset size\[[5](https://arxiv.org/html/2607.01646#bib.bib302),[15](https://arxiv.org/html/2607.01646#bib.bib299)\]\. Thus, to overcome training time and memory limitations, researchers have designed various distributed training strategies, including data\-, tensor\-, pipeline\-, and sequence\-parallelism\[[4](https://arxiv.org/html/2607.01646#bib.bib301)\]\.

Data parallelismreplicates a model across processors\. Training then iterates five steps: 1\) I/O, 2\) forward compute, 3\) gradient evaluation, 4\) gradient exchange, and 5\) variable update\.3D\-parallelismpartitions a model across processors by layering data\-parallelism over model\- and pipeline\-parallelism\. Processors are partitioned into groups \(Data Parallel Rank\) and each group hosts a complete copy of the model\. Inside each group, the model is spread across nodes by layers to form the pipeline stages\. Inside each node, layers \(tensors\) are evenly distributed across multiple processors in what is referred to asmodelortensorparallelism\. With the data\-parallel approach, the only communications are the broadcast at the start of training and the gradient exchange in each iteration\. In contrast, 3D\-parallelism requires intra\- and inter\-node communication for both forward and backward computation\.Long sequence support, a fourth dimension in LLM training, enables learning from larger contexts\. Researchers have designed sequence\-parallelism with various partitioning and communication strategies over the query, key, and value matrices\[[20](https://arxiv.org/html/2607.01646#bib.bib305),[22](https://arxiv.org/html/2607.01646#bib.bib304),[23](https://arxiv.org/html/2607.01646#bib.bib303)\]\. Among existing methods, ring attention achieves the lowest memory overhead by using an outer loop of query blocks \(across GPUs\) and inner loops for key\-value blocks \(within a GPU\)\. These distributed LLM training strategies complicate the design of fault\-tolerant LLM training systems, as they are prone to suboptimal recovery decisions and unnecessary overhead in the error\-free phase\.

### II\-BErrors in Systems

In computer systems, hardware transient faults \(i\.e\., soft errors\)\[[30](https://arxiv.org/html/2607.01646#bib.bib306),[3](https://arxiv.org/html/2607.01646#bib.bib307)\]can be triggered by high energy particles from cosmic radiation\[[9](https://arxiv.org/html/2607.01646#bib.bib308)\], shrinking transistors, and low voltage operation\[[10](https://arxiv.org/html/2607.01646#bib.bib310)\]\. These hardware faults manifest as bit\-flips and can range from a single bit to multiple bits and even to hundreds of lines of memory cells\[[3](https://arxiv.org/html/2607.01646#bib.bib307)\]\. We explain the concepts of fault/error/failure\[[36](https://arxiv.org/html/2607.01646#bib.bib311)\]next: A hardwarefaultpropagates in the system; once it reaches the software level, it becomes anerrorthat is visible to the program\. That error further propagates in the program execution, which can lead to three possible outcomes\[[6](https://arxiv.org/html/2607.01646#bib.bib290)\]: 1\) no effect \(i\.e\., correct output\), 2\) silent data corruption, 3\)failuresincluding application crashes and unresponsive systems\. The first two types of outcome are not noticeable to users, although as quantified in prior research\[[45](https://arxiv.org/html/2607.01646#bib.bib288),[14](https://arxiv.org/html/2607.01646#bib.bib287)\], silent data corruption can alter the model parameters produced by training\. For noticeable errors, scientists have designed a comprehensive set of technologies for detecting and tolerating the errors\. As our choices of recovery methods will vary greatly with error type, error characterization and analysis are important for understanding and, more importantly, for detecting and correcting errors efficiently\[[32](https://arxiv.org/html/2607.01646#bib.bib312)\]\. Accurate understanding of errors, and, if possible, accurate prediction of imminent errors and failures, are important if we are to perform timely mitigation actions that avoid data loss and increases system dependability\[[32](https://arxiv.org/html/2607.01646#bib.bib312)\]\.

TABLE I:Comparison of failure recovery approaches\.DeadPoolmaintains continuously recoverable state and enables online recovery via topology repair\.

## IIIRelated Works

Existing fault tolerance solutions can be categorized into two approaches\.*Checkpointing*methods recover model states from recent checkpoints, while*Exclude\-and\-Run*methods exclude faulty devices while allowing healthy workers to continue without interruption\.

Early neural network training approaches employed elastic training methods to handle faults, with frameworks like Horovod\[[35](https://arxiv.org/html/2607.01646#bib.bib297)\]and PyTorch\[[31](https://arxiv.org/html/2607.01646#bib.bib298)\]offering APIs for failure recovery\. However, these methods primarily address data parallelism for models that can fit on a single GPU\. As the size of models grows, so has the likelihood of encountering failures during training\[[17](https://arxiv.org/html/2607.01646#bib.bib270),[42](https://arxiv.org/html/2607.01646#bib.bib294),[26](https://arxiv.org/html/2607.01646#bib.bib295)\]\. EasyScale\[[19](https://arxiv.org/html/2607.01646#bib.bib275)\]and DLRover\[[40](https://arxiv.org/html/2607.01646#bib.bib276)\]take hybrid parallelism into account, then propose resilient training pipelines that can be applied to thousands of GPUs\. Failure rates are even higher when using spot training\[[1](https://arxiv.org/html/2607.01646#bib.bib277),[37](https://arxiv.org/html/2607.01646#bib.bib278)\]in clusters\. Varuna\[[1](https://arxiv.org/html/2607.01646#bib.bib277)\]uses periodic checkpoints to recover from one or more failures, while Bamboo\[[37](https://arxiv.org/html/2607.01646#bib.bib278)\]takes advantage of pipeline parallelism by utilizing bubbles between neighboring stages to create redundancy\. However, both methods introduce additional re\-computing or checkpointing overhead, which can significantly impact training efficiency as failure rates increase\.

Oobleck\[[16](https://arxiv.org/html/2607.01646#bib.bib279)\]achieves a balance between the throughput and fault tolerance by introducing pipeline templates that enable rapid failure recovery and full GPU utilization\. Parcae\[[7](https://arxiv.org/html/2607.01646#bib.bib280)\]leverages a novel metric, liveput, to dynamically reconfigure the parallelization strategy in preemptive environments, maximizing throughput\. Since frequent adjustments to hybrid parallelism can lead to overhead, Slipstream\[[11](https://arxiv.org/html/2607.01646#bib.bib281)\]exploits healthy GPUs to take over computations from failed GPUs in a pipeline stage without changing the strategy\.

Due to the high overhead associated with checkpoint\-based recovery, Checkfreq\[[28](https://arxiv.org/html/2607.01646#bib.bib282)\]introduces a fine\-grained, automated checkpointing mechanism that overlaps computation with saving model states, optimizing checkpoint frequency through runtime profiling\. Gemini\[[41](https://arxiv.org/html/2607.01646#bib.bib265)\]leverages hierarchical memory to create a more efficient checkpointing system, while FastPersist\[[39](https://arxiv.org/html/2607.01646#bib.bib283)\], developed by DeepSpeed, optimizes checkpoint writing from NVMe to SSDs in parallel, significantly improving checkpointing efficiency\. In fault\-tolerant scenarios, Just\-in\-time checkpoint\[[13](https://arxiv.org/html/2607.01646#bib.bib284)\]uses data\-parallel replicas in 3D parallelism and leverages the iterative nature of DNN training to implement a mechanism that allows at most one step of re\-computation in case of failure\. Techniques\[[44](https://arxiv.org/html/2607.01646#bib.bib285),[8](https://arxiv.org/html/2607.01646#bib.bib286),[24](https://arxiv.org/html/2607.01646#bib.bib268)\]that consider model\-specific characteristics \(e\.g\., recommendation models that do not update all parameters simultaneously\) further enhance fault recovery efficiency\. For example, Check\-N\-Run\[[8](https://arxiv.org/html/2607.01646#bib.bib286)\]copies only modified parameters to host memory, and CPR\[[24](https://arxiv.org/html/2607.01646#bib.bib268)\]recovers only the failed GPU using checkpoints while allowing other GPUs to continue working\.

In general, checkpoint\-based methods ensure that the model returns to a globally stable state after a failure, but may introduce additional synchronization overhead that lowers training efficiency, particularly when failure rates are high\. Exclude\-and\-run methods avoid this overhead by dynamically adjusting the parallelization strategy or redistributing computations to functioning workers, but the imbalanced distribution of a mini\-batch across the reduced number of GPUs may lead to straggling processes\. Existing in\-memory approaches such as JIT checkpointing can avoid checkpoint\-interval replay, but their reliance on failure\-time state reconstruction through function overloading can introduce nested recovery logic and unstable behavior under complex failure scenarios\[[18](https://arxiv.org/html/2607.01646#bib.bib318)\]\. As shown in[Table I](https://arxiv.org/html/2607.01646#S2.T1),DeadPooldiffers fundamentally from all three categories: rather than constructing checkpoints at failure time, relying on redundant computation, or reducing the worker set\.DeadPoolmaintains a continuously recoverable state in distributed memory and performs recovery through online node replacement and communication reconfiguration, eliminating full\-job restart and reducing recovery to a bounded topology repair process\.

## IVDisruption Cost Modeling

We model the disruption caused by failures as the sum of three terms:

1. \(1\)The overhead paid during failure\-free execution to maintain recoverability\.
2. \(2\)The latency required to restore execution after a failure\.
3. \(3\)The amount of work that must be replayed after recovery\.

For periodic checkpoint\-restart, if checkpoints are taken everyKKsteps and each step takeststept\_\{\\text\{step\}\}, then the replay cost of a failure isR⋅tstepR\\cdot t\_\{\\text\{step\}\}, whereRRis the number of steps executed since the last checkpoint\. Under a uniform\-failure approximation,𝔼​\[R\]≈\(K−1\)/2\\mathbb\{E\}\[R\]\\approx\(K\-1\)/2\. However, failures in large\-scale training are not necessarily uniform; they may be bursty and system\-dependent\. We therefore also consider an empirical replay distribution derived from failure traces, replacing the uniform approximation with𝔼​\[R\]=∑rr​Pr⁡\(R=r\)\\mathbb\{E\}\[R\]=\\sum\_\{r\}r\\Pr\(R=r\)\.

This yields the expected disruption per failure under checkpoint\-restart:

𝔼​\[Cckpt\]=Coverheadckpt\+𝔼​\[Trestart\]\+tstep⋅𝔼​\[R\]\.\\mathbb\{E\}\[C\_\{\\text\{ckpt\}\}\]=C\_\{\\text\{overhead\}\}^\{\\text\{ckpt\}\}\+\\mathbb\{E\}\[T\_\{\\text\{restart\}\}\]\+t\_\{\\text\{step\}\}\\cdot\\mathbb\{E\}\[R\]\.
We treat replay as wasted work relative to failure\-free execution, even though it is re\-executed as part of resumed training\.

To relate per\-failure disruption to overall training efficiency, we incorporate the system\-level failure rate using the mean time to failure \(MTTF\)\. LetMTTFs​y​s\\mathrm\{MTTF\}\_\{sys\}denote the average time between failures at the system scale, andλ=1/MTTFs​y​s\\lambda=1/\\mathrm\{MTTF\}\_\{sys\}the corresponding failure rate\. The expected disruption per unit time can then be approximated as:

𝔼​\[Crate\]=λ⋅𝔼​\[Cckpt\]\.\\mathbb\{E\}\[C\_\{\\text\{rate\}\}\]=\\lambda\\cdot\\mathbb\{E\}\[C\_\{\\text\{ckpt\}\}\]\.
We further approximate training as alternating between failure\-free execution intervals of lengthMTTFs​y​s\\mathrm\{MTTF\}\_\{sys\}and disruption intervals of expected duration𝔼​\[Cckpt\]\\mathbb\{E\}\[C\_\{\\text\{ckpt\}\}\], yielding the effective training efficiency:

η≈MTTFs​y​sMTTFs​y​s\+𝔼​\[Cckpt\]\.\\eta\\approx\\frac\{\\mathrm\{MTTF\}\_\{sys\}\}\{\\mathrm\{MTTF\}\_\{sys\}\+\\mathbb\{E\}\[C\_\{\\text\{ckpt\}\}\]\}\.
As system scale increases,MTTFs​y​s\\mathrm\{MTTF\}\_\{sys\}typically decreases due to the growing number of failure\-prone components, amplifying the impact of recovery cost\.

FT\-HSDP reports that at O\(100K\) GPUs, failures occur roughly once every 18 minutes, synchronous recovery stalls the entire job for about 10 minutes, and persistent checkpoints are commonly written every 100 steps, with a typical step time of about 20 seconds\. Under the uniform model, this corresponds to an expected replay cost of approximately\(K/2\)⋅tstep≈16\.7\(K/2\)\\cdot t\_\{\\text\{step\}\}\\approx 16\.7minutes, which is comparable to or even exceeds the restart stall itself\. This highlights that, at a large scale, replay and restart lead to low overall training efficiency\.

DeadPoolreduces disruption per failure by eliminating checkpoint\-interval replay and replacing full\-job restart with fast hot\-swapping recovery\. The resulting disruption can be expressed as:

𝔼​\[CDeadPool\]=CoverheadDeadPool\+𝔼​\[Thotswap\]\+tstep⋅𝔼​\[ϕ\],\\mathbb\{E\}\[C\_\{\\text\{\{DeadPool\}\{\}\}\}\]=C\_\{\\text\{overhead\}\}^\{\\text\{\{DeadPool\}\{\}\}\}\+\\mathbb\{E\}\[T\_\{\\text\{hotswap\}\}\]\+t\_\{\\text\{step\}\}\\cdot\\mathbb\{E\}\[\\phi\],
whereϕ∈\[0,1\)\\phi\\in\[0,1\)denotes the fractional progress of the interrupted step\. This formulation captures thatDeadPoolreduces replay to at most the interrupted step and avoids full\-job restart, significantly lowering the disruption per failure\.

## VSystem Design

In this section, we describe howDeadPooloverlaps state protection with training execution and ensures fast, correct recovery without interrupting the global training workflow\.

### V\-AOverview

![Refer to caption](https://arxiv.org/html/2607.01646v1/x1.png)Figure 2:System overview ofDeadPool\. The normal path \(top\) runs a two\-phase offload pipeline overlapped with each training step\. The failure path \(bottom\) detects faults, replaces the failed node, and reconstructs state from in\-memory replicas\.![Refer to caption](https://arxiv.org/html/2607.01646v1/x2.png)Figure 3:Timeline comparison of conventional checkpoint recovery andDeadPool\.DeadPooloverlaps protection with training and reduces disruption after failure\.As shown in Section[IV](https://arxiv.org/html/2607.01646#S4), failure\-induced disruption in large\-scale training is dominated by restart latency and replay cost under checkpoint\-restart\. This motivates a design that eliminates checkpoint\-interval replay and avoids full\-job restart\.DeadPoolachieves this by transforming recovery into an online topology repair problem: instead of restoring execution from a persistent checkpoint,DeadPoolmaintains a continuously recoverable state in distributed memory and replaces failed nodes with healthy ones, allowing training to resume without restarting the job\.

As illustrated in Figure[2](https://arxiv.org/html/2607.01646#S5.F2),DeadPoolcomprises three components that separate normal\-path state protection from failure\-path recovery:

- •Asynchronous Offload Pipeline\(Section[V\-B](https://arxiv.org/html/2607.01646#S5.SS2)\) runs on the normal path once per training iteration\. It copies the full rank state to host memory and replicates optimizer\-state shards to a DP\-ring peer\. Both phases are overlapped with forward, backward, and gradient synchronization, keeping redundancy cost off the critical path\.
- •Error Detection and Isolation\(Section[VI\-B](https://arxiv.org/html/2607.01646#S6.SS2)\) monitors node health via heartbeats and NCCL timeouts\. When a failure is detected, it traps the exception before it propagates to the job manager, holding surviving ranks at a barrier while a replacement node is provisioned\.
- •Recovery Policy\(Section[V\-C](https://arxiv.org/html/2607.01646#S5.SS3)\) executes on the failure path\. It attaches a spare node, rebuilds communication groups, and reconstructs the missing state from in\-memory replicas\. Training resumes from the last completed step with at most one step of re\-execution\.

During normal execution, only the offload pipeline is active, and its cost is hidden within each training iteration\. When a failure occurs, the error detection module triggers the recovery policy, which leverages the remote replicas to restore training without a full job restart\.

Figure[3](https://arxiv.org/html/2607.01646#S5.F3)compares conventional checkpoint\-restart withDeadPoolfrom a timeline perspective\. LetT1T\_\{1\}denote the latest recoverable state before failure\.T2T\_\{2\}is the failure time\.T3T\_\{3\}is the point at which training resumes, andT4T\_\{4\}is the point at which execution catches up to a failure\-free trajectory\. In conventional checkpoint\-restart, periodic backup operations are placed on the training critical path, and a failure introduces two sources of disruption: explicit recovery latency and replay of lost work betweenT1T\_\{1\}andT2T\_\{2\}\. As a result, training cannot resume until the system reloads state and reconstructs execution, and additional time is spent re\-executing previously completed iterations before reachingT4T\_\{4\}\.

In contrast,DeadPooldecouples state protection from the critical path by continuously maintaining recoverable state in memory and overlapping redundancy operations with normal execution\. Upon failure, the system performs online node replacement and reconstructs only the missing state, allowing training to resume nearT3T\_\{3\}with at most one step of re\-execution\. This design significantly reduces both replay cost and end\-to\-end recovery latency, effectively shrinking the disruption window between failure and full recovery\.

### V\-BAsynchronous Offload Pipeline

The offload pipeline runs once per training iteration, overlapped with the forward pass, backward pass, and gradient synchronization of the same iteration as illustrated in[Figure 4](https://arxiv.org/html/2607.01646#S5.F4)\. It proceeds in two phases whose payloads differ by design\.

##### Device\-to\-host \(D2H\)

The full rank state, which comprises the optimizer\-state shard, the local model\-parameter shard, and a metadata envelope \(step index, parallelism configuration, RNG states\), is packed into a contiguous buffer and copied asynchronously to pinned host memory on a dedicated CUDA stream\. Retaining the complete rank state on the host enables fast local recovery \(e\.g\., after a transient GPU error\) without any network transfer\.

##### Host\-to\-host \(H2H\)

![Refer to caption](https://arxiv.org/html/2607.01646v1/x3.png)Figure 4:Illustration of the asynchronous offload pipeline across pipeline\-parallel stages\. Computation \(forward and backward passes\) is interleaved with gradient synchronization, while device\-to\-host \(D2H\) offload and host\-to\-host \(H2H\) replication progress concurrently on separate execution paths\. The timeline highlights the overlap between compute and communication throughout an iteration\.Only the optimizer\-state shard and the metadata envelope are transmitted to the peer node in a ring topology, which is the rank sharing the same tensor\-parallel and pipeline\-parallel positions but with a different data\-parallel index\. Model parameters are excluded: upon node failure, the replacement rank can reconstruct its parameter shard from healthy ranks at the same TP/PP coordinate via a targeted collective, so replicating parameters to a peer in advance would consume network bandwidth without reducing recovery latency\. A background worker thread waits on the D2H completion event and then sends the optimizer payload to the ring peer via chunked MPI over the DP\-local communicator\.

This payload is asymmetric, full rank state for D2H versus optimizer\-only for H2H, because of the bandwidth difference of the system\. Local PCIe transfers are fast, whereas inter\-node network bandwidth is more constrained\. By transmitting only the optimizer shard over the network, the H2H phase reduces its transfer volume, increasing the likelihood that both phases complete within a single training iteration\.

### V\-CRecovery Policy and Correctness Guarantees

DeadPoolhandles failures by determining whether the affected node can continue participating in distributed execution\.

Failure Model\.We consider two classes of failures\. \(1\)*Transient failures*, where the affected node remains operational and its state remains valid\. \(2\)*Permanent failures*, where a node or its state is permanently lost and can no longer contribute its assigned shards\. In the latter case, the system replaces the failed node with a spare node and reconstructs the missing state\.

Failures that may compromise training semantics, such as software bugs, numerical errors, or silent data corruption, are treated as outside the scope ofDeadPooland handled by an external fallback mechanism\.

System Invariants\.To ensure correctness under failures,DeadPoolmaintains the following invariants:

- •I1 \(Topology Consistency\)\.All involved nodes agree on a single communication topology and shard placement\.
- •I2 \(Shard Completeness\)\.For every logical shard, there exists exactly one valid owner among the involved nodes\.
- •I3 \(Optimizer Availability\)\.The optimizer state for every shard is available from in\-memory replicas\.
- •I4 \(Step Atomicity\)\.Training progresses in discrete steps; partial progress from a failed step is not committed\.

During transient failures, the LLM training system preserves all invariants\. These failures can be recovered in place without modifying involved nodes or the communication topology\.

Recovery Procedure\.For permanent failures,DeadPoolrecovers in three stages\. First, the system selects a spare node and updates the set of involved nodes\. This change invalidates the pre\-failure communication topology\. Second, all involved nodes are quiesced at a safe boundary, and the system re\-establishes a consistent communication topology and shard placement across all nodes\. This step ensures that Invariants I1 and I2 hold before execution resumes\. Third, the system reconstructs the missing state on the spare node\. Model parameters are reconstructed from healthy nodes that share the same partition, while optimizer state is restored from in\-memory replicas\. This step ensures that Invariant I3 holds\. Finally, execution resumes from the last completed step, ensuring that Invariant I4 is preserved\.

##### Tolerating Multi\-Node Failures via Multi\-Peer Replication\.

By default,DeadPoolreplicates each optimizer\-state shard to one DP\-ring neighbor, tolerating any single\-node failure\. To guard against concurrent multi\-node failures,DeadPoolcan be configured to replicate each shard tokkdistinct DP peers, with the additional H2H cost scaling linearly inkkbut remaining overlappable with computation for a sufficiently smallkk\.

A shard is irrecoverably lost only if its owner and allkkreplica holders fail within the same step\. We quantify this risk using published failure statistics from Meta’s 100K\-GPU deployment, which reports one interruption roughly every 18 minutes\[[34](https://arxiv.org/html/2607.01646#bib.bib315)\]\. With a step time of approximately 20 s, this corresponds to a per\-node per\-step failure probability on the order ofp≈10−6p\\approx 10^\{\-6\}, which is an aggressive estimate that we adopt as an upper bound for our analysis\.

Independent failures\. When replica placement is topology\-unaware, we assume node failures are independent\. The probability that a specific shard loses allk\+1k\{\+\}1copies in a single step ispk\+1p^\{k\+1\}\. Applying a union bound overDDdata\-parallel shards, the per\-step probability of any shard becoming unrecoverable is:

Ploss≤D⋅pk\+1\.P\_\{\\text\{loss\}\}\\leq D\\cdot p^\{k\+1\}\.\(1\)WithD=128D=128andk=1k=1, this yieldsPloss≤1\.28×10−10P\_\{\\text\{loss\}\}\\leq 1\.28\\times 10^\{\-10\}per step, or a cumulative probability of1\.28×10−51\.28\\times 10^\{\-5\}over a10510^\{5\}\-step run—fewer than one event per 100 full training runs\. Settingk=2k=2reduces the cumulative probability to1\.28×10−111\.28\\times 10^\{\-11\}, which is effectively zero\.

Correlated failures with topology\-aware placement\. The independent\-failure analysis breaks down when a shared infrastructure component—such as a top\-of\-rack \(ToR\) switch, a network cable, or a power distribution unit—fails and takes down multiple co\-located nodes simultaneously\. If a shard’s owner and its replica reside under the same switch, a single switch failure can destroy both copies\.

To mitigate this,DeadPoolcan leverage cluster topology information to place thekkreplicas on nodes underkkdistinct switches or racks, ensuring that a single infrastructure event affects at most one copy of any given shard\. Letqqdenote the per\-domain per\-step failure probability \(e\.g\., a ToR switch failure rate\)\. With replicas distributed across independent failure domains, the per\-shard loss probability becomesqk\+1q^\{k\+1\}\. Even under a relatively high domain failure rate ofq=10−4q=10^\{\-4\},k=1k=1yields a cumulative risk ofD⋅q2×105=0\.128D\\cdot q^\{2\}\\times 10^\{5\}=0\.128over10510^\{5\}steps—non\-negligible for safety\-critical runs\. Settingk=2k=2reduces this toD⋅q3×105=1\.28×10−5D\\cdot q^\{3\}\\times 10^\{5\}=1\.28\\times 10^\{\-5\}, rendering correlated shard loss negligible\.

In practice,k=1k=1with topology\-aware placement suffices for the vast majority of production scenarios, whilek=2k=2provides an additional safety margin for deployments with frequent infrastructure\-level failures\.

## VIImplementation

We implementDeadPoolas a hybrid design that combines a data\-plane extension to Megatron\-LM with an external control plane\. The data plane integrates directly with Megatron\-LM’s distributed execution model, including checkpoint metadata, sharded optimizer abstractions, and 3D\-parallel process\-group organization, leaving the failure\-free execution path unchanged when the mechanism is disabled\.

The control plane is implemented as a lightweight standalone service that coordinates failure detection, node replacement, and recovery orchestration across all involved nodes\. It provides a persistent coordination layer independent of the training processes, enabling recovery to proceed even when a subset of training nodes fail\.

The implementation spans four key components:

1. \(1\)the program entry path for initializing failure\-aware execution and registering with the control plane,
2. \(2\)the training\-step scheduler for enforcing safe quiescence and recovery boundaries,
3. \(3\)the checkpoint and optimizer wrapper for maintaining in\-memory recoverable state, and
4. \(4\)an asynchronous MPI\-based transport service for state replication and recovery\.

We implementDeadPoolas an opt\-in extension inside Megatron\-LM\. The in\-process design has two advantages over an external wrapper: it reuses Megatron’s distributed\-optimizer serialization and 3D\-parallel process\-group layout directly, avoiding a redundant checkpoint stack; and when the mechanism is disabled, no interception layer is present in the training loop, so the failure\-free path incurs zero overhead\.

### VI\-AIn\-Memory State Protection

![Refer to caption](https://arxiv.org/html/2607.01646v1/x4.png)Figure 5:Ping\-pong double buffering across iterations\. Two local buffers alternate between committed and staging states, while peer buffers alternate between committed and receiving\. Computation overlaps with data movement \(D2H and H2H\), and buffer roles switch every iteration\.This component maintains a consistent, immediately recoverable snapshot of each rank’s training state in host memory at all times\. As shown in[Figure 5](https://arxiv.org/html/2607.01646#S6.F5)offload manager uses a ping\-pong scheme for the local snapshot buffer: two host\-side buffers alternate roles across iterations, one holding the committed snapshot from iterationk−1k\{\-\}1and the other serving as the staging target for iterationkk\. If a failure interrupts the current offload or the subsequent peer transfer, the in\-progress buffer may be incomplete, but the previous buffer remains intact and immediately usable for recovery\. This design guarantees that a valid recovery point is always present in host memory without requiring the offload to be atomic\. The buffer received from the ring predecessor is maintained separately and follows the same alternation logic\.

The training loop exposes a single synchronization point: a wait immediately before the optimizer step that guarantees the peer\-resident replica is fully committed before the optimizer mutates the protected state\. In the common case, both D2H and H2H transfers complete within the iteration’s compute window, so this wait observes an already\-finished offload and adds no exposed stall\.

TABLE II:Error classification and recovery actions inDeadPool\.
### VI\-BError Classification

Following the failure model defined in Section[V\-C](https://arxiv.org/html/2607.01646#S5.SS3),DeadPoolimplements a classification layer that maps observed failures to recovery actions as illustrated in[Table II](https://arxiv.org/html/2607.01646#S6.T2)\. The classification is driven by runtime signals that indicate loss of node participation or loss of reliable state\. Device\-level failures \(e\.g\., GPU memory errors\), system\-level failures \(e\.g\., host crashes or reboots\), and persistent communication failures are treated as node\-replacement events, as they prevent the node from safely contributing its assigned shards\. Transient communication failures are initially handled as locally recoverable when the node remains responsive, but are elevated if they lead to sustained communication failure or node unavailability\. When a failure is classified as locally recoverable,DeadPoolperforms a local reset by clearing the node’s state and reinitializing it as a replacement node\. From the system perspective, this is equivalent to a special case of node replacement where the failed node re\-enters the system after reset\. For clarity, we treat both cases uniformly as hot\-swapping, and focus the following discussion on node replacement\.

To evaluate the system under controlled yet realistic failure conditions, we inject errors during live training runs using timer\-based triggers\. At predefined execution points, the timer activates different fault types, including fail\-stop events \(via process termination\), NCCL communication errors and storage\-related errors\. This unified injection mechanism enables reproducible evaluation of both abrupt node failures and runtime error conditions within the same framework\.

### VI\-CControl Plane

This component provides a fault\-isolated coordination substrate for involved nodes, maintaining globally consistent recovery metadata throughout failure handling and node replacement\.DeadPoolimplements the control plane as a standalone TCPStore\-based service running as an independent process\. It maintains a key\-value store to recover metadata, including failure notifications, recovery epochs, node assignments, and synchronization signals\. Because the control plane is decoupled from training processes, it remains available when a subset of nodes fails, allowing surviving and replacement nodes to continue coordination\.

The control plane supports dynamic node participation, allowing nodes to join during recovery without requiring global re\-initialization\. Each involved node publishes a recovery descriptor that encodes its logical role, including shard ownership and parallel partition placement\. These descriptors are used to derive deterministic mappings between source and replacement nodes based on logical shard identity\.

### VI\-DRecovery Scheduling and Topology Reconfiguration

This component orchestrates the recovery process across involved nodes, ensuring consistent execution progression during node replacement and topology reconstruction\. When a failure requires node replacement, the runtime initiates a recovery epoch and coordinates all involved nodes through the control plane, satisfying I1 mentioned in[V\-C](https://arxiv.org/html/2607.01646#S5.SS3)\. Each node captures its local recovery state in memory and synchronizes before recovery\. Nodes are paused at a step boundary to establish a consistent execution point\.

A spare node is activated and incorporated into the set of involved nodes\. The runtime reconstructs communication groups and shard placement based on the updated topology, and all nodes resume execution under the same communication configuration\. Recovery epochs are serialized, so that a new recovery begins only after the current one completes\.

### VI\-EState Restoration and Transport

After topology reconfiguration,DeadPoolreconstructs the state of failed shards on the spare node\. Model parameters are restored by transferring shards from source nodes that share the same logical partition, while optimizer state is restored directly from in\-memory replicas maintained during training, avoiding access to persistent checkpoints\.

To ensure correct reconstruction under dynamic rank reassignment, source and replacement nodes are matched using logical shard identity derived from recovery descriptors rather than post\-failure rank indices\. This guarantees that each shard is reconstructed from the correct source regardless of how ranks are reassigned, corresponding to I2 and I3\. State transfer is implemented by an asynchronous MPI\-based transport layer, which allows communication to overlap with recovery preparation on the receiving node and reduces recovery latency\.

### VI\-FTraining Resume

Once model parameters and optimizer state are restored, the spare node assumes the role of the failed node and joins the set of involved nodes, ensuring the I4 completeness\. Training resumes under the reconstructed communication topology from the last completed step, and any partial work from the interrupted step is discarded to preserve consistency\. If a subsequent failure occurs,DeadPoolinitiates a new recovery epoch following the same procedure\.

## VIIExperimental Evaluation

We evaluateDeadPoolalong three dimensions\. First, we quantify its checkpoint overhead in the error\-free case\. Second, we measure the recovery cost of conventional checkpoint/restart under failures\. Third, using an error model, we compare the end\-to\-end time\-to\-solution ofDeadPoolagainst checkpoint\-based recovery under realistic failure scenarios\. Across the configurations we study,DeadPoolconsistently reduces total wall\-clock time\.

### VII\-AExperimental Setup

#### VII\-A1Platforms

We evaluateDeadPoolon two production supercomputers: Perlmutter at NERSC and Vista at TACC\.

Perlmutter is an HPE Cray EX system\. We use its GPU nodes, with each containing one 64\-core AMD EPYC 7763 CPU, four NVIDIA A100 GPUs, and 256 GB of DDR4 host memory\. The four GPUs are interconnected via third\-generation NVLink\. Each node is equipped with four HPE Slingshot 11 NICs, each providing 200 Gb/s, for an aggregated bandwidth of 800 Gb/s per node\. GPU–CPU and NIC–CPU connectivity are both provided through PCIe 4\.0\. The Perlmutter interconnect uses a three\-hop dragonfly topology\.

Vista is an NVIDIA Grace\-based supercomputer at TACC\. We use Grace–Hopper \(GH\) nodes, each containing one 72\-core NVIDIA Grace CPU and one NVIDIA H200 GPU with 96 GB of HBM3 memory, connected through the Grace–Hopper superchip interconnect\. Each node also provides 116 GB of LPDDR host memory and connects to the system fabric via NVIDIA NDR InfiniBand at 400 Gb/s\. Vista employs a fat\-tree topology\.

#### VII\-A2Training Configuration

We train GPT\-style transformer models from 0\.6 B to 65 B parameters, varying the memory footprint by two orders of magnitude\.

Unless otherwise noted, all experiments use full\-precision \(FP32\) training: model parameters, main gradients, and Adam optimizer states \(first and second moments, together with the FP32 master copy\) are stored intorch\.float32, with BF16 and FP16 disabled\. This setting maximizes the per\-rank state volume and therefore represents a demanding configuration forDeadPool’s offload path\.

We use Megatron\-LM with 3D parallelism and ZeRO\-2 distributed optimizer\. Unless otherwise noted, all experiments use the 1F1B pipeline schedule\. We evaluateDeadPoolacross multiple parallel configurations from 8 to 512 GPUs\.

### VII\-BDeadPool’s performance

We design our evaluation to answer two key questions aboutDeadPool:

1. \(1\)Does per\-iteration in\-memory checkpointing introduce any runtime overhead under scaling?
2. \(2\)Does hot\-swapping recovery remain efficient as system scale increases?

To this end, we conduct two sets of experiments\. Firstly, we evaluate the runtime overhead ofDeadPoolunder weak scaling, examining whether enabling per\-iteration checkpointing affects training step time as model size and cluster size grow\. Secondly, we measure the recovery latency of hot\-swapping across different cluster scales to characterize its scalability and identify dominant cost components\.

#### VII\-B1Per\-iteration Checkpoint Overhead under Scaling

We first evaluate whetherDeadPoolintroduces any checkpoint overhead in the error\-free case\.[Figure 6](https://arxiv.org/html/2607.01646#S7.F6)shows weak scaling for the 2\.3B GPT model under a fixed parallel topology, with TP=4 and PP=4 held constant\. As the system scales from 32 to 256 GPUs, we increase the global batch size proportionally to the data\-parallel degree, keeping the per\-rank workload unchanged\. To report per\-iteration measurement, we discard the first 10 iterations as warm\-up, remove the slowest five percent of the remaining iterations, and average the rest\. Across all scales,DeadPoolclosely tracks the baseline\. On eight nodes, the baseline averages 2442\.13 ms whileDeadPoolaverages 2470\.15 ms\. On 16 nodes, the two are nearly identical \(2560\.35 ms vs\. 2561\.75 ms\)\. On 32 and 64 nodes,DeadPoolis marginally faster \(2696\.62 ms vs\. 2701\.87 ms and 3122\.86 ms vs\. 3134\.81 ms, respectively\)\. The differences are small and non\-monotonic, with no trend of increasing overhead as the system grows, indicating thatDeadPoolintroduces negligible runtime overhead during error\-free execution\.

![Refer to caption](https://arxiv.org/html/2607.01646v1/x5.png)Figure 6:Same\-topology weak scaling results for the 2\.3B model with tensor parallelism \(TP=4\) and pipeline parallelism \(PP=4\)\. The x\-axis shows the number of nodes \(8, 16, 32, and 64\), and the y\-axis reports the per\-iteration training time in milliseconds\. The blue bars \(baseline\) correspond to the original training configuration withoutDeadPool, while the orange dotted bars \(DeadPool\) show performance withDeadPoolenabled\.![Refer to caption](https://arxiv.org/html/2607.01646v1/x6.png)Figure 7:Per\-iteration training time across six 3D parallel topologies on 4 nodes \(16 GPUs\) under weak\-scaling settings\. The x\-axis lists the training topology, and the y\-axis shows elapsed time per iteration in milliseconds\. Colors denote model size\. For each model size within a topology, the solid bar represents the baseline configuration and the hatched bar representsDeadPool\.In addition to the fixed\-topology weak\-scaling results above, we also evaluateDeadPoolacross a range of 3D parallel layouts on 16 GPUs and 8 GPUs\.[Figure 7](https://arxiv.org/html/2607.01646#S7.F7)compares the baseline andDeadPoolacross six topologies and four model sizes\. Across all configurations,DeadPoolclosely tracks the baseline\. Although a few cases show small positive or negative deviations, these differences are non\-systematic and are consistent with normal run\-to\-run and node\-allocation variability, rather than a persistent overhead introduced by per\-iteration checkpointing\. Together with the same\-topology weak\-scaling results, these results show thatDeadPoolintroduces no observable checkpoint overhead both as system scale increases and across diverse parallel topologies\.

![Refer to caption](https://arxiv.org/html/2607.01646v1/x7.png)Figure 8:Per\-iteration training time for 7B, 21B, and 65B models on Vista\.To verify thatDeadPool’s zero\-overhead property generalizes across hardware platforms, we further evaluate it with larger models on Vista’s Grace–Hopper nodes using 64 nodes\.[Figure 8](https://arxiv.org/html/2607.01646#S7.F8)compares per\-iteration training time for 7B, 21B, and 65B models\. Across all three scales,DeadPoolclosely tracks the baseline: at 7B,DeadPoolaverages 1854\.5 ms versus the baseline’s 1912\.8 ms; at 21B, 3311\.3 ms versus 3479\.4 ms; and at 65B, the two are effectively identical \(7316\.2 ms vs\. 7312\.0 ms\)\. The small differences in either direction are system noise\. These results confirm thatDeadPool’s zero\-overhead property holds for models an order of magnitude larger than those tested on Perlmutter, and generalizes across GPU architectures \(A100 vs\. H200\) and interconnect fabrics \(Slingshot vs\. InfiniBand\)\.

#### VII\-B2DeadPoolCommunication Breakdown

![Refer to caption](https://arxiv.org/html/2607.01646v1/x8.png)Figure 9:Per\-iteration communication breakdown ofDeadPool\. \(Left\) Same\-topology weak scaling with TP=4 and PP=4, where the x\-axis shows the node count and the y\-axis shows elapsed time per iteration in milliseconds\. \(Right\) 2 nodes runs across three 3D parallel topologies, with the y\-axis showing elapsed time per iteration in milliseconds\. Bars are decomposed into forward/backward computation \(F/B\), device\-to\-host transfer \(D2H\), and host\-to\-host transfer \(H2H\)\.We first examine the communication breakdown under the same\-topology weak\-scaling setup used in the main text\.[Figure 9](https://arxiv.org/html/2607.01646#S7.F9)\(Left\) shows that as the system scales from 32 to 256 GPUs with TP=4 and PP=4 fixed, forward/backward computation remains the dominant component of iteration time, while both D2H and H2H offload costs decrease with scale\. This behavior is expected because increasing the data\-parallel degree reduces the amount of optimizer state that must be offloaded per rank\. Consequently, the offload path occupies only a small fraction of the available compute window, making it straightforward to overlap communication with ongoing computation\. This explains whyDeadPoolintroduces negligible checkpoint overhead in weak\-scaling\.

We next consider a more communication\-intensive setting: 2\-node \(8\-GPU\) runs across several 3D parallel topologies\. In this setting, each rank carries a larger checkpoint payload than in the larger\-scale weak\-scaling experiments, making overlap more challenging\.[Figure 9](https://arxiv.org/html/2607.01646#S7.F9)\(Right\) shows that although D2H and H2H costs are higher in absolute terms, they still remain well below the 5\.6–6\.4 s forward/backward compute window across all tested topologies\. Thus, even in this more demanding small\-scale case, the offload path still has sufficient room to be overlapped with computation\. Together, these results show thatDeadPoolbenefits naturally from weak scaling, but remains effective even when the per\-rank offload volume is higher\.

![Refer to caption](https://arxiv.org/html/2607.01646v1/figures/model_size_recovery_gpu.png)Figure 10:Breakdown of recovery latency across model scales with measured GPU memory utilization on 128 GPUs\.![Refer to caption](https://arxiv.org/html/2607.01646v1/figures/gpu_scaling_recovery_time.png)Figure 11:Breakdown of recovery latency and end\-to\-end recovery time across cluster scales for GPT\-7B model under fixed high per\-GPU memory utilization\.
#### VII\-B3Hot\-swapping Recovery Latency across Scales

We evaluate the end\-to\-end recovery latency ofDeadPoolacross both model and system scales\. Our goal is to understand how hot\-swapping latency depends on model size, cluster size, and memory pressure\.

We first vary model sizes on a fixed 128\-GPU cluster with a fixed parallel layout\. As shown in[Figure 10](https://arxiv.org/html/2607.01646#S7.F10), recovery latency remains stable across models ranging from 0\.6B to 50B parameters, with per\-GPU memory utilization increasing from 12\.1% to over 96\.6%\. This demonstrates that recovery time is independent of model scale and memory footprint\.

We then scale the cluster size in a weak\-scaling setup, where per\-GPU workload and memory usage are held constant\. As shown in[Figure 11](https://arxiv.org/html/2607.01646#S7.F11), recovery latency increases from 17\.1 s to 29\.8 s as the system scales from 32 to 512 GPUs\. These results show thatDeadPool’s recovery latency is independent of model size and memory pressure, and grows only with system\-level communication reconfiguration This increase closely follows the growth of topology reconfiguration time, while parameter and optimizer\-state restoration remain nearly constant across scales\. A breakdown of the recovery path shows that communicator reconstruction \(e\.g\., NCCL rebuild\) dominates recovery latency, whereas state restoration contributes only a small and stable portion\.

In comparison, restarting from checkpoint in our experiments takes on average 150 s, including distributed initialization, checkpoint loading, and process synchronization, whileDeadPoolcompletes recovery via hot\-swapping in under 40 s\. This corresponds to a reduction of over 3\.7×\\timesin recovery latency\. Notably, communicator reconstruction corresponds to a subset of the initialization steps required in checkpoint\-restart, where restarting the training job requires fulltorch\.distributedinitialization and additional overheads\. In contrast,DeadPoollimits recovery to communication reconfiguration without full re\-initialization, resulting in a more lightweight recovery path\. As system scale increases and failures become more frequent, the relative advantage ofDeadPoolis expected to further increase\.

### VII\-CReal\-World Benefit Quantification

We quantify the impact ofDeadPoolunder realistic failure conditions by combining measured recovery costs from a real training run with failure events generated from empirically observed failure rates\. We use the failure rate reported in Meta’s large\-scale training study\[[33](https://arxiv.org/html/2607.01646#bib.bib4)\]to model the occurrence of failures, and apply Monte Carlo sampling to generate failure times for our experiment\.

We consider a 2\.3B training job on 64 nodes for 100K steps, with persistent checkpoints taken every 5K steps \(8\.9 s per checkpoint\), corresponding to a total runtime of 2\.41 days \(2\.08 s per step\)\. Based on the sampled failure process, two failure events occur at 18 and 2366 steps after the most recent checkpoint, respectively\.

To resume to the original training progress,DeadPoolrequires 32\.6 s and 29\.8 s for the two failures, respectively\. In contrast, checkpoint\-restart requires 150 s per recovery in addition to replaying lost work since the last checkpoint\. Overall,DeadPoolreduces the failure\-induced recovery time to5\.3%of checkpoint\-restart, corresponding to a18\.8×\\timesreduction in time to resume training progress\. As system scale increases and failures become more frequent, the relative advantage ofDeadPoolbecomes more pronounced, since it avoids both repeated checkpoint loading and full\-job restarts\.

## VIIIConclusion and Future Work

In this paper, we presentDeadPool, a fault\-tolerant training system for large\-scale LLMs that enables recovery via online hot\-swapping rather than full job restarts\. By continuously replicating optimizer\-state shards in memory and reconstructing distributed communication with a spare node,DeadPooltransforms failure recovery into an online topology repair problem\. We show thatDeadPoolenables per\-iteration checkpointing without introducing overhead on the training critical path, making fine\-grained state protection practical at scale\. Our evaluation further demonstrates that hot\-swapping achieves efficient and scalable recovery: recovery latency grows only gradually with cluster size and is dominated by communicator reconstruction, while the cost of state restoration remains nearly constant\. Together, these results indicate thatDeadPooldecouples recovery performance from model size and avoids the replay and restart costs inherent in checkpoint\-based approaches\. This makesDeadPoola practical and scalable solution for fault\-tolerant LLM training in large\-scale systems\.

Future work includes improving control plane fault tolerance, extendingDeadPoolto additional training frameworks, and validating its performance at larger scales\.

## References

- \[1\]S\. Athlur, N\. Saran, M\. Sivathanu, R\. Ramjee, and N\. Kwatra\(2022\)Varuna: scalable, low\-cost training of massive deep learning models\.InProceedings of the Seventeenth European Conference on Computer Systems,pp\. 472–487\.External Links:[Link](https://api.semanticscholar.org/CorpusID:243847496)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[2\]L\. Bautista\-Gomez, S\. Tsuboi, D\. Komatitsch, F\. Cappello, N\. Maruyama, and S\. Matsuoka\(2011\)FTI: high performance fault tolerance interface for hybrid systems\.InProceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis,SC ’11,New York, NY, USA\.External Links:ISBN 9781450307710,[Link](https://doi.org/10.1145/2063384.2063427),[Document](https://dx.doi.org/10.1145/2063384.2063427)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p5.1)\.
- \[3\]M\. V\. Beigi, Y\. Cao, S\. Gurumurthi, C\. Recchia, A\. Walton, and V\. Sridharan\(2023\)A systematic study of ddr4 dram faults in the field\.In2023 IEEE International Symposium on High\-Performance Computer Architecture \(HPCA\),pp\. 991–1002\.External Links:[Document](https://dx.doi.org/10.1109/HPCA56546.2023.10071066)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1)\.
- \[4\]T\. Ben\-Nun and T\. Hoefler\(2019\)Demystifying parallel and distributed deep learning: an in\-depth concurrency analysis\.ACM Computing Surveys52\(4\),pp\. 1–43\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1145/3320060)Cited by:[§II\-A](https://arxiv.org/html/2607.01646#S2.SS1.p1.1)\.
- \[5\]R\. Bommasani, D\. A\. Hudson, E\. Adeli, R\. Altman,et al\.\(2021\)On the opportunities and risks of foundation models\.Preprint arXiv:2108\.07258\.External Links:[Link](https://arxiv.org/abs/2108.07258)Cited by:[§II\-A](https://arxiv.org/html/2607.01646#S2.SS1.p1.1)\.
- \[6\]D\. A\. G\. De Oliveira, L\. L\. Pilla, M\. Hanzich, V\. Fratin, F\. Fernandes, C\. Lunardi, J\. M\. Cela, P\. O\. A\. Navaux, L\. Carro, and P\. Rech\(2017\)Radiation\-induced error criticality in modern hpc parallel accelerators\.In2017 IEEE International Symposium on High Performance Computer Architecture \(HPCA\),pp\. 577–588\.External Links:[Document](https://dx.doi.org/10.1109/HPCA.2017.41)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1.2)\.
- \[7\]J\. Duan, Z\. Song, X\. Miao, X\. Xi, D\. Lin, H\. Xu, M\. Zhang, and Z\. Jia\(2024\)Parcae: proactive,Liveput\-Optimized DNN Training on Preemptible Instances\.In21st USENIX Symposium on Networked Systems Design and Implementation \(NSDI 24\),pp\. 1121–1139\.External Links:[Link](https://arxiv.org/abs/2403.14097)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p3.1)\.
- \[8\]A\. Eisenman, K\. K\. Matam, S\. Ingram, D\. Mudigere, R\. Krishnamoorthi, K\. Nair, M\. Smelyanskiy, and M\. Annavaram\(2022\)Check\-N\-Run: a checkpointing system for training deep learning recommendation models\.In19th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 22\),pp\. 929–943\.External Links:[Link](https://www.usenix.org/conference/nsdi22/presentation/eisenman)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[9\]V\. Fratin, D\. Oliveira, C\. Lunardi, F\. Santos, G\. Rodrigues, and P\. Rech\(2018\)Code\-dependent and architecture\-dependent reliability behaviors\.In2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks \(DSN\),Vol\.,pp\. 13–26\.External Links:[Document](https://dx.doi.org/10.1109/DSN.2018.00015)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1)\.
- \[10\]S\. Ganapathy, J\. Kalamatianos, B\. M\. Beckmann, S\. Raasch, and L\. G\. Szafaryn\(2019\)Killi: runtime fault classification to deploy low voltage caches without MBIST\.In25th IEEE International Symposium on High Performance Computer Architecture, HPCA 2019, Washington, DC, USA, February 16\-20, 2019,pp\. 304–316\.External Links:[Link](https://doi.org/10.1109/HPCA.2019.00046),[Document](https://dx.doi.org/10.1109/HPCA.2019.00046)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1)\.
- \[11\]S\. Gandhi, M\. Zhao, A\. Skiadopoulos, and C\. Kozyrakis\(2024\)SlipStream: adapting pipelines for distributed training of large dnns amid failures\.arXiv preprint arXiv:2405\.14009\.External Links:[Document](https://dx.doi.org/10.48550/arXiv.2405.14009)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p3.1)\.
- \[12\]A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Vaughan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.External Links:[Link](https://arxiv.org/abs/2407.21783)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p1.1)\.
- \[13\]T\. Gupta, S\. Krishnan, R\. Kumar, A\. Vijeev, B\. Gulavani, N\. Kwatra, R\. Ramjee, and M\. Sivathanu\(2024\)Just\-in\-time checkpointing: low cost error recovery from deep learning training failures\.InProceedings of the Nineteenth European Conference on Computer Systems,pp\. 1110–1125\.External Links:[Document](https://dx.doi.org/10.1145/3627703.3650085)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[14\]Y\. He, M\. Hutton, S\. Chan, R\. De Gruijl, R\. Govindaraju, N\. Patil, and Y\. Li\(2023\)Understanding and mitigating hardware failures in deep learning training systems\.InProceedings of the 50th Annual International Symposium on Computer Architecture,pp\. 1–16\.External Links:[Document](https://dx.doi.org/10.1145/3579371.3589105)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1.2)\.
- \[15\]J\. Hoffmann, S\. Borgeaud, A\. Mensch, E\. Buchatskaya, T\. Cai, E\. Rutherford, D\. de Las Casas, L\. A\. Hendricks, J\. Welbl, A\. Clark, T\. Hennigan, E\. Noland, K\. Millican, G\. van den Driessche, B\. Damoc, A\. Guy, S\. Osindero, K\. Simonyan, E\. Elsen, J\. W\. Rae, O\. Vinyals, and L\. Sifre\(2022\)Training compute\-optimal large language models\.arXiv preprint arXiv:2203\.15556\.External Links:[Link](https://arxiv.org/abs/2203.15556)Cited by:[§II\-A](https://arxiv.org/html/2607.01646#S2.SS1.p1.1)\.
- \[16\]I\. Jang, Z\. Yang, Z\. Zhang, X\. Jin, and M\. Chowdhury\(2023\)Oobleck: resilient distributed training of large models using pipeline templates\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 382–395\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613152)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p3.1)\.
- \[17\]M\. Jeon, S\. Venkataraman, A\. Phanishayee, J\. Qian, W\. Xiao, and F\. Yang\(2019\)Analysis of large\-scale multi\-tenant gpu clusters for dnn training workloads\.External Links:1901\.05758,[Link](https://arxiv.org/abs/1901.05758)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[18\]E\. Karrels, L\. Huang, Y\. Kan, I\. Arora, Y\. Wang, D\. S\. Katz, W\. Gropp, and Z\. Zhang\(2023\)Fine\-grained policy\-driven i/o sharing for burst buffers\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,SC ’23,New York, NY, USA\.External Links:ISBN 9798400701092,[Link](https://doi.org/10.1145/3581784.3607041),[Document](https://dx.doi.org/10.1145/3581784.3607041)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p5.1)\.
- \[19\]M\. Li, W\. Xiao, H\. Yang, B\. Sun, H\. Zhao, S\. Ren, Z\. Luan, X\. Jia, Y\. Liu, Y\. Li, W\. Lin, and D\. Qian\(2023\)EasyScale: elastic training with consistent accuracy and improved utilization on gpus\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–14\.External Links:[Document](https://dx.doi.org/10.1145/3581784.3607054)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[20\]S\. Li, F\. Xue, C\. Baranwal, Y\. Li, and Y\. You\(2022\)Sequence parallelism: long sequence training from system perspective\.External Links:2105\.13120,[Link](https://arxiv.org/abs/2105.13120)Cited by:[§II\-A](https://arxiv.org/html/2607.01646#S2.SS1.p2.1)\.
- \[21\]X\. Lian, S\. A\. Jacobs, L\. Kurilenko, M\. Tanaka, S\. Bekman, O\. Ruwase, and M\. Zhang\(2025\)Universal checkpointing: a flexible and efficient distributed checkpointing system for large\-scale dnn training with reconfigurable parallelism\.InProceedings of the 2025 USENIX Conference on Usenix Annual Technical Conference,USENIX ATC ’25,USA\.External Links:ISBN 978\-1\-939133\-48\-9,[Document](https://dx.doi.org/10.48550/arXiv.2406.18820)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p2.1)\.
- \[22\]H\. Liu and P\. Abbeel\(2024\)Blockwise parallel transformers for large context models\.Advances in Neural Information Processing Systems36\.External Links:[Document](https://dx.doi.org/10.5555/3666122.3666508)Cited by:[§II\-A](https://arxiv.org/html/2607.01646#S2.SS1.p2.1)\.
- \[23\]H\. Liu, M\. Zaharia, and P\. Abbeel\(2023\)Ring attention with blockwise transformers for near\-infinite context\.External Links:2310\.01889,[Link](https://arxiv.org/abs/2310.01889)Cited by:[§II\-A](https://arxiv.org/html/2607.01646#S2.SS1.p2.1)\.
- \[24\]K\. Maeng, S\. Bharuka, I\. Gao, M\. C\. Jeffrey, V\. Saraph, B\. Su, C\. Trippel, J\. Yang, M\. Rabbat, B\. Lucia, and C\. Wu\(2020\)CPR: understanding and improving failure tolerant training for deep learning recommendation with partial recovery\.External Links:2011\.02999,[Link](https://arxiv.org/abs/2011.02999)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[25\]A\. Maurya, R\. Underwood, M\. M\. Rafique, F\. Cappello, and B\. Nicolae\(2024\)Datastates\-llm: lazy asynchronous checkpointing for large language models\.InProceedings of the 33rd international symposium on high\-performance parallel and distributed computing,pp\. 227–239\.External Links:[Document](https://dx.doi.org/10.1145/3625549.3658685)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p2.1)\.
- \[26\]MetaSeq\(2022\)OPT\-175b baselines logbook\.Note:[https://github\.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B\_Logbook\.pdf](https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p1.1),[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[27\]J\. Mohan, A\. Phanishayee, and V\. Chidambaram\(2021\-02\)CheckFreq: frequent, Fine\-Grained DNN checkpointing\.In19th USENIX Conference on File and Storage Technologies \(FAST 21\),pp\. 203–216\.External Links:ISBN 978\-1\-939133\-20\-5,[Link](https://www.usenix.org/conference/fast21/presentation/mohan)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p2.1)\.
- \[28\]J\. Mohan, A\. Phanishayee, and V\. Chidambaram\(2021\)CheckFreq: Frequent,Fine\-Grained DNN Checkpointing\.In19th USENIX Conference on File and Storage Technologies \(FAST 21\),pp\. 203–216\.External Links:[Link](https://par.nsf.gov/biblio/10286595)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[29\]D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, and B\. Catanzaro\(2021\)Efficient large\-scale language model training on gpu clusters using megatron\-lm\.arXiv preprint arXiv:2104\.04473\.External Links:[Link](https://arxiv.org/abs/2104.04473)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p1.1),[§I](https://arxiv.org/html/2607.01646#S1.p5.1)\.
- \[30\]B\. Nie, D\. Tiwari, S\. Gupta, E\. Smirni, and J\. H\. Rogers\(2016\)A large\-scale study of soft\-errors on gpus in the field\.In2016 IEEE International Symposium on High Performance Computer Architecture, HPCA 2016, Barcelona, Spain, March 12\-16, 2016,pp\. 519–530\.External Links:[Link](https://doi.org/10.1109/HPCA.2016.7446091),[Document](https://dx.doi.org/10.1109/HPCA.2016.7446091)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1)\.
- \[31\]A\. Paszke, S\. Gross, F\. Massa, A\. Lerer, J\. Bradbury, G\. Chanan, T\. Killeen, Z\. Lin, N\. Gimelshein, and L\. Antiga\(2019\)Pytorch: an imperative style, high\-performance deep learning library\.Advances in neural information processing systems32,pp\. 8026–8037\.External Links:[Link](https://arxiv.org/abs/1912.01703)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p5.1),[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[32\]R\. Pinciroli, L\. Yang, J\. Alter, and E\. Smirni\(2021\)Lifespan and failures of ssds and hdds: similarities, differences, and prediction models\.IEEE Transactions on Dependable and Secure Computing\.External Links:[Document](https://dx.doi.org/10.1109/TDSC.2021.3131571)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1.2)\.
- \[33\]O\. Salpekar, R\. Varma, K\. Yu, V\. Ivanov, Y\. Wang, A\. Sharif, M\. Si, S\. Xu, F\. Tian, S\. Zheng, T\. Rice, A\. Garg, S\. Peng, S\. Siravara, W\. Fu, R\. de Castro, A\. Gangidi, A\. Obraztsov, S\. Narang, S\. Edunov, M\. Naumov, C\. Tang, and M\. Oldham\(2026\)Training LLMs with fault Tolerant HSDP on 100,000 GPUs\.External Links:2602\.00277,[Link](https://arxiv.org/abs/2602.00277)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p1.1),[§VII\-C](https://arxiv.org/html/2607.01646#S7.SS3.p1.1)\.
- \[34\]O\. Salpekar, R\. Varma, K\. Yu, V\. Ivanov, Y\. Wang, A\. Sharif, M\. Si, S\. Xu, F\. Tian, S\. Zheng, T\. Rice, A\. Garg, S\. Peng, S\. Siravara, W\. Fu, R\. de Castro, A\. Gangidi, A\. Obraztsov, S\. Narang, S\. Edunov, M\. Naumov, C\. Tang, and M\. Oldham\(2026\)Training llms with fault tolerant hsdp on 100,000 gpus\.External Links:2602\.00277,[Link](https://arxiv.org/abs/2602.00277)Cited by:[§V\-C](https://arxiv.org/html/2607.01646#S5.SS3.SSS0.Px1.p2.2)\.
- \[35\]A\. Sergeev and M\. D\. Balso\(2018\)Horovod: fast and easy distributed deep learning in TensorFlow\.arXiv preprint arXiv:1802\.05799\.External Links:[Link](https://arxiv.org/abs/1802.05799)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[36\]M\. Snir, R\. W\. Wisniewski, J\. A\. Abraham, S\. V\. Adve, S\. Bagchi, P\. Balaji, J\. Belak, P\. Bose, F\. Cappello, B\. Carlson, A\. A\. Chien, P\. Coteus, N\. A\. DeBardeleben, P\. C\. Diniz, C\. Engelmann, M\. Erez, S\. Fazzari, A\. Geist, R\. Gupta, F\. Johnson, S\. Krishnamoorthy, S\. Leyffer, D\. Liberty, S\. Mitra, T\. Munson, R\. Schreiber, J\. Stearley, and E\. V\. Hensbergen\(2014\)Addressing failures in exascale computing\.The International Journal of High Performance Computing Applications28\(2\),pp\. 129–173\.External Links:[Document](https://dx.doi.org/10.1177/1094342014522573)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1)\.
- \[37\]J\. Thorpe, P\. Zhao, J\. Eyolfson, Y\. Qiao, Z\. Jia, M\. Zhang, R\. Netravali, and G\. H\. Xu\(2023\)Bamboo: making preemptible instances resilient for affordable training of large DNNs\.In20th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 23\),pp\. 497–513\.External Links:[Link](https://arxiv.org/abs/2204.12013)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p2.1),[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[38\]B\. Wan, M\. Han, Y\. Sheng, Y\. Peng, H\. Lin, M\. Zhang, Z\. Lai, M\. Yu, J\. Zhang, Z\. Song, X\. Liu, and C\. Wu\(2025\)ByteCheckpoint: a unified checkpointing system for large foundation model development\.InProceedings of the 22nd USENIX Symposium on Networked Systems Design and Implementation,NSDI ’25,USA\.External Links:ISBN 978\-1\-939133\-46\-5,[Link](https://arxiv.org/abs/2407.20143)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p2.1)\.
- \[39\]G\. Wang, O\. Ruwase, B\. Xie, and Y\. He\(2024\)FastPersist: Accelerating Model Checkpointing in Deep Learning\.arXiv preprint arXiv:2406\.13768\.External Links:[Link](https://arxiv.org/abs/2406.13768)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[40\]Q\. Wang, T\. Lan, Y\. Tang, Z\. Huang, Y\. Du, H\. Zhang, J\. Sha, H\. Lu, Y\. Zhou, K\. Zhang, and M\. Tang\(2024\)DLRover\-rm: resource optimization for deep recommendation models training in the cloud\.External Links:2304\.01468,[Link](https://arxiv.org/abs/2304.01468)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[41\]Z\. Wang, Z\. Jia, S\. Zheng, Z\. Zhang, X\. Fu, T\. E\. Ng, and Y\. Wang\(2023\)Gemini: fast failure recovery in distributed training with in\-memory checkpoints\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 364–381\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613145)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[42\]Q\. Weng, W\. Xiao, Y\. Yu, W\. Wang, C\. Wang, J\. He, Y\. Li, L\. Zhang, W\. Lin, and Y\. Ding\(2022\)MLaaS in the wild: workload analysis and scheduling in Large\-Scale heterogeneous GPU clusters\.In19th USENIX Symposium on Networked Systems Design and Implementation \(NSDI 22\),pp\. 945–960\.External Links:[Link](https://www.usenix.org/conference/nsdi22/presentation/weng)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p2.1)\.
- \[43\]B\. Wu, L\. Xia, Q\. Li, K\. Li, X\. Chen, Y\. Guo, T\. Xiang, Y\. Chen, and S\. Li\(2023\)TRANSOM: an Efficient Fault\-Tolerant System for Training LLMs\.External Links:2310\.10046,[Link](https://arxiv.org/abs/2310.10046),[Document](https://dx.doi.org/10.48550/arXiv.2310.10046)Cited by:[§I](https://arxiv.org/html/2607.01646#S1.p2.1)\.
- \[44\]T\. Zhang, K\. Liu, J\. Kosaian, J\. Yang, and R\. Vinayak\(2023\)Efficient fault tolerance for recommendation model training via erasure coding\.Proceedings of the VLDB Endowment16\(11\),pp\. 3137–3150\.External Links:[Document](https://dx.doi.org/10.14778/3611479.3611514)Cited by:[§III](https://arxiv.org/html/2607.01646#S3.p4.1)\.
- \[45\]Z\. Zhang, L\. Huang, R\. Huang, W\. Xu, and D\. S\. Katz\(2019\)Quantifying the impact of memory errors in deep learning\.In2019 IEEE International Conference on Cluster Computing \(CLUSTER\),pp\. 1–12\.External Links:[Document](https://dx.doi.org/10.1109/CLUSTER.2019.8890989)Cited by:[§II\-B](https://arxiv.org/html/2607.01646#S2.SS2.p1.1.2)\.