Simulating fault tolerance with stage skipping in pipeline-parallel training [R]

Our most recent work at Templar explores fault tolerance in Crucible, our distributed pre-training platform. The goal is to keep healthy workers training when another pipeline stage goes offline. Crucible combines data-parallel replicas with pipeline parallelism. Each replica holds a copy of the model, split into stages on separate workers. SparseLoCo exchanges compressed updates between replicas, while pipeline compression reduces the communication across stage boundaries. We combine those methods with stage skipping. When an inner stage goes offline, activations and gradients bypass it for multiple steps. Healthy stages keep processing tokens instead of waiting for recovery. The bypass omits the unavailable stage’s computation. The simulations use a 178M model, eight replicas and four stages per replica. At a 1% per-replica failure probability per global step, validation loss stayed close to the no-failure baseline, even though each simulated outage removed a stage for six global steps. Each configuration is compared with its own no-failure run. Fixed projections shared across layers improve robustness further when using pipeline compression. This suggests that shared projectors align representations across stage boundaries, making bypasses less disruptive. The alignment explanation remains a hypothesis. These results point toward training on a broader pool of compute, including unreliable workers and spot instances. This is a simulation of the learning effects of stage failures, rather than a measurement of physical worker replacement or production cost savings. The article includes the setup, comparisons and figures: https://www.tplr.ai/publications/blog/skipping-stages-with-fixed-projections