From nnScaler to Large-Scale Training: Some Thoughts on the Boundaries of a Versatile Training System

This article originates from a technical talk I recently prepared. The specific topic was “Distributed Training Systems: From Research to Production.” When turning it into a blog post, I didn’t want to write a complete training tutorial, nor did I want to miss the chance to organize my understanding of the boundaries of training systems over the past few years.

There are already many articles on Zhihu covering DP, TP, PP, FSDP, ZeRO, Megatron, and DeepSpeed. I’d like to discuss more judgment-oriented questions: When a versatile training system moves from a research prototype to real training, which issues become more critical? Why does a seemingly natural general-purpose system encounter boundaries in the real large-model training ecosystem?

My background is roughly: I participated in the early stages of OneFlow, then mainly worked on nnScaler / AutoDist and related work, and also supported multiple training runs for several public model projects. More recently, I’ve encountered system issues related to larger-scale training, post-training, and reinforcement learning workflows. Many of the judgments below come from these experiences and are heavily personal.

1. Distributed Training Is Not Simply “Making a Program Run Faster”

In single-GPU or single-node training optimization, we usually face a relatively fixed training program: model structure, parameters, activations, gradients, optimizer states, checkpoints, and loss computation are all localized. Optimization often involves local modifications (kernel, memory, fusion, layout, batching) on this fixed program.

Distributed training is not simply running the same program on more GPUs; it changes the way the program’s states exist:

• Parameters can be replicated or sharded;
• Gradients can be all-reduced or reduce-scattered;
• Activations can be recomputed, checkpointed, or sharded along sequence/context dimensions;
• Optimizer states can be replicated, sharded, or offloaded;
• RNG states, data order, loss normalization, global norm, and checkpoint/resume all need to maintain training semantics.

Therefore, a so-called “parallelization plan” is not just “using the names DP/TP/PP/EP/CP.” A truly executable plan must at least answer:

• Which tensor dimensions are sharded?
• At each moment, which ranks hold which states?
• Where do communication operations occur?
• Which tensors are materialized, recomputed, or released?
• Do checkpoint, resume, RNG, hooks, and global norm still preserve the same semantics?

Today, I prefer to understand a parallelization plan as an executable contract: it is both a performance plan and a semantic promise. If this promise is not clearly articulated, the system will face many correctness and trust issues down the road.

2. Why Automatic Parallelism Felt Natural in 2022

When I first started working with training systems, a key impression was: manually specifying how to shard every operation, while extremely expressive, was painful for users and often didn’t lead to good solutions.

Based on some experience gained at OneFlow, I felt that searching for automatic parallel strategies was a natural direction. Around 2022, large models had not yet become the mainstream narrative (like ChatGPT), but model sizes were already increasing significantly, GPU memory on V100/A100 was limited, and algorithms began to demand scale. In other words, the demand for distributed training was emerging, though many did not yet see it as the core problem it is today.

The initial idea of AutoDist was straightforward: given a model and a hardware environment, profile computation and communication costs, and then search for a good execution plan. The optimization had two main goals:

• Search fast;
• Search well.

This approach shared direction with systems like Alpa: treat parallelization as an optimization problem, and use analysis, cost models, and heuristic search to reduce search cost. In a research environment, this direction was clean and attractive.

Later, AutoDist was integrated into nnScaler training. nnScaler provided a compiler stack: a frontend that captured PyTorch programs, a middle layer for graph transformation and plan generation, and a backend that generated executable PyTorch code.

At that time, we strongly believed in the PyTorch ecosystem. We thought that a system tightly integrated with PyTorch, rather than requiring users to migrate to a completely new framework, would have greater practical value.

I still think this judgment is correct. However, the difficulties that followed went far beyond “how to find a fast plan.”

3. From Research System to Real Training: Trust Is the Hardest to Earn

System builders tend to focus on speed first: how much step time is reduced, how much memory is saved, and how well it scales to more GPUs. But from the perspective of model and algorithm users, the top priority is often not speed, but: Is your system correct?

This sounds simple, but it is extremely difficult in practice.

Theoretically, users want bitwise reproducibility: same input, same environment, same random seed, and identical results across multiple runs. However, reality is much more complex:

• Floating-point accumulation order can change;
• Some kernels are non-deterministic by nature;
• Deterministic modes often incur significant performance penalties;
• NCCL collective algorithms and implementation details can affect results;
• Performance seen by profilers may depend on software versions, shapes, and hardware states;
• Neural networks can tolerate some errors, and small errors may only show up in loss curves or evaluations after many steps.

In the early days of nnScaler, we relied on empirical parity tests in practice: tensor/value comparisons, short-range loss curves, alignment with reference runs, and then asking users to perform larger-scale validation. This approach was not perfect, but it was close to how many training frameworks test in practice.

Later, I gradually realized that correctness is not an appendix; it is a research-worthy problem on its own. nnScaler generated an executable plan and execution program, but users truly need to answer: “Why should I trust this program?” This naturally led to subsequent work like TrainVerify, which pushed correctness for distributed training from engineering experience to formal verification.

This was the last major lesson I learned from nnScaler: A training system is not enough if it just runs and runs fast. It must make users genuinely trust that the transformed execution still preserves training semantics.

4. Flexibility Is Not Free

Compared to systems like FSDP, Megatron, and DeepSpeed, nnScaler/AutoDist had the advantage of a larger search space and stronger generality. In theory, a larger search space can find better solutions for given model and hardware environments.

But over time, I realized that flexibility is not free.

A larger search space means:

• More plans to generate, test, and explain;
• More graph transformations and runtime states;
• More ways for codegen to go wrong;
• More parity and verification cases;
• More difficulty for users to locate issues when problems arise.

In research papers, generality is often a strength. But in real-world training, the cost of generality concretely manifests in development, debugging, validation, and user support. A system that tries to support everything may end up supporting each scenario to only “not reassuring” degree.

This is not to say general-purpose systems have no value, but one must ask: In which scenarios is generality really worth the heavy cost?

Today, I lean toward the view that systems like nnScaler fit best in scenarios where:

• The model architecture is still evolving rapidly;
• Users do not want to handcraft complex strategies;
• The system can derive or search for correct strategies from model semantics;
• The workload is relatively static, and shapes/control flow will not break compiler assumptions.

Once model architectures converge, or mainstream training revolves around a small set of hardware/software stacks, the appeal of dedicated systems increases. For example, LLaMA-like models, MoE models, and specific long-context solutions — if existing stacks like Megatron-style, FSDP, verl, or specialized trainers can support them stably, users are unwilling to absorb additional learning and debugging costs for a general compiler stack.

5. Real Large-Scale Training Continues to Push System Boundaries

If nnScaler taught me about complexity at the compiler/trainer level, my more recent exposure to larger-scale training has given me another insight: the boundaries of a distributed training system are constantly pushed wider by real workloads.

At much larger scales, training systems are no longer just a framework-internal problem. They involve:

• Storage and data loader throughput;
• Checkpoint save/load and recovery costs;
• Cluster schedulers;
• Machine validation, repair, and reinsertion pipelines;
• Network topology, collective performance, and stragglers;
• Fault tolerance;
• Logging, metrics, profiling, and diagnostics;
• Inference engines;
• Co-design between model architecture and training algorithms.

At this point, the optimization target changes. At small scale, we naturally look at step time, TFLOPS of a kernel, or memory profile of a layer. At large scale, the more important units become the metrics of a long-running training campaign:

• Tokens per day;
• Effective training time;
• Recovery cost from failures;
• Debugging turnaround time;
• Speed of locating the real cause when anomalies occur during training.

Public system works like MegaScale, Robust LLM Training Infrastructure, FALCON, XPUTimer, SuperBench, and L4 are worth reading, not just because they report large-scale cluster experience, but because they extend the concept of “training system” from inside the framework to a closed loop of cluster, network, diagnostics, and operations.

6. Post-Training and RL Make Inference Part of the Training System

Another shift comes from post-training and reinforcement learning.

Traditional pre-training is often imagined as a relatively clean loop: forward, backward, update, plus checkpoints and evaluation. But in RL / post-training, the training loop becomes a closed cycle:

• Rollout;
• Environment;
• Reward;
• Filtering;
• Training;
• Evaluation;
• Resuming rollout.

In this context, the inference engine is no longer just a deployment component after training; it directly affects training efficiency. Techniques like rollout routing, load balancing, prefix-aware routing, speculative decoding, MTP, etc., can all become part of the training system.

Moreover, some system optimizations are not semantically neutral. For example, asynchronous RL is attractive from a system utilization perspective, but from an algorithm perspective it can introduce staleness and off-policy effects. Without co-design with the algorithm side, it is easy to encounter the situation: “the system runs faster, but the target hasn’t actually improved.”

The lesson here is: in post-training / RL scenarios, system optimization must not only ask “can it be faster?”, but also “does it preserve the algorithm’s contract?”

7. Fault Tolerance Becomes a Training System Problem

As pre-training scales continue to grow, machine failures are no longer an exception but a norm.

Most existing training stacks are still fundamentally synchronous: if one rank fails, the entire job may hang, fail, or be forced to restart. Checkpoint/restart is a necessary capability, but when training is large enough and long enough, the restart cost itself significantly impacts effective training time.

Therefore, a more desirable direction is not “train assuming machines don’t fail,” but to make fault tolerance a part of the overall design. For example, work like FT-HSDP explores isolating failed replicas in large-scale synchronous training, allowing healthy replicas to continue and rejoin after recovery.

This is not exactly the same as traditional “operational reliability for training.” At sufficiently large scales, fault tolerance itself becomes a semantic and performance problem of the system.

8. The Most Valuable Capability Now May Be Verification / Testing / Diagnostics

A major reason large-scale training is difficult is that the entire stack is evolving rapidly:

• GPUs and network devices change;
• CUDA, NCCL, PyTorch, Transformer Engine, Megatron change;
• Model architecture variants: MoE, long context, MLA, MTP, new attention variants;
• Workloads are shifting: pre-training, post-training, RL, online generation, inference-heavy loops.

These changes make “directly reusing others’ experiences and code” increasingly risky. Even if a solution works on another organization’s model, hardware combination, software version, and cluster topology, it does not mean it will work in your own environment.

Therefore, I increasingly feel that the value of verification, testing, and diagnostics is underestimated.

A good diagnostic system should focus on answering:

• Is it a model implementation problem or a training framework problem?
• Is it a kernel problem or a communication problem?
• Is it a data problem or a checkpoint/restore problem?
• Is it a few slow/failing machines or a global configuration issue?
• Is it performance drift or correctness drift?

Often the most expensive thing in large-scale training is not a failure itself, but not knowing why it failed. A system that can locate the real cause faster can significantly improve engineering efficiency.

This is one of the long-term valuable directions I see for future training systems: not just generating a plan, but making the training process more verifiable, diagnosable, and recoverable.

9. Scaling Laws Still Hold, and the Training System Market Will Stratify

Let me offer a more biased viewpoint.

Assuming scaling laws continue to hold at the frontier, frontier pre-training will keep pushing toward larger models, larger data, and larger compute resources. The number of organizations that can fully participate in the frontier pre-training loop will shrink, because it requires not just GPUs, but data, algorithms, engineering, stability, load, diagnostics, and long-term training experience.

But this does not mean open-source models or medium/small models will disappear. It is more likely we will see stratification:

• Frontier organizations train increasingly large flagship models;
• They also release smaller dense models for edge devices, developer ecosystems, and research communities;
• They release efficient / quantized variants to cover more scenarios with related service costs;
• Occasionally they release large MoE or open-weight models as ecosystem entry points.

This stratification will in turn affect training systems.

If public model architectures gradually converge — e.g., mostly LLaMA-like backbones with local variations like attention, MoE, long context, MTP — then writing a dedicated trainer for a specific architecture or modifying existing stacks will become cheaper. Coding agents are also further reducing the cost of developing glue code, adapters, and baseline trainers.

This will compress the value space of pure generality. A system that only emphasizes “I can support more models” may be less convincing than before, because users can often quickly adapt a working baseline based on public code.

But this does not mean training systems have no value. On the contrary, the truly hard parts will persist:

• Scalability;
• Checkpoint/restore;
• Long-term stability;
• Fault tolerance;
• Communication performance;
• Diagnostics;
• Rapid adaptation to real workloads.

So my judgment now is: the value of a general-purpose distributed training system will not disappear, but its value function will shift. Lasting value may not equal “automatically generating a parallelism plan” itself, but rather making the complex training process correct, diagnosable, recoverable, and able to be quickly configured for real workloads.

10. A Small Revelation for Doing Research

This article is not arguing that a particular system direction is definitely right, or that some route is now worthless. Rather, I want to express: when doing systems research, one must observe demand signals and ecosystem changes very keenly.

Around 2022, automatic parallelism was a very natural problem: model architectures were more diverse, the cost of hand-writing strategies was high, and hardware memory was a clear bottleneck. Today, if model architectures, hardware stacks, and training pipelines are all shifting, the value of the same system direction will also change.

For researchers, this is not necessarily bad. It simply reminds us:

• Don’t just ask what a system can do in a general sense;
• Also ask in which scenarios that generality truly provides value;
• Don’t only look at performance numbers;
• Also look at correctness, observability, debuggability, operational cost;
• Don’t study an isolated layer in isolation;
• Understand its real upstream and downstream constraints.

Personally, I now lean more toward this judgment:

The core value of a distributed training system is shifting from “finding a parallelism plan” to “making the complex training process verifiable, diagnosable, recoverable, and sustainably evolvable.”

This judgment is definitely incomplete and may have biases. If you work on large-scale training, post-training, inference systems, cluster scheduling, networking, checkpoints, verification, or diagnostics, feel free to add different observations.

Author: Zhu Yi
https://zhuanlan.zhihu.com/p/2056045172111177252