Training Variable Long Sequences with Data-Centric Parallel

# Training Variable Long Sequences with Data-Centric Parallel
Source: [https://arxiv.org/html/2608.07524](https://arxiv.org/html/2608.07524)
Geng Zhang, Xuanlei Zhao∗, Kai Wang†, Yang You† National University of Singapore \{zhangg,xuanlei,kai\.wang,youy\}@comp\.nus\.edu\.sg

###### Abstract

Training deep learning models on variable long sequences poses significant computational challenges\. Existing methods force a difficult trade\-off between efficiency and ease\-of\-use\. Simple approaches use static configurations that cause workload imbalance low efficiency, while complex methods introduces significant complexity and code change for new models\. To break this trade\-off, we introduce Data\-Centric Parallel \(DCP\)\. Its core principle is to let the data itself drive the runtime\. It achieves this by dynamically adjusting direct runtime settings \(e\.g\., parallel size, gradient accumulation, recomputation\) based on each batch’s sequence length\. Empirical results demonstrate that our method achieves up to a 2\.88×\\timesspeedup on 32 H200 GPUs\. Designed for generalization, it can be integrated into any model with 10 lines of code\. We anticipate this simple yet effective approach will serve as a robust baseline and facilitate future advancements in distributed training for variable long sequences\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/dcp_compare.png)Figure 1:Comparison of parallel methods for variable sequences training including bucket parallel, packed parallel and data\-centric parallel \(ours\)\.D​iDirefers to theii\-t​hthdevice\.The capacity to process long sequences is a crucial driver for a growing number of deep learning applications\. This trend is evident across diverse fields, including video generation\(Brookset al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib8); Zhenget al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib1); Polyaket al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib45); Konget al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib56)\), image generation\(Esseret al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib18)\), multi\-modal perception\(Chenet al\.,[2024b](https://arxiv.org/html/2608.07524#bib.bib3); Wanget al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib2)\), text generation\(Touvronet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib5); Baiet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib4)\), and scientific computing\(Jumperet al\.,[2021](https://arxiv.org/html/2608.07524#bib.bib6)\)\.

However, training on such data presents significant computational challenges: 1\)Long sequence: the substantial length of sequences consumes lots of GPU memory, requiring sequence parallelism to partition one sequence across multiple devices to reduce the memory cost\. 2\)Variable length: The inherent wide variation in sequence lengths, as illustrated in Figure[4](https://arxiv.org/html/2608.07524#S2.F4), leads to severe workload imbalances during distributed training, especially when combined with sequence parallel\.

Parallel methods for training on variable\-length sequences can be categorized into three classes as shown in Figure[1](https://arxiv.org/html/2608.07524#S1.F1)\. Bucket parallel\(Esseret al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib18)\)sets a fixed parallel size for all sequences, and decrease the batch size for slow batches to reduce the imbalance, as indicated in Figure[2](https://arxiv.org/html/2608.07524#S1.F2)\. Nevertheless, this naive solution introduces two problems\. First, the batch size for very long sequences is often inherently limited, providing minimal flexibility for tuning and achieving the desired load

![Refer to caption](https://arxiv.org/html/2608.07524v1/x1.png)Figure 2:Workload balance in bucket parallel for various sequence lengths\. Dashed lines indicate the batch size\.balance\. Second, it does not consider the computational efficiency of short sequences when reducing batch size, leaving significant speed loss\.

Packed parallel\(Dehghaniet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib17)\)improves load balance by packing multiple sequences into a batch instead of reducing batches, but this introduces communication overhead and requires extra changes for sequence level operations\. Compiler\-based parallel methods\(Geet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib53)\)apply a compiler to automatically optimize for efficient plans, but this requires significant code changes and makes new model adaptation difficult\.

Existing methods present a difficult trade\-off between efficiency and ease\-of\-use and\. We argue that the core limitation that preventing simple works from being effective is their dependence on predetermined runtime settings \(e\.g\., parallel size\), which significantly limits both communication cost and workload imbalance\. By directly confronting this, it is possible to attain high efficiency without the need of more heavyweight systems\. This leads us to the central question of our work:how can we let the data itself drive the runtime in a simple yet effective way?

To address this challenge, we propose Data\-Centric Parallel \(DCP\), the first method that achieve both efficient and ease\-of\-use method for training variable long sequences\. Instead of using a fixed setting, DCP dynamically adjusts the runtime settings such as parallel strategy, gradient accumulation, and recomputation based each data’s sequence length\. By minimizing the communication cost for each batch while balancing the workload, DCP significantly improves training throughput\.

DCP comprises of two strategies: DCP\-inter and DCP\-intra\. DCP\-inter utilizes gradient accumulation to balance workload instead of reducing batch size\. DCP\-intra further speedup by minimizing recomputation\. The extra memory cost is tackled by carefully adjusting the sequence parallel and batch size with ignorable cost\.

Empirical results show that DCP effectively improve throughput for variable long sequences training by up to 2\.88×\\timeswith 32 H200 GPUs across 3 datasets on 2 models\. Designed for generalization, DCP can be easily adapted for any models with at most 10 lines of code change as shown in Appendix[D](https://arxiv.org/html/2608.07524#A4)\. We anticipate the simple yet effective DCP will serve as a robust baseline and facilitate future advancements for distributed training with variable sequence lengths\.

## 2Challenges in training variable long sequences

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/clip_sequence_length_distribution.png)Figure 3:Visualization of sequence length distribution of Panda\-80M\.
![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/imbalance.png)Figure 4:Imbalance ratio analysis across 140 datasets based on bucket parallel\.

### 2\.1Large variance in Data Length

The primary challenge is the large variance in sequence lengths in real\-world datasets\. For example, Figure[4](https://arxiv.org/html/2608.07524#S2.F4)demonstrates it with the Panda\-70M\(Chenet al\.,[2024a](https://arxiv.org/html/2608.07524#bib.bib14)\)dataset, which contains 70 million videos\. As a result, the training system must handle extreme diversity in workload\.

A naive solution is to group data of similar lengths into one training step\. However, this approach harms model quality because it disrupts the independent and identically distributed sampling crucial for stable convergence\(Waltz,[1984](https://arxiv.org/html/2608.07524#bib.bib16); Brookset al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib8)\)\. Therefore, an effective training system must handle this extreme length diversity without compromising statistical efficiency\.

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/sp_cost.png)Figure 5:Weak scaling communication overhead for different sizes of sequence parallel on various sequence length for Transformer\-1D\.
### 2\.2Communication Cost for Sequence Parallel

For the longest sequences in a dataset, a large SP size is inevitable to avoid out\-of\-memory errors\. This maximum required parallelism often dictates the global SP configuration for the entire training\.

#### Large SP is necessary for long sequences\.

For the longest sequences, a large SP size is inevitable to avoid out of memory error, and decides the global sequence parallel size\.

#### Large SP is harmful for shorter sequences

\. This large SP becomes a major bottleneck for shorter sequences as shown in Figure[5](https://arxiv.org/html/2608.07524#S2.F5), which often dominate datasets\. For these sequences, the computation on each device is less, but the fixed cost of communication required to exchange information between devices remains high\. This communication overhead can easily dominate the total processing time, drastically reducing training efficiency\.

This creates a dilemma: a large, fixed SP size is wasteful for the common short sequences, while a small size cannot accommodate the necessary long sequences\.

### 2\.3Workload Imbalance for Distributed Training

When using Data Parallelism \(DP\), the high variance in sequence lengths causes severe workload imbalance as shown in Figure[4](https://arxiv.org/html/2608.07524#S2.F4)\. In a DP system, each worker processes a different batch of data, but all workers must synchronize before starting the next step\.

Due to length variation, a worker assigned a batch of long sequences will take significantly longer to complete its computation\. In contrast, workers with short sequences finish quickly and are forced to sit idle, waiting for the slowest worker to complete its task\. This effect leads to severe under\-utilization of computational resources, as expensive accelerators waste cycles waiting\. The overall training throughput is bottlenecked by the slowest worker in each step, a problem that is exacerbated as the number of workers in the cluster grows\.

### 2\.4Trade\-off between Balance and Communication

An intuitive approach to mitigate workload imbalance is to adjust batch sizes statically: use smaller batches for long sequences and larger ones for short sequences to equalize the processing time per step\. However, this strategy is ineffective and introduces a costly new trade\-off\.

#### Small batch size for long sequences\.

For the long sequences, memory constraints already force the batch size to the minimum, so it cannot be reduced further\. Even if there is a few batches for long sequences, after they reduce the batch size, for all other sequences, they need to use a batch size smaller than what the GPU can hold leads to the under\-utilization of expensive hardware\.

#### Communication overhead for reduced batch sizes\.

More critically, this approach creates a new bottleneck\. By using smaller batches to balance workload in each, the total number of training steps needed to process the dataset increases substantially\. Since every iteration requires a costly global synchronization step to collect all parameters’ gradients, this strategy trades per\-iteration workload imbalance for a massive increase in total communication costs\. The result is often no net performance gain, or even a regression\.

Takeaway:workload imbalance,communication cost, andcomputation efficiencyare the three key factors to effectively train variable long sequences\.

## 3Data\-Centric Parallel

### 3\.1Overview

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/dcp_overview3.png)Figure 6:The overview of DCP\. It first collects speed and memory cost by fast dual\-layer profile\. Then maximizes efficiency by dynamically adjusting parallel size, batch size, and gradient checkpointing based on sequence length\.As shown in Figure[6](https://arxiv.org/html/2608.07524#S3.F6), Data\-Centric Parallel \(DCP\) dynamically adjust the parallel, batch size and gradient checkpointing to maximize the efficiency\. Specifically, we first group different sequences in to several groups with similar sizes\. And for each group, we use a fast profiling to get the speed and memory cost for different batch sizes and sequence parallel sizes\. Then, we apply one of the following two strategies to dynamically adjust the settings of based on data:

#### DCP\-inter

optimize load balance by gradient accumulation\. Prior to each iteration, it selects the optimal batch size and sequence parallel size for the incoming batch’s sequence length based on profile\. Instead of reducing batch size, this method enable better balance with no cost\.

#### DCP\-intra

further exploits dynamic recomputation for better speed\. This is motivated by our analysis that for short sequences, gradient checkpointing introduces considerable unnecessary computational overhead\. Based on this insight, DCP\-intra implements a strategic trade\-off: it partially deactivates gradient checkpointing\. To manage the increase in memory consumption, it then dynamically adjusts sequence parallel and batch size with ignorable cost\.

### 3\.2Problem Formulation

Givennnbatches\{d1,d2,…,dn\}\\\{d\_\{1\},d\_\{2\},\.\.\.,d\_\{n\}\\\}with variable sequence lengths\{s1,s2,…,sn\}\\\{s\_\{1\},s\_\{2\},\.\.\.,s\_\{n\}\\\}for each training iteration, DCP dynamically adapts training configuration of each batch according to its sequence length to improve the training throughput\. The key is to jointlyimprove the throughputandbalance the execution timefor different sequence lengths in each training iteration\. Formally, it is translated to two optimization targets:

max​\(bi∗sipi∗T​\(di\)\),s\.t\.​∑i=1npi=W,\\displaystyle\\text\{max\}\(\\frac\{b\_\{i\}\*s\_\{i\}\}\{p\_\{i\}\*T\(d\_\{i\}\)\}\),\\text\{s\.t\.\}\\sum\_\{i=1\}^\{n\}\{p\_\{i\}\}=W,\(1\)min​∑i=1n\(maxj=1n​T​\(dj\)−T​\(di\)\)/W\.\\displaystyle\\text\{min\}\\sum\_\{i=1\}^\{n\}\{\(\\text\{max\}\_\{j=1\}^\{n\}T\(d\_\{j\}\)\-T\(d\_\{i\}\)\)\}/W\.\(2\)wherebib\_\{i\},pip\_\{i\}andT​\(ni\)T\(n\_\{i\}\)are the batch size, sequence parallel size, and execution time fornin\_\{i\}, andWWis the total number of GPUs\.

### 3\.3DCP\-INTER

DCP\-inter first determines the best parallel settings for each sequence length based on profiling\. It then exploits gradient accumulation to balance the execution time across data batches in each iteration instead of reducing batch size\. Such simple but effective strategy maintains a high throughput for slow batches and fill the idle time of fast batches by running multiple batches\.

Givennnbatches, DCP\-inter aims at finding the number of accumulation stepsgig\_\{i\}for each batch basedT​\(di\)T\(d\_\{i\}\)to meet balance and throughput\.T​\(di\)T\(d\_\{i\}\)is obtained by a fast profiling to be throughput\-optimal for eachbib\_\{i\}\. To achieve optimal workload balance, Equation[2](https://arxiv.org/html/2608.07524#S3.E2)can be rewritten as:

min​∑i=1n\(maxj=1n​\(gj∗T​\(dj\)\)−gi∗T​\(di\)\)/W\.\\displaystyle\\text\{min\}\\sum\_\{i=1\}^\{n\}\{\(\\text\{max\}\_\{j=1\}^\{n\}\(g\_\{j\}\*T\(d\_\{j\}\)\)\-g\_\{i\}\*T\(d\_\{i\}\)\)\}/W\.\(3\)To search forgig\_\{i\}meeting this target, DCP\-inter iterates all possible number of accumulation steps for every batch and evaluates each combination ofgig\_\{i\}fornnbatches using Equation[3](https://arxiv.org/html/2608.07524#S3.E3)as the performance model to find the optimal setup for thennbatches in each iteration, detailed in Appendix[C](https://arxiv.org/html/2608.07524#A3)\.

### 3\.4DCP\-INTRA

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/dcp_ckpt.png)Figure 7:The overview of DCP\-intra\. It applies less recomputation for shorter sequences and increase sequence parallel size for extra memory cost\.For long sequence training, the activation checkpointing strategy is often fully applied to reduce the memory cost of GPU\(Yuanet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib47)\)\. However, with variable sequence lengths, we have spare GPU memory to enable less recomputation layers for shorter sequences to reduce the execution timeT​\(bi\)T\(b\_\{i\}\)Also, with larger sequence parallel size, the memory cost per GPU decreases, leaving more space for less recomputation to improve the throughput in Equation[1](https://arxiv.org/html/2608.07524#S3.E1)\. Based on the two observations, we proposes DCP\-intra\.

For aLLlayer model, the execution timeT​\(bi\)T\(b\_\{i\}\)and the memory overheadM​\(bi\)M\(b\_\{i\}\)forbib\_\{i\}are:

T​\(di\)=Tf​\(di\)\+Tb​\(di\)\+Tf′​\(di\)\+TC,\\displaystyle T\(d\_\{i\}\)=T\_\{f\}\(d\_\{i\}\)\+T\_\{b\}\(d\_\{i\}\)\+T\_\{f\}^\{\\prime\}\(d\_\{i\}\)\+T\_\{C\},\(4\)Tf′​\(di\)=ri∗tf​\(di\),\\displaystyle T\_\{f\}^\{\\prime\}\(d\_\{i\}\)=r\_\{i\}\*t\_\{f\}\(d\_\{i\}\),\(5\)M​\(di\)=\(L−ri\)∗m​\(di\)\+L∗mi​n​p​\(di\)\+MC,\\displaystyle M\(d\_\{i\}\)=\(L\-r\_\{i\}\)\*m\(d\_\{i\}\)\+L\*m\_\{inp\}\(d\_\{i\}\)\+M\_\{C\},\(6\)whereTf​\(bi\)T\_\{f\}\(b\_\{i\}\)andTb​\(bi\)T\_\{b\}\(b\_\{i\}\)are the forward and backward execution time, andTCT\_\{C\}is the constant time cost by other components like embedding, loss computation, etc\.Tf′​\(bi\)T\_\{f\}^\{\\prime\}\(b\_\{i\}\)is the recomputation execution time which denotes recomputing the forward passes ofrir\_\{i\}layers before the backward pass, with each layer usingtf​\(bi\)t\_\{f\}\(b\_\{i\}\)time\.

The memory overheadM​\(bi\)M\(b\_\{i\}\)comprises of three terms\. The first is the memory overhead costed by layers that does not recompute\. These layers need to stash all the intermediate activations with a cost ofm​\(bi\)m\(b\_\{i\}\)for every layer for backward\. The second is the necessary memory overhead for input activationsmi​n​p​\(bi\)m\_\{inp\}\(b\_\{i\}\)for each layer\. The third termMCM\_\{C\}includes a constant memory costed by model parameters, optimizers and intermediate buffers\(Rajbhandariet al\.,[2020](https://arxiv.org/html/2608.07524#bib.bib9); Yuanet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib47)\)\.

The procedure is described in Algorithm[1](https://arxiv.org/html/2608.07524#alg1)\. To reduce the execution time, for each sequence lengthsis\_\{i\}, it first runs a fast profiling for all possible batch sizebib\_\{i\}and sequence parallel sizepip\_\{i\}to get the execution timeT​\(di\)T\(d\_\{i\}\)and the memory overheadM​\(di\)M\(d\_\{i\}\)withrir\_\{i\}set toLL\. Then, according to Equation[6](https://arxiv.org/html/2608.07524#S3.E6), it computes the minimalrir\_\{i\}under the GPU memory capacityMMas the optimal recomputation forbib\_\{i\}andpip\_\{i\}\. Finally, we can compare the throughput according to Equation[1](https://arxiv.org/html/2608.07524#S3.E1)for all possible batch size and parallel size to obtain the throughput\-optimal settings\.

Algorithm 1DCP\-intra algorithm1:Input:Memory capacity

MM
2:Input:Model layers

LL
3:Input:Candidates

CCfrom profiling \(

bb,

pp\): \(

tft\_\{f\},

mm,

T\(b,p\)T\_\{\(b,p\)\},

M\(b,p\)M\_\{\(b,p\)\}\)

4:Output:Optimal batch size

bib\_\{i\}, sequence parallel size

pip\_\{i\}, recompute layer

rir\_\{i\}
5:

m​i​n​\_​t​i​m​e←∞min\\\_time\\leftarrow\\infty
6:Optimal result

\(bi,pi,ri\)\(b\_\{i\},p\_\{i\},r\_\{i\}\)←\\leftarrowNone

7:for

\(b,p\)∈C\(b,p\)\\in Cdo

8:

\(tf,m,T\(b,p\),M\(b,p\)\)←C​\[\(b,p\)\]\(t\_\{f\},m,T\_\{\(b,p\)\},M\_\{\(b,p\)\}\)\\leftarrow C\[\(b,p\)\]
9:for

r←1r\\leftarrow 1to

LLdo⊳\\trianglerightCalculate minimal recompute layerrrwith spare GPU memory

10:

Mo​p​t←M\(b,p\)\+r∗mM\_\{opt\}\\leftarrow M\_\{\(b,p\)\}\+r\*m
11:if

Mo​p​t≤MM\_\{opt\}\\leq Mthen

12:break

13:endif

14:endfor

15:

c​u​r​\_​t​i​m​e←T\(b,p\)−r∗tfcur\\\_time\\leftarrow T\_\{\(b,p\)\}\-r\*t\_\{f\}⊳\\trianglerightCalculate execution timec​u​r​\_​t​i​m​ecur\\\_timewith more memory

16:if

c​u​r​\_​t​i​m​e<m​i​n​\_​t​i​m​ecur\\\_time<min\\\_timethen

17:

m​i​n​\_​t​i​m​e←c​u​r​\_​t​i​m​emin\\\_time\\leftarrow cur\\\_time
18:

\(bi,pi,ri\)←\(b,p,r\)\(b\_\{i\},p\_\{i\},r\_\{i\}\)\\leftarrow\(b,p,r\)
19:endif

20:endfor

### 3\.5Dual Layer Profiling

DCP\-intra and DPC\-inter rely on a profiling process to set up an initial configurations\. Based on the observation that a transformer model often consists of multiple repeating layers, the profiling process only run two layers for each batch size and sequence parallel size\. The first layer is used to warm up, and the forward execution timetf​\(bi\)t\_\{f\}\(b\_\{i\}\), backward execution timetb​\(bi\)t\_\{b\}\(b\_\{i\}\)and memory overheadm​\(bi\)m\(b\_\{i\}\)for the second layer are recorded to estimate these metrics for the whole model\. The result also includes the constant timeTCT\_\{C\}and memory overheadMCM\_\{C\}\. With the four parts, the full execution time is estimated asL∗\(2∗tf​\(di\)\+tb​\(di\)\)\+TCL\*\(2\*t\_\{f\}\(d\_\{i\}\)\+t\_\{b\}\(d\_\{i\}\)\)\+T\_\{C\}\. The total memory overhead is estimated by Equation[6](https://arxiv.org/html/2608.07524#S3.E6), withrir\_\{i\}set toLL\.

## 4Experiments

### 4\.1Experiment Setup

#### Models\.

We test on two typical transformer architectures, Transformer\-1D and Transformer\-2D\. Transformer\-1D model is of 5B parameters, follow common architecture as in LLM\(Dubeyet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib58)\)\. Transformer\-2D has 1\.2B parameters, which applies attention sequentially across 2D sequences\. This is common in models for extreme long sequences such as protein prediction\(Jumperet al\.,[2021](https://arxiv.org/html/2608.07524#bib.bib6)\)and video generation\(Zhenget al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib1)\), detailed in Appendix[B](https://arxiv.org/html/2608.07524#A2)\.

#### Datasets\.

To assess our method’s performance across diverse sequence length distribution, we synthesize three datasets including: 1\) short: dominated by short sequences; 2\) balanced: balanced across a wide range of lengths; 3\) long: dominated by long sequences, detailed in Appendix[A](https://arxiv.org/html/2608.07524#A1)\.

#### Baselines\.

We takes bucket parallel\(Esseret al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib18)\)as our baseline, which fix parallel size and adjust batch size according to profiling results to balance workload\.

#### Testbed\.

Our experiments are conducted on a cluster of NVIDIA H200 GPUs\. Each node contains 8 GPUs connected via 900GB/s NVLink, with an 8×\\times400Gbs InfiniBand network for internode communication\. Although H200 GPUs have 141GB of memory, we limit usage to 80GB to demonstrate our method’s generalizability to more common GPUs like the H100 or A100\.

#### Implementation details\.

We adopt DeepSpeed Ulysses\(Jacobset al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib36)\)for sequence parallel and ZeRO\-1\(Rajbhandariet al\.,[2020](https://arxiv.org/html/2608.07524#bib.bib9)\)for data parallel, and enable Flash Attention\(Daoet al\.,[2022](https://arxiv.org/html/2608.07524#bib.bib42)\)for all experiments\. Note that our methods are compatible with all sequence parallel methods\.

### 4\.2End\-to\-end Evaluation

![Refer to caption](https://arxiv.org/html/2608.07524v1/x2.png)\(a\)Comparison on Transformer\-1D\.
![Refer to caption](https://arxiv.org/html/2608.07524v1/x3.png)\(b\)Comparison on Transformer\-2D\.

Figure 8:End\-to\-end throughput and imbalance comparison between different approaches for training Transformer\-1D and Transformer\-2D with 32 GPUs across 3 datasets\.#### Throughput\.

presents an end\-to\-end comparison of DCP\-inter and DCP\-intra methods against the baselines\. The evaluation is conducted on Transformer\-1D and Transformer\-2D models using 32 H200 GPUs across three distinct synthesized datasets\.

Based on the results, we make the following observations: 1\) DCP\-inter consistently outperforms the baseline, delivering speedups of up to 2\.70×\\timesand 1\.68×\\timesfor two models, respectively\. 2\) DCP\-intra, which leverages dynamic checkpointing, provides additional performance gains over DCP\-inter, achieving a total speedup of up to 2\.88×\\times\. 3\) The improvements of our methods are most pronounced on datasets dominated by shorter sequences\.

#### Imbalance\.

As shown in Figure[8](https://arxiv.org/html/2608.07524#S4.F8), our methods significantly reduce the workload imbalance observed across training tasks\. We define the imbalance ratio as the average proportion of time a GPU is idle while waiting for computations on other devices\. The results indicate that the baseline method suffers from a high degree of imbalance that also change significantly with the data distribution\. In contrast, our methods maintain a consistently low imbalance ratio across all conditions, which is a direct result of our flexible strategy\.

#### Effect of datasets distribution\.

Our methods achieve more performance gains on datasets dominated by shorter sequences\. Because our method mitigates these issues by reducing sequence parallel, which lowers communication costs\. Furthermore, unlike bucket parallel reduces the batch size for balance, our method maintains a full batch size, leading to better computation efficiency\.

#### Effect of model architecture\.

Our method performs better on Transformer\-1D because the computation time is significantly longer as it uses 1D attention compared with 2D attention, so the imbalance will be more severe\.

![Refer to caption](https://arxiv.org/html/2608.07524v1/x4.png)\(a\)Results for Transformer\-1D\.
![Refer to caption](https://arxiv.org/html/2608.07524v1/x5.png)\(b\)Results for Transformer\-2D\.

Figure 9:Scalability evaluation of different methods with different models\.

### 4\.3Scaling Ability

Scaling ability is of critical importance to training variable long sequences, as the communication cost, and especially workload imbalance will increase significantly as the scale of devices increases\. Because larger\-scale systems often encounter a greater diversity of sequence lengths\.

As shown in Figure[9](https://arxiv.org/html/2608.07524#S4.F9), we present a weak\-scaling comparison of our method against the baselines\. The results yield two primary observations: 1\) The bucket parallel baseline exhibits a significant decay in throughput as it scales, due to its inherent trade\-offs between load balance and computational efficiency\. 2\) In contrast, DCP demonstrates substantial scalability improvements, achieving near\-linear scaling on both Transformer\-1D and Transformer\-2D models\.

The consistent scaling performance of DCP establishes it as a highly effective and robust solution for training variable long sequences at scale, regardless of model architecture and data distribution\.

### 4\.4Ablation Study

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/dcp_intra_speedup.png)Figure 10:The speedup of DCP\-intra over DCP\-inter across various sequence lengths on Transformer\-1D\.As shown in Figure[10](https://arxiv.org/html/2608.07524#S4.F10), we evaluated the speedup of DCP\-intra compared with DCP\-inter across various sequence lengths\. The results demonstrate that for most sequence lengths, our method achieves 20\-25% speedup for sequence length less than 200k\. This performance gain directly corresponds to the overhead of gradient checkpointing, which our method effectively eliminates\. Furthermore, by allowing reasonable adjustments to the batch size and sequence parallel degree, our method can address extra memory cost with ignorable computational overhead\.

## 5Related Work

### 5\.1Parallel Training

#### Data parallelism

is a commonly adopted technique to enable distributed training of neural networks\(Xinget al\.,[2015](https://arxiv.org/html/2608.07524#bib.bib48)\)\. It replicates the model to different workers and split the global data batch to multiple batches\. With the advent of transformer models\(Brownet al\.,[2020](https://arxiv.org/html/2608.07524#bib.bib49); Touvronet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib5); Yanget al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib12)\), the memory capacity of a single GPU cannot accommodate the high memory demand\. ZeRO\(Rajbhandariet al\.,[2020](https://arxiv.org/html/2608.07524#bib.bib9)\)reduces the memory redundancy from optimizer states, gradients and model parameters across different workers to enable large scale model training\.

#### Sequence parallelism

is used for distributed training with long sequence length\. DeepSpeed Ulysess\(Jacobset al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib36)\)partitions the input activations, query, key and value for the attention module along the sequence dimension, with each sequence parallel worker possessing one split\. After getting the local output, another all\-to\-all operation is used to recover the attention heads and split along sequence dimension\. Ring attention also partitions the query, key and value along the sequence dimension\(Liet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib50); Fang and Zhao,[2024](https://arxiv.org/html/2608.07524#bib.bib51)\)\. To get the attention output, the key and value splits of different workers are exchanged via p2p communication in a ring pattern to perform and accumulate partial computation with the local query split\. Sequence parallelism of Megatron\-LM\(Korthikantiet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib34)\)splits both activations and model parameters across sequence parallel workers\.\. It uses all\-gather operation to recover the full input sequence and reduce\-scatter operation to reduce and split the output sequence\.

### 5\.2Activation Checkpointing

Activation checkpointing trades off memory cost against computation during model training\(Chenet al\.,[2016](https://arxiv.org/html/2608.07524#bib.bib41)\)\. Concretely, during the forward pass for a sequence of modules of a model, certain intermediate activations need to be stashed for gradient computation during backward pass\. Activation checkpointing only stashes the input activation for the sequence of modules, and computes the forward pass to materialize the activations needed for the backward pass\. Existing works either utilize heuristics to design static strategies\(Chenet al\.,[2016](https://arxiv.org/html/2608.07524#bib.bib41); Narayananet al\.,[2021](https://arxiv.org/html/2608.07524#bib.bib33); Korthikantiet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib34)\), or automatically searches for adaptive strategies that are optimal in terms of certain optimization targets\(Jainet al\.,[2020](https://arxiv.org/html/2608.07524#bib.bib52); Sunet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib40); Yuanet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib47)\)\.

### 5\.3Dynamic Parallelisms for Variable Sequence Length

Variable sequence length emerges as a new topic for large language models\. HotSPa addresses this during training by dynamically switching parallelism degrees \(data, sequence, and pipeline\) based on runtime sequence length\(Geet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib53)\), while Tenplex focuses on elasticity by supporting dynamic parallelism updates when GPU counts change\(Wagenländeret al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib54)\)\. Both prioritize reducing communication overhead from moving model parameters and optimizer states\. On the inference side, LoongServe applies elastic sequence parallelism, similarly minimizing overhead from KV cache movement during parallelism changes\(Wuet al\.,[2024](https://arxiv.org/html/2608.07524#bib.bib55)\)\. Despite these efforts, variable sequence length in transformer model training remains underexplored\.

## 6Conclusion

This paper introduces Data\-Centric Parallel \(DCP\), a framework that resolves the trade\-off between efficiency and ease\-of\-use when training deep learning models on variable long sequences\. Unlike traditional methods that force a choice between inefficient static configurations and complex, model\-specific code, DCP breaks this by letting the data itself drive the runtime\. By dynamically adjusting settings like parallel size and gradient accumulation based on each batch’s sequence length, DCP ensures more effective hardware utilization\. Our empirical results demonstrate up to a 2\.88x speedup on 32 H200 GPUs\. A key advantage is its simplicity; DCP can be integrated into any model with only ten lines of code, promoting wide adoption\. We believe this effective method will serve as a robust baseline for distributed training and facilitate future advancements in this area\.

#### Limitation\.

This method is currently subject to two primary limitations\. First, its application is restricted to Transformer\-based models\. Extending DCP to other architectures would require new schedulers and profiling strategies\. Second, it’s designed for a single model architecture and cannot be applied to systems of multiple, distinct networks\.

#### Future works\.

Future work could enhance this method by developing predictive models to proactively select a wider range of dynamic runtime parameters, moving beyond the current profiling based adjustments\. Furthermore, exploring finer\-grained, intra\-batch parallelism could offer more efficiency with highly diverse sequence lengths, further minimizing padding\-related overhead\. The scope of DCP can also be broadened to other architectures\.

## References

- J\. Bai, S\. Bai, Y\. Chu, Z\. Cui, K\. Dang, X\. Deng, Y\. Fan, W\. Ge, Y\. Han, F\. Huang,et al\.\(2023\)Qwen technical report\.arXiv preprint arXiv:2309\.16609\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1)\.
- T\. Brooks, B\. Peebles, C\. Holmes, W\. DePue, Y\. Guo, L\. Jing, D\. Schnurr, J\. Taylor, T\. Luhman, E\. Luhman, C\. Ng, R\. Wang, and A\. Ramesh \(2024\)Video generation models as world simulators\.External Links:[Link](https://openai.com/research/video-generation-models-as-world-simulators)Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1),[§2\.1](https://arxiv.org/html/2608.07524#S2.SS1.p2.1)\.
- T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell,et al\.\(2020\)Language models are few\-shot learners\.Advances in neural information processing systems33,pp\. 1877–1901\.Cited by:[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px1.p1.1)\.
- T\. Chen, B\. Xu, C\. Zhang, and C\. Guestrin \(2016\)Training deep nets with sublinear memory cost\.arXiv preprint arXiv:1604\.06174\.Cited by:[§5\.2](https://arxiv.org/html/2608.07524#S5.SS2.p1.1)\.
- T\. Chen, A\. Siarohin, W\. Menapace, E\. Deyneka, H\. Chao, B\. E\. Jeon, Y\. Fang, H\. Lee, J\. Ren, M\. Yang,et al\.\(2024a\)Panda\-70m: captioning 70m videos with multiple cross\-modality teachers\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13320–13331\.Cited by:[§2\.1](https://arxiv.org/html/2608.07524#S2.SS1.p1.1)\.
- Z\. Chen, J\. Wu, W\. Wang, W\. Su, G\. Chen, S\. Xing, M\. Zhong, Q\. Zhang, X\. Zhu, L\. Lu,et al\.\(2024b\)Internvl: scaling up vision foundation models and aligning for generic visual\-linguistic tasks\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 24185–24198\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1)\.
- T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)Flashattention: fast and memory\-efficient exact attention with io\-awareness\.Advances in Neural Information Processing Systems35,pp\. 16344–16359\.Cited by:[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px5.p1.1)\.
- M\. Dehghani, B\. Mustafa, J\. Djolonga, J\. Heek, M\. Minderer, M\. Caron, A\. Steiner, J\. Puigcerver, R\. Geirhos, I\. M\. Alabdulmohsin,et al\.\(2023\)Patch n’pack: navit, a vision transformer for any aspect ratio and resolution\.Advances in Neural Information Processing Systems36,pp\. 2252–2274\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p5.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px1.p1.1)\.
- P\. Esser, S\. Kulal, A\. Blattmann, R\. Entezari, J\. Müller, H\. Saini, Y\. Levi, D\. Lorenz, A\. Sauer, F\. Boesel,et al\.\(2024\)Scaling rectified flow transformers for high\-resolution image synthesis\.InForty\-first international conference on machine learning,Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1),[§1](https://arxiv.org/html/2608.07524#S1.p3.1),[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px3.p1.1)\.
- J\. Fang and S\. Zhao \(2024\)A unified sequence parallelism approach for long context generative ai\.arXiv preprint arXiv:2405\.07719\.Cited by:[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px2.p1.1)\.
- H\. Ge, F\. Fu, H\. Li, X\. Wang, S\. Lin, Y\. Wang, X\. Nie, H\. Zhang, X\. Miao, and B\. Cui \(2024\)Enabling parallelism hot switching for efficient training of large language models\.InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles,pp\. 178–194\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p5.1),[§5\.3](https://arxiv.org/html/2608.07524#S5.SS3.p1.1)\.
- S\. A\. Jacobs, M\. Tanaka, C\. Zhang, M\. Zhang, R\. Y\. Aminadabi, S\. L\. Song, S\. Rajbhandari, and Y\. He \(2024\)System optimizations for enabling training of extreme long sequence transformer models\.InProceedings of the 43rd ACM Symposium on Principles of Distributed Computing,pp\. 121–130\.Cited by:[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px5.p1.1),[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px2.p1.1)\.
- P\. Jain, A\. Jain, A\. Nrusimha, A\. Gholami, P\. Abbeel, J\. Gonzalez, K\. Keutzer, and I\. Stoica \(2020\)Checkmate: breaking the memory wall with optimal tensor rematerialization\.Proceedings of Machine Learning and Systems2,pp\. 497–511\.Cited by:[§5\.2](https://arxiv.org/html/2608.07524#S5.SS2.p1.1)\.
- J\. Jumper, R\. Evans, A\. Pritzel, T\. Green, M\. Figurnov, O\. Ronneberger, K\. Tunyasuvunakool, R\. Bates, A\. Žídek, A\. Potapenko,et al\.\(2021\)Highly accurate protein structure prediction with alphafold\.nature596\(7873\),pp\. 583–589\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px1.p1.1)\.
- W\. Kong, Q\. Tian, Z\. Zhang, R\. Min, Z\. Dai, J\. Zhou, J\. Xiong, X\. Li, B\. Wu, J\. Zhang,et al\.\(2024\)HunyuanVideo: a systematic framework for large video generative models\.arXiv preprint arXiv:2412\.03603\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1)\.
- V\. A\. Korthikanti, J\. Casper, S\. Lym, L\. McAfee, M\. Andersch, M\. Shoeybi, and B\. Catanzaro \(2023\)Reducing activation recomputation in large transformer models\.Proceedings of Machine Learning and Systems5,pp\. 341–353\.Cited by:[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px2.p1.1),[§5\.2](https://arxiv.org/html/2608.07524#S5.SS2.p1.1)\.
- S\. Li, F\. Xue, C\. Baranwal, Y\. Li, and Y\. You \(2023\)Sequence parallelism: long sequence training from system perspective\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 2391–2404\.Cited by:[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px2.p1.1)\.
- Y\. Luo, X\. Ren, Z\. Zheng, Z\. Jiang, X\. Jiang, and Y\. You \(2023\)Came: confidence\-guided adaptive memory efficient optimization\.arXiv preprint arXiv:2307\.02047\.Cited by:[Appendix C](https://arxiv.org/html/2608.07524#A3.p3.2)\.
- D\. Narayanan, M\. Shoeybi, J\. Casper, P\. LeGresley, M\. Patwary, V\. Korthikanti, D\. Vainbrand, P\. Kashinkunti, J\. Bernauer, B\. Catanzaro,et al\.\(2021\)Efficient large\-scale language model training on gpu clusters using megatron\-lm\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–15\.Cited by:[§5\.2](https://arxiv.org/html/2608.07524#S5.SS2.p1.1)\.
- A\. Polyak, A\. Zohar, A\. Brown, A\. Tjandra, A\. Sinha, A\. Lee, A\. Vyas, B\. Shi, C\. Ma, C\. Chuang,et al\.\(2024\)Movie gen: a cast of media foundation models\.arXiv preprint arXiv:2410\.13720\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1)\.
- S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. He \(2020\)Zero: memory optimizations toward training trillion parameter models\.InSC20: International Conference for High Performance Computing, Networking, Storage and Analysis,pp\. 1–16\.Cited by:[§3\.4](https://arxiv.org/html/2608.07524#S3.SS4.p3.4),[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px5.p1.1),[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px1.p1.1)\.
- Z\. Sun, H\. Cao, Y\. Wang, G\. Feng, S\. Chen, H\. Wang, and W\. Chen \(2024\)AdaPipe: optimizing pipeline parallelism with adaptive recomputation and partitioning\.InProceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3,pp\. 86–100\.Cited by:[§5\.2](https://arxiv.org/html/2608.07524#S5.SS2.p1.1)\.
- H\. Touvron, T\. Lavril, G\. Izacard, X\. Martinet, M\. Lachaux, T\. Lacroix, B\. Rozière, N\. Goyal, E\. Hambro, F\. Azhar,et al\.\(2023\)Llama: open and efficient foundation language models\.arXiv preprint arXiv:2302\.13971\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1),[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px1.p1.1)\.
- M\. Wagenländer, G\. Li, B\. Zhao, L\. Mai, and P\. Pietzuch \(2024\)Tenplex: dynamic parallelism for deep learning using parallelizable tensor collections\.InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles,pp\. 195–210\.Cited by:[§5\.3](https://arxiv.org/html/2608.07524#S5.SS3.p1.1)\.
- D\. Waltz \(1984\)A theory of the learnable\.Communications of the ACM27\(11\)\.Cited by:[§2\.1](https://arxiv.org/html/2608.07524#S2.SS1.p2.1)\.
- P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1)\.
- B\. Wu, S\. Liu, Y\. Zhong, P\. Sun, X\. Liu, and X\. Jin \(2024\)Loongserve: efficiently serving long\-context large language models with elastic sequence parallelism\.InProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles,pp\. 640–654\.Cited by:[§5\.3](https://arxiv.org/html/2608.07524#S5.SS3.p1.1)\.
- E\. P\. Xing, Q\. Ho, W\. Dai, J\. Kim, J\. Wei, S\. Lee, X\. Zheng, P\. Xie, A\. Kumar, and Y\. Yu \(2015\)Petuum: a new platform for distributed machine learning on big data\.InProceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,pp\. 1335–1344\.Cited by:[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px1.p1.1)\.
- Z\. Yang, J\. Teng, W\. Zheng, M\. Ding, S\. Huang, J\. Xu, Y\. Yang, W\. Hong, X\. Zhang, G\. Feng,et al\.\(2024\)Cogvideox: text\-to\-video diffusion models with an expert transformer\.arXiv preprint arXiv:2408\.06072\.Cited by:[§5\.1](https://arxiv.org/html/2608.07524#S5.SS1.SSS0.Px1.p1.1)\.
- Y\. You, J\. Li, S\. Reddi, J\. Hseu, S\. Kumar, S\. Bhojanapalli, X\. Song, J\. Demmel, K\. Keutzer, and C\. Hsieh \(2020\)Large batch optimization for deep learning: training bert in 76 minutes\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Syx4wnEtvH)Cited by:[Appendix C](https://arxiv.org/html/2608.07524#A3.p3.2)\.
- T\. Yuan, Y\. Liu, X\. Ye, S\. Zhang, J\. Tan, B\. Chen, C\. Song, and D\. Zhang \(2024\)Accelerating the training of large language models using efficient activation rematerialization and optimal hybrid parallelism\.In2024 USENIX Annual Technical Conference \(USENIX ATC 24\),pp\. 545–561\.Cited by:[§3\.4](https://arxiv.org/html/2608.07524#S3.SS4.p1.1),[§3\.4](https://arxiv.org/html/2608.07524#S3.SS4.p3.4),[§5\.2](https://arxiv.org/html/2608.07524#S5.SS2.p1.1)\.
- Z\. Zheng, X\. Peng, T\. Yang, C\. Shen, S\. Li, H\. Liu, Y\. Zhou, T\. Li, and Y\. You \(2024\)Open\-sora: democratizing efficient video production for all\.arXiv preprint arXiv:2412\.20404\.Cited by:[§1](https://arxiv.org/html/2608.07524#S1.p1.1),[§4\.1](https://arxiv.org/html/2608.07524#S4.SS1.SSS0.Px1.p1.1)\.

Training Variable Long Sequences with Data\-Centric Parallel

Appendix

We organize our appendix as follows:

- •Section[A](https://arxiv.org/html/2608.07524#A1): Dataset configuration\.
- •Section[B](https://arxiv.org/html/2608.07524#A2): Model settings\.
- •Section[C](https://arxiv.org/html/2608.07524#A3): Implementation details of DCP\-inter\.
- •Section[D](https://arxiv.org/html/2608.07524#A4): API usage\.

## Appendix ADataset Configuration

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/seq_length_distribution_logx_short.png)Figure 11:Distribution of short sequences dominated dataset\.![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/seq_length_distribution_logx_balance.png)Figure 12:Distribution of balanced dataset\.![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/seq_length_distribution_logx_long.png)Figure 13:Distribution of long sequences dominated dataset\.The sequence length distributions for our three evaluation datasets—short\-sequence\-dominated, balanced, and long\-sequence\-dominated—are visualized in Figures[11](https://arxiv.org/html/2608.07524#A1.F11),[12](https://arxiv.org/html/2608.07524#A1.F12), and[13](https://arxiv.org/html/2608.07524#A1.F13), respectively\. We define dataset dominance by computational cost, not by the raw count of sequences\. Consequently, in the long\-sequence\-dominated dataset, the running time contributed by a few long sequences is orders of magnitude greater than that of the more numerous short sequences, thereby dictating the overall training time\.

## Appendix BModel settings

Table 1:Model settings of Transformer\-1D and Transformer\-2D\.
## Appendix CImplementation Details of DCP\-Inter

There are two additional considerations for DCP\-inter when setting the number of gradient accumulation steps\.

First, as this search procedure should run for each training iteration at runtime, it should be lightweight and fast to ensure not slow down the actual training process\.

Second, while extremely large number of accumulation steps can make the workload imbalance negligible, this will also leads to unstable convergence speed and longer training time\[Youet al\.,[2020](https://arxiv.org/html/2608.07524#bib.bib46), Luoet al\.,[2023](https://arxiv.org/html/2608.07524#bib.bib15)\]\. Therefore, DCP\-inter limitsgig\_\{i\}to a range\[Gm​i​n,Gm​a​x\]\[G\_\{min\},G\_\{max\}\]to relieve the two problems, detailed described

Gm​a​xG\_\{max\}: Within an iteration, find the batch with the longest processing time\. The maximum time for this iteration is calculated by multiplying the duration of that batch by this value\.

Gm​i​nG\_\{min\}: If the gas for every batch within an iteration is less than this value, then the gas for each batch is scaled up by a factor of \[Gm​i​nG\_\{min\}/ gas\]\.

## Appendix DAPI Usage

An example to demonstrate that our method can be used within 10 lines of code is illustrated in Figure[14](https://arxiv.org/html/2608.07524#A4.F14)\.

![Refer to caption](https://arxiv.org/html/2608.07524v1/figs/code.png)Figure 14:The API usage to enable our methods within 10 lines of code\.