Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
Summary
This paper introduces DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that uses generation-axis pipeline parallelism and trainer-assisted generation to improve throughput by 1.56-2.10x over existing systems.
arXiv:2606.24369v1 Announce Type: new
Abstract: Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs). In parallel, diffusion-oriented RL algorithms, e.g., DanceGRPO and FlowGRPO, have rapidly expanded the scope of RL from language reasoning to diffusion-based visual and flow-based generation. However, efficient RL systems for diffusion generative LLMs remain underexplored. Existing implementations, e.g., veRL-Omni, still rely on colocated execution, which simplifies synchronization but couples rollout and training resources, limits heterogeneous deployment, and constrains independent scaling.
To this end, we introduce DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that supports flexible resource allocation, accommodates heterogeneous GPUs, and facilitates efficient task scheduling. To maximally reduce the execution bubbles in the disaggregated architecture, we propose: 1) a generation-axis pipeline (GAP) and time-step parallelism (TSP) in the diffusion architecture to enable finer-grained pipelining between rollout and training; 2) an elastic trainer-assisted generation (TAG) approach to enable the trainer GPU resources to dynamically assist in executing rollout generations; and 3) a tightly one-step constrained asynchronous strategy to further utilize the tail bubble in the pipeline. Extensive experiments are conducted on three hardware testbeds with 16-32 GPUs using HunyuanVideo-13B, Wan2.1-14B, FLUX.1-12B, and QwenImage-20B generative models. Experimental results show that DigenRL achieves 1.56-2.10x throughput improvements over state-of-the-art diffusion RL systems, veRL-Omni and GenRL.
View Cached Full Text
Cached at: 06/24/26, 07:46 AM
# Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation Source: [https://arxiv.org/html/2606.24369](https://arxiv.org/html/2606.24369) Sijie WangSchool of Computer Science and Technology, Harbin Institute of Technology, Shenzhen[25b951105@stu\.hit\.edu\.cn](https://arxiv.org/html/2606.24369v1/mailto:[email protected])Zhengyu QingSchool of Computer Science and Technology, Harbin Institute of Technology, Shenzhen[210110609@stu\.hit\.edu\.cn](https://arxiv.org/html/2606.24369v1/mailto:[email protected]),Zhiqiang TanSchool of Computer Science and Technology, Harbin Institute of Technology, Shenzhen[25B951103@stu\.hit\.edu\.cn](https://arxiv.org/html/2606.24369v1/mailto:[email protected]),Yiming YinData Science and Analytics Thrust, The Hong Kong University of Science and Technology \(Guangzhou\)[yyin464@connect\.hkust\-gz\.edu\.cn](https://arxiv.org/html/2606.24369v1/mailto:[email protected]),Yeqing ZhangHuawei Technologies Ltd\.[zhangyeqing@huawei\.com](https://arxiv.org/html/2606.24369v1/mailto:[email protected]),Yaoyuan WangHuawei Technologies Ltd\.[wangyaoyuan1@huawei\.com](https://arxiv.org/html/2606.24369v1/mailto:[email protected]),Qiang WangSchool of Computer Science and Technology, Harbin Institute of Technology, Shenzhen[qiang\.wang@hit\.edu\.cn](https://arxiv.org/html/2606.24369v1/mailto:[email protected]),Xiaowen ChuData Science and Analytics Thrust, The Hong Kong University of Science and Technology \(Guangzhou\)[xwchu@ust\.hk](https://arxiv.org/html/2606.24369v1/mailto:[email protected])andShaohuai ShiSchool of Computer Science and Technology, Harbin Institute of Technology, Shenzhen[shaohuais@hit\.edu\.cn](https://arxiv.org/html/2606.24369v1/mailto:[email protected]) ###### Abstract\. Reinforcement learning \(RL\) has become a dominant post\-training paradigm, driving the emergence of high\-performance RL systems such as veRL for autoregressive large language models \(LLMs\)\. In parallel, diffusion\-oriented RL algorithms \(e\.g\., DanceGRPO and FlowGRPO\), have rapidly expanded the scope of RL from language reasoning to diffusion\-based visual and flow\-based generation\. However, efficient RL systems for diffusion generative LLMs remain underexplored\. Existing implementations \(e\.g\., veRL\-Omni\) still rely on colocated execution, which simplifies synchronization but couples rollout and training resources, limits heterogeneous deployment and constrains independent scaling\. To this end, we introduce DigenRL, a disaggregated RL framework for diffusion\-based generative LLMs that supports flexible resource allocation, accommodates heterogeneous GPUs, and facilitates efficient task scheduling\. To maximally reduce the execution bubbles in the disaggregated architecture, we propose 1\) a generation\-axis pipeline \(GAP\) and time\-step parallelism \(TSP\) in the diffusion architecture to enable finer\-grained pipelining between rollout and training, 2\) an elastic trainer\-assisted generation \(TAG\) approach to enable the trainer GPU resources to dynamically assist in executing rollout generations, and 3\) a tightly one\-step constrained asynchronous strategy to further utilize the tail bubble in the pipeline\. Extensive experiments are conducted on three hardware testbeds with 16\-32 GPUs using HunyuanVideo\-13B, Wan2\.1\-14B, FLUX\.1\-12B, QwenImage\-20B generative models\. Experimental results show that DigenRL achieves 1\.56\-2\.10×\\timesthroughput improvements over state\-of\-the\-art diffusion RL systems, veRL\-Omni and GenRL\. diffusion reinforcement learning, disaggregated post\-training, pipeline scheduling, resource scheduling ††copyright:none††ccs:Computing methodologies Reinforcement learning††ccs:Computer systems organization Distributed architectures††ccs:Software and its engineering Scheduling## 1\.Introduction Table 1\.Comparison of RL frameworks for diffusion models\.FeatureveRLGenRLveRL\-OmniDigenRLDiffusion\-RL×\\times✓\\checkmark✓\\checkmark✓\\checkmarkMulti\-model Support×\\timesPartialPartial✓\\checkmarkAlgorithm Extensibility✓\\checkmark✓\\checkmarkLimited✓\\checkmarkDisaggregated TrainingPartial×\\times×\\times✓\\checkmarkAsynchronous RL✓\\checkmark×\\times×\\times✓\\checkmarkReinforcement learning \(RL\) has re\-emerged as one of the central paradigms for post\-training large language models \(LLMs\)\(Ouyanget al\.,[2022](https://arxiv.org/html/2606.24369#bib.bib14); Guoet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib37)\)\. In preference alignment, complex reasoning, and agentic training, RL typically boosts final model quality using proximal policy optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.24369#bib.bib13)\), group relative policy optimization \(GRPO\)\(DeepSeek\-AI,[2025](https://arxiv.org/html/2606.24369#bib.bib33)\), or its variants \(e\.g\., DanceGRPO\(Xueet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib8)\), FlowGRPO\(Liuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib7)\)\)\. The basic workflow of RL for LLMs proceeds iteratively through: 1\) sampling with the current policy model \(generator or rollout\), 2\) evaluating the sampled outputs \(reward model or critic\), and 3\) updating the policy based on these samples \(actor update\)\. The latter two steps are often grouped into a single trainer component, which computes rewards and losses to adjust the policy, after which the updated policy is synchronized with the generator\. As a result, the generator and trainer share strong data dependencies, demand significant computation, and in practice often become the primary performance bottleneck\. This has motivated the development of high\-performance RL systems such as DeepSpeed\-Chat\(Yaoet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib38)\), veRL\(Shenget al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib3)\), AReaL\(Fuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib5)\), StreamRL\(Zhonget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib4)\), OpenRLHF\(Huet al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib15)\), LlamaRL\(Wuet al\.,[2025a](https://arxiv.org/html/2606.24369#bib.bib16)\), ROLL\(Wanget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib17)\), etc\. These systems illustrate that the training throughput of RL post\-training is influenced not only by the choice of learning algorithms \(e\.g\., PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.24369#bib.bib13)\), GRPO\(Shaoet al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib34)\), etc\.\), but also strongly by the way resources are organized and tasks are scheduled within the system\. Recent RL systems \(e\.g\., AReaL\(Fuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib5)\), StreamRL\(Zhonget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib4)\), and Laminar\(Shenget al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib35)\), OrchestrRL\(Tanet al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib40)\), WEAVE\(Wanget al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib54)\)\) tend to exploit a disaggregated architecture due to its flexibility for resource allocation, asynchronous rollouts, and long\-tail generation repacking, which are particularly effective for autoregressive LLM post\-training \(AR\-RL\)\. However, for the RL post\-training of diffusion\-based visual generative models \(Diffusion\-RL\)\(Xueet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib8); Liuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib7)\), current optimized systems \(e\.g\., Long\-RL\(Chenet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib27)\), GenRL\(ModelTC,[2025](https://arxiv.org/html/2606.24369#bib.bib28)\)and veRL\-Omni\(verl\-omni Contributors,[2026](https://arxiv.org/html/2606.24369#bib.bib24)\)\) still adopt a colocated architecture\. This may be because 1\) rollout workers have relatively uniform workloads, and 2\) rollout and training share the same backend \(e\.g\., PyTorch\-FSDP\(Zhaoet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib36)\)\), avoiding efficiency issues\. Yet, we argue that Diffusion\-RL still needs a disaggregated architecture to enable more fine\-grained parallelism and task scheduling, thereby making better use of hardware resources and eventually increasing overall system throughput\. However, disaggregated Diffusion\-RL naturally incurs substantial pipeline bubbles, as shown in Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(b\)\(c\), because the trainer and generator must alternatively wait for one another, leaving compute devices underutilized and prolonging the overall training process\. As illustrated in Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(b\), the disaggregated architecture introduces additional bubbles relative to the collocated architecture in Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(a\)\. Therefore, to enable disaggregated Diffusion\-RL to fully exploit GPU resources during training, it is necessary to design fine\-grained pipeline scheduling strategies that minimize these bubbles as much as possible\. In this paper, we introduce DigenRL, a disaggregated Diffusion\-RL system for improving post\-training throughput\. To achieve this goal, we propose the following novel techniques\. First, we design a diffusion\-based parallelism \(§[3\.2](https://arxiv.org/html/2606.24369#S3.SS2)\) according to the diffusion inference and training characteristics with 1\)*generation\-axis pipeline \(GAP\)*execution in the generator and 2\)*time\-step parallelism \(TSP\)*in the trainer\. GAP and TSP together support finer\-grained generation and training task pipelining and reduce the bubble time\. Second, to further reduce the bubbles when the trainer waits for the generator, we propose*elastic trainer\-assisted generation \(TAG\)*to allow the trainer to execute part of the micro\-batch generation workloads when it would otherwise be idle, as shown in Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(d\)\. Third, to better exploit the final micro\-batch bubble, which in theory cannot be removed in synchronous Diffusion\-RL, we propose a*trajectory\-consistent stale synchronization \(TCSS\)*scheme\. TCSS allows the generator to progress by at most one policy version while ensuring that each denoising trajectory is produced entirely under a single, fixed policy snapshot\. The feature comparison of our DigenRL with existing RL systems is shown in Table[1](https://arxiv.org/html/2606.24369#S1.T1)\. We conduct extensive experiments compared with state\-of\-the\-art Diffusion\-RL systems \(veRL\-Omni\(verl\-omni Contributors,[2026](https://arxiv.org/html/2606.24369#bib.bib24)\)and GenRL\(ModelTC,[2025](https://arxiv.org/html/2606.24369#bib.bib28)\)\) using various visual generative models \(HunyuanVideo\-13B\(Konget al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib20)\), Wan2\.1\-14B\(Wanet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib21)\), FLUX\.1\-12B\(Black Forest Labs,[2024](https://arxiv.org/html/2606.24369#bib.bib25)\), QwenImage\-20B\(Wuet al\.,[2025b](https://arxiv.org/html/2606.24369#bib.bib22)\)\) on two homogeneous 32\-GPU testbeds and one heterogeneous 16\-GPU testbed\. Experimental results \(§[5](https://arxiv.org/html/2606.24369#S5)\) show that our DigenRL achieves1\.56×1\.56\\times–2\.1×2\.1\\timesspeedups over veRL\-Omni and GenRL\. Figure 1\.Comparison of four execution modes\. We omit the reward stage in the figures as it occupies only a small fraction of the total time\. \(a\) Colocated execution: all compute resources are assigned to the one stage at a time, with workers sharing model states and data\. \(b\) Naive disaggregated execution: the resources are partitioned to two groups \(Generator and Trainer\), which suffers from strong inter\-stage dependencies\. \(c\) Disaggregated execution with micro\-batch pipelining: a batch of data is split to multiple micro\-batches to enable the generator and the trainer to be executed in a pipeline manner\. \(d\) Our proposed Trainer\-Assisted Generation \(TAG\): pipeline bubbles can be reduced by allowing idle trainer resources to participate in generation\.A four\-panel timeline compares colocated execution, naive disaggregation, micro\-batch pipelining, and TAG, highlighting how each design changes generator and trainer idle time\. ## 2\.Preliminaries and Motivations ### 2\.1\.Diffusion\-RL Workflow Existing Diffusion\-RL algorithms such as DanceGRPO\(Xueet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib8)\)and Flow\-GRPO\(Liuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib7)\)are typically executed in a colocated fashion: the same worker group, or tightly coupled device pool, is responsible for both sample generation and trainer\-side policy updates\. Under this workflow, the generator uses the current policyπk\\pi\_\{k\}to produce a denoising trajectory\(Hoet al\.,[2020](https://arxiv.org/html/2606.24369#bib.bib43); Dhariwal and Nichol,[2021](https://arxiv.org/html/2606.24369#bib.bib44); Lipmanet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib45)\) τi=\{\(xi,t,xi,t−1,ℓi,told\)\}t=1T,xi,t−1∼πk\(⋅∣xi,t,ci\),\\tau\_\{i\}=\\\{\(x\_\{i,t\},x\_\{i,t\-1\},\\ell^\{\\text\{old\}\}\_\{i,t\}\)\\\}\_\{t=1\}^\{T\},\\qquad x\_\{i,t\-1\}\\sim\\pi\_\{k\}\(\\cdot\\mid x\_\{i,t\},c\_\{i\}\),wherexi,tx\_\{i,t\}is the latent at the denoising steptt,cic\_\{i\}is the condition or prompt, and ℓi,told=logπk\(xi,t−1∣xi,t,ci\)\\ell^\{\\text\{old\}\}\_\{i,t\}=\\log\\pi\_\{k\}\(x\_\{i,t\-1\}\\mid x\_\{i,t\},c\_\{i\}\)is the step\-level log\-probability recorded during rollout\. After the final samplexi,0x\_\{i,0\}is decoded by decoder\(Kingma and Welling,[2013](https://arxiv.org/html/2606.24369#bib.bib49); van den Oordet al\.,[2017](https://arxiv.org/html/2606.24369#bib.bib50)\), a reward model\(Xuet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib52); Kirstainet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib53)\)evaluates its final reward ri=R\(xi,0,ci\)\.r\_\{i\}=R\(x\_\{i,0\},c\_\{i\}\)\.Then, the trainer consumes these trajectories as one training micro\-batch\. In recent Diffusion\-RL algorithms like DanceGPRO and Flow\-GRPO\(Xueet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib8); Liuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib7)\), final rewards are first converted into advantages before policy optimization\. For example, for a group of samples generated from the same prompt, one may use \(1\)A^i=ri−μgσg\+ϵ,\\hat\{A\}\_\{i\}=\\frac\{r\_\{i\}\-\\mu\_\{g\}\}\{\\sigma\_\{g\}\+\\epsilon\},whereμg\\mu\_\{g\}andσg\\sigma\_\{g\}denote the mean and standard deviation of rewards within the group\. Figure 2\.Comparison between per\-timestep loss computation and our proposed time\-step parallelism \(TSP\)\.The figure contrasts separate per\-timestep loss computation with timestep\-batched training, where selected timesteps are grouped into a larger parallel training unit\.The trainer then recomputes the step\-level log\-probability under the current policy and forms an importance ratio \(2\)ρi,t\(θ\)=exp\(logπθ\(xi,t−1∣xi,t,ci\)−ℓi,told\)\.\\rho\_\{i,t\}\(\\theta\)=\\exp\\\!\\Big\(\\log\\pi\_\{\\theta\}\(x\_\{i,t\-1\}\\mid x\_\{i,t\},c\_\{i\}\)\-\\ell^\{\\text\{old\}\}\_\{i,t\}\\Big\)\. Policy optimization is typically performed only on a selected subset of denoising steps to reduce training cost\(Xueet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib8); Liuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib7)\)\. Let𝒮i⊆\{1,…,T\}\\mathcal\{S\}\_\{i\}\\subseteq\\\{1,\\dots,T\\\}denote the selected training timesteps for sampleii\. A PPO/GRPO\-style clipped objective can then be written as ℒpg\\displaystyle\\mathcal\{L\}\_\{\\text\{pg\}\}=−1∑i\|𝒮i\|∑i∑t∈𝒮iℒi,t,\\displaystyle=\-\\frac\{1\}\{\\sum\_\{i\}\|\\mathcal\{S\}\_\{i\}\|\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{S\}\_\{i\}\}\\mathcal\{L\}\_\{i,t\},ℒi,t\\displaystyle\\mathcal\{L\}\_\{i,t\}=min\(ρi,t\(θ\)A^i,clip\(ρi,t\(θ\),1−ϵ,1\+ϵ\)A^i\)\.\\displaystyle=\\min\\\!\\Big\(\\rho\_\{i,t\}\(\\theta\)\\hat\{A\}\_\{i\},\\;\\operatorname\{clip\}\\\!\\big\(\\rho\_\{i,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\big\)\\hat\{A\}\_\{i\}\\Big\)\.Under a fixed policy snapshot and a fixed collected trajectory batch, this objective is additive over the selected timesteps\. If we define the per\-step loss term asℓi,t\(θ\)\\ell\_\{i,t\}\(\\theta\), then ℒpg=1Z∑i∑t∈𝒮iℓi,t\(θ\),∇θℒpg=1Z∑i∑t∈𝒮i∇θℓi,t\(θ\),\\mathcal\{L\}\_\{\\text\{pg\}\}=\\frac\{1\}\{Z\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{S\}\_\{i\}\}\\ell\_\{i,t\}\(\\theta\),\\qquad\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{\\text\{pg\}\}=\\frac\{1\}\{Z\}\\sum\_\{i\}\\sum\_\{t\\in\\mathcal\{S\}\_\{i\}\}\\nabla\_\{\\theta\}\\ell\_\{i,t\}\(\\theta\),whereZ=∑i\|𝒮i\|Z=\\sum\_\{i\}\|\\mathcal\{S\}\_\{i\}\|\. Therefore, timestep\-parallel training does not assume that denoising decisions are independent\. The workflow is shown in Figure[2](https://arxiv.org/html/2606.24369#S2.F2)\(a\); each time step can only begin after the previous time step has completed\. ### 2\.2\.Disaggregated Diffusion\-RL Figure 3\.Scalability comparison across different stages\.Compared with the colocated architecture, in which the generator and trainer jointly use the entire pool of GPU resources \(Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(a\)\), the disaggregated architecture splits the GPUs into two non\-overlapping sets: a generator group \(GenGroup\) and a trainer group \(TrainGroup\)\. The generator and trainer then take turns executing their workloads \(Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(b\)\)\. During each RL training iteration, the generator produces a micro\-batch of training data, which the trainer consumes to update the model weights; these updated weights are subsequently synchronized back to the generator\. The disaggregated architecture in Diffusion\-RL is useful in scenarios when the workloads of the generator and trainer vary and the GPU resources are heterogeneous\(Shenget al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib35); Zhonget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib4); Wuet al\.,[2026b](https://arxiv.org/html/2606.24369#bib.bib18)\)\. In Diffusion\-RL, the generator still advances sequentially along the denoising chain, while the optimization of the trainer can parallelize computations across multiple selected timesteps, which easily causes workload imbalance between the generator and trainer\. As shown in Figure[3](https://arxiv.org/html/2606.24369#S2.F3), when training FLUX\.1\-12B on a 32\-GPU cluster, the trainer maintains good scalability as the number of GPUs increases because it is compute\-bound\. In contrast, the generator needs to process timestep generation sequentially, so it often fails to fully utilize all available compute resources as the resource scale grows\. ### 2\.3\.Motivations As shown in Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(c\), naively pipelining generation tasks and training tasks in Diffusion\-RL with multiple micro\-batches still causes many bubbles\. First, the nature of data dependencies between the generator and trainer easily causes pipeline bubbles when their execution times are not comparable\. Although one can assign different numbers of GPUs to the generator and trainer based on their respective workloads to balance their execution times, this is difficult in practice because RL algorithms such as DanceGRPO and Flow\-GRPO involve hyperparameters \(e\.g\., the time\-step fraction for training\) that influence the wall\-clock time of both generation and training\. On the other hand, when Diffusion\-RL is executed on a heterogeneous GPU cluster, its runtime performance can fluctuate because of variations in hardware capabilities, including compute power, memory capacity/bandwidth, and interconnect bandwidth/latency\. Thus, we propose \(1\) leveraging the computational properties of diffusion models during both inference and training to shorten bubble time, and \(2\) adaptively using the trainer’s bubble time to support the generation process\. Second, because synchronous Diffusion\-RL inherently incurs a bubble at the final micro\-batch, this last idle period is theoretically unavoidable\. This observation motivates us to decouple generation and training into an asynchronous execution scheme, so that the remaining bubble time can be further utilized while still maintaining convergence guarantees\. ## 3\.DigenRL: System Design Figure 4\.The system overview of DigenRL\.A system diagram presents DigenRL with separate generator and trainer workers, queues between stages, and scheduling modules for pipelining, TAG, and stale synchronization\.### 3\.1\.System overview The overall system design of DigenRL is shown in Figure[4](https://arxiv.org/html/2606.24369#S3.F4)\. For any Diffusion\-RL post\-training workload executed on a GPU cluster, DigenRL allocates distinct GPUs for the generator and the trainer\. Given user\-specified mini\-batch and micro\-batch sizes, DigenRL leverages the generation\-axis pipeline \(GAP\) module to pipeline generator and trainer operations along the generation dimension, thereby achieving finer\-grained task pipelining\. In the training phase, DigenRL adopts timestep parallelism \(TSP\) rather than conventional data parallelism to boost GPU utilization\. During runtime, the trainer\-assisted generation \(TAG\) module adaptively decides when the trainer should assist the generator with rollouts, aiming to shorten the generation time under a fixed mini\-batch budget\. Finally, when the one\-step off\-policy option is enabled, our TCSS leverages the last microbatch bubble to produce samples with at most a one\-step stale policy, further enhancing system throughput\. ### 3\.2\.Diffusion\-based Parallelism Figure 5\.Comparison of different micro\-batch partitioning strategies\. \(a\) Partitioning along the batch dimension\. \(b\) DigenRL partitions work along the generation dimension, producing more fine\-grained micro\-batches and enabling better overlap\.Two partitioning layouts contrast coarse batch\-dimension splits with finer generation\-dimension micro\-batches used by DigenRL\.#### 3\.2\.1\.Generation\-Axis Pipeline \(GAP\) Existing AR RL systems form micro\-batches mainly along the prompt/sequence dimension and balance them by token length\. This is effective for AR workloads, where RL training batches often contain hundreds or thousands of samples\. In Diffusion RL, however, a train batch often contains only dozens, or even tens, of samples\. Naively splitting along the batch dimension therefore produces too few pipeline units under small\-batch Diffusion RL settings, as shown in Figure[5](https://arxiv.org/html/2606.24369#S3.F5)\(a\)\. DigenRL therefore introduces GAP, a micro\-batch formation strategy that partitions work along the generation axis, creating finer pipeline units while preserving the semantics of group reward normalization and policy update\. As illustrated in Figure[5](https://arxiv.org/html/2606.24369#S3.F5), panel \(a\) shows the mainstream partitioning method, which splits work along the batch dimension\. Under this scheme, different GPUs sample different prompts, and reward and advantage can only be computed after all samples for one prompt have completed generation\. In the example shown in the figure, there are two GPUs, the training batch size is four, and each prompt is sampled twice\. Under this setup, the system can be partitioned into at most two micro\-batches\. GAP instead partitions work along the sampling dimension, as illustrated in Figure[5](https://arxiv.org/html/2606.24369#S3.F5)\(b\)\. Different GPUs sample the same prompt, and once all GPUs complete sampling, reward and advantage can be computed and used for training\. Under the same configuration, this strategy yields four micro\-batches\. More micro\-batches enable finer\-grained pipeline scheduling and more overlap\. Our method also does not introduce numerical error\. Before one training batch finishes all generation and training work, model weights are not updated; only gradient accumulation is performed\. Since gradients are vectors, changing the order of accumulation does not alter the final result\. For example, in Figure[5](https://arxiv.org/html/2606.24369#S3.F5), panel \(a\) accumulates the gradient of Prompt 1, Sample 0 at the second step, whereas panel \(b\) accumulates the gradient of Prompt 0, Sample 1 at the second step\. Although the accumulation order changes, the final gradient remains unchanged\. #### 3\.2\.2\.Timestep Parallelism \(TSP\) Building on GAP, we further introduce Timestep Pipeline \(TSP\)\. As discussed in Section[2\.1](https://arxiv.org/html/2606.24369#S2.SS1), the latent, next latent, and other per\-timestep context are already known during training\. Thus, once the rollout trajectory is fixed, the trainer can form a timestep batch, evaluate multiple selected timesteps in parallel, and aggregate them into the same update as shown in Figure[2](https://arxiv.org/html/2606.24369#S2.F2)\(b\)\. This is theoretically equivalent to sequentially accumulating the same per\-step losses under the same policy snapshot\. This consideration becomes even more critical in sharded training setups like FSDP\(Zhaoet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib36)\)and ZeRO\(Rajbhandariet al\.,[2020](https://arxiv.org/html/2606.24369#bib.bib41)\), where parameters are split across devices and must be all\-gathered before each layer is computed\. If the chosen timesteps are handled strictly one after another, each all\-gather effectively benefits only a single timestep\. In contrast, grouping several timesteps into a batch allows a single gathered parameter view to be reused for more computation, which amortizes the communication cost and enhances both parallelism and overall hardware utilization\. One might wonder how TSP differs from simply enlarging the training micro\-batch size, since both approaches increase the computation per forward/backward pass to better saturate the hardware\. There are two key distinctions\. First, for a fixed micro\-batch size, TSP offers a higher potential speedup because it exposes parallelism not only across the batch dimension but also across the timestep dimension\. Second, pipeline scheduling inherently balances the number of micro\-batches against their size\. Using smaller micro\-batches increases the number of pipeline units and allows more fine\-grained overlap, but typically leads to poorer hardware utilization and lower compute efficiency\. In practice, one must select a micro\-batch size that trades off overlap efficiency against compute efficiency\. TSP mitigates this trade\-off: even when the generation\-axis micro\-batch size is small, timestep batching maintains high compute efficiency, while the increased number of micro\-batches still supports fine\-grained pipeline overlap\. In combination, GAP and TSP largely eliminate the conflict between overlap efficiency and compute efficiency\. Figure 6\.Impact of different numbers of TAG micro\-batches on end\-to\-end time\.A set of timelines compares using different counts of TAG micro\-batches and shows how too few or too many assisted micro\-batches affect overall latency\. ### 3\.3\.Elastic Trainer\-Assisted Generation \(TAG\) In practical training, generation is usually slower than training\. Each sample typically requires multiple rollouts, whereas the trainer often updates on only a subset of the generated samples and only a subset of their timesteps\. Moreover, our timestep\-parallel training further shortens trainer\-side execution time\. This creates a new inefficiency: as illustrated in Figure[6](https://arxiv.org/html/2606.24369#S3.F6)\(a\), when one micro\-batch finishes training, the next micro\-batch may still be under generation, leaving the trainer idle because no new micro\-batch is yet available for update\. In disaggregated AR RL systems, a common solution is to repartition resources: more resources are assigned to the slower generation stage so that generator and trainer runtimes become closer, thereby increasing overlap\. However, this strategy is less suitable for Diffusion RL\. As discussed earlier, Diffusion RL typically uses much smaller training batches than AR RL, so resource repartitioning can easily create workload imbalance across GPUs\. In practice, resource allocation in Diffusion RL is therefore often constrained by training batch size and cannot be adjusted as freely\. AR RL also suffers from imperfect load balance, but because its batch size is much larger, the imbalance is usually acceptable and does not translate into severe resource waste\. For this reason, we favor a workload\-balanced resource allocation strategy for Diffusion RL, while allowing the trainer to take over part of the generation workload when it would otherwise be idle\. We call this mechanism*Elastic Trainer\-Assisted Generation \(TAG\)*, as illustrated in Figure[6](https://arxiv.org/html/2606.24369#S3.F6)\. Figure 7\.TAG execution workflow\.A workflow timeline shows trainer\-assisted generation filling trainer\-side idle bubbles before normal trainer updates resume\.TAG execution workflow\.A practical RL system must decide how to construct resource pools, and which components should be disaggregated or colocated\. Taking VeRL\-Omni as an example, the system mainly consists of a generator and a reward model backed by vLLM\-Omni, and a trainer backed by FSDP\. After separating the generator from the trainer, we colocate the reward model with the generator in the same resource pool, because the trainer has high memory demand during training and is better suited to occupying an exclusive resource pool\. However, colocating scoring with generation under pipelined execution introduces another issue: dynamically arriving tasks may compete for the same compute resource\. For example, after the trainer finishes a TAG task, the system may request scoring for that generated result\. If the generator is still executing another generation task, the scoring task and generation task would contend for the same resource pool, sharply reducing efficiency\. We therefore designed a buffering and synchronization mechanism to coordinate task execution\. As shown in Figure[7](https://arxiv.org/html/2606.24369#S3.F7), the driver process maintains two queues\. The Scoring Queue \(SQ\) buffers generated images, videos, and other intermediate states waiting for reward evaluation, while the Update Queue \(UQ\) buffers reward\-annotated samples waiting for trainer\-side update\. At the beginning of training, the driver first performs centralized data preprocessing and assigns micro\-batches to the generator and trainer\. Once a micro\-batch finishes generation, it is pushed into SQ\. After each task, the generator checks whether SQ is empty: if it is empty, the generator continues generation; otherwise, it takes images or videos from SQ for scoring\. The scored samples are then pushed into UQ and consumed by the trainer\. This gives scoring higher priority than generation, allowing the trainer to receive rewards earlier and proceed with update more quickly\. As for Trainer, it may perform a context switch depending on the backend configuration before each TAG execution\. For example, if generation and training use different backends, the trainer loads the generation model before TAG and offloads the training model to save memory, then reverses this operation after TAG completes\. After each task, the trainer checks UQ: if UQ is non\-empty, it runs a trainer\-side update; otherwise, the driver may assign a TAG task according to the TAG scheduling policy\. Since each update micro\-batch only accumulates gradients without changing model weights and we call the optimizer step only after all micro\-batches have been trained, the overall training process preserves on\-policy semantics\. This workflow avoids severe efficiency degradation caused by resource contention, while also preventing the trainer from waiting idly for new training samples\. A remaining question is when TAG should be applied\. We will introduce our search strategy in Section[4](https://arxiv.org/html/2606.24369#S4)\. ### 3\.4\.Trajectory\-Consistent Stale Synchronization \(TCSS\) As shown in Figure[1](https://arxiv.org/html/2606.24369#S1.F1)\(d\), the generator idle time during the final trainer\-side micro\-batch update is unavoidable under strict synchronization\. The generator cannot start producing the next train batch until the latest trainer weights have been synchronized\. This barrier remains even after micro\-batch pipelining and TAG, because it is caused by the global\-step consistency requirement rather than by coarse scheduling granularity\. To address the remaining resource waste caused by strict synchronization, we build on TAG and introduce*Trajectory\-Consistent Stale Sync*\(TCSS\), a trajectory\-consistent one\-step stale synchronization strategy for disaggregated Diffusion RL\. Instead of enforcing a strict global\-step barrier after the last trainer micro\-batch update, the generator is allowed to start a bounded number of next\-step micro\-batches using the previous policy snapshot\. Once the updated policy is synchronized, subsequent micro\-batches use the new policy\. Importantly, TCSS differs from asynchronous AR schemes such as AReaL’s partial rollout\(Fuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib5)\)\. In AR generation, a response can be naturally split along token prefixes, so different parts of a response may be produced under different policy versions\. Directly applying such version mixing to diffusion is riskier because each sample is a tightly coupled denoising trajectory; mixing policy versions inside the same trajectory can amplify step\-level log\-probability mismatch and blur credit assignment across old and new denoising decisions\. TCSS therefore mixes policy versions only across micro\-batches, never within a single denoising trajectory\. This design removes the unavoidable synchronization bubble while preserving trajectory\-level policy consistency\. Figure[8](https://arxiv.org/html/2606.24369#S3.F8)\(a\) illustrates that, under synchronous TAG, an unavoidable bubble appears at the global\-step boundary because the generator must wait until the trainer finishes synchronizing the latest weights before continuing generation\. Figure[8](https://arxiv.org/html/2606.24369#S3.F8)\(b\) shows our TCSS design\. We define*staleness*as the maximum fraction of stale samples allowed within one global step, and allow the generator to produce a bounded number of micro\-batches using the stale policy under this staleness constraint\. In this way, the otherwise idle bubble can be used to start generation for the next global step, reducing the next step’s end\-to\-end latency\. Asynchronism can affect training rewards because some micro\-batches are generated by an old policy, so the staleness ratio should not be set too large\. At the same time, GRPO already constrains excessive deviation between old and new policies in the loss throughclip\(ρi,t\(θ\),1−ϵ,1\+ϵ\)A^i\\operatorname\{clip\}\\\!\\big\(\\rho\_\{i,t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\\big\)\\hat\{A\}\_\{i\}\. Therefore, an appropriate staleness ratio can still preserve stable reward improvement\. Figure 8\.Comparison between TAG and TCSS\.A timeline comparison shows synchronous TAG leaving a final synchronization bubble, while TCSS starts bounded stale micro\-batches during that otherwise idle period\.#### 3\.4\.1\.TCSS Algorithm Algorithm 1Trajectory\-Consistent Stale Sync1:Initialize trainer workers 𝒯\\mathcal\{T\}, generator workers 𝒢\\mathcal\{G\}, and async queues 2: s←0s\\leftarrow 0, ssync←−1s\_\{\\mathrm\{sync\}\}\\leftarrow\-1 3: ℛ,𝒰,𝒬upd←∅\\mathcal\{R\},\\mathcal\{U\},\\mathcal\{Q\}\_\{\\mathrm\{upd\}\}\\leftarrow\\emptyset 4:while s<Ss<Sdo 5:if \|ℛ\|<Cgen∧sgen−supd<W\|\\mathcal\{R\}\|<C\_\{\\mathrm\{gen\}\}\\land s\_\{\\mathrm\{gen\}\}\-s\_\{\\mathrm\{upd\}\}<Wthen 6: m←NextMicroBatch\(𝒟,s\)m\\leftarrow\\mathrm\{NextMicroBatch\}\(\\mathcal\{D\},s\) 7: w←SelectWorker\(m,𝒢,𝒯\)w\\leftarrow\\mathrm\{SelectWorker\}\(m,\\mathcal\{G\},\\mathcal\{T\}\) 8: r←GenAsync\(m,w\)r\\leftarrow\\mathrm\{GenAsync\}\(m,w\) 9: ℛ←ℛ∪\{r\}\\mathcal\{R\}\\leftarrow\\mathcal\{R\}\\cup\\\{r\\\} 10:endif 11:if r←FutureReturned\(ℛ\)r\\leftarrow\\mathrm\{FutureReturned\}\(\\mathcal\{R\}\)is not Nonethen 12: τ←Get\(r\)\\tau\\leftarrow\\mathrm\{Get\}\(r\) 13: A←ComputeAdvantage\(τ\)A\\leftarrow\\mathrm\{ComputeAdvantage\}\(\\tau\) 14: ℬ←SelectBestOfN\(τ,A\)\\mathcal\{B\}\\leftarrow\\mathrm\{SelectBestOfN\}\(\\tau,A\) 15: 𝒬upd←𝒬upd∪ℬ\\mathcal\{Q\}\_\{\\mathrm\{upd\}\}\\leftarrow\\mathcal\{Q\}\_\{\\mathrm\{upd\}\}\\cup\\mathcal\{B\} 16: ℛ←ℛ∖\{r\}\\mathcal\{R\}\\leftarrow\\mathcal\{R\}\\setminus\\\{r\\\} 17:endif 18:while 𝒬upd≠∅∧Free\(𝒯\)\\mathcal\{Q\}\_\{\\mathrm\{upd\}\}\\neq\\emptyset\\land\\mathrm\{Free\}\(\\mathcal\{T\}\)do 19: b←Pop\(𝒬upd\)b\\leftarrow\\mathrm\{Pop\}\(\\mathcal\{Q\}\_\{\\mathrm\{upd\}\}\) 20: u←UpdateAsync\(b,𝒯\)u\\leftarrow\\mathrm\{UpdateAsync\}\(b,\\mathcal\{T\}\) 21: 𝒰←𝒰∪\{u\}\\mathcal\{U\}\\leftarrow\\mathcal\{U\}\\cup\\\{u\\\} 22:endwhile 23:if u←FutureReturned\(𝒰\)u\\leftarrow\\mathrm\{FutureReturned\}\(\\mathcal\{U\}\)is not Nonethen 24: MarkCompleted\(u\)\\mathrm\{MarkCompleted\}\(u\) 25: 𝒰←𝒰∖\{u\}\\mathcal\{U\}\\leftarrow\\mathcal\{U\}\\setminus\\\{u\\\} 26:endif 27:while StepComplete\(ssync\+1\)\\mathrm\{StepComplete\}\(s\_\{\\mathrm\{sync\}\}\+1\)do 28: ssync←ssync\+1s\_\{\\mathrm\{sync\}\}\\leftarrow s\_\{\\mathrm\{sync\}\}\+1 29: SyncWeights\(𝒯,𝒢\)\\mathrm\{SyncWeights\}\(\\mathcal\{T\},\\mathcal\{G\}\) 30: s←ssync\+1s\\leftarrow s\_\{\\mathrm\{sync\}\}\+1 31:endwhile 32:endwhile Here,CgenC\_\{\\mathrm\{gen\}\}denotes the maximum number of concurrent generation futures\.ssis the current global training step, andSSis the total number of training steps\.ssyncs\_\{\\mathrm\{sync\}\}records the latest step whose trainer weights have been synchronized to the generator workers\.sgens\_\{\\mathrm\{gen\}\}andsupds\_\{\\mathrm\{upd\}\}denote the latest dispatched generation step and the latest trainer\-update step, respectively, andWWbounds the maximum allowed policy staleness between generation and update\. The core logic appears in lines 5–9\. Instead of using a global step as the outer scheduling unit, TCSS runs a continuous while\-loop that allows micro\-batches from different global steps to enter execution as long as the staleness boundWWis satisfied\. When the condition holds, the scheduler selects an idle trainer or generator worker and launches a generation task, thereby maximizing resource utilization\. Lines 23–31 handle completion and synchronization\. When a trainer\-side micro\-batch finishes, TCSS marks it as completed\. Once all micro\-batches of a global step are completed, the scheduler temporarily stops issuing trainer work for that step and performs weight synchronization when the generator is ready\. ## 4\.Recursive Bubble\-Triggered TAG Search Figure 9\.Recursive bubble\-triggered TAG search\. At each emerging bubble, the scheduler either inserts one TAG micro\-batch or skips the bubble\.A decision tree illustrates recursive scheduling choices at pipeline bubbles, where each branch either inserts a TAG micro\-batch or skips the bubble\.A straightforward option is to insert several TAG micro\-batches before the trainer\-side update starts\. This can exploit the trainer\-side bubble at the beginning of each training step and, by delaying the update, allows the update queue to accumulate enough ready samples, avoiding the case where the trainer waits for the generator at the very beginning\. We call this approachStatic TAG\. However, inserting too many TAG micro\-batches can make the trainer become the long tail: the trainer performs excessive TAG work, starts update too late, and eventually makes the generator wait for the trainer\. Inserting too few TAG micro\-batches, on the other hand, underutilizes trainer\-side bubbles\. Therefore, we prefer to adjust TAG decisions dynamically during execution\. We propose a recursive global search strategy, calledBubble\-Triggered TAG Search\(BTS\), to decide whether each emerging bubble should be filled by TAG\. Because inserting a TAG micro\-batch can fill the current trainer\-side bubble, but it may also delay the training of subsequent micro\-batches as shown in Figure[6](https://arxiv.org/html/2606.24369#S3.F6); in other words, blindly inserting TAG at every bubble can be locally beneficial but globally suboptimal\. Algorithm 2Recursive Bubble\-Triggered TAG Search1:Input:initial schedule state S0S\_\{0\} 2:Output:minimum latency T∗T^\{\*\}and TAG strategy Ω∗\\Omega^\{\*\} 3: \(T∗,Ω∗\)←Search\(S0,∅\)\(T^\{\*\},\\Omega^\{\*\}\)\\leftarrow\\mathrm\{Search\}\(S\_\{0\},\\emptyset\) 4:return \(T∗,Ω∗\)\(T^\{\*\},\\Omega^\{\*\}\) 5:procedure Search\(S,Ω\)\\mathrm\{Search\}\(S,\\Omega\) 6: S←Advance\(S\)S\\leftarrow\\mathrm\{Advance\}\(S\) 7:if Finished\(S\)\\mathrm\{Finished\}\(S\)then 8:return \(FinishTime\(S\),Ω\)\(\\mathrm\{FinishTime\}\(S\),\\Omega\) 9:endif 10: b←EmergingBubble\(S\)b\\leftarrow\\mathrm\{EmergingBubble\}\(S\) 11: \(T∗,Ω∗\)←\(\+∞,∅\)\(T^\{\*\},\\Omega^\{\*\}\)\\leftarrow\(\+\\infty,\\emptyset\) 12: Sskip←SkipBubble\(S,b\)S\_\{\\mathrm\{skip\}\}\\leftarrow\\mathrm\{SkipBubble\}\(S,b\) 13: \(Tskip,Ωskip\)←Search\(Sskip,Ω∘Skip\(b\)\)\(T\_\{\\mathrm\{skip\}\},\\Omega\_\{\\mathrm\{skip\}\}\)\\leftarrow\\mathrm\{Search\}\(S\_\{\\mathrm\{skip\}\},\\Omega\\circ\\mathrm\{Skip\}\(b\)\) 14:if Tskip<T∗T\_\{\\mathrm\{skip\}\}<T^\{\*\}then 15: \(T∗,Ω∗\)←\(Tskip,Ωskip\)\(T^\{\*\},\\Omega^\{\*\}\)\\leftarrow\(T\_\{\\mathrm\{skip\}\},\\Omega\_\{\\mathrm\{skip\}\}\) 16:endif 17: m←NextTAG\(S,b\)m\\leftarrow\\mathrm\{NextTAG\}\(S,b\) 18:if m≠∅m\\neq\\emptysetthen 19: Stag←InsertTAG\(S,b,m\)S\_\{\\mathrm\{tag\}\}\\leftarrow\\mathrm\{InsertTAG\}\(S,b,m\) 20: \(Ttag,Ωtag\)←Search\(Stag,Ω∘Insert\(b,m\)\)\(T\_\{\\mathrm\{tag\}\},\\Omega\_\{\\mathrm\{tag\}\}\)\\leftarrow\\mathrm\{Search\}\(S\_\{\\mathrm\{tag\}\},\\Omega\\circ\\mathrm\{Insert\}\(b,m\)\) 21:if Ttag<T∗T\_\{\\mathrm\{tag\}\}<T^\{\*\}then 22: \(T∗,Ω∗\)←\(Ttag,Ωtag\)\(T^\{\*\},\\Omega^\{\*\}\)\\leftarrow\(T\_\{\\mathrm\{tag\}\},\\Omega\_\{\\mathrm\{tag\}\}\) 23:endif 24:endif 25:return \(T∗,Ω∗\)\(T^\{\*\},\\Omega^\{\*\}\) 26:end procedure Figure[9](https://arxiv.org/html/2606.24369#S4.F9)illustrates the decision tree\. Instead of fixing the number of TAG micro\-batches in advance, the scheduler advances execution until a new emerging bubble appears, and then decides whether to fill this bubble with one TAG micro\-batch or skip it\. A skipped bubble is marked inactive and will not be reconsidered\. The recursion terminates in two cases\. First, even if a bubble appears, there may be no remaining micro\-batch that can be assigned to TAG, as illustrated by node 2 in Figure[9](https://arxiv.org/html/2606.24369#S4.F9)\. Second, with branch\-and\-bound pruning, a branch can be stopped early once its latency lower bound already exceeds the current best latency, as illustrated by node 5\. In practical training, we apply TAG to the first bubble by default, i\.e\., the position of micro\-batch 1 in Figure[9](https://arxiv.org/html/2606.24369#S4.F9)\. This provides online profiling measurements for the generation time of the next micro\-batch on the generator, as well as the generation and training times on the trainer\. Training then proceeds normally\. Whenever a new bubble appears, the driver process invokes Algorithm[2](https://arxiv.org/html/2606.24369#alg2)to solve for the globally optimal strategy and decide whether the current bubble should be filled by TAG\. Although Diffusion RL does not exhibit the same long\-tail effect as AR workloads, and micro\-batch runtimes are usually same, small runtime variations can still be introduced by hardware instability, reward computation, and cold\-start distortion during online profiling\. These errors may accumulate as execution progresses, so a one\-time global search is insufficient\. Instead, whenever a bubble is encountered, we update the runtime estimates using historical online profiling measurements, use the averaged values as the new search input, and recompute the strategy to correct drift caused by measurement error\. If the execution exposesKKemerging bubbles, Algorithm[2](https://arxiv.org/html/2606.24369#alg2)has worst\-case complexityO\(2K\)O\(2^\{K\}\), orO\(B2K\)O\(B2^\{K\}\)when each state simulation costsO\(B\)O\(B\)forBBmicro\-batches\. However, the homogeneous micro\-batch model allows us to substantially reduce the effective search space\. Since all micro\-batches have the same generation, TAG, and update costs, their identities are irrelevant to future scheduling decisions\. We therefore compress a schedule state into aggregate counters, including the number of pending micro\-batches, ready\-to\-update micro\-batches, completed micro\-batches, and the current generator/trainer timestamps\. Different decision paths that reach the same compressed state share the same optimal continuation\. We memorize the optimal result for each compressed state and reuse it when the state is encountered again\. For example, nodes 2 and 3 in Figure[9](https://arxiv.org/html/2606.24369#S4.F9)follow different decision paths, but they reach equivalent compressed states and therefore have the same remaining optimal latency\. Figure[10](https://arxiv.org/html/2606.24369#S4.F10)reports the runtime of a single search\. The y\-axis varies the number of micro\-batches, where a larger value implies a larger search space\. The x\-axis varies the ratio between the update time and the generation time of one micro\-batch, where a smaller ratio indicates more frequent pipeline bubbles\. Even in the most challenging setting\(B=128,t/gG=0\.1\)\(B=128,\\;t/g\_\{G\}=0\.1\), the search overhead remains negligible compared with generation and training\. Figure 10\.Recursive TAG search time\.A performance plot reports recursive TAG search overhead as the number of micro\-batches and the update\-to\-generation time ratio vary\. ## 5\.Evaluation ### 5\.1\.Experimental Settings ##### Testbed\. Experiments are mainly carried out on: 1\) Testbed A, a 32\-GPU cluster comprising four interconnected 8\-GPU nodes over a 200 Gb/s network, with each 48 GB GPU providing 336 third\-generation Tensor Cores and up to 38\.7 FP32 TFLOPS / 309\.7 Tensor TFLOPS peak performance\. 2\) Testbed B, a 32\-GPU cluster comprising four interconnected 8\-GPU nodes over a 400 Gb/s network, with each 80 GB GPU providing 528 fourth\-generation Tensor Cores and up to 67 FP32 TFLOPS / 1,979 BF16/FP16 Tensor TFLOPS peak performance\. 3\) Testbed C, a 16\-GPU cluster comprising two interconnected 8\-GPU nodes over a 200 Gb/s network, with each 96 GB GPU providing 752 fifth\-generation Tensor Cores and up to 126 FP32 TFLOPS / 503\.8 BF16/FP16 Tensor Core TFLOPS peak performance\. Figure 11\.QwenImage end\-to\-end time comparison with VeRL\-Omni, colocated execution and DigenRL\.Figure 12\.HunyuanVideo end\-to\-end time comparison with colocated execution and DigenRL\. ##### Baselines\. Our end\-to\-end evaluation consists of four parts\. First, to validate the effectiveness of DigenRL, we integrate the Diffusion RL algorithm FlowGRPO and support the QwenImage 20B model\. We compare its end\-to\-end time against VeRL\-Omni, a state\-of\-the\-art open\-source Diffusion RL framework\. For fairness, DigenRL uses the same inference backend, vLLM\-Omni, and training backend, FSDP, as VeRL\-Omni\. Second, to verify that disaggregated execution outperforms colocated execution, we further integrate HunyuanVideo with DanceGRPO and Wan2\.1 14B with FlowGRPO into DigenRL, and compare their end\-to\-end time against their native colocated execution\. Third, to evaluate DigenRL under heterogeneous resources, we compare it with the colocated execution of HunyuanVideo and show that it can still maintain high throughput\. Finally, we conduct ablation experiments to break down the gains from each optimization\. We also verify that, when bounding the number of stale samples and limiting their impact on training, TCSS maintains a reward curve comparable to on\-policy training\. ### 5\.2\.End\-to\-End Time Comparison with Open\-Source Frameworks For open\-source framework comparisons, GenRL is mainly characterized by its support for diverse models and algorithms, but provides relatively limited system\-level optimization\. Meanwhile, our measurements show that GenRL performs similarly to native colocation on Wan2\.1, so the colocation comparison in §[5\.3](https://arxiv.org/html/2606.24369#S5.SS3)can serve as a performance reference between DigenRL and GenRL\. Therefore, we mainly compare end\-to\-end time with VeRL\-Omni\. Meanwhile, to ensure a fair on\-policy comparison, we disable TCSS\. Our experiments are conducted on Testbed B with a training batch size of 32 and an image resolution of 512 × 512\. We use intra\-node FSDP full sharding and inter\-node data parallelism \(DP\)\. Since FlowGRPO for QwenImage uses Qwen3\-VL\-8B\-Instruct\(Baiet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib51)\)as the reward model, we colocate the generator and reward model to balance GPU memory pressure\. Generation and scoring are then coordinated through the two\-queue scheduling mechanism described in §[3\.3](https://arxiv.org/html/2606.24369#S3.SS3)\. Across all cases, DigenRL substantially outperforms both native colocation and VeRL\-Omni\. Compared with VeRL\-Omni, DigenRL achieves speedups ranging from 1\.56×\\timesto 2\.21×\\times\. A key reason is that vLLM\-Omni schedules multiple requests through asynchronous queuing: as the batch size grows, scheduling queues and long\-tail requests increasingly limit efficiency, and larger batches can even reduce overall throughput\. This effect appears in both VeRL\-Omni and native colocation\. DigenRL mitigates the issue by splitting large batches into smaller micro\-batches and pipelining them while controlling concurrency\. Another reason is that image training does not impose severe memory pressure during the training stage, allowing us to use a high degree of TSP parallelism\. As a result, DigenRL achieves higher efficiency in both the generation and training stages, which explains its substantial gains over VeRL\-Omni and native colocation\. Figure 13\.Wan2\.1 14B end\-to\-end time comparison with colocated execution and DigenRL\.Figure 14\.FLUX\.1 12B 32 GPUs end\-to\-end time comparison with colocated execution and DigenRL\. ### 5\.3\.End\-to\-End Time Comparison with Native Colocation Figure[12](https://arxiv.org/html/2606.24369#S5.F12)and Figure[13](https://arxiv.org/html/2606.24369#S5.F13)compare DigenRL with native colocated Diffusion RL execution\. The HunyuanVideo experiments are conducted on Testbed A, and the Wan2\.1 14B experiments are conducted on Testbed B\. We split the nodes evenly between the generator and trainer\. The model uses FSDP as the backend, and the training batch size is 32\. We use common sampling\-step counts of 20, 30, and 40, and timestep fractions of 0\.4 and 0\.6, which denotes the proportion of denoising timesteps used for training\. The resolution is352×352×25352\\times 352\\times 25\. For HunyuanVideo, DigenRL achieves end\-to\-end speedups ranging from 1\.04×\\timesto 1\.37×\\times\. When TCSS is enabled, the speedups increase to 1\.13×\\times–1\.49×\\times\. The gains are larger on 16 GPUs because scaling to 32 GPUs introduces additional driver\-side scheduling pressure and GPU–CPU communication overhead, which partially offset the benefits of our optimizations\. A similar trend appears on FLUX\.1 12B, where the speedup improves from 1\.06×\\timesto 1\.21×\\times\. Because FLUX is an image generation model and has a shorter computation time, the scheduling overhead incurred on 32 GPUs becomes more prominent in the end\-to\-end runtime\. For Wan2\.1 14B, DigenRL achieves end\-to\-end speedups ranging from 1\.13×\\timesto 1\.34×\\times\. When TCSS is enabled, the speedups increase to 1\.21×\\times–1\.45×\\times\. More importantly, it achieves similar gains on both 16 and 32 GPUs, mainly because Testbed B provides stronger CPU scheduling capability and higher CPU\-GPU communication bandwidth than Testbed A, which mitigates the scheduling and communication pressure caused by scaling up\. Another supporting observation is that DigenRL gains increase with the number of sampling steps, because computation occupies a larger fraction of the end\-to\-end time and therefore partially amortizes communication and scheduling overhead\. Overall, the performance gains come from two sources: TSP reduces trainer\-side update time, while disaggregation with pipelining provides better scalability and overlap than native colocation\. Figure 15\.HunyuanVideo end\-to\-end time comparison on heterogeneous resources\.Figure 16\.TCSS reward curve\. ### 5\.4\.End\-to\-End Time Comparison with Heterogeneous Resources We conduct this set of experiments using eight GPUs from Testbed A and eight GPUs from Testbed C, connected through 200 Gb/s HDR InfiniBand\. We choose HunyuanVideo as the evaluation model because RL for video diffusion models typically imposes higher compute and memory requirements, making the impact of heterogeneous resources more pronounced\. As shown in Figure[15](https://arxiv.org/html/2606.24369#S5.F15), DigenRL improves speedup over colocated execution from 1\.46×\\timesto 1\.85×\\times\. The overall speedup increases with the number of sampling steps and the timestep fraction, as larger values of these parameters amplify performance differences across heterogeneous hardware\. Figure 17\.End\-to\-end speedup of different optimization methods on HunyuanVideo\.A multi\-panel evaluation figure compares Hunyuan and Flux experiment results across the scheduling variants studied in the paper\.Figure 18\.End\-to\-end speedup of different optimization methods on Wan2\.1 14B\.A multi\-panel evaluation figure compares Wan 14B experiment results across the scheduling variants studied in the paper\.Colocated execution is less effective under heterogeneous resources due to two key drawbacks\. First, the more capable side often remains idle while waiting for the weaker side to complete the current stage, resulting in suboptimal resource utilization\. Second, the training workload is constrained by the memory capacity of the weaker side, limiting support for higher\-resolution videos and images\. In contrast, DigenRL is well suited to this heterogeneous training setting\. By assigning stronger compute resources to the trainer, it provides a higher memory ceiling for training, while allowing idle trainer resources to assist generation and thereby improving overall resource utilization\. ### 5\.5\.Training Effectiveness of TCSS Figure[16](https://arxiv.org/html/2606.24369#S5.F16)shows the reward curves of on\-policy training and TCSS when training FLUX\.1\-dev 12B\. By strictly controlling staleness and limiting the impact of stale policies during training, TCSS maintains a steadily increasing reward even when the staleness threshold is 0\.5\. ### 5\.6\.Ablation Study Figure[17](https://arxiv.org/html/2606.24369#S5.F17)shows the speedup brought by the optimizations introduced on top of naive disaggregation\. GAP improves the speedup from 1\.52×\\timesto 1\.84×\\times\. The 0\.6 cases benefit more than the 0\.4 cases because the training stage overlapped by GAP accounts for a larger fraction of the end\-to\-end time\. Furthermore, TSP reduces the training time of each micro\-batch, increasing the speedup from 1\.62×\\timesto 2\.14×\\times\. At this point, micro\-batch training is no longer the bottleneck; generation becomes the dominant cost\. TAG then alleviates this generation bottleneck by utilizing idle trainer resources, further improving the speedup from 2\.28×\\timesto 3\.05×\\times, while TCSS increases it from 2\.49×\\timesto 3\.34×\\times\. The additional gain from TCSS is relatively small because TAG with already achieves high resource utilization; TCSS mainly extracts the remaining bubbles at synchronization boundaries\. For Wan2\.1\-14B, as shown in Figure[18](https://arxiv.org/html/2606.24369#S5.F18), GAP improves the speedup from 1\.58×\\timesto 1\.70×\\times\. TSP further increases it from 1\.71×\\timesto 2\.10×\\times, TAG improves it from 2\.27×\\timesto 2\.79×\\times, and TCSS increases it from 2\.44×\\timesto 2\.85×\\times\. ## 6\.Related Work RL post\-training systems\.Recent RL post\-training systems for autoregressive foundation models have made large\-scale policy optimization increasingly practical\. General\-purpose systems such as veRL\(Shenget al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib3)\), OpenRLHF\(Huet al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib15)\), ROLL\(Wanget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib17)\), and LlamaRL\(Wuet al\.,[2025a](https://arxiv.org/html/2606.24369#bib.bib16)\)provide modular execution substrates for algorithms such as PPO\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.24369#bib.bib13)\), GRPO\(Anonymous,[2025](https://arxiv.org/html/2606.24369#bib.bib9)\), DPO\(Rafailovet al\.,[2023](https://arxiv.org/html/2606.24369#bib.bib39)\), and ReMax\(Liet al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib42)\), and expose distributed training backends for large models\. Disaggregated RL systems \(AReaL\(Fuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib5)\), StreamRL\(Zhonget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib4)\), Laminar\(Shenget al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib35)\), RollArt\(Gaoet al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib46)\), Weave\(Wuet al\.,[2026a](https://arxiv.org/html/2606.24369#bib.bib47)\), OrchestrRL\(Tanet al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib40)\), etc\.\) for post\-training text\-only LLMs or multimodal LLMs have been proposed to improve the training throughput in different aspects\. AReaL\(Fuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib5)\)eliminates the synchronization bottleneck of traditional RL pipelines through fully asynchronous training\. StreamRL\(Zhonget al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib4)\)further improves disaggregated RL execution by addressing pipeline and workload\-skew bubbles\. Laminar\(Shenget al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib35)\)enhances resource utilization through asynchronous overlap between rollout and training\. RollArt\(Gaoet al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib46)\)focuses on scalable multi\-task agentic RL training, while Weave\(Wuet al\.,[2026a](https://arxiv.org/html/2606.24369#bib.bib47)\)improves cluster efficiency via co\-scheduling of disaggregated RL workloads\. OrchestrRL\(Tanet al\.,[2026](https://arxiv.org/html/2606.24369#bib.bib40)\)jointly optimizes compute and network resource orchestration for large\-scale disaggregated RL deployments\. RLBoost\(Wuet al\.,[2026b](https://arxiv.org/html/2606.24369#bib.bib18)\)exploits otherwise idle training resources through partial response seeding\. These systems mainly target autoregressive decoding, where requests can often be partitioned or resumed along token prefixes\. DigenRL instead targets Diffusion\-RL, where a sample is generated through a tightly coupled denoising trajectory and training must preserve step\-level log\-probabilities, rewards, and policy\-version consistency\. RL for diffusion and flow models\.RL has likewise been explored in the context of diffusion and flow\-based generative models\. DDPO\(Blacket al\.,[2024](https://arxiv.org/html/2606.24369#bib.bib48)\)casts denoising as a sequential decision\-making problem, while DanceGRPO\(Xueet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib8)\), Flow\-GRPO\(Liuet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib7)\), MixGRPO\(Anonymous,[2025](https://arxiv.org/html/2606.24369#bib.bib9)\), and Long\-RL\(Chenet al\.,[2025](https://arxiv.org/html/2606.24369#bib.bib27)\)adapt group\-relative or flow\-matching RL objectives for image and video synthesis\. More recent toolkits, including GenRL\(ModelTC,[2025](https://arxiv.org/html/2606.24369#bib.bib28)\)and veRL\-Omni\(verl\-omni Contributors,[2026](https://arxiv.org/html/2606.24369#bib.bib24)\), offer software infrastructures for RL in visual generation\. However, these approaches concentrate mainly on objectives, reward shaping, and model coverage, and continue to rely on a colocated architecture\. DigenRL is orthogonal to these efforts: instead of proposing a new RL objective, it offers a high\-performance execution substrate for existing Diffusion\-RL methods based on a disaggregated architecture\. ## 7\.Conclusion This paper introduces DigenRL, a disaggregated RL system for post\-training diffusion\-based visual generative LLMs\. We demonstrate that current colocated Diffusion\-RL designs unnecessarily bind together sample generation and trainer\-side updates, and that a straightforward decoupling of these phases leads to pipeline stalls and synchronization\-induced idle time\. In contrast, DigenRL removes such bubbles via generation\-axis pipelining to support more fine\-grained task pipelining, leverages timestep\-parallel training to raise training GPU utilization, adopts elastic trainer\-assisted generation to opportunistically use trainer idle periods to speed up generation, and applies trajectory\-consistent stale synchronization to exploit the final microbatch bubble for additional throughput gains\. Across a variety of models, GRPO algorithms, hyperparameters, and hardware setups, our evaluation shows that DigenRL achieves higher end\-to\-end training throughput than state\-of\-the\-art Diffusion\-RL systems such as veRL\-Omni and GenRL\. ## References - Anonymous \(2025\)MixGRPO: unlocking flow\-based grpo efficiency with mixed ode\-sde\.Note:arXiv preprint arXiv:2507\.21802[https://arxiv\.org/abs/2507\.21802](https://arxiv.org/abs/2507.21802)Cited by:[§6](https://arxiv.org/html/2606.24369#S6.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - S\. Bai, Y\. Cai, R\. Chen,et al\.\(2025\)Qwen3\-vl technical report\.Note:arXiv preprint arXiv:2511\.21631[https://arxiv\.org/abs/2511\.21631](https://arxiv.org/abs/2511.21631)Cited by:[§5\.2](https://arxiv.org/html/2606.24369#S5.SS2.p2.1)\. - Black Forest Labs \(2024\)FLUX\.1\.Note:[https://blackforestlabs\.ai/announcing\-black\-forest\-labs/](https://blackforestlabs.ai/announcing-black-forest-labs/)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p4.2)\. - K\. Black, M\. Janner, Y\. Du, I\. Kostrikov, and S\. Levine \(2024\)Training diffusion models with reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 4965–4987\.Cited by:[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - Y\. Chen, W\. Huang, B\. Shi, Q\. Hu, H\. Ye, L\. Zhu, Z\. Liu, P\. Molchanov, J\. Kautz, X\. Qi, S\. Liu, H\. Yin, Y\. Lu, and S\. Han \(2025\)Scaling rl to long videos\.Note:arXiv preprint arXiv:2507\.07966[https://openreview\.net/forum?id=TxedB8hI5O](https://openreview.net/forum?id=TxedB8hI5O)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - DeepSeek\-AI \(2025\)DeepSeek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.External Links:2501\.12599Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1)\. - P\. Dhariwal and A\. Q\. Nichol \(2021\)Diffusion models beat gans on image synthesis\.Note:arXiv preprint arXiv:2105\.05233[https://arxiv\.org/abs/2105\.05233](https://arxiv.org/abs/2105.05233)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.1)\. - W\. Fu, J\. Gao, X\. Shen, C\. Zhu, Z\. Mei, C\. He, S\. Xu, G\. Wei, J\. Mei, J\. Wang, T\. Yang, B\. Yuan, and Y\. Wu \(2025\)AReaL: a large\-scale asynchronous reinforcement learning system for language reasoning\.Note:arXiv preprint arXiv:2505\.24298[https://arxiv\.org/abs/2505\.24298](https://arxiv.org/abs/2505.24298)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§3\.4](https://arxiv.org/html/2606.24369#S3.SS4.p3.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - W\. Gao, Y\. Zhao, T\. Wu, S\. Xiong, W\. Wang, D\. An, L\. Cao, D\. Muhtar, Z\. Liu, H\. Zhao, J\. Huang, S\. Yang, Y\. Li, W\. Su, J\. Wang, L\. Qu, B\. Zheng, and W\. Wang \(2026\)RollArt: disaggregated multi\-task agentic rl training at scale\.InProceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation \(OSDI ’26\),Seattle, WA, USA\.Cited by:[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - D\. Guo, D\. Yang, H\. Zhang, J\. Song, P\. Wang, Q\. Zhu, R\. Xu, R\. Zhang, S\. Ma, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1)\. - J\. Ho, A\. Jain, and P\. Abbeel \(2020\)Denoising diffusion probabilistic models\.Note:arXiv preprint arXiv:2006\.11239[https://arxiv\.org/abs/2006\.11239](https://arxiv.org/abs/2006.11239)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.1)\. - J\. Hu, X\. Wu, W\. Shen, J\. K\. Liu, Z\. Zhu, W\. Wang, S\. Jiang, H\. Wang, H\. Chen, B\. Chen, W\. Fang, Xianyu, Y\. Cao, H\. Xu, and Y\. Liu \(2024\)OpenRLHF: an easy\-to\-use, scalable and high\-performance rlhf framework\.Note:arXiv preprint arXiv:2405\.11143[https://arxiv\.org/abs/2405\.11143](https://arxiv.org/abs/2405.11143)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - D\. P\. Kingma and M\. Welling \(2013\)Auto\-encoding variational bayes\.Note:arXiv preprint arXiv:1312\.6114[https://arxiv\.org/abs/1312\.6114](https://arxiv.org/abs/1312.6114)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.5)\. - Y\. Kirstain, A\. Polyak, U\. Singer, S\. Matiana, J\. Penna, and O\. Levy \(2023\)Pick\-a\-pic: an open dataset of user preferences for text\-to\-image generation\.Note:arXiv preprint arXiv:2305\.01569[https://arxiv\.org/abs/2305\.01569](https://arxiv.org/abs/2305.01569)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.5)\. - W\. Kong, Q\. Tian, Z\. Zhang,et al\.\(2024\)HunyuanVideo: a systematic framework for large video generative models\.Note:arXiv preprint arXiv:2412\.03603[https://arxiv\.org/abs/2412\.03603](https://arxiv.org/abs/2412.03603)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p4.2)\. - Z\. Li, T\. Xu, Y\. Zhang, Z\. Lin, Y\. Yu, R\. Sun, and Z\. Luo \(2024\)ReMax: a simple, effective, and efficient reinforcement learning method for aligning large language models\.InInternational Conference on Machine Learning,pp\. 29128–29163\.Cited by:[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - Y\. Lipman, R\. T\. Q\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le \(2023\)Flow matching for generative modeling\.Note:arXiv preprint arXiv:2210\.02747[https://arxiv\.org/abs/2210\.02747](https://arxiv.org/abs/2210.02747)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.1)\. - J\. Liu, G\. Liu, J\. Liang, Y\. Li, J\. Liu, X\. Wang, P\. Wan, D\. Zhang, and W\. Ouyang \(2025\)Flow\-grpo: training flow matching models via online rl\.Note:arXiv preprint arXiv:2505\.05470[https://arxiv\.org/abs/2505\.05470](https://arxiv.org/abs/2505.05470)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.8),[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p3.2),[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - ModelTC \(2025\)GenRL: reinforcement learning framework for visual generation\.Note:[https://github\.com/ModelTC/GenRL](https://github.com/ModelTC/GenRL)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§1](https://arxiv.org/html/2606.24369#S1.p4.2),[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - L\. Ouyang, J\. Wu, X\. Jiang,et al\.\(2022\)Training language models to follow instructions with human feedback\.Note:arXiv preprint arXiv:2203\.02155[https://arxiv\.org/abs/2203\.02155](https://arxiv.org/abs/2203.02155)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1)\. - R\. Rafailov, A\. Sharma, E\. Mitchell, S\. Ermon, C\. D\. Manning, and C\. Finn \(2023\)Direct preference optimization: your language model is secretly a reward model\.Note:arXiv preprint arXiv:2305\.18290[https://arxiv\.org/abs/2305\.18290](https://arxiv.org/abs/2305.18290)Cited by:[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - S\. Rajbhandari, J\. Rasley, O\. Ruwase, and Y\. He \(2020\)ZeRO: memory optimizations toward training trillion parameter models\.InProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis \(SC\),pp\. 1–16\.Note:[https://arxiv\.org/abs/1910\.02054](https://arxiv.org/abs/1910.02054)Cited by:[§3\.2\.2](https://arxiv.org/html/2606.24369#S3.SS2.SSS2.p2.1)\. - J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.Note:arXiv preprint arXiv:1707\.06347[https://arxiv\.org/abs/1707\.06347](https://arxiv.org/abs/1707.06347)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1)\. - G\. Sheng, Y\. Tong, B\. Wan, W\. Zhang, C\. Jia, X\. Wu, Y\. Wu, X\. Li, C\. Zhang, Y\. Peng,et al\.\(2026\)Laminar: a scalable asynchronous rl post\-training framework\.InProceedings of the 21st European Conference on Computer Systems,pp\. 400–422\.Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.24369#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - G\. Sheng, C\. Zhang, Z\. Ye, X\. Wu, W\. Zhang, R\. Zhang, Y\. Peng, H\. Lin, and C\. Wu \(2024\)HybridFlow: a flexible and efficient rlhf framework\.Note:arXiv preprint arXiv:2409\.19256[https://arxiv\.org/abs/2409\.19256](https://arxiv.org/abs/2409.19256)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - X\. Tan, Y\. Feng, Y\. Zhou, Y\. Jiang, Y\. Zhu, and H\. Xu \(2026\)OrchestrRL: dynamic compute and network orchestration for disaggregated rl\.Inthe 40th ACM Special Interest Group on Data Communication \(SIGCOMM\),Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - A\. van den Oord, O\. Vinyals, and K\. Kavukcuoglu \(2017\)Neural discrete representation learning\.Note:arXiv preprint arXiv:1711\.00937[https://arxiv\.org/abs/1711\.00937](https://arxiv.org/abs/1711.00937)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.5)\. - verl\-omni Contributors \(2026\)Verl\-omni: a multimodal reinforcement learning extension for verl\.Note:[https://github\.com/verl\-project/verl\-omni](https://github.com/verl-project/verl-omni)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§1](https://arxiv.org/html/2606.24369#S1.p4.2),[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - T\. Wan, A\. Wang, B\. Ai,et al\.\(2025\)Wan: open and advanced large\-scale video generative models\.Note:arXiv preprint arXiv:2503\.20314[https://arxiv\.org/abs/2503\.20314](https://arxiv.org/abs/2503.20314)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p4.2)\. - W\. Wang, S\. Xiong, G\. Chen, W\. Gao, S\. Guo, Y\. He, J\. Huang, J\. Liu, Z\. Li, X\. Li,et al\.\(2025\)Reinforcement learning optimization for large\-scale learning: an efficient and user\-friendly scaling library\.Note:arXiv preprint arXiv:2506\.06122[https://arxiv\.org/abs/2506\.06122](https://arxiv.org/abs/2506.06122)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - W\. Wang, S\. Xiong, G\. Chen, W\. Gao, S\. Guo, Y\. He, J\. Huang, J\. Liu, Z\. Li, X\. Li,et al\.\(2026\)WEAVE: efficient co\-scheduling for disaggregated RL post\-training\.Note:arXiv preprint arXiv:2602\.11143[https://arxiv\.org/abs/2602\.11143](https://arxiv.org/abs/2602.11143)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1)\. - B\. Wu, S\. Wang, Y\. Tang, J\. Ding, E\. Helenowski, L\. Tan, T\. Xu, T\. Gowda, Z\. Chen, C\. Zhu, X\. Tang, Y\. Qian, B\. Zhu, and R\. Hou \(2025a\)LlamaRL: a distributed asynchronous reinforcement learning framework for efficient large\-scale llm training\.Note:arXiv preprint arXiv:2505\.24034[https://arxiv\.org/abs/2505\.24034](https://arxiv.org/abs/2505.24034)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - C\. Wu, J\. Li, J\. Zhou,et al\.\(2025b\)Qwen\-image technical report\.Note:arXiv preprint arXiv:2508\.02324[https://arxiv\.org/abs/2508\.02324](https://arxiv.org/abs/2508.02324)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p4.2)\. - T\. Wu, L\. Cao, Y\. Wei, W\. Gao, Y\. Zhao, D\. An, S\. Xiong, Z\. Lv, J\. Huang, S\. Yang, Y\. Yu, J\. Wang, L\. Qu, and W\. Wang \(2026a\)Weave: efficient co\-scheduling for disaggregated rl post\-training\.InProceedings of the 20th USENIX Symposium on Operating Systems Design and Implementation \(OSDI ’26\),Seattle, WA, USA\.Cited by:[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - Y\. Wu, X\. Liu, H\. Zheng, J\. Gu, B\. Chen, Z\. M\. Mao, A\. Krishnamurthy, and I\. Stoica \(2026b\)RLBoost: harvesting preemptible cloud resources for cost\-efficient reinforcement learning on llms\.Note:arXiv preprint arXiv:2510\.19225[https://arxiv\.org/abs/2510\.19225](https://arxiv.org/abs/2510.19225)Cited by:[§2\.2](https://arxiv.org/html/2606.24369#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\. - J\. Xu, X\. Liu, Y\. Wu, Y\. Tong, Q\. Li, M\. Ding, J\. Tang, and Y\. Dong \(2023\)ImageReward: learning and evaluating human preferences for text\-to\-image generation\.Note:arXiv preprint arXiv:2304\.05977[https://arxiv\.org/abs/2304\.05977](https://arxiv.org/abs/2304.05977)Cited by:[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.5)\. - Z\. Xue, J\. Wu, Y\. Gao, F\. Kong, L\. Zhu, M\. Chen, Z\. Liu, W\. Liu, Q\. Guo, W\. Huang, and P\. Luo \(2025\)DanceGRPO: unleashing grpo on visual generation\.Note:arXiv preprint arXiv:2505\.07818[https://arxiv\.org/abs/2505\.07818](https://arxiv.org/abs/2505.07818)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p1.8),[§2\.1](https://arxiv.org/html/2606.24369#S2.SS1.p3.2),[§6](https://arxiv.org/html/2606.24369#S6.p2.1)\. - Z\. Yao, R\. Y\. Aminabadi, O\. Ruwase, S\. Rajbhandari, X\. Wu, A\. A\. Awan, J\. Rasley, M\. Zhang, C\. Li, C\. Holmes,et al\.\(2023\)Deepspeed\-chat: easy, fast and affordable rlhf training of chatgpt\-like models at all scales\.arXiv preprint arXiv:2308\.01320\.Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1)\. - Y\. Zhao, A\. Gu, R\. Varma, L\. Luo, C\. Huang, M\. Xu, L\. Wright, H\. Shojanazeri, M\. Ott, S\. Shleifer,et al\.\(2023\)PyTorch fsdp: experiences on scaling fully sharded data parallel\.Proceedings of the VLDB Endowment16\(12\),pp\. 3848–3860\.Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§3\.2\.2](https://arxiv.org/html/2606.24369#S3.SS2.SSS2.p2.1)\. - Y\. Zhong, Z\. Zhang, X\. Song, H\. Hu, C\. Jin, B\. Wu, N\. Chen, Y\. Chen, Y\. Zhou, C\. Wan, H\. Zhou, Y\. Jiang, Y\. Zhu, and D\. Jiang \(2025\)StreamRL: scalable, heterogeneous, and elastic rl for llms with disaggregated stream generation\.Note:arXiv preprint arXiv:2504\.15930[https://arxiv\.org/abs/2504\.15930](https://arxiv.org/abs/2504.15930)Cited by:[§1](https://arxiv.org/html/2606.24369#S1.p1.1),[§1](https://arxiv.org/html/2606.24369#S1.p2.1),[§2\.2](https://arxiv.org/html/2606.24369#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.24369#S6.p1.1)\.
Similar Articles
$R^2$-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
R²-dLLM introduces spatio-temporal redundancy reduction techniques that cut diffusion LLM decoding steps by up to 75% while preserving generation quality, addressing a key deployment bottleneck.
@robertnishihara: A great example of the importance of disaggregation in RL. From the paper LLM generation alternates between prefill and…
Robert Nishihara highlights a paper on disaggregating RL workloads, showing that using compute-optimized H800s for prefill and bandwidth-optimized H20s for decode can cut rollout times by 21-51% and 47% respectively, emphasizing that no single hardware type fits all stages.
Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
This paper proposes PAT, an adaptive tensor parallelism method that dynamically reconfigures TP during the generation stage of synchronous RLHF training to mitigate long-tail generation bottlenecks. Evaluations on LLaMA3.1-8B and Qwen3-14B show reductions in generation latency by up to 34.6% and end-to-end iteration latency by up to 27.2%.
RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework
RAD-2 presents a unified generator-discriminator framework for autonomous driving that combines diffusion-based trajectory generation with RL-optimized reranking, achieving 56% collision rate reduction compared to diffusion-based planners. The approach introduces techniques like Temporally Consistent Group Relative Policy Optimization and BEV-Warp simulation environment for efficient large-scale training.
Dynamic-dLLM: Dynamic Cache-Budget and Adaptive Parallel Decoding for Training-Free Acceleration of Diffusion LLM
This paper proposes Dynamic-dLLM, a training-free framework that accelerates diffusion large language models by dynamically allocating cache-update budgets and calibrating decoding thresholds, achieving over 3x speedup on models like LLaDA and Dream while maintaining performance.