TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling

# TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
Source: [https://arxiv.org/html/2608.10402](https://arxiv.org/html/2608.10402)
Yanyu Ren1∗†, Xizheng Wang3∗, Xiao Liu1,2∗, Bowen Lv1†, Hanchen Zhang1†, Shudan Zhang1†, Hanyu Lai1†, Shuai Wang3, Li Chen3, Dan Li1, Jie Tang1 1Tsinghua University2Z\.AI3Zhongguancun Laboratory

###### Abstract

Reinforcement learning \(RL\) for large language models is moving toward multi\-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times\. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead\. We presentTideRL, a readiness\-aware elastic RL system with Continuous Task Batching, Resource\-Aware Ref\-Actor Pipelining, and Elastic Resource Scaling\. CTB preserves useful rollout state,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals\. Across text\-only and multi\-modal agentic workloads,TideRLimproves RL training goodput by up to 5\.6×\\timesover synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance\. It also improves KV cache hit rate by 1\.58×\\times, reduces per\-step training time by up to 44\.3%, and cuts total waiting time by up to 77\.6%\.

11footnotetext:Equal contribution\.22footnotetext:Work done while these authors interned at Z\.AI\.## 1Introduction

Large Language Models \(LLMs\) are rapidly evolving from static text generators into autonomous agents capable of solving complex, real\-world problems\. In these agentic workloads, models do not merely produce a single\-turn answer; instead, they iteratively interact with external, distributed environments, such as web browsers, smartphones, or computer OSes\[webarena,osworld,androidlab\]\. This introduces a multi\-turn execution flow\. An agent generates an action, transmits it over the network to the environment, waits for the execution results, and resumes generation based on the newly returned observations\. Because LLM outputs are non\-deterministic, execution trajectories of the same task may differ drastically in turn count, context length, and wall\-clock duration, producing workloads with extreme variance across both spatial and temporal dimensions\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/concept_arch.png)Figure 1:The dataflow and workflow of different RL strategies on agentic tasks\.To enhance these autonomous capabilities, researchers turn to reinforcement learning \(RL\)\. While RL has shown immense success in single\-turn reasoning tasks\[openrlhf,grpo,ppo\], applying it to multi\-turn agentic scenarios requires orchestrating massive, networked machine learning clusters\. A typical RL workflow goes through four phases\. In therolloutphase, a batch of agentic tasks runs concurrently, requesting actions from rollout workers and observations from environment workers to generate interactive trajectories in multiple turns\. Later, the RL system evaluates these trajectories in therewardandreferencephase\. Finally, the system uses the output from the reward and the reference to update the model weights in theactorphase\.

However, a small number of long\-tail tasks in the rollout phase may block the pipeline and leave expensive GPU resources underutilized\. For agentic RL, the right objective is goodput, which we measure through training throughput: the rollout’d tokens that reach the Trainer and advance an update counts\. GPU waiting and repeated prefill recomputation are pure overhead\. To maximize goodput, asynchronous systems\[streamrl,areal\]separate these components onto different GPUs, unlike synchronous systems like VeRL\[verl\]that force rollout\-train barriers\. On a subset of GPUs,rollout worker groupsperform continuous generation as producers\. On the remaining GPUs,trainer worker groups, which colocate the reference and actor models, pull data from the global buffer to compute weight updates, as consumers\. This decoupling theoretically allows generation and training to proceed concurrently, so that the rollout phase of long\-tail tasks can take place simultaneously with the training phase of shorter tasks\.

While effective for single\-turn tasks, current asynchronous architectures exhibit severe systemic inefficiencies when processing multi\-turn agentic workloads characterized by high concurrency and variable context lengths\. We identify three critical challenges:

- ∙\\bulletKV cache preemption during intermittent interactions\.Standard inference engines rely on request\-level continuous batching, assuming each request is self\-contained\. However, multi\-turn tasks proceed as a sequence of dependent requests separated by idle periods while waiting for external feedback\. During these pauses, the engine greedily evicts its KV cache \(the memory footprint of historical tokens\) to serve requests from other concurrent tasks\. Upon resumption, the system suffers a catastrophic cache miss as Figure[2](https://arxiv.org/html/2608.10402#S2.F2)shows, forcing redundant and computationally expensive recomputation \(prefill\) for an increasingly long historical context\.
- ∙\\bulletStalls and thrashing in reference inference\.Existing systems share GPU resources between actor and reference workers within the trainer, enforcing strict stage dependencies\. To avoid memory exhaustion, they either wait for large global batches to arrive before initiating updates, or frequently alternate between reference and actor models \(which share the same architecture but have different weights\) on the same GPUs for each micro\-batch\. The former creates massive computation bubbles that leave GPUs idle for up to 81% of the step time, while the latter introduces severe model\-swapping thrashing overhead\.
- ∙\\bulletRigidity of resource allocation under shifting bottlenecks\.Agentic RL operates as a producer\-consumer pipeline\. Because task complexity varies drastically, the system bottleneck dynamically shifts\. A batch of complex tasks stalls the trainer as it waits for slow rollout data, while simple tasks create a large ready backlog at the beginning of a training step and then keep feeding new micro\-batches rapidly\. Relying on static resource partitioning between rollout and trainer worker groups, existing systems cannot react to either the initial ready backlog or the steady ready interval, structurally capping global throughput during these macroscopic workload shifts, as shown in Figure[4](https://arxiv.org/html/2608.10402#S2.F4)\.

These intertwined challenges expose architectural flaws of existing systems, raising a critical research question:How can we build a unified, elastic training architecture that maximizes RL goodput, as measured by training throughput, under multi\-turn agentic execution?The key is not to optimize rollout, training, and resource allocation as isolated modules\. Multi\-turn RL needs a shared readiness view: Rollout must preserve the task state that will become useful soon, Trainer must choose a ref\-actor execution mode that matches how ready data arrives, and the cluster scheduler must move GPUs toward the side that removes the current goodput loss\. Answering this requires co\-designing the multi\-turn memory management, distributed execution pipeline, and dynamic cluster resource allocation\.

To this end, we identify two architectural blind spots in existing frameworks\. First, systems such as AReaL and StreamRL expose asynchrony, but still treat the rollout\-trainer split as a mostly fixed operating point; they cannot use the current ready backlog and arrival pace to jointly choose the ref\-actor execution strategy and the GPU split\. Second, standard inference engines allocate memory at therequestlevel\. Memory management must be elevated to thetasklevel to gain global visibility over the multi\-turn lifecycle, actively coordinating context pinning across distributed rollout ranks\. Together, these changes make individual data parallel \(DP\) ranks schedulable resources rather than static members of a worker group\.

Motivated by these insights, we presentTideRL, an elastic and asynchronous distributed system tailored for multi\-turn agentic RL goodput\.TideRLsolves these challenges through three co\-designed components:

- ∙\\bulletContinuous Task Batching \(CTB\)replaces request\-level continuous batching on each rollout rank with task\-level admission, preemption, and resumption\. CTB tracks per\-rank token footprints in real time and applies a semantic\-aware priority hierarchy—prioritizing evaluation boundaries, near\-complete GRPO groups, active trajectories, and long contexts\. This keeps useful rollout state resident until it can feed the global buffer, providing the stable producer side thatRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}and ERS rely on\.
- ∙\\bulletResource\-Aware Ref\-Actor Pipelining \(𝐑𝐀2​𝐏\\mathbf\{RA\}^\{2\}\\mathbf\{P\}\)resolves the stall\-thrashing dilemma through two complementary execution modes selected by two data\-readiness signals:Ready\-at\-Start\(RAS\), the amount of micro\-batch work already present when a training step begins, andTime Per Ready Micro\-batch\(TPRM\), the interval at which additional micro\-batches become available\. In decoupled mode, Ref and Actor run on separate GPUs in a streaming micro\-batch pipeline; a restructured computation graph defers loss computation into the backward pass, enabling Ref forward, Actor forward, and Actor backward to overlap with zero swap overhead\. In colocated mode,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}uses ready\-batch aggregation and zero\-copy shared\-memory transfer to amortize unavoidable swapping across ready bursts\. Together,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}turns readiness into goodput under both startup backlog pressure and steady per\-micro\-batch arrival pressure\.
- ∙\\bulletElastic Resource Scaling \(ERS\)exploitsRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}’s dual\-mode structure to migrate individual ranks between rollout and training on the fly\. ERS reads RAS and TPRM from the global buffer: a small RAS with slow TPRM indicates rollout starvation and triggers colocatedRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}plus more Rollout ranks, whereas a large RAS or fast TPRM indicates trainer\-side pressure and triggers decoupledRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}plus more Trainer capacity\. Crucially, ERS piggybacks weight migration onto the natural cache\-flush boundary that on\-policy RL already mandates at every sync, achieving zero\-overhead elasticity: no pipeline suspension, no NCCL group rebuild, no wasted KV cache\.

We implementTideRLwith a highly modular architecture, ensuring seamless compatibility with diverse multi\-modal and text\-only tasks, and easy integration with mainstream serving and training backends\. We deploy the system on a 4\-node cluster with 32 NVIDIA H100 GPUs and evaluate models with different architectures\. Our evaluations demonstrate large performance gains over existing RL frameworks\. End\-to\-end,TideRLraises RL training goodput, measured by training throughput, by 1\.8–7\.0×\\timeson text\-only tasks \(reducing time\-to\-convergence by 51\.1%\) and improves multi\-modal goodput by over 33%, reducing training time by 62\.2% with similar performance\. Component studies close the loop from motivation to design: CTB improves generation goodput by preserving task state,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}reduces per\-step training time by up to 44\.3% under different readiness patterns, and ERS cuts total idle waiting time by 87\.4% by schedulingRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}modes and GPU ranks from the same readiness signals\.

## 2Background and Motivation

We first summarize the agentic RL workflow and then motivate the three system bottlenecks thatTideRLtargets\.

### 2\.1Agentic Reinforcement Learning

Agentic RL workloads in this paper follow Group Relative Policy Optimization \(GRPO\)\[grpo\], whose execution alternates between trajectory generation and model update\.

The Agentic RL Pipeline\.Unlike traditional single\-turn chat optimization, agentic RL requires models to interact with environments iteratively\. As depicted in Figure[1](https://arxiv.org/html/2608.10402#S1.F1)\(b\), a standard agentic GRPO pipeline consists of four sequential stages to train a new model𝒜\\mathcal\{A\}from the original model𝒪\\mathcal\{O\}:

- ∙\\bulletRollout Generation:Rollout111For clarity and brevity,Rollout,Ref, andActordenote the worker group of rollout worker, reference worker, and actor worker, respectively, andEnvdenotes environment, throughout the remainder of this paper\.hosts𝒜\\mathcal\{A\}to generate actions\. Crucially, this involvesmulti\-turninteractions: the model generates an action, pauses to wait for environment execution, receives the observation, and generates the next action\. For a given task, a group ofGGdistinct multi\-turn trajectories is sampled\. Each trajectory may comprise different turns and lengths of messages\.
- ∙\\bulletReward Calculation:Once the trajectories in a group are completed, the environment evaluates them\. GRPO determines the advantageAiA\_\{i\}by standardizing the rewardsrir\_\{i\}strictly based on the performance of other trajectories within the same group:Ai=\(ri−mean​\(𝐫\)\)/std​\(𝐫\)A\_\{i\}=\(r\_\{i\}\-\\text\{mean\}\(\\mathbf\{r\}\)\)/\\text\{std\}\(\\mathbf\{r\}\)\.
- ∙\\bulletReference Calculation:Ref computes the base log\-probabilities on𝒪\\mathcal\{O\}of generated trajectories\. This provides a Kullback\-Leibler \(KL\) divergence penalty to prevent the policy from over\-optimizing the reward\.
- ∙\\bulletActor Update:Actor executes the forward and backward passes on𝒜\\mathcal\{A\}to update its weights \(i\.e\., update the policy\)\. Once updated, the latest weights are synchronized back to Rollout for the next iteration\.

TheTrainercomprises Ref, Actor and the reward calculation, and consumes the trajectories from Rollout\.

Asynchronous RL\.Traditional synchronous architectures enforce a strict alternation between the rollout and training stages\. This coupling causes severe GPU idling, as the training must wait for the long\-tail task to complete\[openrlhf,rlhfuse\]\. To break this resource coupling, modern systems typically adopt an asynchronous execution paradigm\[streamrl,areal\]\. As depicted in Figure[1](https://arxiv.org/html/2608.10402#S1.F1)\(c\), they decouple generation and training into physically independent workers\. This allows Rollout to continuously generate new groups of responses while Trainer consumes historical data in the background, significantly boosting hardware throughput\. However, this asynchrony introducespolicy staleness, as Rollout generates data using older model weights\. Since algorithms like GRPO are fundamentally on\-policy, excessive off\-policy data severely degrade training stability\. Consequently, asynchronous systems must strictly bound this staleness gap, requiring timely weight synchronization to prevent performance degradation\.

### 2\.2Key Challenges in Asynchronous Agentic RL

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/bg_kvcache_hit_rate.png)Figure 2:Multi\-turn agentic tasks experience severe cache misses in one WebShop training step\. The data is collected on one NVIDIA H100 GPU running Qwen2\.5\-7B\[qwen25\]\.![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/bg_resource_utilization.png)Figure 3:Trainer experiences significant stalls and thrashing due to long\-tail tasks\. Even without CPU offloading, model alternation still incurs non\-negligible GPU context\-switching overhead, as shown in[˜6\.5](https://arxiv.org/html/2608.10402#S6.SS5)\.![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/bg_resource_util.png)Figure 4:Rollout Wait Time \(RWT\) and Training Wait Time \(TWT\) during the first 100 steps under varying fraction of GPUs assigned to Rollout\. These waits expose the two readiness signals that driveRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}and ERS\. The experiment is carried out on one node with 8 NVIDIA H100 GPUs, running the WebShop\[webshop\]agents\.Although asynchronous RL frameworks overlap generation and training, their execution traces under multi\-turn agentic workloads reveal three ways the system loses RL goodput\.

C1: Catastrophic KV Cache Preemption in Rollouts\.Observation:Shifting our focus to the Rollout phase, we monitored the KV cache hit rate and generation throughput during a single training step\. Figure[2](https://arxiv.org/html/2608.10402#S2.F2)exhibits a typical three\-phase fluctuation rather than stable cache utilization\. At the beginning of the step, the hit rate increases as a large influx of new short sequences is initially cached\. However, the hit rate subsequently plummets and remains at only∼12%\\sim 12\\%, dragging the throughput down to∼650\\sim 650tokens/s⋅\\cdotinstance\. A spike in the hit rate occurs at the very end of the training step\.

Root Cause:The fundamental flaw lies in the mismatch between request\-level scheduling and multi\-turn semantics\. While recent scheduling optimizations target single\-turn reasoning tasks, they fall short in multi\-turn agentic workloads\. For instance, Seer’s\[seer\]chunk\-level scheduling yields minimal benefits because agentic outputs per request are typically short, and StreamRL’s\[streamrl\]length\-probing mechanism fails to capture the intermittent nature of environment interactions\[agserve,continuum\]\. When waiting for environment feedback, the underlying inference engine, driven by standard continuous batching to maximize concurrency, schedules new requests, thereby preempting the tasks’ KV caches\. The engine suffers massive cache misses as the observation returns, and must recompute the rapidly growing historical context\. The late recovery in hit rate is merely an artifact of dropping concurrency: as the global batch nears completion, no new tasks are admitted, allowing the few straggling tasks to safely retain their cache without preemption\.

C2: Stalls and Thrashing in Training\.Observation:We examined the Trainer’s execution trace, and observed a dilemma between computation stalls and I/O thrashing\. In Figure[3](https://arxiv.org/html/2608.10402#S2.F3)\(a\), standard execution forces the Trainer to stall significantly while waiting for a full global batch to complete, a delay severely exacerbated by long\-tail tasks\. Conversely, if we attempt to mitigate this idle time by streaming micro\-batches, the system falls into another trap: massive I/O thrashing that degrades training GPU goodput, as illustrated in Figure[3](https://arxiv.org/html/2608.10402#S2.F3)\(b\)\.

Root Cause:This dilemma is structurally rooted in the memory\-intensive nature of on\-policy algorithms like GRPO\. The Trainer must execute reference calculations and actor updates sequentially using two distinct models\. Under standard execution, the system waits for all rollout tasks in a global batch to finish, making the Trainer’s idle time bottlenecked by the slowest long\-tail samples\. To avoid these macroscopic stalls, the system could adopt a streaming mode to process micro\-batches as soon as they arrive\. However, existing systems \(e\.g\., VeRL\[verl\]and AsyncFlow\[asyncflow\]\) cannot run both models on the same GPUs simultaneously due to the runtime memory consumed by training activations\. Consequently, streaming micro\-batches forces the system to frequently load and offload model weights between the CPU and GPU over PCIe, replacing the stall with a model\-swapping bottleneck\.

C3: Shifting Bottlenecks Invalidating Static Allocation\.

Observation:To investigate whether global resource tuning could improve agentic RL goodput, we tracked the Rollout Wait Time \(RWT\) and Training Wait Time \(TWT\) across 100 steps under various static Rollout GPU ratios\. SinceTideRLmeasures goodput through training throughput, these waits matter because they directly consume time without advancing useful training\. Fundamentally, RWT measures the idle duration Rollout workers stall upon reaching off\-policy staleness limits and waiting for new weights, whereas TWT quantifies the Trainer’s starvation period awaiting sufficient trajectory accumulation\. These two waits correspond to the Trainer’s data\-readiness state: high TWT means the step begins with too few ready micro\-batches and receives later ones slowly, while high RWT means Rollout has produced a backlog that the Trainer cannot consume quickly enough\. As illustrated in Figure[4](https://arxiv.org/html/2608.10402#S2.F4), the system bottleneck oscillates wildly\. A lower rollout ratio \(0\.125\) consistently starves the Trainer \(high TWT\), while a higher ratio \(0\.5\) generates data too fast, causing Rollout to stall \(high RWT\)\. Most crucially, even within a single optimal fixed ratio \(e\.g\., 0\.25\), RWT and TWT frequently cross paths\.

Root Cause:This invalidation of static partitioning stems from the extreme variance in agentic trajectory execution times, further compounded by the unpredictable load of intermediate evaluation tasks\. Unlike single\-turn chat, multi\-turn environments cause the computational bottleneck to dynamically flip between generation and training, so a static GPU quota cannot preserve goodput across steps\. It cannot react to either the micro\-batch backlog already ready at a training boundary or the per\-micro\-batch ready interval after the step starts\. These two signals determine whether the trainer should spend GPUs on a decoupled Ref\-Actor pipeline or release them to Rollout through colocated execution\. Although StreamRL\[streamrl\]attempts to address this by dynamically increasing the size of Rollout, it lacks real\-time reaction due to high initialization overheads\. Furthermore, this scaling approach is incompatible with the fixed\-resource environments prevalent in large\-scale production clusters, such as those operated by Alibaba and Microsoft\[antman,pollux,gao2024empirical\]\. As a result, the cluster is left severely underutilized throughout the training lifecycle, with goodput lost to both data starvation and off\-policy throttling\.

Summary\.The paradigm shift toward multi\-turn agentic RL exposes fundamental architectural mismatches\. Request\-level scheduling leads to severe cache preemption \(C1\), rigid stage dependencies create computation bubbles \(C2\), and static partitioning fails to adapt to shifting bottlenecks \(C3\), all of which reduce training throughput and therefore goodput\. Addressing these inefficiencies necessitates a holistic system redesign, spanning from computation graph execution to physical resource allocation and memory management\.

## 3TideRLOverview

To address these challenges, we proposeTideRL, an elastic asynchronous RL system tailored for multi\-turn agentic workloads\.TideRLcoordinates three decisions that existing systems handle separately: which tasks stay resident in Rollout, how ready trajectories are consumed by the Trainer, and how GPUs move between the two sides\.

### 3\.1Design Rationale

The design follows the dataflow of agentic RL\. CTB keeps Rollout productive without losing task context;RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}consumes ready micro\-batches without waiting for a full global batch or thrashing between Ref and Actor; ERS adjusts resources so the selectedRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}strategy matches the current readiness pattern\. The three components are intentionally coupled by one feedback loop: CTB controls when trajectories become ready,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}exposes how expensive it is to consume the current ready stream, and ERS moves ranks to reduce the dominant wait revealed by that stream\.

Continuous Task Batching \(CTB\)\.To resolve KV cache preemption in multi\-turn interactions \(C1\),TideRLelevates Rollout scheduling from requests to tasks\. CTB tracks each task’s token footprint, admits tasks only when a rank has enough KV cache headroom, pauses tasks according to RL\-aware priorities, and resumes them with worker affinity\. This preserves useful context while maintaining high concurrency\.

Resource\-Aware Ref\-Actor Pipelining \(𝐑𝐀2​𝐏\\mathbf\{RA\}^\{2\}\\mathbf\{P\}\)\.To address Trainer stalls and model alternation thrashing \(C2\),TideRLstreams generated trajectories directly to the Trainer\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}then chooses between two Ref\-Actor execution strategies\. When ready micro\-batches are abundant or arrive quickly,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}decouples Ref and Actor onto different GPUs to overlap reference computation with policy updates\. When ready micro\-batches are scarce,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}colocates Ref and Actor, uses ready\-batch aggregation and zero\-copy transfer to amortize model alternation, and leaves more GPUs available for Rollout\.

Elastic Resource Scaling \(ERS\)\.To absorb shifting bottlenecks \(C3\),TideRLabandons static resource partitioning\. ERS acts as the runtime scheduler for the twoRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}strategies: it provisions more Rollout ranks and selects colocatedRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}when the Trainer is data\-starved to reduce TWT, and it switches to decoupledRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}with additional Trainer capacity when ready data accumulates to reduce RWT\. This keeps Ref\-Actor execution aligned with the current producer\-consumer imbalance\.

### 3\.2System Architecture and Workflow

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/sys_arch.png)Figure 5:The overall architecture and workflow ofTideRL\. Environments and reward computations are disaggregated to an external CPU cluster\. The system features a continuous data flow orchestrated by CTB on Rollout ranks,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}on Trainer nodes, and an overarching ERS scheduler\.Inspired by AgentRL\[agentrl\],TideRLadopts a disaggregated architecture where environment maintenance and reward computation are offloaded to an external CPU cluster\. As illustrated in Figure[5](https://arxiv.org/html/2608.10402#S3.F5), this isolates GPU instances purely for rollout and training\. We trace the continuous lifecycle of a task and its trajectory to illustrateTideRL’s workflow\.

A task’s lifecycle begins in the background, where CTB manages the admission of new agentic tasks based on Actor KV cache availability and strictly bounds off\-policy staleness\. Once dispatched to a Rollout rank, the Actor and Env engage in multi\-turn interactions\. During this highly concurrent multi\-task generation, CTB monitors the real\-time token footprint, pausing or resuming tasks to achieve higher throughput and prevent memory exhaustion\.

Upon completing all interaction turns, the Env calculates group\-relative rewards and pushes the group to a global rollout buffer\. The Trainer pulls ready trajectories from the buffer and usesRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}to update the policy with low or no model\-swapping overhead\. Once the training step completes, Trainer synchronizes the new weights to Rollout\.

The overarching ERS scheduler monitors the utilization and readiness state of the rollout buffer\. If trajectory generation outpaces training, ERS dynamically commands a subset of Rollout ranks to safely offload their Actor weights and reconfigures these freed nodes into the trainer group; if training starves, it moves trainer\-side ranks back to Rollout and letsRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}run in the colocated mode\.

TideRLalso considers evaluation tasks\. For the evaluation of theii\-th step’s model, CTB organizes them concurrently with training data rollout in the\(i\+1\)\(i\+1\)\-th step\. CTB aggregates the metrics just before the weight synchronization of the\(i\+1\)\(i\+1\)\-th step\. This ensures metrics are reported accurately without interrupting the continuous trajectory flow\.

## 4Continuous Task Batching

CTB makes Rollout scheduling task\-aware\. Instead of treating every environment turn as an independent request, CTB tracks each task’s growing context, admits new tasks only when a Rollout rank has enough KV cache headroom, pauses tasks before memory is exhausted, and resumes paused tasks with cache locality whenever possible\.

### 4\.1Token\-Aware Admission Control

In highly concurrent agentic RL, dispatching tasks by raw request counts creates severe imbalance because tasks consume very different amounts of KV cache\. CTB instead treats each Data Parallel \(DP\) rank as a token budget pool and admits tasks according to measured context size\.

To estimate a task’s initial footprint, CTB sends a lightweight probing task for each trajectory group and records the token usage of its first turn, following the idea of Seer\[seer\]\. During dispatch, CTB checks the current token utilization of each rank and admits a task only if the remaining headroom can accommodate the probed footprint\. After admission, CTB updates the task’s actual token usage at the start of every interaction turn, so later scheduling decisions reflect the growing context rather than the initial estimate\.

CTB also enforces three admission constraints:

- ∙\\bulletEnvironment concurrency:Active tasks cannot exceed the parallel capacity of the external environments\.
- ∙\\bulletPolicy staleness:CTB throttles dispatch when Rollout is likely to generate more data than Trainer can consume within the staleness bound\.
- ∙\\bulletTask diversity:CTB penalizes assigning too many identical task types to the same batch, spreading environment\-side stragglers across ranks\.

Input:Active

𝒯a​c​t\\mathcal\{T\}\_\{act\}, Paused

𝒯p​a​u​s​e\\mathcal\{T\}\_\{pause\}, New

𝒯n​e​w\\mathcal\{T\}\_\{new\}, Rank budgets

ℬ\\mathcal\{B\}1:while

𝒯a​c​t∪𝒯p​a​u​s​e∪𝒯n​e​w≠∅\\mathcal\{T\}\_\{act\}\\cup\\mathcal\{T\}\_\{pause\}\\cup\\mathcal\{T\}\_\{new\}\\neq\\emptysetdo

2:Update token footprint for

t∈𝒯a​c​tt\\in\\mathcal\{T\}\_\{act\}via turn probing

3:foreach rank

rrdo

4:whileTokenUsage\(

rr\)

\>ℬr\>\\mathcal\{B\}\_\{r\}do

5:

te​v​i​c​t←argmint∈𝒯a​c​tr​Priority​\(t\)t\_\{evict\}\\leftarrow\\text\{argmin\}\_\{t\\in\\mathcal\{T\}\_\{act\}^\{r\}\}\\text\{Priority\}\(t\)\{Tiers 1\-4\}

6:Preempt

te​v​i​c​tt\_\{evict\}from

𝒯a​c​tr\\mathcal\{T\}\_\{act\}^\{r\}to

𝒯p​a​u​s​e\\mathcal\{T\}\_\{pause\}
7:endwhile

8:whileTokenUsage\(

rr\)

<ℬr<\\mathcal\{B\}\_\{r\}do

9:if

𝒯p​a​u​s​e≠∅\\mathcal\{T\}\_\{pause\}\\neq\\emptysetand fits in

ℬr\\mathcal\{B\}\_\{r\}then

10:

tr​e​s←argmaxt∈𝒯p​a​u​s​e​Priority​\(t\)t\_\{res\}\\leftarrow\\text\{argmax\}\_\{t\\in\\mathcal\{T\}\_\{pause\}\}\\text\{Priority\}\(t\)
11:Resume

tr​e​st\_\{res\}on

rr\(enforcing group affinity\)

12:elseif

𝒯n​e​w≠∅\\mathcal\{T\}\_\{new\}\\neq\\emptysetand satisfies constraintsthen

13:Admit

tn​e​wt\_\{new\}to

rrbased on probed footprint

14:else

15:break

16:endif

17:endwhile

18:endfor

19:endwhile

Algorithm 1Semantic\-Aware CTB Scheduling
### 4\.2Semantic\-Aware Pausing and Resuming

As tasks progress through multiple turns, their contexts may outgrow a rank’s token budget\. CTB handles this by pausing selected tasks before KV cache pressure causes uncontrolled eviction\. Algorithm[1](https://arxiv.org/html/2608.10402#alg1)summarizes the loop: update active footprints, pause low\-priority tasks when a rank exceeds its budget, and fill recovered headroom by resuming paused tasks before admitting new ones\.

The priority score combines RL progress with system cost:

P​\(t\)=ω1⋅𝕀eval\+ω2⋅Gcompletion\+ω3⋅𝕀active\+ω4⋅LcontextP\(t\)=\\omega\_\{1\}\\cdot\\mathbb\{I\}\_\{\\mathrm\{eval\}\}\+\\omega\_\{2\}\\cdot G\_\{\\mathrm\{completion\}\}\+\\omega\_\{3\}\\cdot\\mathbb\{I\}\_\{\\mathrm\{active\}\}\+\\omega\_\{4\}\\cdot L\_\{\\mathrm\{context\}\}
Here,𝕀eval\\mathbb\{I\}\_\{\\mathrm\{eval\}\}marks evaluation tasks,GcompletionG\_\{\\mathrm\{completion\}\}is the completion ratio of the task’s GRPO group \(e\.g\.,Nfinished/NtotalN\_\{\\mathrm\{finished\}\}/N\_\{\\mathrm\{total\}\}\),𝕀active\\mathbb\{I\}\_\{\\mathrm\{active\}\}marks tasks currently executing, andLcontextL\_\{\\mathrm\{context\}\}is the current token length\. Table[1](https://arxiv.org/html/2608.10402#S4.T1)summarizes why each term matters\.

Table 1:Taxonomy of CTB’s semantic\-aware preemption hierarchy\.MetricRationalePriorityTask TypeEvaluation tasks require anchored weights; stalling them blocks global weight synchronization\.HighestGroup Comp\.GRPO requires full groups to calculate advantages; fragmented groups stall gradient updates\.HighExec\. StatePreempting actively running tasks destroys in\-flight compute cycles and prior investments\.Med\.Context Len\.Evicting massive historical contexts causes catastrophic redundant prefill upon resumption\.LowWhen capacity is restored, CTB resumes paused tasks before admitting new tasks\. It also enforcesworker affinitybased on group IDs, placing resumed tasks on ranks that are most likely to retain their shared prefixes\. This preserves useful KV cache state and avoids unnecessary prefix recomputation\.

## 5Resource\-Aware Ref\-Actor Collaboration

The Trainer must process rollout data as soon as it becomes useful, but agentic workloads make data readiness highly uneven\. At some steps, many micro\-batches are already waiting in the global buffer; at others, the Trainer starts almost empty and receives new micro\-batches slowly\.

TideRLuses RAS \(startup ready backlog\) and TPRM \(later ready interval\) in two layers\. First,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}provides two Ref\-Actor execution strategies: a decoupled streaming strategy for high RAS or short TPRM, and a colocated aggregation strategy for low RAS and long TPRM\. Second, ERS schedules between these twoRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}strategies by moving ranks between Rollout and Trainer\. This section first explains the twoRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}strategies and then shows how ERS selects and provisions them\.

### 5\.1𝐑𝐀2​𝐏\\mathbf\{RA\}^\{2\}\\mathbf\{P\}Execution Strategies

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/rap_ds.png)Figure 6:TheRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}pipeline for decoupled streaming mode\. The loss computation for Actor ranks is moved from the forward pass into the backward pass\.Decoupled Streaming Mode\.When RAS is large or TPRM is short, the Trainer has enough ready work to keep dedicated Ref and Actor resources busy\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}therefore avoids model\-swapping overhead by allocating the Reference \(Ref\) and Actor models onto distinct GPUs\. Since Ref only performs forward passes,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}groups one Ref rank with multiple Actor ranks to form a streaming micro\-batch pipeline\.

To maximize concurrency and avoid blocking the Actor, we restructure the computation graph\. In standard implementations, the loss computation is included in the Actor’s forward pass, requiring Actor to wait for the Ref to output the reference log probabilities \(ref\_log\_probs\) before the forward pass\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}breaks this dependency by deferring the Actor’s loss computation \(including the KL divergence penalty\) to its backward pass\. As shown in Figure[6](https://arxiv.org/html/2608.10402#S5.F6), the Ref model sequentially computesref\_log\_probs\(F0,F1,F2F\_\{0\},F\_\{1\},F\_\{2\}\), while the Actor performs its forward passes \(F0,F3,F6F\_\{0\},F\_\{3\},F\_\{6\}\) independently\. The Actor only consumes theref\_log\_probswhen initiating the backward passes \(B0,B3,B6B\_\{0\},B\_\{3\},B\_\{6\}\)\. This realignment ensures overlap between reference log prob computation and policy updates with much lower synchronization bubbles\.

Optimized Colocated Mode\.When RAS is small and TPRM is long, a dedicated Ref rank would spend much of the step waiting for data\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}then colocates the Ref and Actor models on the same GPUs, releasing the extra GPUs to Rollout where they can shorten TPRM for later steps\. To keep colocation efficient,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}introduces two optimizations, as depicted in Figure[3](https://arxiv.org/html/2608.10402#S2.F3)\(c\)\.

First, we implementready\-batch aggregation\. Instead of alternating between models for every single micro\-batch \(e\.g\.,R1→A1→R2→A2R\_\{1\}\\rightarrow A\_\{1\}\\rightarrow R\_\{2\}\\rightarrow A\_\{2\}\),RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}monitors the global rollout buffer\. If multiple micro\-batches are ready, it aggregates their execution\. The system loads the Ref model once to process all ready batches \(R2,3R\_\{2,3\}\), and then swaps to the Actor to perform updates \(A2,3A\_\{2,3\}\)\. This amortizes the I/O cost of model alternation\.

Second,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}leverageszero\-copy transmission\. Because the Ref and Actor reside on the same physical node, the input sequences and the generatedref\_log\_probsare passed directly via GPU shared memory, bypassing network serialization and redundant memory allocations\.

Analytical Mode Selection\.The two strategies target different bottlenecks\. Decoupled mode pays a pipeline\-fill cost but removes model swaps, so it is best when high RAS amortizes the fill cost or short TPRM keeps the pipeline continuously fed\. Colocated mode keeps fewer Trainer GPUs active and uses ready\-batch aggregation to amortize occasional swaps, so it is best when low RAS and long TPRM would otherwise leave a decoupled Ref rank idle\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}compares the decoupled latency cost \(Cd​e​cC\_\{dec\}\) and colocated latency cost \(Cc​o​lC\_\{col\}\) under these two signals; Appendix[A](https://arxiv.org/html/2608.10402#A1)details the cost model\.

Micro\-Batch Dispatching\.To maximize throughput and prevent stragglers,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}jointly optimizeshowa micro\-batch is formed and allocated across DP ranks\.

RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}employs the same micro\-batching strategy for both execution modes\. Early micro\-batches are accumulated from the rollout buffer until they contain sufficient tokens to fully saturate the compute capacity of all DP ranks\. Conversely, the last micro\-batch of a global batch is deliberately kept as small as possible\. This intentionally minimizes the duration of the final forward and backward passes, thereby reducing the pipeline flush bubble at the end of the training step\.

RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}further applies distinct dispatching strategies to handle the heterogeneous micro\-batch size:

- ∙\\bulletColocated Mode\.In this mode, GPU execution is purely sequential\. The primary goal is to balance the total compute load across all training ranks\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}utilizes a zig\-zag \(longest\-processing\-time\-first\) allocation strategy, sorting sequences by length and distributing them iteratively in a zig\-zag pattern\. This ensures that the sum of sequence lengths assigned to each rank remains nearly identical\.
- ∙\\bulletDecoupled Mode\.In the decoupled mode, the overall completion time is bottlenecked by the execution time of the final micro\-batch\. Building upon our small\-final\-batch sizing strategy,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}sorts and allocates the sequences such that thelastsequence fed into the pipeline is the absolute smallest\. This minimizes the tail latency, allowing the last active rank to finish its backward pass at the earliest possible time\.

### 5\.2ERS Scheduling of𝐑𝐀2​𝐏\\mathbf\{RA\}^\{2\}\\mathbf\{P\}Strategies

ERS closes the loop betweenRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}’s mode choice and the producer\-consumer imbalance\. It reallocates GPUs on the fly so that Rollout can generate data fast enough and Trainer can consume ready micro\-batches with the rightRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}strategy\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/ers_arch.png)Figure 7:The ERS architecture\. The ERS coordinator dynamically reallocates GPU resources between functional roles\.Readiness\-Aware Plan Generation\.To accommodate volatile generation rates,TideRLadapts the elastic batching strategy, where the Trainer consumes any accumulated batch size falling within an interval\[Bm​i​n,Bi​d​e​a​l\]\[B\_\{min\},B\_\{ideal\}\]\. Here,Bm​i​nB\_\{min\}is the minimum batch size required for an effective training step, whileBi​d​e​a​lB\_\{ideal\}represents the full utilization of GPU HBM and computing resources\. ERS generates scaling plans immediately after the Trainer finishes an Actor update, before the new weights are broadcast \(sync\_params\)\.

Despite CTB andRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}, static resource allocation cannot perfectly align the generation and consumption throughputs\. The mismatch appears as TWT when the Trainer starves for ready trajectories, and as RWT when Rollout is throttled by stale weights or an overfilled buffer\. ERS monitors the global buffer to estimate the RAS and TPRM signals introduced above, and converts them into two complementary actions: scaling Rollout up to reduce TWT, and scaling Trainer up to reduce RWT\.

Directly using instantaneous arrivals yields a noisy signal skewed by trajectory variance even within one step\. To avoid reactive thrashing, the ERS coordinator \(Figure[7](https://arxiv.org/html/2608.10402#S5.F7)\) estimates these trends through two queue metrics: the generation and consumption deficits\.

When Rollout resources are insufficient, generation slows\. The next training step begins with a small RAS and observes a long TPRM, so the Trainer is starved both at startup and during the step, increasing TWT\. ERS detects this via thegeneration deficitmetric \(the global batch volume<Bi​d​e​a​l<B\_\{ideal\}\)\. It then selects colocatedRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}and reallocates the GPUs previously used by decoupled Ref ranks to Rollout\. This gives generation more capacity, directly reducing TWT, while the Trainer avoids wasting dedicated Ref GPUs on an empty stream\.

Conversely, when Rollout is over\-provisioned, ready data accumulates\. The next training step starts with a large RAS, and continued fast generation shortens TPRM\. If the Trainer cannot drain this data fast enough, Rollout eventually waits for buffer clearance and fresh weights, increasing RWT\. ERS detects thisconsumption deficitthrough the Head\-of\-Line \(HoL\) latency of the oldest micro\-batch in the global buffer, together with the current ready volume\. If the HoL latency exceeds a threshold \(τ×Ts​t​e​p\\tau\\times T\_\{step\}, whereTs​t​e​pT\_\{step\}is the average step duration\) or RAS exceeds the decoupled\-mode break\-even point, ERS selects decoupledRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}and reclaims a Rollout rank to provision a dedicated Ref rank\. The Trainer can then drain the backlog without repeated model swaps, directly reducing RWT\.

Beyond standard training,TideRLhandles boundary conditions viapreemptive generation allocation\. During cold starts or evaluation phases, generation dominates\. Rather than waiting for RAS to remain low and TPRM to become long, the coordinator preemptively maximizes the Rollout group size, allocating maximum hardware to bootstrap the buffer quickly\.

TideRLadoptsadaptive data retentionto prevent starvation cycles following a Rollout scale\-down\. If a scaled\-down Rollout cannot sustain the Trainer’s consumption, the Trainer adaptively reduces its fetch size towardBm​i​nB\_\{min\}\. By pacing consumption and retaining a micro\-batch reservoir, ERS extends the step duration, granting Rollout time to accumulate a larger batch for the next iteration\.

Seamless Scaling Operations\.Traditional elastic frameworks suspend training to migrate model weights across nodes\.TideRLeliminates this latency by exploiting the mathematical properties of on\-policy RL\.

Cache\-Free Task Migration via CTB Coordination\.Iterative model weight updates render historical KV caches mathematically incompatible with the new policy\. Thus, caches across Rollout ranks must be flushed at the post\-update boundary\. Piggybacking on this invalidation cycle, task migration becomes cache\-free\.TideRLbypasses physical KV cache transfers\. When ERS closes a Rollout rank, the CTB scheduler automatically reassigns the active tasks on it to the remaining active ranks\. Upon resumption, these tasks perform a fresh prefill using the synchronized new version of weights on their new Rollout ranks\.

Latency\-Hiding Role Switching\.To ensure dynamic resource reallocation does not block the training loop,TideRLoverlaps PCIe weight\-swapping operations with thesync\_paramsbroadcast\. When scaling down a Reference rank \(to reassign it to Rollout\), ERS offloads the reference model to the CPU while waiting for the Actor backward pass to complete\. The reactivated Rollout rank then joins thesync\_paramsbroadcast to load the Actor\. Conversely, when scaling down a Rollout rank, it discards the stale Rollout model and skips the weight download\. Instead, it utilizes the synchronization time window to load the Reference model concurrently with the other ranks’ broadcast\. Through this operational alignment,TideRLadjusts its functional GPU distribution without adding latency to the critical path\.

## 6Evaluation

We implementTideRLwith∼\\sim16000 lines of Python code, supporting mainstream training \(e\.g\., Megatron\[megatron\], PyTorch FSDP\[fsdp\]\) and serving \(vLLM\[vllm\], SGLang\[sglang\]\) frameworks\. We disclose the implementation details in Appendix[B](https://arxiv.org/html/2608.10402#A2)\. In this section, we use vLLM as the rollout backend and Megatron as the training backend\.

We evaluateTideRLacross text\-only and multi\-modal tasks, model sizes, and readiness regimes\. Our key findings are:

- ∙\\bulletTideRLachieves over 5\.6×\\timesRL training throughput on text\-only tasks and reduces training time by 51\.1% to reach similar task performance\.
- ∙\\bulletFor multi\-modal tasks,TideRLimproves RL throughput by over 33% and reduces training time by 62\.2% for similar task performance\.
- ∙\\bulletCTB improves KV cache hit rate by 1\.58×\\timesand generation throughput by 1\.15×\\timesby mitigatingC1\.
- ∙\\bulletRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}reduces per\-step training time by up to 44\.3% by selecting the proper ref\-actor execution mode for different readiness patterns \(C2\)\.
- ∙\\bulletERS uses readiness signals to reallocate GPUs between rollout and training, reducing total waiting time by up to 77\.6% \(C3\)\.

### 6\.1Methodology

Testbed Configuration\.We evaluateTideRLon a physically disaggregated cluster that completely separates the RL training backend from the environment simulation frontend\. The training testbed consists of four compute nodes\. Each node is equipped with 8×\\timesNVIDIA H100 GPUs interconnected via NVLink, a 64\-core Intel Xeon CPU, and 1\.5 TB of RAM memory\. To support high\-throughput distributed checkpointing and rapid parameter synchronization, a shared JuiceFS\-backed NFS is deployed across the training cluster\.

Workloads and Tasks\.To comprehensively evaluateTideRLacross diverse agentic scenarios, we select a mixture of standard text\-based and complex multimodal tasks\. All the tasks simulate varying lengths of trajectories, and are widely adopted by researchers\[agentbench,visualagentbench\]\. For text\-based workloads, we employ a hybrid task suite comprising WebShop\[webshop\]and AlfWorld\[alfworld\]\. For multi\-modal workloads, we utilize OSWorld\[osworld\]and ScienceBoard\[sciboard\]\. They require the agent to process high\-resolution screenshots and interact with graphical user interfaces \(GUIs\), heavily stressing the prefill and context\-carriage capacities of Rollout\. In the meantime, multi\-modal tasks allow a larger allowed turns of interactions\. We disclose the different behaviors of these workloads in Table[4](https://arxiv.org/html/2608.10402#A3.T4)of Appendix[C](https://arxiv.org/html/2608.10402#A3)\. We conduct an evaluation every 20 training steps to evaluate the training performance\.

Disaggregated Environment\.We utilize a dedicated bare\-metal Kubernetes cluster provisioned with 1024 CPU cores to run environments\. This cluster exposes APIs to orchestrate environment lifecycles, enabling the Rollout workers to start, interact with, and close external simulation instances without blocking the model execution threads\.

Models\.We evaluate the system using a wide spectrum of state\-of\-the\-art open\-weights models to demonstrate its architectural generality\. For text\-only tasks, we employ the Qwen\-2\.5 series \(7B and 14B,\[qwen25\]\)\. For multimodal tasks, we utilize advanced multi\-modal language models, Qwen\-3\-VL \(4B,\[qwen3vl\]\), Qwen\-3\.5 \(9B,\[qwen35\]\)\. These models have diverse architectures\. Qwen\-3\.5 also employs linear attention and multi\-token prediction\[mtp\]\. This diverse selection of model scales and architectures allows us to analyze the system’s performance and memory management efficiency across different compute\-to\-memory\-bandwidth regimes\.

Baselines\.To evaluateTideRLagainst the relevant design points, we compare it with three RL training frameworks:

- ∙\\bulletVeRL\[verl\]: A highly optimized synchronous RL framework that enforces strict phase barriers between generation and training\. It serves as our primary synchronous baseline to demonstrate the massive idle overheads inherent in coupled architectures\.
- ∙\\bulletAReaL\[areal\]: A foundational asynchronous RL framework that physically decouples the Rollout and Trainer workers\. It explicitly utilizes the active partial rollout mechanism introduced by APRIL\[april\]to manage off\-policy staleness and improve concurrency, representing standard request\-level asynchronous scheduling\.
- ∙\\bulletStreamRL\[streamrl\]: An advanced asynchronous framework featuring stream generation support\. By streaming trajectories directly to the Trainer without waiting for the entire global batch, it overlaps pipeline execution and represents the current state\-of\-the\-art in asynchronous RL scheduling\. For fair comparison, StreamRL always utilize the maximum\-allowed GPUs\.

All frameworks use the same GPU budget, task stream, model checkpoints, staleness bound, global batch configuration, and rollout/training backends whenever the framework supports them\. For fixed\-partition asynchronous baselines, we sweep the rollout GPU ratio over the same candidate set used in Figure[4](https://arxiv.org/html/2608.10402#S2.F4)and report the best\-performing setting for each workload\.TideRLstarts from the same candidate ratio, but ERS may reassign ranks during execution\. We choose not to include Seer\[seer\]as a baseline because its synchronous architecture is represented by VeRL, and its suffix decoding optimization increases rollout time on our multi\-turn workloads due to draft\-verification overheads, as discussed in[˜7](https://arxiv.org/html/2608.10402#S7)\.

Metrics\.We use training throughput as the primary system metric: the number of generated tokens that are eventually consumed by the Trainer per second\. This excludes stale or discarded rollout tokens and therefore measures useful progress for on\-policy RL\. We also report task performance, KV cache hit rate, generation throughput, RWT, and TWT to explain where the end\-to\-end gains come from\. In other words, training throughput is our operational measure of RL training goodput\.

### 6\.2End\-to\-End Performance on Text\-Only Models

We runTideRLand the baselines for 100 training steps\. Unless otherwise stated, fixed\-partition asynchronous systems use the best static rollout ratio from our sweep; on this workload, that ratio is 0\.25\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/thrp_text.png)Figure 8:TideRLachieves the highest throughput for different models on text\-only tasks across four frameworks\.Throughput\.Figure[8](https://arxiv.org/html/2608.10402#S6.F8)reports training throughput across models and frameworks\.

Compared with the synchronous VeRL baseline,TideRLachieves a 5\.6×\\timesspeedup\. VeRL is bottlenecked by the slowest trajectories in each global batch: a wrong action can require extra environment turns and substantially extend rollout time\. Its colocated synchronous execution also repeatedly swaps Rollout and Trainer states at every step, further reducing useful training throughput\.

AReaL improves over VeRL by decoupling generation from training, but its best fixed partition is still a single compromise across different readiness regimes\. When evaluation or long\-tail tasks make readiness sparse, Trainer stalls; when simple tasks make readiness dense, Rollout stalls behind the Trainer\.TideRLreacts to these regimes through ERS and the twoRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}modes, achieving a 1\.8×\\timesthroughput improvement over AReaL\.[˜6\.5](https://arxiv.org/html/2608.10402#S6.SS5)breaks down this effect in the RAS/TPRM plane\.

StreamRL reduces global\-batch waiting by streaming micro\-batches, but on text workloads each DP rank can receive more than 16 micro\-batches per step\. This bursty stream triggers frequent Ref\-Actor model alternation and dominates the saved stall time; the 14B run does not complete within six hours\.TideRLkeeps the streaming benefit while avoiding model thrashing withRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}, yielding over 7×\\timesspeedup in this setting\.

Training Performance\.We further exhibit the Best\-of\-N \(BoN,\[bon\]\) reward and the pass rate after the 100\-th training step in Figure[9](https://arxiv.org/html/2608.10402#S6.F9)\. The strictly synchronized VeRL has all the training data perfectly on policy with zero staleness\. As expected, it exhibits the best training performance\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/perf_text.png)Figure 9:The BoN reward and pass rate for different models on text\-only tasks\.Among the asynchronous RL frameworks,TideRLachieves the best task performance and remains close to VeRL, with a BoN reward deficit of 0\.01 and a pass\-rate deficit of 0\.5%\. It reaches this performance using only 48\.9% of VeRL’s wall\-clock time\.

Compared with StreamRL and AReaL,TideRLgenerates and consumes useful trajectories faster, so more of its training data is consumed within the zero\-staleness window\. The other asynchronous baselines more often fall back to one\-step\-stale data, which explains whyTideRLis closer to the synchronous learning curve while retaining much higher throughput\.

### 6\.3End\-to\-End Performance on Multi\-Modal Models

We runTideRLand the baselines on multi\-modal tasks and evaluate both throughput and task performance\. Compared with text\-only workloads, multi\-modal trajectories have longer environment interactions, larger observations, and more variable rollout/training balance, as detailed in Appendix[C](https://arxiv.org/html/2608.10402#A3)\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/thrp_mm.png)Figure 10:TideRLachieves the highest training throughput on multi\-modal tasks across four RL frameworks\.Throughput\.Figure[10](https://arxiv.org/html/2608.10402#S6.F10)shows thatTideRLachieves the highest training throughput on multi\-modal tasks\.

VeRL suffers more severely in this setting because long GUI interactions and high\-variance observations amplify global\-batch tail latency\.

All asynchronous frameworks improve over VeRL because sparse micro\-batch arrivals make overlap more valuable\. This regime also reduces StreamRL’s model\-swapping pressure compared with text tasks\. Even after tuning AReaL’s fixed ratio to provide sufficient Trainer capacity,TideRLachieves 6\.02×\\timesthroughput over VeRL and more than 1\.33×\\timesover the asynchronous baselines\. The gain is smaller than in text\-only tasks because larger Trainer ranks reduce the number of ranks that ERS can migrate, but readiness\-aware scheduling still avoids the worst fixed\-partition stalls\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/perf_mm.png)Figure 11:The BoN reward and pass rate for different models on multi\-modal tasks\.Training Performance\.Figure[11](https://arxiv.org/html/2608.10402#S6.F11)reports the BoN reward and pass rate after 40 training steps, whereTideRLconsistently outperforms the three asynchronous baselines\. However, due to the high variance in training workloads, ERS generates deliberately conservative scaling plans\. Because Trainer scale\-up operations are bounded to one rank per step, the Trainer eventually emerges as the long\-term pipeline bottleneck, causing the system to naturally gravitate toward a one\-step off\-policy staleness\. While this staleness introduces a slight performance degradation compared to the fully synchronous VeRL,TideRLcompletes the 40 steps in only 37\.8% of the wall\-clock time, justifying the trade\-off between algorithmic equivalence and training throughput\.

### 6\.4Improvement Breakdown

To quantify the contribution of eachTideRLcomponent, Table[2](https://arxiv.org/html/2608.10402#S6.T2)presents an ablation study on OSWorld with Qwen\-3\-VL 4B\. We use StreamRL as the baseline because it already streams rollout data to the Trainer, then addRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}, ERS, and CTB cumulatively\. The table reports average throughput over the first 10 steps, including the initial evaluation step used to verify model correctness\.

Table 2:Throughput improvement breakdown on OSWorld with Qwen\-3\-VL 4B\.MethodThroughput \(k token/s\)ImprovementBaseline20\.2–\+RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}23\.110\.4%\+ERS31\.835\.7%\+CTB33\.34\.8%StreamRL’s main bottleneck is frequent model swapping across many micro\-batches\.RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}removes PCIe transfers and GPU context\-switching overhead from this path, improving Trainer throughput\. ERS further adjusts GPU allocation: after the first step, it scales Rollout down and Trainer up, adding 35\.7% improvement on top ofRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}by reducing both RWT and TWT\. CTB improves the Rollout side by limiting harmful concurrency and reducing prefix recomputation, especially in the first step where many evaluation tasks run together\. This yields a 54\.8% first\-step throughput improvement and a 4\.8% overall improvement\.

### 6\.5Extended Evaluation and Micro\-Benchmarks

We next isolate how each component addresses the bottlenecks in[˜2\.2](https://arxiv.org/html/2608.10402#S2.SS2)\.

CTB improves KV cache hit rate and rollout throughput\.We perform a step of 1,312 WebShop tasks on one H100\. Figure[13](https://arxiv.org/html/2608.10402#S6.F13)compares CTB with fixed concurrency settings of 1024 \(maximum allowed, F10\) and 512 \(F9\)\. Figure[13](https://arxiv.org/html/2608.10402#S6.F13)shows that CTB improves cache hit rate by1\.58×1\.58\\timesand generation throughput by1\.15×1\.15\\times, reducing step duration by 15\.6%\. The vanilla scheduler’s frequent preemption drives the cache hit rate as low as 0\.9%, while CTB keeps it above 6\.0%, close to the ideal upper limit\. Although hierarchical caching such as SGLang HiCache\[hicache\]can expand cache capacity, CTB remains necessary because it avoids excessive PCIe transfers and prefix recomputation\. We omit HiCache in our deployment due to initialization overhead and host\-memory contention, since the global trajectory buffer and offloaded Trainer model already exhaust the available CPU RAM\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/ctb_micro_metrics.png)Figure 12:CTB beats F10 and F9 on both metrics\.
![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/ctb_kvcache_hit_rate.png)Figure 13:CTB achieves higher throughput and hit rate, in comparison with Figure[2](https://arxiv.org/html/2608.10402#S2.F2)\.

![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/rap_utils.png)Figure 14:RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}\(CM, DM\) has higher throughput under ready micro\-batch execution\.
![Refer to caption](https://arxiv.org/html/2608.10402v1/figs/ers_utils.png)Figure 15:ERS relieves RWT and TWT by reacting to readiness imbalance\.

𝐑𝐀2​𝐏\\mathbf\{RA\}^\{2\}\\mathbf\{P\}reduces stall and thrashing\.Figure[15](https://arxiv.org/html/2608.10402#S6.F15)shows the overhead for processing eight ready micro\-batches on four H100 GPUs with data parallelism only\. Each micro\-batch has 8,192 tokens\. This setup isolates the high\-RAS regime where the Trainer begins with enough work to expose model alternation overhead\. We compareRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}’s colocated mode \(CM\) and decoupled mode \(DM\) against vanilla streaming with \(VSO\) and without \(VS\) model offloading\. PCIe transfers and GPU context switching introduce significant overhead in the vanilla designs, whileRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}reduces overhead by up to 44\.3%\. All four methods avoid full\-global\-batch stalls because they process ready micro\-batches immediately\.

ERS relieves both Rollout and Trainer waiting\.Figure[15](https://arxiv.org/html/2608.10402#S6.F15)reports the sum of RWT and TWT in the first 100 WebShop steps on one 8×\\timesNVIDIA H100 node, compared with the fixed rollout ratios in Figure[4](https://arxiv.org/html/2608.10402#S2.F4)\. Among fixed settings, a rollout ratio of 0\.25 gives the best throughput, but it still suffers high RWT in normal steps and high TWT during intermediate evaluation\. These phases correspond to alternating large\-RAS/short\-TPRM and small\-RAS/long\-TPRM regimes\. ERS moves GPUs between Rollout and Trainer and selects the matchingRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}mode, reducing total TWT and RWT by 68\.6–77\.6%\.

## 7Discussion

Table 3:Single\-instance rollout for Qwen\-2\.5\-7B profiling on WebShop\. One step comprises 32 train groups with 8 samples each and 200 eval groups with 4 samples each\.ConfigurationTime \(s\)Acc\. RateAcc\. LenOverheadCTB Off \(w/o SD\)548\.5\-\-\-CTB Off \(w/ SD\)575\.440\.1%2\.47\+4\.9%CTB On \(w/o SD\)460\.4\-\-\-CTB On \(w/ SD\)495\.443\.7%2\.83\+7\.6%

Incompatibility of Suffix Decoding\.While suffix decoding \(SD,\[suffix\]\) effectively accelerates single\-turn inference\[seer\], our profiling shows that it is detrimental to highly variable multi\-turn agentic workloads under our model scale\. As demonstrated in Table[3](https://arxiv.org/html/2608.10402#S7.T3), enabling SD increases total rollout time regardless of CTB status\. Complex environmental interactions yield low acceptance rates \(∼40%\\sim\\\!40\\%\) and short acceptance lengths \(∼2\.8\\sim\\\!2\.8tokens\), so draft\-verification overhead outweighs speculative gains\. Consequently, SD is excluded from our implementation, though future SD mechanisms could be integrated intoTideRLwhen they are beneficial for a workload\.

Fault Tolerance and Recovery\.In large\-scale agentic RL, node failures are inevitable\.TideRLprovides robust fault tolerance without requiring global pipeline restarts\. If a Rollout rank fails, the CTB scheduler immediately quarantines the node by halting new task dispatches, while the Trainer seamlessly excludes it from the asynchronoussync\_paramsbroadcast\. Furthermore,TideRLleverages asynchronous distributed checkpointing\[distcheckpoint,bytecheckpoint\]to periodically persist policy weights, ensuring rapid state recovery and minimal interruption to the global training loop\.

Algorithmic Effectiveness\.InTideRL, tokens within a single multi\-turn trajectory might be generated by slightly different policy versions as weights update mid\-rollout\. Consistent with findings in asynchronous RL literature \(e\.g\., APRIL\[april\]and AReaL\[areal\]\), this does not degrade algorithmic convergence\. The KL penalty natively regularizes the policy against such version drifts\. As demonstrated in[˜6](https://arxiv.org/html/2608.10402#S6),TideRLachieves highly competitive reward growth and learning efficiency compared to strict synchronous baselines\.

Limitations and Future Work\.WhileTideRL’s algorithm design tolerates policy version drift during asynchronous training, evaluation tasks demand an anchored, static model version to ensure metric consistency\. Currently, enforcing this strict version synchronization for evaluation can temporarily disrupt the asynchronous pipeline momentum\. Future work could address this by integrating a relay weight synchronization mechanism \(e\.g\., Laminar\[laminar\]\)\. This would allow the system to maintain decoupled, frozen model snapshots specifically for evaluation tasks without blocking the trainer, provided that the multi\-version footprint in host memory can be carefully optimized to be kept with the reference model and large image files in trajectories simultaneously\.

## 8Related Work

Distributed RL Systems for LLMs\.RL systems for LLMs have primarily optimized GPU utilization under coupled or partially decoupled execution\. Synchronous frameworks such as VeRL and RLHFuse\[verl,rlhfuse\]colocate generation and training and therefore inherit strict phase barriers\. Production frameworks such as SLIME\[slime\]and OpenRLHF\[openrlhf\]integrate distributed training backends with high\-throughput inference engines, but still rely on fixed execution roles during a training job\. Recent asynchronous systems overlap rollout and training and reduce long\-tail stalls\[streamrl,asyncflow,laminar\]; AReaL further bounds policy staleness with active partial rollouts\[areal,april\]\. These systems expose asynchrony, but they do not jointly decide the ref\-actor execution mode and the rollout\-trainer GPU split from data\-readiness signals\.TideRLtargets this missing control loop: CTB shapes ready\-data production,RA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}provides two consumption strategies, and ERS schedules both ranks andRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}modes as readiness shifts\. Speculative decoding\[seer,specactor\], memory sampling\[infinitesampling\], and hardware\-specific dataflows\[mindspeedrl\]are orthogonal optimizations that can be integrated when they are beneficial for the workload\.

LLM Inference for Agentic Workloads\.Foundational inference engines optimize throughput via continuous batching\[vllm,orca\], RadixAttention\-based prefix sharing\[sglang,cachedattention\], and prefill\-decode disaggregation\[distserve,megascale\-infer\]\. Agent\-centric serving systems model multi\-turn sessions explicitly, adding session\-aware scheduling\[agserve,helium\], KV\-cache pinning\[continuum\], and congestion control\[concur\]to reduce cache evictions during tool\-execution pauses\. These techniques improve serving throughput, but agentic RL adds algorithmic dependencies absent in serving: GRPO groups must complete before advantages are valid, evaluation tasks must preserve an anchored model version, and rollout staleness must be bounded by training progress\. CTB incorporates these dependencies into task admission, pausing, and resumption, so cache residency decisions serve the training loop rather than only request throughput\.

Distributed Training Architectures\.Standard large\-scale training relies on 3D parallelism for weight updates\[megatron,gpipe,pipedream,chimera\]\. DistTrain\[disttrain\]disaggregates the visual encoder from the language backbone for multi\-modal training\. General\-purpose elastic frameworks\[pollux,elasticflow,coddl\]reallocate resources through co\-adaptive batch sizes and throughput modeling, but their scaling operations usually suspend jobs, migrate large model states, or rebuild communication groups\. Those costs are too high for agentic RL, where the bottleneck can change across adjacent steps\.TideRLbuilds on existing training backends but aligns role switching with the natural post\-update synchronization point: stale rollout KV caches are invalidated anyway, and weight movement can be hidden undersync\_params\. This makes rank\-level elasticity practical inside a fixed\-size cluster allocation\.

## 9Conclusion

We presentTideRL, an asynchronous system for multi\-turn agentic RL\.TideRLkeeps Rollout efficient with task\-level CTB, consumes ready trajectories with the twoRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}Ref\-Actor strategies, and uses ERS to move GPUs between Rollout and Trainer as bottlenecks shift\. On real testbeds, it improves training throughput by over 1\.8×\\timeson text\-only tasks and over 33% on multi\-modal tasks compared with existing asynchronous systems\.

## References

## Appendix

## Appendix AAnalytical Model for𝐑𝐀2​𝐏\\mathbf\{RA\}^\{2\}\\mathbf\{P\}Selection

To decide the optimal execution mode inRA2​P\\mathrm\{RA\}^\{2\}\\mathrm\{P\}, we use an analytical model to evaluate the total latency to process a global batch ofNNmicro\-batches\. Lettr​e​ft\_\{ref\}andta​c​t​\_​ft\_\{act\\\_f\}denote the forward pass time for the Ref and Actor models respectively, andta​c​t​\_​bt\_\{act\\\_b\}denote the Actor’s backward pass time\. Letts​w​a​pt\_\{swap\}be the overhead of model swapping\. Given the ordered micro\-batch ready times𝒯=\{a1,a2,…,aN\}\\mathcal\{T\}=\\\{a\_\{1\},a\_\{2\},\\dots,a\_\{N\}\\\}relative to the start of the training step, RAS isN0=\|\{i∣ai≤0\}\|N\_\{0\}=\|\\\{i\\mid a\_\{i\}\\leq 0\\\}\|, and TPRM is represented by the post\-start gapsΔi=ai−ai−1\\Delta\_\{i\}=a\_\{i\}\-a\_\{i\-1\}fori\>N0\+1i\>N\_\{0\}\+1\.

Cost of Decoupled Streaming Mode\.In the decoupled mode, models are distributed across separate GPUs, completely eliminatingts​w​a​pt\_\{swap\}\. However, idle bubbles are not entirely absent; they exist aspipeline fill bubblesduring the startup phase\.

As illustrated in the timeline \(see Figure[6](https://arxiv.org/html/2608.10402#S5.F6)\), multiple Actor ranks initiate their forward passes concurrently\. Assuming an Actor group size ofMM, Actorii\(0≤i<M0\\leq i<M\) finishes its forward pass att≈ta​c​t​\_​ft\\approx t\_\{act\\\_f\}\. However, the single Ref model processes the reference forward passes sequentially\. Theref\_log\_probsfor Actoriiare only generated att≈\(i\+1\)⋅tr​e​ft\\approx\(i\+1\)\\cdot t\_\{ref\}\. Consequently, Actoriimust wait for a duration ofi⋅tr​e​fi\\cdot t\_\{ref\}\(assumingta​c​t​\_​f≈tr​e​ft\_\{act\\\_f\}\\approx t\_\{ref\}\) before initiating its backward passBiB\_\{i\}\.

Beyond this initial pipeline flush overhead, the steady\-state throughput is governed by the maximum of each ready gap and the Actor’s computation time \(ta​c​t​\_​f\+ta​c​t​\_​bt\_\{act\\\_f\}\+t\_\{act\\\_b\}\)\. Micro\-batches that are already ready at the step boundary have zero ready gap, so a largeN0N\_\{0\}immediately amortizes the pipeline fill bubble\. The total latency can be approximated as:

Cd​e​c≈∑i=0M−1\(i⋅tr​e​f\)⏟Pipeline Fill Bubble\+∑i=1Nmax⁡\(δi,ta​c​t​\_​f\+ta​c​t​\_​b\)C\_\{dec\}\\approx\\underbrace\{\\sum\_\{i=0\}^\{M\-1\}\(i\\cdot t\_\{ref\}\)\}\_\{\\text\{Pipeline Fill Bubble\}\}\+\\sum\_\{i=1\}^\{N\}\\max\(\\delta\_\{i\},t\_\{act\\\_f\}\+t\_\{act\\\_b\}\)whereδi=0\\delta\_\{i\}=0fori≤N0i\\leq N\_\{0\},δN0\+1=max⁡\(aN0\+1,0\)\\delta\_\{N\_\{0\}\+1\}=\\max\(a\_\{N\_\{0\}\+1\},0\), andδi=ai−ai−1\\delta\_\{i\}=a\_\{i\}\-a\_\{i\-1\}otherwise\.

Cost of Optimized Colocated Mode\.In the colocated mode, the dynamic ready\-batch aggregation dictates that the aggregation sizekkis strictly non\-deterministic and monotonically adjusts based on real\-time task arrivals\. Therefore, the execution latency cannot be expressed as a closed\-form static formula\. Instead, the total timeCc​o​lC\_\{col\}is determined algorithmically by simulating the scheduler’s progression\.

The scheduler maintains a global clockTT, initialized toT=0T=0\. The evaluation proceeds as follows for the unprocessed micro\-batches starting at indexj=1j=1:

1. 1\.Wait for Data:If the current timeT<ajT<a\_\{j\}, the system idles until the next micro\-batch arrives:T←ajT\\leftarrow a\_\{j\}\.
2. 2\.Determine Aggregation \(kk\):The scheduler counts all ready micro\-batches that have arrived by timeTT\. Letkkbe the number of batches such that their arrival timeaj\+k−1≤Ta\_\{j\+k\-1\}\\leq T, capped by the maximum memory capacity\.
3. 3\.Execute Ref Model:The system incurs a swap overhead, then sequentially processeskkreference passes\. The clock updates: T←T\+ts​w​a​p\+k⋅tr​e​fT\\leftarrow T\+t\_\{swap\}\+k\\cdot t\_\{ref\}
4. 4\.Execute Actor Model:The system incurs another swap, followed by the Actor’s forward and backward passes\. The clock updates: T←T\+ts​w​a​p\+k⋅\(ta​c​t​\_​f\+ta​c​t​\_​b\)T\\leftarrow T\+t\_\{swap\}\+k\\cdot\(t\_\{act\\\_f\}\+t\_\{act\\\_b\}\)
5. 5\.Advance Index:The pointer advances \(j←j\+kj\\leftarrow j\+k\) and the cycle repeats untilj\>Nj\>N\.Cc​o​lC\_\{col\}evaluates to the final clock timeTT\.

Qualitative Mode Analysis\.Based on the formulated cost models, the selection between the two execution modes hinges on bothN0N\_\{0\}and the post\-start ready gaps\. A largeN0N\_\{0\}means the Trainer already has a startup burst to process, so the initial pipeline fill bubbles inCd​e​cC\_\{dec\}are rapidly amortized\. Short post\-start ready gaps then keep the decoupled pipeline saturated\. The decoupled mode provides superior throughput in these regimes by strictly eliminating thets​w​a​pt\_\{swap\}overhead, making it the optimal choice when maximizing processing speed is the priority and GPU resources are sufficient\.

Conversely, whenN0N\_\{0\}is small and post\-start ready gaps are long, data readiness dominates the overall latency\. Under these conditions, the colocated mode naturally absorbs the model\-swapping penalty within the physical generation delays \(Step 1 of the colocated algorithm\)\. Consequently, the colocated mode maintains comparable end\-to\-end latency to the decoupled pipeline while effectively halving the required GPU footprint, maximizing overall resource efficiency in bottlenecked regimes\.

## Appendix BImplementation Details

### B\.1Environment Interfacing and Lifecycle Management

TideRLinteracts with external task environments through a standardized set of lightweight RESTful\-like APIs\. The three primary endpoints are,/start,/observation, and/end\.

In asynchronous agentic RL, tasks may enter a long pause by CTB, causing standard environments to falsely detect inactivity and terminate the session\. To solve this,TideRLintroduces two critical supplementary endpoints:/pauseand/resume\. These APIs define a protected temporal window\. Upon eviction by the CTB scheduler, the/pausesignal explicitly halts the environment’s internal timeout counters, guaranteeing that long\-running sessions and their associated external states \(e\.g\., Docker containers or browser contexts\) remain intact until execution resumes\.

### B\.2Rollout Engine and CTB Observability

To execute token\-aware Continuous Task Batching \(CTB\), the system must accurately track sequence lengths across the cluster\.TideRLimplements dual\-mode length awareness:

- ∙\\bulletActive Reporting:The user\-defined agentic loop proactively reports token consumption and trajectory metadata back to the global scheduler at each interaction turn\.
- ∙\\bulletEngine Hijacking:For non\-intrusive tracking,TideRLdynamically hijacks the scheduling APIs of the underlying Rollout execution engines\. This extracts real\-time GPU scheduling metadata directly from individual Rollout ranks, avoiding reliance on rough heuristics\.

Further,TideRLnatively interfaces with the Prometheus endpoints exposed by SGLang and vLLM\. This integration provides granular observability into KV cache utilization, prefill latencies, and decode throughput, serving as foundational metrics for system profiling and performance tuning\.

### B\.3Trainer Modifications and Sequence Dispatching

TideRLexecutes customized data dispatching and micro\-batch scheduling to accommodate the asynchronous nature of agentic trajectories\. We also include trainer metrics, such as loss to our Prometheus, so that users can easily monitor if it is algorithmically correct, just like SLIME\[slime\]\.

- ∙\\bulletAsynchronous Micro\-Batching \(Megatron Core\):We modified Megatron Core to decouple forward and backward execution from the strict requirement of a fully assembled global batch\. This permits the pipeline to initiate execution on partial micro\-batches immediately upon arrival\. Notably, we retain the original 1F1B \(One\-Forward\-One\-Backward\) pipeline schedule without modification\. BecauseTideRLdoes not employ Virtual Pipeline Parallelism \(VPP\), the absolute pipeline completion time is strictly determined by the arrival and execution of the final micro\-batch; altering the schedule interleaving yields no structural latency reduction\.
- ∙\\bulletLogical DP Grouping \(FSDP\):In architectures like PyTorch FSDP, different FSDP ranks within a same DP group mandate collective data fetching and synchronized gradient reduction\. To prevent intra\-group stragglers,TideRLpartitions physical DP ranks intological DP ranks\. Sequences of highly similar lengths are tightly packed and fed to logical ranks within the same DP group, ensuring symmetric processing times and eliminating synchronization barriers\.

## Appendix CDetailed Task Specifications

To comprehensively evaluateTideRLacross diverse agentic scenarios, we detail the interaction constraints and trajectory characteristics for our workloads\. These are categorized into text\-only and multi\-modal tasks based on the nature of their observations and interactions\.

### C\.1Text\-Only Workloads

For standard text\-based tasks, we enforce a standardized upper bound of 20 interaction turns and a maximum context window of 8192 tokens per trajectory\. Table[4](https://arxiv.org/html/2608.10402#A3.T4)summarizes the specific interaction statistics for these benchmarks, integrating environment\-specific data from systematic evaluations\.

- ∙\\bulletWebShopsimulates a real\-world e\-commerce experience with approximately one million products\. These tasks average 13 interaction turns and approximately 3000 tokens per trajectory, reflecting high\-density HTML\-based observations\.
- ∙\\bulletAlfWorld\-WebShop SuiteincludesAlfWorldalong with the WebShop dataset\. AlfWorld is an embodied household agent, ordering agents to accomplish household tasks, such as finding items\. We group the two types, and try to make the agent capable of different tasks\.
- ∙\\bulletEnvironmental Complexity: While our evaluation enforces a 20\-turn limit for these tasks, inherent complexity varies\. For example,AlfWorldandWebShoppresents totally different interaction patterns\. What’s more, as the training goes, we observe that the average turns and the average sequence lengths tend to decrease as the training step increases\.

### C\.2Multi\-Modal Workloads

Complex multi\-modal and long\-horizon tasks, such asOSWorldandScienceBoard, are permitted up to 50 interaction turns to accommodate the increased complexity of GUI\-based navigation\. To balance visual fidelity with context efficiency, we enforce a sliding window strategy for visual context retention across both environments\. Specifically, screenshots are appended sequentially at each turn\. Once a trajectory accumulates ten images, the system preemptively prunes the historical visual context, retaining only the five most recent images\. This mechanism splits the trajectory and strictly bounds the dynamic image allowance within a limit per step, avoiding overly long trajectories\. Images are captured at a resolution of1280×7201280\\times 720\. In the meantime, it also takes longer for multi\-modal tasks to start an environment, which is another reason for them to have a lower throughput\.

- ∙\\bulletOSWorldrequires agents to process high\-resolution screenshots and interact with general\-purpose graphical user interfaces \(GUIs\), such as Microsoft Office\. We run the task with 5–9 image allowance\.
- ∙\\bulletScienceBoardis a comprehensive benchmark designed to evaluate multimodal autonomous agents in realistic scientific workflows\. These tasks demand that agents navigate specialized scientific software GUIs and process complex domain\-specific visual data \(e\.g\., molecular structures or data plots\)\. We run the task with 8–15 image allowance\.

Table 4:Task Trajectory StatisticsWorkloadAvg\. RoundsAvg\. TokensImage AllowanceText\-Only TasksWebShop8→\\to53400→\\to2400N/AAlfWorld16→\\to117400N/AMulti\-Modal TasksOSWorld35→\\to15120005–9 \(Sliding\)SciBoard39→\\to2924000→\\to230008–15 \(Sliding\)