EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents

# EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents
Source: [https://arxiv.org/html/2607.09773](https://arxiv.org/html/2607.09773)
Taofeng Xue\*,†\\daggerMeituanChong Peng\*,†\\daggerMeituanJinrui Ding\*MeituanSicheng FanMeituanFudan UniversityJiale HongMeituanShanghai Jiao Tong UniversityYufei GaoMeituanZhejiang UniversityXiaocheng ZhangMeituanLinsen GuoMeituanXin YangMeituanDengchang ZhaoMeituanXiandi MaMeituanYuchen XieMeituanPeng PeiMeituanXunliang CaiMeituanXipeng QiuFudan University

###### Abstract

Computer\-use agents must solve long\-horizon tasks through repeated interaction with partially observable, multimodal desktop environments\. Although imitation learning and offline trajectory refinement provide strong priors, static traces cannot cover the causal feedback loop of real computer use: each action changes the screen state, future action space, and recovery options\. EvoCUA\-1\.5 extends self\-evolving computer\-use agents from offline experience learning to online reinforcement learning, where policies interact with executable sandbox environments and improve from verifiable task outcomes\. Online RL in this setting requires more than directly reusing single\-turn language\-RL recipes\. Multi\-turn interaction introduces context\-managed observations, sparse terminal rewards, variable\-length trajectories, and slow environment feedback\. EvoCUA\-1\.5 addresses these challenges with Step\-Level Policy Optimization \(STEPO\), which preserves trajectory\-level advantage balance after decomposition into step\-level samples; policy\-aware filtering and pass\-rate calibration over verifiable synthesized tasks; Dynamic Tri\-Adaptive Curriculum \(DTAC\), which combines learnable tasks, difficult positive replay, and controlled infeasible\-task exposure; and a fully asynchronous RL infrastructure with staleness control and mini\-group batching\. Experiments show that these components improve training stability and downstream performance\. EvoCUA\-1\.5 achieves 63\.2% success on OSWorld\-Verified, outperforming comparable 32B/35B\-scale open\-weight baselines and even approaching models with significantly larger parameter counts\. Overall, EvoCUA\-1\.5 provides a practical framework for scaling online RL in multi\-turn computer\-use agents\.

††footnotetext:\*Equal contribution\.†\\daggerCorresponding authors\.![Refer to caption](https://arxiv.org/html/2607.09773v1/x1.png)Figure 1:Performance comparison on OSWorld\-Verified\. EvoCUA\-1\.5 achieves 63\.2% success rate, outperforming comparable 32B/35B\-scale open\-weight models\.## 1Introduction

Developing generalist computer\-use agents that can operate graphical user interfaces \(GUIs\) is a critical step toward autonomous computer use\. Unlike specialized automation tools, these agents must perceive complex visual contexts, infer task\-relevant states from partial observations, and execute long\-horizon workflows across heterogeneous applications\. Recent native vision\-language models \(VLMs\) have made substantial progress in unifying perception, reasoning, and action within end\-to\-end computer\-use agents\(Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10); ByteDance Seed Team,[2025](https://arxiv.org/html/2607.09773#bib.bib17); Wanget al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib3),[a](https://arxiv.org/html/2607.09773#bib.bib4)\)\. Nevertheless, reliable performance in realistic computer\-use environments remains difficult, particularly for tasks that require multi\-step exploration, recovery from mistakes, and adaptation to dynamic interface states\.

Current progress is still largely driven by imitation learning or offline refinement on fixed interaction traces\. Such static data provides useful behavioral priors, but it cannot fully capture the causal feedback loop that characterizes real computer use: each action changes the environment state, which in turn determines the next observation and future action space\. Consequently, models trained primarily on offline trajectories are constrained by the coverage of previously collected experience and may struggle with long\-tail states, delayed consequences, and failures that emerge only during interaction\. This limitation motivates a shift from static trajectory scaling to*online experience scaling*, in which the policy directly interacts with executable environments, receives verifiable feedback, and improves from newly generated experience\.

EvoCUA\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)explored self\-evolution from offline synthetic experience\. EvoCUA\-1\.5 extends this line of work by introducing online reinforcement learning for multi\-turn computer\-use agents\. We identify several design choices that make online RL effective in computer\-use settings\. We find that directly transplanting single\-turn RL recipes to computer\-use agents is insufficient\. A computer\-use trajectory contains multiple decisions, and each decision is conditioned on a context\-management policy that may evict, fold, or summarize earlier observations\. Training samples must therefore be constructed after context management rather than obtained by simply masking a static trajectory\. This seemingly implementation\-level detail changes the optimization problem: a trajectory\-level reward is expanded into multiple step\-level samples, and naive GRPO\-style advantage reuse can overweight long trajectories and destabilize learning\.

Beyond the objective itself, online RL for computer\-use agents must also address the efficiency and stability of the full training loop\. Because terminal rewards are sparse and trajectories are expensive to generate, poorly calibrated task distributions can waste rollout compute or produce high\-variance updates\. Tasks that are already too easy provide little contrast, tasks far beyond the current policy mostly yield failed rollouts, and noisy validators can convert environmental feedback into misleading gradients\. Meanwhile, the system bottleneck is more severe than in text\-only RL: each interaction step involves screenshot rendering, action execution, environment transition, and reward verification, making experience production substantially slower than model optimization\. Stable and efficient learning therefore requires coordinated design across objectives, task selection, curriculum sampling, and asynchronous infrastructure\.

This paper presentsEvoCUA\-1\.5, an online RL framework for multi\-turn computer\-use agents\. We formalize computer\-use interaction with explicit reasoning and context management, and then introduce an integrated set of optimization, data, curriculum, and systems mechanisms for stable online optimization\. First, we proposeStep\-Level Policy Optimization \(STEPO\), which computes advantages at the trajectory level and redistributes them across turn\-level samples to preserve group\-level balance after trajectory decomposition\. Second, building on the verifiable task synthesis introduced in EvoCUA, we develop a stricter data filtering and calibration procedure for online RL, using sandbox feasibility checks, executable validators, model\-judge trajectory analysis, and policy\-dependent pass\-rate statistics to maintain high\-SNR training data\. Third, we proposeDynamic Tri\-Adaptive Curriculum \(DTAC\), which dynamically combines variance\-adaptive sampling, difficulty\-adaptive positive replay, and controlled sampling of infeasible tasks\. Finally, we build an asynchronous online RL infrastructure that decouples rollout generation from policy updates through a staleness\-controlled data buffer\.

Our main contributions are summarized as follows:

- •Step\-level policy optimization\.We proposeStep\-Level Policy Optimization \(STEPO\)to address the mismatch between trajectory\-level rewards and turn\-level training samples\. By distributing each trajectory advantage across its decomposed steps, STEPO improves training efficiency over directly applying GRPO to multi\-turn computer\-use trajectories\.
- •Policy\-aware data filtering and calibration\.We introduce a high\-SNR online data filtering mechanism built on EvoCUA’s verifiable task synthesis, selecting tasks according to sandbox feasibility, validator reliability, and model\-dependent pass\-rate statistics\.
- •Adaptive curriculum learning\.We proposeDynamic Tri\-Adaptive Curriculum \(DTAC\)to dynamically balance learnable tasks, difficult positive replay, and infeasible\-task sampling, improving stability under sparse rewards and changing policy capabilities\.
- •Asynchronous interaction infrastructure\.We build a staleness\-aware asynchronous training infrastructure that decouples experience generation from policy updates, reducing GPU idle time caused by slow environment interaction and variable\-length trajectories\.

Together, these components turn online computer\-use interaction into a scalable training signal\. Our experiments and ablations show that the resulting framework improves both optimization stability and downstream OSWorld\-style performance, while also revealing important practical pitfalls, including model\-dependent data selection and potential reward hacking from process reward models\.

## 2Preliminaries

EvoCUA\-1\.5 inherits the basic computer\-use agent formulation from EvoCUA\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)\. We therefore recap only the notation required for online RL and context management\. Given a natural\-language instructiongg, the agent interacts with a partially observable computer\-use environment\. At steptt, the underlying computer statests\_\{t\}is rendered as a screenshot observationot=Render​\(st\)o\_\{t\}=\\mathrm\{Render\}\(s\_\{t\}\)\. Conditioned on the retained interaction historyhth\_\{t\}, the policy generates an explicit reasoning traceztz\_\{t\}and an executable actionata\_\{t\}:

πθ​\(zt,at∣ht,ot\)\.\\pi\_\{\\theta\}\(z\_\{t\},a\_\{t\}\\mid h\_\{t\},o\_\{t\}\)\.\(1\)After executingata\_\{t\}, the environment transitions to the next state and eventually returns a sparse terminal reward from an executable validatorVgV\_\{g\}:

R​\(sT;g\)≜𝕀​\[Vg​\(sT\)=True\]\.R\(s\_\{T\};g\)\\triangleq\\mathbb\{I\}\[V\_\{g\}\(s\_\{T\}\)=\\mathrm\{True\}\]\.\(2\)The key distinction in EvoCUA\-1\.5 is not the base computer\-use interaction formulation, but how online RL should construct training samples and assign trajectory\-level rewards under multi\-turn context management\.

### 2\.1Context Management in Multi\-Turn Computer\-Use Agents

![Refer to caption](https://arxiv.org/html/2607.09773v1/figs/context_management.png)Figure 2:Context management in multi\-turn computer\-use agent training\. The full trajectory history is transformed into a managed context before each decision: earlier turns are compressed into lightweight action history, while recent turns retain full multimodal information\. Consequently, training data must be constructed at the step level after context management\.Before steptt, the complete interaction history is

htf​u​l​l=\{g,o0,z0,a0,…,ot−1,zt−1,at−1\}\.h\_\{t\}^\{full\}=\\\{g,o\_\{0\},z\_\{0\},a\_\{0\},\\ldots,o\_\{t\-1\},z\_\{t\-1\},a\_\{t\-1\}\\\}\.\(3\)Feeding this full history directly to the model is impractical for long\-horizon computer\-use tasks, because screenshots and reasoning traces quickly consume the context window and introduce redundant visual information\. Modern computer\-use agents therefore rely on context\-management strategies such as sliding windows, preserved thinking, folding, and summarization\(Wanget al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib3); Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10); Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)\. As illustrated in Figure[2](https://arxiv.org/html/2607.09773#S2.F2), this work focuses on the common sliding\-window setting\. Given a window sizewwandt\>wt\>w, the managed context stores earlier turns as compact action history and retains full observation–thought–action triples only for the most recent turns:

ht\(w\)=\{g,a0,…,at−w−1⏟compact action history,ot−w,zt−w,at−w,…,ot−1,zt−1,at−1\}\.h\_\{t\}^\{\(w\)\}=\\\{g,\\underbrace\{a\_\{0\},\\ldots,a\_\{t\-w\-1\}\}\_\{\\text\{compact action history\}\},o\_\{t\-w\},z\_\{t\-w\},a\_\{t\-w\},\\ldots,o\_\{t\-1\},z\_\{t\-1\},a\_\{t\-1\}\\\}\.\(4\)
This context transformation has a direct implication for training\. After stepttis generated, its reasoning and action are no longer merely prediction targets; they become part of the managed input for stept\+1t\+1, while older information may be evicted or compressed\. A multi\-turn trajectory therefore cannot be converted into a single static training sequence with a fixed input–output mask\. To match inference\-time behavior, we construct one training example after each context\-management operation\. Under the sliding\-window strategy, every executable turn becomes a separate step\-level sample\. The remaining question is how to assign a trajectory\-level terminal reward to these step\-level samples, which motivates the optimization design in the next section\.

### 2\.2Learning Objective

For a group ofGGrollouts sampled from the same instruction or task family, each trajectoryτi\\tau\_\{i\}contains\|Ti\|\|T\_\{i\}\|executable turns and receives a terminal rewardRi∈\{0,1\}R\_\{i\}\\in\\\{0,1\\\}\. Online RL improves the policy by contrasting successful and failed trajectories produced by the current or recent policy\. The central challenge is to assign trajectory\-level rewards to step\-level samples without distorting group\-level advantage normalization or overweighting long trajectories\.

## 3Step\-Level Policy Optimization

Objective design is a central challenge for online reinforcement learning in computer\-use environments\. Unlike a single response conditioned on a fixed prompt, a computer\-use trajectory consists of a sequence of interdependent decisions, each conditioned on the evolving visual state and on the agent’s context\-management policy\. The reward, however, is typically observed only at the trajectory level after execution\. This section introduces Step\-Level Policy Optimization \(STEPO\), a group\-based RL objective adapted to this mismatch between trajectory\-level feedback and turn\-level training samples\.

### 3\.1Bias Induced by Naive GRPO under Trajectory Decomposition

Group Relative Policy Optimization \(GRPO\) is an appealing post\-training objective because it obviates a separate critic while normalizing rewards within a rollout group\(Shaoet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib2); Guoet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib1)\)\. Given a group ofGGtrajectories with terminal rewardsR1,…,RGR\_\{1\},\\ldots,R\_\{G\}, the trajectory\-level advantage is defined as

Ai=Ri−mean​\(𝐑\)std​\(𝐑\)\+ϵ,∑i=1GAi≈0\.A\_\{i\}=\\frac\{R\_\{i\}\-\\mathrm\{mean\}\(\\mathbf\{R\}\)\}\{\\mathrm\{std\}\(\\mathbf\{R\}\)\+\\epsilon\},\\qquad\\sum\_\{i=1\}^\{G\}A\_\{i\}\\approx 0\.\(5\)This zero\-mean property is defined over trajectories: before decomposition, each sampled trajectory contributes once to the group update\. In a single\-turn language task, this is consistent with assigningAiA\_\{i\}to all tokens in the only response\. In contrast, a computer\-use trajectory is naturally expanded into step\-level training samples,

τi=\{\(xi,t,yi,t\)\}t=1\|Ti\|,\\tau\_\{i\}=\\\{\(x\_\{i,t\},y\_\{i,t\}\)\\\}\_\{t=1\}^\{\|T\_\{i\}\|\},\(6\)wherexi,tx\_\{i,t\}denotes the context\-managed observation at steptt, andyi,ty\_\{i,t\}denotes the corresponding reasoning trace and executable action\. This expansion is necessary because the output at stepttbecomes part of the input at stept\+1t\+1, while earlier observations may be evicted, summarized, or compacted by the context\-management module\. Consequently, the policy must be optimized on the post\-context\-management view available at each decision point, rather than on a single static trajectory sequence\.

A naive application of GRPO after decomposition would assign the same trajectory advantageAiA\_\{i\}to every decomposed step\. For clarity, abstracting away clipping and KL regularization, the resulting update takes the form

𝒥naive​\(θ\)=𝔼​\[∑i=1G∑t=1\|Ti\|1\|yi,t\|​∑k=1\|yi,t\|Ai​ri,t,k​\(θ\)\],\\mathcal\{J\}\_\{\\mathrm\{naive\}\}\(\\theta\)=\\mathbb\{E\}\\left\[\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}\\frac\{1\}\{\|y\_\{i,t\}\|\}\\sum\_\{k=1\}^\{\|y\_\{i,t\}\|\}A\_\{i\}r\_\{i,t,k\}\(\\theta\)\\right\],\(7\)whereri,t,k​\(θ\)r\_\{i,t,k\}\(\\theta\)is the token\-level importance sampling ratio\. Aggregating the repeated advantage terms over steps shows that the effective coefficient assigned to trajectoryiibecomes

∑t=1\|Ti\|Ai=\|Ti\|​Ai\.\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}A\_\{i\}=\|T\_\{i\}\|A\_\{i\}\.\(8\)Thus, naive GRPO no longer corresponds to the original trajectory\-balanced objective\. Instead, it optimizes a length\-weighted surrogate in which longer trajectories induce larger\-magnitude updates\.

Equivalently, the normalized advantages are no longer balanced over the actual training samples\. The sample\-level mean advantage becomes

A¯sample=∑i=1G∑t=1\|Ti\|Ai∑i=1G\|Ti\|=∑i=1G\|Ti\|​Ai∑i=1G\|Ti\|,\\bar\{A\}\_\{\\mathrm\{sample\}\}=\\frac\{\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}A\_\{i\}\}\{\\sum\_\{i=1\}^\{G\}\|T\_\{i\}\|\}=\\frac\{\\sum\_\{i=1\}^\{G\}\|T\_\{i\}\|A\_\{i\}\}\{\\sum\_\{i=1\}^\{G\}\|T\_\{i\}\|\},\(9\)which is generally nonzero even though∑iAi≈0\\sum\_\{i\}A\_\{i\}\\approx 0\. This induces a systematic bias whenever trajectory lengths vary within a rollout group\. The issue is particularly consequential in computer\-use tasks, where trajectory length often correlates with task difficulty, recovery behavior, and failure modes\.

### 3\.2Step\-Level Policy Optimization

STEPO corrects this distortion by preserving the trajectory\-level advantage mass while still optimizing on step\-level samples\. For each trajectory, we first compute the GRPO\-style advantageAiA\_\{i\}and then distribute it uniformly across the steps in that trajectory:

A^i,tSTEPO=Ai\|Ti\|=Ri−mean​\(𝐑\)\(std​\(𝐑\)\+ϵ\)​\|Ti\|\.\\hat\{A\}\_\{i,t\}^\{\\mathrm\{STEPO\}\}=\\frac\{A\_\{i\}\}\{\|T\_\{i\}\|\}=\\frac\{R\_\{i\}\-\\mathrm\{mean\}\(\\mathbf\{R\}\)\}\{\(\\mathrm\{std\}\(\\mathbf\{R\}\)\+\\epsilon\)\|T\_\{i\}\|\}\.\(10\)This assignment conserves the original trajectory\-level advantage:

∑t=1\|Ti\|A^i,tSTEPO=Ai\.\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}\\hat\{A\}\_\{i,t\}^\{\\mathrm\{STEPO\}\}=A\_\{i\}\.\(11\)Summing over the full rollout group gives

∑i=1G∑t=1\|Ti\|A^i,tSTEPO=∑i=1GAi≈0,\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}\\hat\{A\}\_\{i,t\}^\{\\mathrm\{STEPO\}\}=\\sum\_\{i=1\}^\{G\}A\_\{i\}\\approx 0,\(12\)thereby restoring the group\-balance property after trajectory decomposition\.

For a turn\-level output sequenceyi,ty\_\{i,t\}that contains both the reasoning trace and the executable action, we optimize the following clipped policy\-gradient objective:

𝒥STEPO​\(θ\)=𝔼​\[∑i=1G∑t=1\|Ti\|1\|yi,t\|​∑k=1\|yi,t\|min⁡\(ri,t,k​\(θ\)​A^i,t,clip​\(ri,t,k​\(θ\),1−ϵc,1\+ϵc\)​A^i,t\)\],\\begin\{split\}\\mathcal\{J\}\_\{\\mathrm\{STEPO\}\}\(\\theta\)=\\mathbb\{E\}\\Bigg\[\\sum\_\{i=1\}^\{G\}\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}\\frac\{1\}\{\|y\_\{i,t\}\|\}\\sum\_\{k=1\}^\{\|y\_\{i,t\}\|\}\\min\\big\(r\_\{i,t,k\}\(\\theta\)\\hat\{A\}\_\{i,t\},\\mathrm\{clip\}\(r\_\{i,t,k\}\(\\theta\),1\-\\epsilon\_\{c\},1\+\\epsilon\_\{c\}\)\\hat\{A\}\_\{i,t\}\\big\)\\Bigg\],\\end\{split\}\(13\)whereri,t,k​\(θ\)r\_\{i,t,k\}\(\\theta\)is the token\-level probability ratio between the current policy and the rollout policy, andϵc\\epsilon\_\{c\}is the clipping coefficient\. As in PPO and GRPO, a KL penalty to a reference policy can be added to further regularize the update\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.09773#bib.bib20); Shaoet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib2)\)\.

### 3\.3Length Preference and Practical Implications

STEPO also introduces a mild length\-dependent inductive bias\. For successful trajectories, shorter solutions receive larger per\-step positive advantages, which encourages concise execution\. For failed trajectories, longer attempts receive smaller\-magnitude per\-step negative advantages, allowing the policy to continue exploration before terminating or giving up\. This bias is desirable in our setting because computer\-use tasks often benefit from concise successful execution while still requiring tolerance for exploratory recovery under failure\. Nevertheless, it should be monitored in environments where repeated actions can manipulate the reward signal or exploit the validator\.

## 4Policy\-Aware Data Filtering and Curriculum Learning

A second challenge is deciding which interaction data should be used for online RL\. For candidate generation, EvoCUA\-1\.5 follows the verifiable task synthesis pipeline introduced in EvoCUA\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\): tasks are generated from environment priors and atomic computer\-use abilities, and are paired with sandbox configurations and executable validators\. Starting from this synthesized task pool, our focus is the downstream data\-selection process: filtering unreliable or uninformative tasks, calibrating task difficulty against the current policy, and scheduling tasks into a policy\-aware curriculum for online RL\.

### 4\.1From Synthesized Tasks to RL\-Ready Data

A synthesized task is not necessarily suitable for online RL\. Even when both the instruction and validator are executable, the task may be too easy for the current model, too difficult to yield informative positive trajectories, or too noisy due to brittle environment states\. We therefore convert synthesized tasks into RL\-ready data through three filtering stages\.

##### Verifiability and feasibility check\.

Each task is executed in the sandbox environment using model rollouts\. The executable validator provides terminal success labels, while trajectory analysis helps identify task\-definition errors\. We distinguish genuine policy failures from ambiguous instructions, invalid initial states, overly strict validators, missing software capabilities, and inherently infeasible goals\. Tasks with unreliable configurations or validators are revised or removed so that noisy task definitions are not mistaken for ordinary policy failures\.

##### Atomic\-ability coverage control\.

The synthesis pipeline associates each task with one or more atomic computer\-use abilities, allowing us to track both ability coverage and the amount of data available for each ability\. During filtering, we consider not only task\-level quality and pass rate, but also whether the retained pool covers the target ability set with sufficient examples per ability\. This prevents online RL from over\-representing abilities that are easy to synthesize or validate while under\-training abilities that have fewer candidate tasks but remain important for general computer use\.

##### Policy\-aware pass\-rate calibration\.

For each candidate task, we estimate the pass rate of the current policy using grouped rollouts\. Tasks whose pass rates are close to0or11are less informative for online RL: near\-zero tasks rarely produce useful successful trajectories, whereas near\-one tasks are already largely solved\. We therefore prioritize tasks in an intermediate, learnable range, where successful and failed trajectories appear within the same rollout group\. Because task difficulty is policy\-dependent, this calibration is repeated as training progresses\.

![Refer to caption](https://arxiv.org/html/2607.09773v1/figs/atomic_ability_wordcloud.png)Figure 3:Atomic\-ability coverage in the synthesized task pool\. EvoCUA\-1\.5 builds on EvoCUA’s verifiable task synthesis pipeline and further filters the task pool through sandbox feasibility checks, validator inspection, pass\-rate calibration, and trajectory analysis\.

### 4\.2Dynamic Tri\-Adaptive Curriculum

The filtering stage produces an RL\-ready task pool, but the usefulness of each task changes as the policy improves\. A task that is informative early in training may later become trivial, whereas a difficult task may become valuable once the policy begins to solve it occasionally\. Static sampling can therefore waste computation on tasks that are either already mastered or not yet learnable\. To adapt the data distribution over time, we introduce Dynamic Tri\-Adaptive Curriculum \(DTAC\), which constructs each training batch from three complementary channels:

ℬ\(t\)=ℬVAS\(t\)∪ℬAdaPR\(t\)∪ℬICS\(t\)\.\\mathcal\{B\}^\{\(t\)\}=\\mathcal\{B\}\_\{\\mathrm\{VAS\}\}^\{\(t\)\}\\cup\\mathcal\{B\}\_\{\\mathrm\{AdaPR\}\}^\{\(t\)\}\\cup\\mathcal\{B\}\_\{\\mathrm\{ICS\}\}^\{\(t\)\}\.\(14\)Here,ℬVAS\(t\)\\mathcal\{B\}\_\{\\mathrm\{VAS\}\}^\{\(t\)\},ℬAdaPR\(t\)\\mathcal\{B\}\_\{\\mathrm\{AdaPR\}\}^\{\(t\)\}, andℬICS\(t\)\\mathcal\{B\}\_\{\\mathrm\{ICS\}\}^\{\(t\)\}denote the mini\-batch components produced by Variance\-Adaptive Sampling \(VAS\), Difficulty\-Adaptive Positive Replay \(AdaPR\), and Infeasibility\-Controlled Sampling \(ICS\), respectively\. These channels target complementary regions of the data distribution: currently informative tasks, hard\-but\-solvable tasks with rare positive trajectories, and infeasible tasks that should be controlled separately\.

Figure[4](https://arxiv.org/html/2607.09773#S4.F4)illustrates how DTAC combines these channels during curriculum construction\.

![Refer to caption](https://arxiv.org/html/2607.09773v1/figs/CL-diagram-v1.png)Figure 4:Overview of Dynamic Tri\-Adaptive Curriculum \(DTAC\)\. Each training batch combines tasks selected by Variance\-Adaptive Sampling, hard\-but\-learnable positive replay examples from AdaPR, and a controlled fraction of infeasible tasks from ICS\.#### 4\.2\.1Variance\-Adaptive Sampling

The main sampling channel adapts the task distribution according to the current policy’s difficulty on each task\. In grouped rollouts, tasks with pass rates close to11are already largely solved and provide limited marginal learning signal, whereas tasks with pass rates close to0rarely expose successful trajectories\. Tasks with intermediate pass rates are more informative because the same rollout group contains both successes and failures, yielding clearer relative comparisons for policy optimization\.

This intuition can be formalized through the uncertainty of a binary success label\. If rollout success is modeled as a Bernoulli variable with task pass ratePP, its entropy is

H​\(P\)=−P​log⁡P−\(1−P\)​log⁡\(1−P\),H\(P\)=\-P\\log P\-\(1\-P\)\\log\(1\-P\),\(15\)which is maximized atP=0\.5P=0\.5and decreases asPPapproaches0or11\. Thus, medium\-difficulty tasks have the highest outcome uncertainty and tend to provide the strongest contrastive learning signal within a rollout group\.

Because each training step observes only a limited number of rollouts per task, the raw pass rate can be noisy\. We therefore maintain an exponential moving average \(EMA\) of each task’s pass rate:

PE​M​A\(t\)=β​PE​M​A\(t−1\)\+\(1−β\)​P\(t\),P\_\{EMA\}^\{\(t\)\}=\\beta P\_\{EMA\}^\{\(t\-1\)\}\+\(1\-\\beta\)P^\{\(t\)\},\(16\)whereP\(t\)P^\{\(t\)\}is the observed group pass rate at training stepttandβ\\betais the smoothing coefficient\. The EMA accumulates historical difficulty estimates and prevents the sampling weight from changing abruptly due to accidental successes or failures in a single rollout group\.

For efficient sampling, we use the variance of the Bernoulli reward as a normalized proxy for task informativeness\. With binary terminal rewards,r∼Bern​\(P\)r\\sim\\mathrm\{Bern\}\(P\)and

Var​\(r\)=P​\(1−P\)\.\\mathrm\{Var\}\(r\)=P\(1\-P\)\.\(17\)This variance has the same qualitative shape as the entropy above: it peaks atP=0\.5P=0\.5and vanishes near0or11\. We therefore define the VAS sampling weight as

w\(t\)=4​PE​M​A\(t\)​\(1−PE​M​A\(t\)\),w^\{\(t\)\}=4P\_\{EMA\}^\{\(t\)\}\(1\-P\_\{EMA\}^\{\(t\)\}\),\(18\)where the factor44normalizes the weight to\[0,1\]\[0,1\]with maximum value11atPE​M​A\(t\)=0\.5P\_\{EMA\}^\{\(t\)\}=0\.5\. As training progresses, mastered tasks and persistently unsolved tasks are automatically down\-weighted, while tasks that enter a partially solvable regime receive higher sampling probability\. VAS therefore avoids fixed difficulty labels and continuously focuses online RL on tasks that are currently most learnable for the policy\.

#### 4\.2\.2Difficulty\-Adaptive Positive Replay

VAS assigns low sampling weights to tasks whose pass rates are close to either0or11\. This is desirable for tasks that are already mastered, but a low pass rate does not necessarily imply that a task is uninformative\. For difficult tasks that the policy solves only occasionally, rare successful or partially successful trajectories can provide valuable learning signals\. If such tasks rely only on VAS, they may be sampled too infrequently because their overall pass rates remain low, making it difficult for the model to consolidate progress on high\-difficulty tasks\.

To compensate for this effect, we introduce Difficulty\-Adaptive Positive Replay \(AdaPR\)\. AdaPR focuses on tasks whose smoothed pass rates fall within a hard\-but\-learnable interval:

Pl​o​w≤PE​M​A\(t\)≤Ph​i​g​h\.P\_\{low\}\\leq P\_\{EMA\}^\{\(t\)\}\\leq P\_\{high\}\.\(19\)Tasks in this interval are challenging for the current policy but not completely out of reach: the model has already produced successful trajectories in some rollout groups, indicating that the task contains actionable positive signal\. We add successful or partially successful rollout groups from these tasks to the AdaPR buffer and sample an additional fraction of them in later training steps\.

To prevent replay from over\-amplifying a small number of hard examples, each buffered item is assigned a maximum replay count\. This cap allows the model to revisit rare positive trajectories from difficult tasks while preventing the overall training distribution from being dominated by a few replayed groups\. In combination, VAS improves the efficiency of the global sampling distribution, while AdaPR provides additional consolidation for hard\-but\-solvable tasks that would otherwise be under\-sampled\.

#### 4\.2\.3Infeasibility\-Controlled Sampling

Finally, infeasible tasks require different treatment from merely difficult tasks\. Infeasibility may arise from environment restrictions, invalid goals, missing software capabilities, or intentionally impossible user requests\. Treating these tasks as ordinary failures can distort the curriculum, whereas removing them entirely weakens the model’s ability to recognize when continued exploration is unproductive\. We therefore maintain a separate infeasible\-task set𝒟i​n​f\\mathcal\{D\}\_\{inf\}and sample it through an independent channel:

\|ℬICS\(t\)\|=⌊ρi​n​f​\|ℬ\(t\)\|⌋,\|\\mathcal\{B\}\_\{\\mathrm\{ICS\}\}^\{\(t\)\}\|=\\lfloor\\rho\_\{inf\}\|\\mathcal\{B\}^\{\(t\)\}\|\\rfloor,\(20\)whereρi​n​f\\rho\_\{inf\}is a small fixed ratio\. This controlled channel exposes the model to infeasible cases at a stable rate, teaching it when to report failure or terminate exploration without allowing infeasible tasks to dominate the main optimization signal\.

## 5Asynchronous Online RL Infrastructure

Large\-scale online RL for computer\-use agents is bottlenecked primarily by environment interaction rather than by policy optimization alone\. Unlike text\-only rollouts, a computer\-use trajectory requires repeated visual rendering, action execution, environment stabilization, state transition, and validator checks\. Each step may take several seconds, and trajectory length varies with task difficulty, application latency, and recovery behavior\. A synchronous RL pipeline therefore produces substantial idle time: training GPUs wait for slow environment rollouts, while rollout workers may be blocked by policy updates or unavailable model weights\.

### 5\.1Architecture Overview

To decouple slow environment interaction from GPU\-intensive optimization, EvoCUA\-1\.5 adopts a fully asynchronous rollout–buffer–training architecture, as shown in Figure[5](https://arxiv.org/html/2607.09773#S5.F5)\. The system is organized around three components\.

- •Rollout workers: rollout workers execute the current or recent policy with an inference engine, interact continuously with sandbox environments, and generate multi\-turn computer\-use trajectories\. Each trajectory records observations, reasoning traces, actions, validator results, and policy metadata required for later filtering and optimization\.
- •Data buffer: a central buffer receives trajectories from rollout workers and stores step\-level samples constructed after context management\. It tracks policy version, task identity, rollout\-group membership, trajectory length, rewards, and other statistics used for curriculum sampling and staleness control\.
- •Training workers: training workers consume group\-aligned samples from the buffer, compute the online RL objective, update the policy, and periodically publish refreshed weights for subsequent rollout generation\.

![Refer to caption](https://arxiv.org/html/2607.09773v1/figs/async_rl_infra.png)Figure 5:Asynchronous online RL infrastructure\. Rollout workers continuously generate computer\-use trajectories, a staleness\-aware buffer stores step\-level data, and training workers consume group\-aligned samples for policy optimization\.
### 5\.2Staleness\-Controlled Buffer

Asynchrony improves throughput, but it also introduces policy lag: trajectories in the buffer may have been generated by an older policy than the one currently being optimized\. If this lag becomes too large, the training distribution can drift away from the current policy and weaken the near\-on\-policy assumption of online RL\. To control this effect, each sample is tagged with the policy version used during rollout\. The buffer retains only samples whose policy version lies within a predefined staleness window and discards samples that become too old\. This design preserves most of the throughput benefit of asynchronous rollout while limiting off\-policy drift\.

The buffer also preserves rollout\-group structure\. For each instruction, multiple trajectories are sampled as a group so that relative advantages can be computed over comparable rollouts\. After context management, each multi\-turn trajectory is decomposed into multiple step\-level samples, but these samples must retain their original group membership\. This metadata is required by both STEPO, which redistributes trajectory\-level advantages to step\-level samples, and mini\-group batching, which keeps complete rollout groups intact during optimization\.

### 5\.3Adaptive GPU Reallocation

The balance between rollout generation and policy optimization changes throughout training\. When tasks become harder or environments respond slowly, fresh rollout groups may arrive more slowly than training workers can consume them\. In this regime, keeping GPUs reserved for training can lead to idle optimization workers and an underfilled buffer\. EvoCUA\-1\.5 therefore supports adaptive GPU reallocation: when the buffer lacks enough fresh groups, the system temporarily offloads training weights to CPU memory and reallocates GPU capacity to rollout inference\. Once sufficient fresh data has accumulated, the training worker is restored and policy updates resume\. This mechanism reduces prolonged GPU idle time and helps stabilize the ratio between experience production and optimization\.

### 5\.4Mini\-Group Batching

Single\-turn RL typically determines the number of training samples in a global step from the prompt batch size and the number of rollouts per prompt\. A fixedmini\_batch\_sizeis sufficient as long as it divides the total sample count\. This assumption breaks down for multi\-turn computer\-use agents because each trajectory is decomposed into a variable number of step\-level samples after context management\. For a rollout group of sizeGG, the actual number of step\-level training samples is

Ns​t​e​p=∑i=1G\|Ti\|,N\_\{step\}=\\sum\_\{i=1\}^\{G\}\|T\_\{i\}\|,\(21\)where\|Ti\|\|T\_\{i\}\|is the number of executable turns in trajectoryii\. Since\|Ti\|\|T\_\{i\}\|varies across trajectories, a fixed sample\-count mini\-batch can split a rollout group or mix incomplete groups, causing the normalized advantages within a mini\-batch to lose their zero\-sum structure\.

We therefore replace fixed sample\-count batching withmini\-group batching\. Each mini\-batch contains a fixed number of complete rollout groups, and all step\-level samples derived from those groups are kept together\. This preserves group\-level advantage normalization after trajectory decomposition and avoids a subtle source of bias in multi\-turn RL training\.

## 6Evaluation

We evaluate EvoCUA\-1\.5 from three complementary perspectives: its main benchmark performance, cross\-platform generalization, and the design choices that contribute to effective online RL\. We first report the main result on OSWorld\-Verified and additional results on WindowsAgentArena and MacOSArena, then organize controlled ablations around step\-level and group\-aware optimization, data filtering and curriculum learning, cross\-domain transfer, and reward\-design pitfalls\.

### 6\.1Experimental Setup

##### Models\.

We use Qwen3\-VL\(Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10)\)and EvoCUA\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)variants as base policies, including 8B and 32B models\. Because EvoCUA\-1\.5 is built on an earlier Qwen3\-VL iteration, we focus on controlled comparisons under the same backbone and training budget rather than direct comparisons with the latest foundation models\.

##### Environment and benchmark\.

We evaluate on OSWorld\-style desktop tasks\(Xieet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib5)\), grouped into Office, Daily, and Professional categories\. Each task is executed in a sandbox environment and judged by an executable validator when available\. We report success rate as the primary metric\.

##### Training protocol\.

For online RL, each task is rolled out in groups\. Terminal rewards are binary, and multi\-turn trajectories are decomposed into step\-level samples according to the context\-management policy\. Unless otherwise specified, training uses mini\-group batching, staleness\-controlled asynchronous rollout data, and the same evaluation protocol across ablations\.

### 6\.2Main Result

Table[1](https://arxiv.org/html/2607.09773#S6.T1)reports the main OSWorld\-Verified comparison\. We group models by accessibility and report the maximum interaction budget used for each result, since longer\-horizon computer\-use tasks are sensitive to the number of allowed steps\. EvoCUA\-1\.5\-32B achieves 63\.2% Pass@1 with a 100\-step budget, improving over the previous EvoCUA\-32B result of 56\.7% and establishing a new state of the art among models at the same 32B scale\. It also substantially outperforms earlier open\-weight computer\-use baselines such as OpenCUA\-72B and UI\-TARS\-1\.5\-7B\. Compared with newer generalist open\-weight systems, EvoCUA\-1\.5\-32B remains competitive and even approaches models with significantly larger parameter counts, suggesting that online RL and policy\-aware data selection can narrow the gap to larger frontier computer\-use models\.

Table 1:Main result on the OSWorld\-Verified benchmark\. Models are categorized by accessibility, and Max Steps denotes the interaction budget per task\. We retain the previous EvoCUA results for reference; EvoCUA\-1\.5\-32B improves over EvoCUA\-32B, sets a new state of the art at the 32B scale, and achieves 63\.2% Pass@1 with a 100\-step budget\.ModelTypeMax StepsSuccess Rate \(Pass@1\)Closed\-Weight ModelsOpenAI CUA\(OpenAI,[2025](https://arxiv.org/html/2607.09773#bib.bib13)\)Specialized5031\.3%Step\-GUI\-8B\(Yanet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib11)\)Specialized10040\.2%Qwen3\-VL\-Flash\(Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10)\)General10041\.6%UI\-TARS\-2\-2509\(Wanget al\.,[2025a](https://arxiv.org/html/2607.09773#bib.bib4)\)General10053\.1%Claude\-4\.5\-Sonnet\(Anthropic,[2025](https://arxiv.org/html/2607.09773#bib.bib18)\)General5058\.1%Seed\-1\.8\(ByteDance Seed Team,[2025](https://arxiv.org/html/2607.09773#bib.bib17)\)General10061\.9%Claude\-4\.5\-Sonnet\(Anthropic,[2025](https://arxiv.org/html/2607.09773#bib.bib18)\)General10062\.9%Qwen3\.7\-Plus\(Qwen Team,[2026](https://arxiv.org/html/2607.09773#bib.bib43)\)General10073\.3%Open\-Weight ModelsScaleCUA\-32B\(Liuet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib38)\)Specialized5017\.7%UI\-TARS\-72B\-DPO\(Qinet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib22)\)Specialized5024\.6%OpenCUA\-7B\(Wanget al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib3)\)Specialized10026\.6%UI\-TARS\-1\.5\-7B\(Qinet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib22)\)Specialized10027\.5%Qwen3\-VL\-8B\-Thinking\(Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10)\)General10030\.6%OpenCUA\-32B\(Wanget al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib3)\)Specialized10034\.8%GUI\-Owl\-7B\-Desktop\-RL\(Yeet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib37)\)Specialized1534\.9%Qwen3\-VL\-235B\-A22B Thinking\(Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10)\)General10038\.1%Qwen3\-VL\-32B\-Thinking\(Baiet al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib10)\)General10041\.0%OpenCUA\-72B\(Wanget al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib3)\)Specialized10045\.0%EvoCUA\-8B\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)General5046\.1%GUI\-Owl\-1\.5\(Xuet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib63)\)General5055\.4%EvoCUA\-32B\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)General5056\.7%EvoCUA\-32B\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)General10057\.8%CUA\-GYM\-35B\-A3B\(Wanget al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib42)\)General10062\.1%Kimi\-K2\.5\(Kimi Team,[2026a](https://arxiv.org/html/2607.09773#bib.bib44)\)General10063\.3%Kimi\-K2\.6\(Kimi Team,[2026b](https://arxiv.org/html/2607.09773#bib.bib45)\)General10073\.1%MiniMax\-M3\(MiniMax Team,[2026](https://arxiv.org/html/2607.09773#bib.bib46)\)General10075\.2%EvoCUA\-1\.5\-32B \(Ours\)General10063\.2%

To assess whether the gains transfer beyond OSWorld\-Verified, we further evaluate EvoCUA\-1\.5\-32B on two cross\-platform benchmarks: WindowsAgentArena\(Bonattiet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib47)\)and MacOSArena from MMBench\-GUI\(Wanget al\.,[2025c](https://arxiv.org/html/2607.09773#bib.bib48)\)\. As shown in Table[2](https://arxiv.org/html/2607.09773#S6.T2), EvoCUA\-1\.5\-32B improves over Qwen3\-VL\-32B\-Thinking on both Windows and macOS environments, suggesting that the online RL recipe improves general computer\-use capability rather than overfitting to OSWorld\-style tasks\.

Table 2:Cross\-platform generalization on WindowsAgentArena and MacOSArena\. We report success rate \(%\)\. EvoCUA\-1\.5\-32B improves over Qwen3\-VL\-32B\-Thinking on both benchmarks\.ModelWindowsAgentArenaMacOSArenaQwen3\-VL\-32B\-Thinking42\.917\.5EvoCUA\-1\.5\-32B62\.127\.4
### 6\.3Ablation Studies

After establishing the main benchmark result, we ablate the key components of EvoCUA\-1\.5 under controlled training and evaluation settings\. These studies isolate how trajectory\-to\-step optimization, mini\-group batching, data selection, curriculum scheduling, transfer data, and auxiliary rewards affect online RL for computer\-use agents\.

##### Step\-level optimization\.

Figure[6](https://arxiv.org/html/2607.09773#S6.F6)compares naive multi\-turn GRPO with STEPO on OpenCUA\-32B\. Naive GRPO struggles to improve reward after trajectories are decomposed into step\-level samples, whereas STEPO produces a clearer upward trend\. This result indicates that preserving trajectory\-level advantage mass is important for multi\-turn computer\-use optimization\.

![Refer to caption](https://arxiv.org/html/2607.09773v1/figs/online_agentic_rl_step40.png)Figure 6:Training dynamics of naive multi\-turn GRPO and STEPO on OpenCUA\-32B\. STEPO improves reward more efficiently by preserving trajectory\-level advantages after step\-level decomposition\.The improvement is not attributable to changes in the environment or reward function\. Both methods use the same rollouts and validators; the difference lies in how trajectory\-level rewards are assigned to turn\-level training examples\. This supports our analysis that context management changes the effective RL objective\.

We further ablate the batching granularity used during online RL\. As shown in Table[3](https://arxiv.org/html/2607.09773#S6.T3), fixed sample\-count mini\-batching improves over the base Qwen3\-VL policy, while mini\-group batching provides additional gains across Calc, Impress, Writer, and the aggregate score\. This confirms that preserving complete rollout groups during optimization is important for stable multi\-turn computer\-use RL\.

Table 3:Ablation of mini\-group batching during online RL\. Fixed sample\-count mini\-batching improves over the base Qwen3\-VL\-8B policy, while mini\-group batching further improves Calc, Impress, Writer, and the aggregate score\.SettingCalcImpressWriterAvg\.Qwen3\-VL\-8B21\.3834\.9160\.6034\.28\+ fixed mini\-batching21\.7441\.0465\.9037\.93\+ mini\-group batching24\.6447\.3168\.1742\.38
##### Data filtering\.

We compare two data choices under the same training budget: continuing online RL with the full task set and continuing with a filtered high\-SNR subset\. As shown in Table[4](https://arxiv.org/html/2607.09773#S6.T4), the filtered subset achieves better downstream performance across all reported categories\. This result suggests that, for online RL, the marginal value of a task depends more on current learnability and reward reliability than on dataset size alone\.

Table 4:Full data vs\. filtered high\-SNR data for EvoCUA\-32B\. The filtered subset improves downstream evaluation despite using fewer tasks\.SettingOfficeDailyProfessionalAllEvoCUA\-32B \+ full data56\.9061\.1275\.5155\.51EvoCUA\-32B \+ filtered data58\.5762\.3677\.5558\.00We further test whether a data subset selected for one model can be directly reused for another model\. Table[5](https://arxiv.org/html/2607.09773#S6.T5)shows a negative result: a partial office subset that substantially improves EvoCUA\-8B does not transfer cleanly to EvoCUA\-32B and slightly reduces its overall score\. This indicates that data quality is not an intrinsic property of a task alone; it is defined jointly by the task, the validator, and the current policy capability\.

Table 5:Policy\-dependent data filtering\. A subset effective for EvoCUA\-8B does not necessarily benefit EvoCUA\-32B, showing that online RL data should be recalibrated for each policy\.SettingCalcImpressWriterOverallEvoCUA\-8B29\.7938\.2152\.1736\.78EvoCUA\-8B \+ partial office data42\.5348\.8062\.3157\.26EvoCUA\-32B55\.3259\.4969\.5759\.79EvoCUA\-32B \+ partial office data51\.0663\.6265\.2158\.89
##### Curriculum learning\.

We evaluate DTAC by comparing Qwen3\-VL\-32B trained with and without curriculum learning\. As shown in Table[6](https://arxiv.org/html/2607.09773#S6.T6), DTAC improves the overall score from 53\.42 to 55\.45, with particularly large gains on Daily and Professional tasks\. These categories contain more diverse interaction patterns, suggesting that adaptive sampling helps the model focus on tasks with useful reward contrast\.

Table 6:Effect of Dynamic Tri\-Adaptive Curriculum \(DTAC\)\. Curriculum learning improves overall performance and stabilizes training on diverse categories\.SettingOfficeDailyProfessionalAllQwen3\-VL\-32B w/o CL56\.0858\.0471\.4353\.42Qwen3\-VL\-32B w/ CL56\.9363\.3576\.8755\.45
##### Cross\-domain generalization\.

We examine whether online RL on one domain only improves that domain, or whether it can strengthen reusable atomic abilities\. We train Qwen3\-VL\-8B using office\-domain data and evaluate it on Office, Daily, and Professional categories\. Table[7](https://arxiv.org/html/2607.09773#S6.T7)shows improvements not only on Office but also on out\-of\-domain categories\. This suggests that online RL can reinforce transferable computer\-use abilities, such as menu navigation, structured editing, file operations, and visual verification, that are shared across domains\.

Table 7:Cross\-domain transfer from office\-only online RL data\. Improvements on Daily and Professional categories suggest compositional generalization across atomic computer\-use abilities\.SettingOfficeDailyProfessionalAllQwen3\-VL\-8B36\.0947\.1468\.7536\.86Qwen3\-VL\-8B \+ office data41\.7451\.7977\.2142\.39
##### Process reward models can be misleading\.

Sparse terminal rewards make credit assignment difficult, so process reward models \(PRMs\) are a natural candidate for providing denser feedback\. However, we find that PRMs can become a source of reward hacking when their preferences are not well aligned with final task success\. On easier tasks, PRM feedback often correlates with useful intermediate behavior and can accelerate learning\. On harder tasks, where terminal success is rare, the policy may learn to optimize PRM\-favored patterns without improving end\-to\-end completion\. In such cases, PRM reward increases while final success remains flat or decreases, as illustrated in Figure[7](https://arxiv.org/html/2607.09773#S6.F7)\.

![Refer to caption](https://arxiv.org/html/2607.09773v1/figs/prm.png)Figure 7:Process reward models can provide misleading optimization signals when their preferences are not aligned with final executable success\. In this case, the PRM score increases while end\-to\-end task success does not improve accordingly, indicating potential reward hacking\.This observation does not imply that process supervision is useless\. Rather, it suggests that PRM signals should be used with alignment checks against executable final rewards\. For computer\-use agents, robust credit assignment should be grounded in environment state changes, task milestones, validator progress, or counterfactual local replay, rather than relying only on model\-judged reasoning quality\.

## 7Discussion and Future Directions

EvoCUA\-1\.5 shows that online RL for computer\-use agents is a system–algorithm co\-design problem\. The policy objective, task generator, context\-management strategy, rollout infrastructure, and evaluation benchmark all influence whether interaction becomes a useful learning signal\. We outline several research directions highlighted by this framework\.

##### Scaling environments and tasks\.

Future online RL for computer\-use agents requires broader and more realistic environment coverage\. Current training remains concentrated in a limited set of desktop applications and task families\. Real users operate across websites, local applications, cloud services, file systems, and mixed GUI/CLI workflows\. Expanding environment diversity is necessary, but it must be paired with reliable validators and task calibration; otherwise, additional scale may primarily introduce reward noise\.

##### Combining GUI and CLI interaction\.

GUI and CLI are complementary rather than competing interfaces\. Many realistic tasks benefit from switching between visual interaction and scriptable tool execution\. A future computer\-use agent should learn when to click, when to type, when to call a command, and when to query structured external information\. This raises new RL questions: how should the agent maintain a consistent task state across interfaces, how should rewards attribute success to GUI versus CLI actions, and how should context management combine screenshots, terminal outputs, files, and tool traces?

##### Adaptive context management\.

Sliding\-window context is simple and effective, but it is coarse\. A stronger computer\-use agent should decide which historical observations are task\-critical, which actions can be summarized, and which intermediate thoughts should be preserved\. Importantly, the training\-time context policy must match inference\-time context management; otherwise, the model may learn from contexts it will never observe during deployment\.

##### Higher\-throughput infrastructure\.

Environment interaction remains a major bottleneck\. Future systems should improve sandbox scheduling, state reset, failure recovery, validator execution, and observability\. As model and task scales grow, online RL throughput will depend as much on environment engineering as on GPU count\.

##### Credit assignment beyond PRMs\.

Sparse terminal rewards make it difficult to identify which action caused success or failure\. PRMs provide one possible dense signal, but our observations suggest that they can be exploited when terminal rewards are rare\. More robust credit assignment may come from executable milestones, rubric\-based validator progress, state\-difference analysis, local trajectory replay, or counterfactual action replacement\. For computer\-use agents, credit assignment is not only a reward\-shaping problem, but also a problem of modeling the causal structure of multi\-turn interaction\.

##### Better benchmarks\.

Success rate is important but incomplete\. Two agents with similar final success can differ substantially in step efficiency, error recovery, robustness to dynamic states, context usage, infeasible\-task handling, and external\-tool selection\. Future benchmarks should evaluate these dimensions explicitly and include more open\-ended, multi\-stage, multi\-tool tasks\. Better benchmarks will in turn guide training toward robust and generalizable behavior rather than narrow score optimization\.

## 8Related Work

##### Static\-data and offline training for computer\-use agents\.

Modern computer\-use agents are often initialized from large\-scale demonstrations, synthesized trajectories, or offline interaction logs collected from mobile, web, and desktop environments, such as Android\-in\-the\-Wild, Mind2Web, WebChain, OSWorld, OS\-Atlas, and OpenCUA\([Rawleset al\.,](https://arxiv.org/html/2607.09773#bib.bib49);[Denget al\.,](https://arxiv.org/html/2607.09773#bib.bib50); Fanet al\.,[2026b](https://arxiv.org/html/2607.09773#bib.bib61); Xieet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib5);[Wuet al\.,](https://arxiv.org/html/2607.09773#bib.bib51); Wanget al\.,[2025b](https://arxiv.org/html/2607.09773#bib.bib3)\)\. These resources support supervised fine\-tuning, behavioral cloning, offline preference optimization, and offline reinforcement fine\-tuning, providing priors for visual grounding, action formatting, instruction following, and interface\-specific reasoning\(Qinet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib22); Luoet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib52); Luet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib53); Baiet al\.,[2025a](https://arxiv.org/html/2607.09773#bib.bib54)\)\. Other methods improve static\-data learning through trajectory preference construction, candidate\-action re\-ranking, or search\-based trajectory synthesis\(Baiet al\.,[2025a](https://arxiv.org/html/2607.09773#bib.bib54); Puttaet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib55)\)\. EvoCUA further shows that verifiable task synthesis can provide scalable offline supervision for computer\-use agents\(Xueet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib41)\)\. However, static\-data training is bounded by trace coverage and provides little recovery feedback once a policy deviates from the offline distribution, making compounding errors severe in long\-horizon, non\-stationary computer\-use tasks\.

##### From offline supervision to online learning for computer\-use agents\.

These limitations motivate a shift from static trajectory scaling to online experience scaling\. Online RL closes the interaction loop by letting agents act in executable desktop environments, observe visual state changes, and improve from task\-level feedback\. This setting is attractive because many computer\-use tasks have verifiable outcomes, such as URL transitions, DOM changes, file\-system states, database updates, or application\-side effects\(Zhouet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib56);[Kohet al\.,](https://arxiv.org/html/2607.09773#bib.bib57); Xieet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib5);[Rawleset al\.,](https://arxiv.org/html/2607.09773#bib.bib58)\)\. Yet online RL for computer\-use agents introduces challenges absent from single\-turn language RL: rollouts are multi\-turn and partially observable, actions mix discrete and continuous components, rewards are sparse, and training contexts may differ from raw trajectories due to sliding windows, history folding, or summarization\. Moreover, naive PPO/GRPO\-style objectives can overweight long trajectories after trajectory\-to\-step decomposition, while synchronous rollout generation is inefficient because each interaction step requires rendering, action execution, environment transition, and reward verification\(Schulmanet al\.,[2017](https://arxiv.org/html/2607.09773#bib.bib20); Shaoet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib2)\)\.

##### Methods for online RL of computer\-use agents\.

Recent work addresses these challenges from several directions\. Curriculum\-based methods such as WebRL and WebFactory construct or distill task distributions to keep training near the agent’s capability frontier\([Qiet al\.,](https://arxiv.org/html/2607.09773#bib.bib59); Fanet al\.,[2026a](https://arxiv.org/html/2607.09773#bib.bib60)\)\. MobileRL uses difficulty\-adaptive optimization, shortest\-path reward adjustment, and failure curriculum filtering to reduce easy\-task dominance and redundant\-action reward hacking\(Xuet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib62)\)\. WebAgent\-R1 optimizes complete trajectories to better capture delayed rewards and long\-horizon dependencies\([Weiet al\.,](https://arxiv.org/html/2607.09773#bib.bib64)\)\. Reward\- and exploration\-oriented methods such as GUI\-R1, UI\-R1, and InfiGUI\-G1 design verifiable or adaptive rewards for efficient reinforcement fine\-tuning and precise GUI grounding\(Luoet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib52); Luet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib53); Liuet al\.,[2026](https://arxiv.org/html/2607.09773#bib.bib65)\)\. Hybrid pipelines, including DigiRL, UI\-S1, and DART\-GUI, further improve sample efficiency through offline initialization, trajectory patching, adaptive data curation, replay, or staged optimization\(Baiet al\.,[2024](https://arxiv.org/html/2607.09773#bib.bib66); Luet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib67); Liet al\.,[2025](https://arxiv.org/html/2607.09773#bib.bib68)\)\. In contrast, EvoCUA\-1\.5 treats online RL for multi\-turn computer\-use agents as a coupled algorithm–data–system problem, jointly addressing step\-level credit assignment, policy\-aware high\-SNR task filtering, adaptive curriculum construction, and staleness\-controlled asynchronous training\.

## 9Conclusion

We presented EvoCUA\-1\.5, an online reinforcement learning framework for multi\-turn computer\-use agents\. Starting from the observation that computer\-use interaction is governed by context management, sparse verifiable rewards, slow environment feedback, and highly variable task difficulty, we argued that standard single\-turn RL recipes cannot be directly applied to this setting\. EvoCUA\-1\.5 addresses these challenges through STEPO, high\-SNR data filtering and calibration on top of verifiable synthesized tasks, an asynchronous rollout–buffer–training infrastructure, and Dynamic Tri\-Adaptive Curriculum\.

Our experiments show that these components improve training efficiency and downstream performance under controlled OSWorld\-style evaluations\. More importantly, the study highlights several practical lessons: online RL for computer\-use agents benefits more from learnable high\-SNR tasks than from raw data scale; data selection must be recalibrated for each policy; group structure should be preserved after trajectory decomposition; and dense auxiliary rewards such as PRMs can be exploited when they are not aligned with final executable success\.

EvoCUA\-1\.5 advances the training methodology for computer\-use agents by demonstrating how trajectory\-level objectives, policy\-aware data selection, adaptive curricula, and asynchronous interaction infrastructure can be combined to support online RL for computer\-use agents\. Future progress will require richer environments, stronger GUI–CLI integration, adaptive context management, higher\-throughput infrastructure, and better credit\-assignment methods grounded in environment state changes\. The framework and empirical findings presented in this report provide a foundation for scaling online RL in realistic multi\-turn computer\-use interaction\.

## References

- Anthropic \(2025\)Introducing claude sonnet 4\.5\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-5](https://www.anthropic.com/news/claude-sonnet-4-5)Accessed: 2025\-10\-31Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.7.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.9.1)\.
- H\. Bai, Y\. Zhou, M\. Cemri, J\. Pan, A\. Suhr, S\. Levine, and A\. Kumar \(2024\)DigiRL: training in\-the\-wild device\-control agents with autonomous reinforcement learning\.InProceedings of the 38th International Conference on Neural Information Processing Systems,pp\. 12461–12495\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- H\. Bai, Y\. Zhou, L\. E\. Li, S\. Levine, and A\. Kumar \(2025a\)Digi\-q: learning q\-value functions for training device\-control agents\.arXiv preprint arXiv:2502\.15760\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu \(2025b\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2607.09773#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.09773#S2.SS1.p1.3),[§6\.1](https://arxiv.org/html/2607.09773#S6.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.16.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.19.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.20.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.5.1)\.
- R\. Bonatti, D\. Zhao, F\. Bonacci, D\. Dupont, S\. Abdali, Y\. Li, Y\. Lu, J\. Wagle, K\. Koishida, A\. Bucker, L\. Jang, and Z\. Hui \(2024\)Windows agent arena: evaluating multi\-modal os agents at scale\.External Links:2409\.08264,[Link](https://arxiv.org/abs/2409.08264)Cited by:[§6\.2](https://arxiv.org/html/2607.09773#S6.SS2.p2.1)\.
- ByteDance Seed Team \(2025\)Seed 1\.8\.Note:[https://github\.com/ByteDance\-Seed/Seed\-1\.8/](https://github.com/ByteDance-Seed/Seed-1.8/)GitHub repositoryCited by:[§1](https://arxiv.org/html/2607.09773#S1.p1.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.8.1)\.
- \[7\]X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. SuMind2Web: towards a generalist agent for the web\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- S\. Fan, Q\. Shi, S\. Xu, S\. Cai, T\. Zeng, L\. Ling, Y\. Shang, and D\. Kong \(2026a\)WebFactory: automated compression of foundational language intelligence into grounded web agents\.arXiv preprint arXiv:2603\.05044\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- S\. Fan, R\. Wan, Y\. Leng, G\. Liang, L\. Ling, Y\. Shang, and D\. Kong \(2026b\)WebChain: a large\-scale human\-annotated dataset of real\-world web interaction traces\.arXiv preprint arXiv:2603\.05295\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- D\. Guo, D\. Yang, H\. Zhang, J\. Song, R\. Zhang, R\. Xu, Q\. Zhu, S\. Ma, P\. Wang, X\. Bi,et al\.\(2025\)Deepseek\-r1: incentivizing reasoning capability in llms via reinforcement learning\.arXiv preprint arXiv:2501\.12948\.Cited by:[§3\.1](https://arxiv.org/html/2607.09773#S3.SS1.p1.2)\.
- Kimi Team \(2026a\)Kimi k2\.5: visual agentic intelligence\.External Links:2602\.02276,[Link](https://arxiv.org/abs/2602.02276)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.27.1)\.
- Kimi Team \(2026b\)Kimi k2\.6: advancing open\-source coding\.Note:[https://www\.kimi\.com/blog/kimi\-k2\-6](https://www.kimi.com/blog/kimi-k2-6)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.28.1)\.
- \[13\]J\. Y\. Koh, R\. Lo, L\. Jang, V\. Duvvur, M\. C\. Lim, P\. Huang, G\. Neubig, S\. Zhou, R\. Salakhutdinov, and D\. FriedVisualWebArena: evaluating multimodal agents on realistic visual web tasks\.InICLR 2024 Workshop on Large Language Model \(LLM\) Agents,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px2.p1.1)\.
- P\. Li, Z\. Hu, Z\. Shang, J\. Wu, Y\. Liu, H\. Liu, Z\. Gao, C\. Shi, B\. Zhang, Z\. Zhang,et al\.\(2025\)Efficient multi\-turn rl for gui agents via decoupled training and adaptive data curation\.arXiv preprint arXiv:2509\.23866\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- Y\. Liu, Z\. Liu, S\. Zhu, P\. Li, C\. Xie, J\. Wang, X\. Hu, X\. Han, J\. Yuan, X\. Wang,et al\.\(2026\)Infigui\-g1: advancing gui grounding with adaptive exploration policy optimization\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 32267–32275\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- Z\. Liu, J\. Xie, Z\. Ding, Z\. Li, B\. Yang, Z\. Wu, X\. Wang, Q\. Sun, S\. Liu, W\. Wang, S\. Ye, Q\. Li, X\. Dong, Y\. Yu, C\. Lu, Y\. Mo, Y\. Yan, Z\. Tian, X\. Zhang, Y\. Huang, Y\. Liu, W\. Su, G\. Luo, X\. Yue, B\. Qi, K\. Chen, B\. Zhou, Y\. Qiao, Q\. Chen, and W\. Wang \(2025\)ScaleCUA: scaling open\-source computer use agents with cross\-platform data\.External Links:[Link](https://github.com/OpenGVLab/ScaleCUA)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.12.1)\.
- Z\. Lu, Y\. Chai, Y\. Guo, X\. Yin, L\. Liu, H\. Wang, H\. Xiao, S\. Ren, P\. Zhao, G\. Liu,et al\.\(2026\)Ui\-r1: enhancing efficient action prediction of gui agents by reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 17608–17616\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- Z\. Lu, J\. Ye, F\. Tang, Y\. Shen, H\. Xu, Z\. Zheng, W\. Lu, M\. Yan, F\. Huang, J\. Xiao,et al\.\(2025\)Ui\-s1: advancing gui automation via semi\-online reinforcement learning\.arXiv preprint arXiv:2509\.11543\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- R\. Luo, L\. Wang, W\. He, L\. Chen, J\. Li, and X\. Xia \(2025\)Gui\-r1: a generalist r1\-style vision\-language action model for gui agents\.arXiv preprint arXiv:2504\.10458\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- MiniMax Team \(2026\)MiniMax m3: frontier coding, 1m context, and native multimodality\.Note:[https://www\.minimaxi\.com/blog/minimax\-m3](https://www.minimaxi.com/blog/minimax-m3)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.29.1)\.
- OpenAI \(2025\)Computer\-using agent \(cua\)\.Note:[https://openai\.com/index/computer\-using\-agent/](https://openai.com/index/computer-using-agent/)Accessed: 2025\-10\-01Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.3.1)\.
- P\. Putta, E\. Mills, N\. Garg, S\. Motwani, C\. Finn, D\. Garg, and R\. Rafailov \(2024\)Agent q: advanced reasoning and learning for autonomous ai agents\.arXiv preprint arXiv:2408\.07199\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- \[23\]Z\. Qi, X\. Liu, I\. L\. Iong, H\. Lai, X\. Sun, J\. Sun, X\. Yang, Y\. Yang, S\. Yao, W\. Xu,et al\.WebRL: training llm web agents via self\-evolving online curriculum reinforcement learning\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- Y\. Qin, Y\. Ye, J\. Fang, H\. Wang, S\. Liang, S\. Tian, J\. Zhang, J\. Li, Y\. Li, S\. Huang,et al\.\(2025\)Ui\-tars: pioneering automated gui interaction with native agents\.arXiv preprint arXiv:2501\.12326\.Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.13.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.15.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- Qwen Team \(2026\)Qwen3\.7\-plus: multimodal agent intelligence\.Note:[https://qwen\.ai/blog?id=qwen3\.7\-plus](https://qwen.ai/blog?id=qwen3.7-plus)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.10.1)\.
- \[26\]C\. Rawles, S\. Clinckemaillie, Y\. Chang, J\. Waltz, G\. Lau, M\. Fair, A\. Li, W\. E\. Bishop, W\. Li, F\. Campbell\-Ajala,et al\.AndroidWorld: a dynamic benchmarking environment for autonomous agents\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px2.p1.1)\.
- \[27\]C\. Rawles, A\. Li, D\. Rodriguez, O\. Riva, and T\. P\. LillicrapAndroidInTheWild: a large\-scale dataset for android device control\.InThirty\-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§3\.2](https://arxiv.org/html/2607.09773#S3.SS2.p2.3),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px2.p1.1)\.
- Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. Li, Y\. Wu,et al\.\(2024\)Deepseekmath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3\.1](https://arxiv.org/html/2607.09773#S3.SS1.p1.2),[§3\.2](https://arxiv.org/html/2607.09773#S3.SS2.p2.3),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px2.p1.1)\.
- B\. Wang, D\. Lu, J\. Wang, T\. Bai, S\. Liu, Z\. Zhang, H\. Wang, H\. Hu, T\. Xie, S\. Bai, D\. Liu, Q\. Shen, J\. Lin, and T\. Yu \(2026\)CUA\-gym: scaling verifiable training environments and tasks for computer\-use agents\.External Links:2605\.25624,[Link](https://arxiv.org/abs/2605.25624)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.26.1)\.
- H\. Wang, H\. Zou, H\. Song, J\. Feng, J\. Fang, J\. Lu, L\. Liu, Q\. Luo, S\. Liang, S\. Huang,et al\.\(2025a\)Ui\-tars\-2 technical report: advancing gui agent with multi\-turn reinforcement learning\.arXiv preprint arXiv:2509\.02544\.Cited by:[§1](https://arxiv.org/html/2607.09773#S1.p1.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.6.1)\.
- X\. Wang, B\. Wang, D\. Lu, J\. Yang, T\. Xie, J\. Wang, J\. Deng, X\. Guo, Y\. Xu, C\. H\. Wu,et al\.\(2025b\)Opencua: open foundations for computer\-use agents\.arXiv preprint arXiv:2508\.09123\.Cited by:[§1](https://arxiv.org/html/2607.09773#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.09773#S2.SS1.p1.3),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.14.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.17.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.21.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- X\. Wang, Z\. Wu, J\. Xie, Z\. Ding, B\. Yang, Z\. Li, Z\. Liu, Q\. Li, X\. Dong, Z\. Chen, W\. Wang, X\. Zhao, J\. Chen, H\. Duan, T\. Xie, C\. Yang, S\. Su, Y\. Yu, Y\. Huang, Y\. Liu, X\. Zhang, Y\. Zhang, X\. Yue, W\. Su, X\. Zhu, W\. Shen, J\. Dai, and W\. Wang \(2025c\)MMBench\-gui: hierarchical multi\-platform evaluation framework for gui agents\.External Links:2507\.19478,[Link](https://arxiv.org/abs/2507.19478)Cited by:[§6\.2](https://arxiv.org/html/2607.09773#S6.SS2.p2.1)\.
- \[34\]Z\. Wei, W\. Yao, Y\. Liu, W\. Zhang, Q\. Lu, L\. Qiu, C\. Yu, P\. Xu, C\. Zhang, B\. Yin,et al\.WebAgent\-r1: training web agents via end\-to\-end multi\-turn reinforcement learning\.InICML 2025 Workshop on Computer Use Agents,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- \[35\]Z\. Wu, Z\. Wu, F\. Xu, Y\. Wang, Q\. Sun, C\. Jia, K\. Cheng, Z\. Ding, L\. Chen, P\. P\. Liang,et al\.OS\-atlas: foundation action model for generalist gui agents\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.\(2024\)Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§6\.1](https://arxiv.org/html/2607.09773#S6.SS1.SSS0.Px2.p1.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px2.p1.1)\.
- H\. Xu, X\. Zhang, H\. Liu, J\. Wang, Z\. Zhu, S\. Zhou, X\. Hu, F\. Gao, J\. Cao, Z\. Wang, Z\. Chen, J\. Liao, Q\. Zheng, J\. Zeng, Z\. Xu, S\. Bai, J\. Lin, J\. Zhou, and M\. Yan \(2026\)Mobile\-agent\-v3\.5: multi\-platform fundamental gui agents\.External Links:2602\.16855,[Link](https://arxiv.org/abs/2602.16855)Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.23.1)\.
- Y\. Xu, X\. Liu, X\. Liu, J\. Fu, H\. Zhang, B\. Jing, S\. Zhang, Y\. Wang, W\. Zhao, and Y\. Dong \(2025\)Mobilerl: online agentic reinforcement learning for mobile gui agents\.arXiv preprint arXiv:2509\.18119\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px3.p1.1)\.
- T\. Xue, C\. Peng, M\. Huang, L\. Guo, T\. Han, H\. Wang, J\. Wang, X\. Zhang, X\. Yang, D\. Zhao, J\. Ding, X\. Ma, Y\. Xie, P\. Pei, X\. Cai, and X\. Qiu \(2026\)EvoCUA: evolving computer use agents via learning from scalable synthetic experience\.External Links:2601\.15876,[Link](https://arxiv.org/abs/2601.15876)Cited by:[§1](https://arxiv.org/html/2607.09773#S1.p3.1),[§2\.1](https://arxiv.org/html/2607.09773#S2.SS1.p1.3),[§2](https://arxiv.org/html/2607.09773#S2.p1.7),[§4](https://arxiv.org/html/2607.09773#S4.p1.1),[§6\.1](https://arxiv.org/html/2607.09773#S6.SS1.SSS0.Px1.p1.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.22.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.24.1),[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.25.1),[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px1.p1.1)\.
- H\. Yan, J\. Wang, X\. Huang, Y\. Shen, Z\. Meng, Z\. Fan, K\. Tan, J\. Gao, L\. Shi, M\. Yang,et al\.\(2025\)Step\-gui technical report\.arXiv preprint arXiv:2512\.15431\.Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.4.1)\.
- J\. Ye, X\. Zhang, H\. Xu, H\. Liu, J\. Wang, Z\. Zhu, Z\. Zheng, F\. Gao, J\. Cao, Z\. Lu,et al\.\(2025\)Mobile\-agent\-v3: foundamental agents for gui automation\.arXiv preprint arXiv:2508\.15144\.Cited by:[Table 1](https://arxiv.org/html/2607.09773#S6.T1.1.1.18.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2024\)Webarena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 15585–15606\.Cited by:[§8](https://arxiv.org/html/2607.09773#S8.SS0.SSS0.Px2.p1.1)\.

## Appendix AAdditional Details on Unified Actions

EvoCUA\-1\.5 uses a unified action interface that abstracts platform\-specific event APIs into a compact set of executable primitives\. The action set contains mouse operations, keyboard operations, and control operations\. Mouse operations include click, double click, drag, and scroll with normalized screen coordinates\. Keyboard operations include text input, key press, hotkey, and deletion\. Control operations include finish, wait, and explicit infeasibility reporting\. This unified representation allows the same policy format to be used across different sandbox applications while keeping validators independent of model\-specific decoding details\.

## Appendix BDerivation of STEPO Advantage Conservation

LetAiA\_\{i\}denote the group\-normalized trajectory\-level advantage for trajectoryii\. After context management, trajectoryiiproduces\|Ti\|\|T\_\{i\}\|step\-level training samples\. Naively assigningAiA\_\{i\}to every step gives total trajectory weight\|Ti\|​Ai\|T\_\{i\}\|A\_\{i\}, which over\-weights long trajectories\. STEPO instead assignsAi/\|Ti\|A\_\{i\}/\|T\_\{i\}\|to each step\. Therefore,

∑t=1\|Ti\|Ai\|Ti\|=Ai\.\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}\\frac\{A\_\{i\}\}\{\|T\_\{i\}\|\}=A\_\{i\}\.\(22\)Summing over the group gives

∑i∑t=1\|Ti\|Ai\|Ti\|=∑iAi≈0,\\sum\_\{i\}\\sum\_\{t=1\}^\{\|T\_\{i\}\|\}\\frac\{A\_\{i\}\}\{\|T\_\{i\}\|\}=\\sum\_\{i\}A\_\{i\}\\approx 0,\(23\)because group\-normalized advantages have approximately zero mean\. This property is why STEPO preserves the relative optimization structure of GRPO after trajectory\-to\-step decomposition\.

## Appendix CVariance\-Adaptive Sampling Intuition

For a task with binary terminal reward, rollout success can be modeled asr∼Bern​\(P\)r\\sim\\mathrm\{Bern\}\(P\)\. The reward variance is

Var​\(r\)=P​\(1−P\)\.\\mathrm\{Var\}\(r\)=P\(1\-P\)\.\(24\)WhenPPis close to0or11, the group contains little reward contrast and provides weak relative learning signal\. WhenPPis close to0\.50\.5, successful and failed rollouts are both likely, producing informative comparisons\. DTAC therefore uses the normalized weight

w=4​PE​M​A​\(1−PE​M​A\),w=4P\_\{EMA\}\(1\-P\_\{EMA\}\),\(25\)wherePE​M​AP\_\{EMA\}smooths noisy pass\-rate estimates across training steps\.

## Appendix DImplementation Details

Table[8](https://arxiv.org/html/2607.09773#A4.T8)summarizes the fixed curriculum and buffer hyperparameters used in our implementation\.

Table 8:Implementation hyperparameters for curriculum sampling and staleness control\.HyperparameterValueEMA smoothing coefficientβ\\beta0\.7AdaPR lower pass\-rate boundPl​o​wP\_\{low\}0\.125AdaPR upper pass\-rate boundPh​i​g​hP\_\{high\}0\.375ICS sampling ratioρi​n​f\\rho\_\{inf\}0\.025Staleness threshold3Maximum replay count3
## Appendix EScope and Limitations

EvoCUA\-1\.5 focuses on the core algorithmic and systems principles required for online RL in multi\-turn computer\-use environments\. The reported results are based on controlled OSWorld\-style settings and should not be interpreted as a claim of universal robustness across all desktop applications, websites, or mixed GUI/CLI workflows\. Any released model or leaderboard submission should report configuration changes relative to Table[8](https://arxiv.org/html/2607.09773#A4.T8)\.