CANDID:以候选方案驱动干预,实现本地计算机操作智能体的规范执行
摘要
本文提出了 CANDID,这是一种执行控制框架,将并行测试时扩展重新用于稀疏的、候选驱动的干预,以提高本地计算机操作智能体的可靠性。该框架包含循环恢复过滤机制和要求一致判定为 DONE 的门控策略,并在 OSWorld 上进行了评估。
arXiv:2611.24977v1 Announce Type: new
Abstract: We present CANDID (Candidate-Driven Intervention for Disciplined Execution), an execution-control framework designed to enhance the reliability of local computer-use agents (CUAs) in resource-constrained settings. While local CUAs feature robust baseline proficiency in single-step action prediction, they frequently fail at critical junctions due to rigid action loops or premature DONE decisions. Rather than imposing imprecise cognitive loads through textual reflection, CANDID repurposes parallel test-time scaling into a disciplined, execution-level intervention at sparse, failure-prone points. We introduce a loop-recovery mechanism to filter candidates away from repeated behaviors and a conservative, unanimous DONE gating policy that requires absolute consensus to terminate. Experiments on OSWorld demonstrate that CANDID's structural control significantly improves success rates, outperforming feedback-based alternatives with minimal execution overhead.
查看缓存全文
缓存时间: 2026/11/20 09:04
# CANDID: Candidate-Driven Intervention for Disciplined Execution in Local Computer-Use Agents
Source: [https://arxiv.org/html/2611.24977](https://arxiv.org/html/2611.24977)
Jungwook Choi††thanks:Corresponding author\.Affiliation:\{lwghanyang, choij\}@hanyang\.ac\.kr
###### Abstract
We present CANDID \(Candidate\-Driven Intervention for Disciplined Execution\), an execution\-control framework designed to enhance the reliability of local computer\-use agents \(CUAs\) in resource\-constrained settings\. While local CUAs feature robust baseline proficiency in single\-step action prediction, they frequently fail at critical junctions due to rigid action loops or premature DONE decisions\. Rather than imposing imprecise cognitive loads through textual reflection, CANDID repurposes parallel test\-time scaling into a disciplined, execution\-level intervention at sparse, failure\-prone points\. We introduce a loop\-recovery mechanism to filter candidates away from repeated behaviors and a conservative, unanimous DONE gating policy that requires absolute consensus to terminate\. Experiments on OSWorld demonstrate that CANDID’s structural control significantly improves success rates, outperforming feedback\-based alternatives with minimal execution overhead\.
## 1Introduction
Computer\-use agents \(CUAs\) have recently emerged as a promising paradigm for autonomous interaction with graphical user interfaces \(GUIs\), enabling multimodal agents to execute real\-world desktop and web tasks by understanding screenshots and executing GUI actions\. Recent advances in multimodal foundation models[Qin et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib5);[Wang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib4);[Xue et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib6);[Xu et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib20);[Lin et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib21)and agentic frameworks[Agashe et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib7);[Agashe et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib8);[Gonzalez\-Pumariega et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib9);[Yang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib10)have significantly improved long\-horizon GUI interaction capabilities across increasingly realistic environments[Xie et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib11);[Zhou et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib23);[Bonatti et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib24);[Deng et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib22)\. In particular, modern CUA systems leverage hierarchical planning[Agashe et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib7), memory retrieval[Liu et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib12), and test\-time scaling[Yang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib10)strategies to improve execution robustness and task success rates\.
Deploying CUAs under local, resource\-constrained settings is practically important, and small language model \(SLM\)\-based agentic systems have been increasingly proposed toward this goal[Belcak et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib32)\. Yet such deployment remains highly challenging: unlike large cloud\-based systems that rely on expensive multi\-agent orchestration[Song et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib14);[Yang et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib13);[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1)or proprietary reasoning models[Anthropic \(2026a\)](https://arxiv.org/html/2611.24977#bib.bib15);[Singh et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib18), local CUAs operate with limited model sizes and weaker multimodal reasoning\. Under these conditions, simply increasing execution horizons or enlarging multimodal histories often fails to reliably improve performance\. The central obstacle is that local agents fail in two coupled ways: they fall into repetitive action loops, or terminate prematurely before completing the task\. Moreover, naively scaling the step budget or contextual history merely shifts the failure pattern between the two, rather than resolving both\.
Prior methods for recovering failed agent trajectories primarily rely on textual reflection or verifier feedback[Shinn et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib2);[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1), where the agent inspects its own recent trajectory, generates natural\-language feedback, and incorporates it before revising its next action\. However, we argue that such feedback\-based approaches provide limited benefit for lightweight local CUAs, whose reasoning capabilities are often insufficient either to generate meaningful textual feedback about the failure or to convert given feedback into distinct, executable GUI actions\. These limitations prevent agents from escaping repetitive action loops or improving termination decisions under ambiguous completion states, often causing the model to still repeat its failed behavior under a restated rationale\.
To overcome these limitations, we instead build on a key observation about local CUA behavior: although they struggle to assess trajectory progress, they remain capable of generating multiple plausible actions—which we refer to ascandidates—from the same interface state\. This suggests that the bottleneck lies not in the availability of good candidates, but in committing to one when progress or completion is uncertain\. We therefore propose CANDID, a candidate\-based control framework that treats candidate disagreement as an execution\-control signal, rather than asking the model to reflect on its own failure\. CANDID instantiates this principle through two modules: \(1\) a loop\-recovery module that identifies repetitive behavior through rule\-based detection and selects an alternative candidate, and \(2\) a unanimous DONE gating module that accepts termination only when all sampled candidates agree\. CANDID applies this sparse intervention only at these failure\-prone points, concentrating computation where it is most beneficial while keeping overhead low\.
We evaluate CANDID on the OSWorld benchmark[Xie et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib11)using multiple local CUA models, including Qwen3\-VL[Bai et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib19)and OpenCUA[Wang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib4)variants\. Experimental results show that CANDID consistently improves task success rates across all evaluated models, outperforming both feedback\-based interventions and a test\-time scaling baseline\. Further analyses show that its two control modules address complementary failure modes, and that sparse intervention is more effective than applying candidate selection uniformly throughout execution\.
Our contributions are summarized as follows:
- •We identify repetitive action loops and premature false success as dominant coupled failure modes in local CUAs that cannot be reliably resolved through naive execution scaling or textual feedback\.
- •We propose CANDID, a sparse, candidate\-based control framework that leverages multiple candidates generated by the same local agent at failure\-prone steps, throughloop\-recovery candidate filteringandunanimousDONEgating, without modifying the model\.
- •We demonstrate that CANDID consistently improves task success across multiple local CUAs, outperforming feedback\-based and test\-time scaling baselines, and through targeted ablations we show that these gains stem specifically from CANDID’s sparse, candidate\-driven intervention design with minimal runtime overhead\.
## 2Background and Motivation
### 2\.1Computer\-Use Agents
Figure 1:Operational workflow of a computer\-use agent\. The agent takes the task instruction and a multimodal history ofhhpast \[screenshot, response\] pairs as input to generate Python\-executable GUI actions up to a maximum step budget \(ss\)\.Computer\-use agents \(CUAs\) autonomously operate graphical user interfaces \(GUIs\) in real OS environments, observing screenshots and generating executable actions such as clicking, typing, scrolling, or keyboard inputs\. As illustrated in Fig\.[1](https://arxiv.org/html/2611.24977#S2.F1), at each step the agent receives a task instruction, the current screenshot, and a multimodal history of previous screenshot–response pairs, and predicts the next action in a structured format \(e\.g\.,click\(x,y\)ortypewrite\(‘Tesla’\)\)\. The action is executed in the environment, after which the updated screenshot is observed\. This interaction continues until the agent emits a termination signal such asDONEor reaches the step budgetss, while the history lengthhhcontrols the number of past screenshot–response pairs retained as context\.
Figure 2:Failure modes of local CUAs on the OSWorld benchmark\. Stacked bars show task outcomes: success, repetitive action loop, premature false success, step budget exhaustion without loops, and other failures\. \(a\) Varying recent history depth \(h∈\{0,1,4\}h\\in\\\{0,1,4\\\}\) withs=50s=50\. \(b\) Varying maximum step budget \(s∈\{15,50,100\}s\\in\\\{15,50,100\\\}\) withh=4h=4\.
### 2\.2Failure Modes in Local CUAs
We analyze the behavioral limitations of local CUAs by evaluating Qwen3\-VL\-30B\-A3B and OpenCUA\-7B on OSWorld under varying step budgetss∈\{15,50,100\}s\\in\\\{15,50,100\\\}and history depthsh∈\{0,1,4\}h\\in\\\{0,1,4\\\}\. As shown in Fig\.[2](https://arxiv.org/html/2611.24977#S2.F2), repetitive action loops and premature false success dominate across nearly all settings, while non\-loop exhaustion remains limited except at the tightest budget \(s=15s=15\)\. Qwen3\-VL is more prone to repetitive loops, whereas OpenCUA more often terminates prematurely\. Increasing the history depth or step budget reduces some exhaustion cases but shifts failures toward false completion rather than success\. These results indicate that naive execution scaling does not resolve the dominant failure modes—it only trades one form of failure for another\.
### 2\.3Limitations of Textual Feedback
FeedbackMetricRate \(%\)Loop BreakerRepeated exact action61\.5Repeated type/target64\.3Failure after recovery92\.8VerifierRepeatedDONEafter rejection73\.4Next action repeats recent target44\.3Failure after rejection75\.9
Table 1:Behavioral outcomes after feedback interventions on Qwen3\-VL\-30B\-A3B\. Despite receiving Loop Breaker or Verifier feedback, the agent frequently repeats the same behavior or continues to fail, indicating that feedback alone offers limited recovery capability\.Prior autonomous\-agent frameworks often use prompt\-level verification or self\-reflection to mitigate execution failures[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1);[Pan et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib3);[Shinn et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib2)\. These methods work well with proprietary, large\-scale reasoning models, whose stronger language understanding lets verbal feedback translate into corrective actions\. Inspired by VLAA\-GUI[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1), we implement two representative self\-feedback mechanisms: aLoop Breaker, which detects repetitive behavior and appends a textual recovery reflection, and aVerifier, which rejects prematureDONEpredictions and provides corrective feedback\.
However, such textual reflection offers limited recovery capability for smaller, local\-scale CUA models, as verbal critiques often fail to induce a distinct executable action once the agent enters a localized loop or faces goal ambiguity\. As shown in Table[1](https://arxiv.org/html/2611.24977#S2.T1), Loop Breaker feedback frequently leads the agent to repeat the same action or target and ultimately fail; for instance, after repeatedly attempting to enable a toggle switch, the reflection merely restates the same instruction rather than suggesting an alternative action\. Similarly, after a rejectedDONEprediction, the agent regeneratesDONEor repeats a recent target, again resulting in failure\. These results highlight the limited effectiveness of textual feedback for local CUAs\.
Figure 3:A local agent can generate valid alternative candidate actions at failure\-prone states\. In \(a\), the agent repeatedly pressesCtrl\+Vdespite an empty clipboard, while another candidate types the URL directly\. In \(b\), the agent prematurely declares task completion before modifying the document, while another candidate continues execution by clicking the target text for editing\.
### 2\.4Generating Failure\-Escaping Candidates
Figure 4:Overview of CANDID, our candidate\-based execution\-control framework for local computer\-use agents\. CANDID intervenes at two high\-risk control points: \(a\)Loop Recovery:A loop detector monitors the execution history for repetitive local behavior and triggers structured candidate generation and filtering to escape the loop\. \(b\)Unanimous DONE Gating:When the baseline agent proposesDONE, CANDID samples additional action candidates from the same context and executesDONEonly when all valid candidates agree on termination\.Although local CUAs often remain trapped in repetitive loops or terminate prematurely, these failures do not necessarily indicate an inability to generate useful executable actions\. As illustrated in Fig\.[3](https://arxiv.org/html/2611.24977#S2.F3), sampling multiple actions from the same execution state can reveal executable alternatives that escape the current failure pattern: an agent that repeatedly pressesCtrl\+Vdespite an empty clipboard may instead generate an action that types the URL directly, while an agent that prematurely declares completion may generate a continuation action that resumes document editing\. This suggests that the bottleneck lies not in generating executable alternatives, but in committing to a single action despite disagreement among plausible candidates\. This capability is not limited to isolated examples: when the agent exhibits repetitive\-action behavior, all four sampled candidates are distinct at the exact\-action level in 47\.7% of cases, whereas complete candidate collapse occurs in only 8\.4% \(Appendix[B\.1](https://arxiv.org/html/2611.24977#A2.SS1)\)\.
## 3Method
We propose CANDID, a sparse, candidate\-based execution\-control framework for local CUAs that intervenes at failure\-prone steps to address two coupled failure modes: repetitive action loops and prematureDONEdecisions\. Our key intuition is that local CUAs can often generate multiple plausible candidates from the same interface state but fail by committing to a single one under uncertain execution states\. Instead of sampling throughout execution, CANDID generates a small set of candidates only at these high\-risk points, recovering from loops by selecting an alternative candidate and preventing premature termination by requiring unanimous agreement before acceptingDONE\.
### 3\.1Loop\-Recovery Candidate Selection
Fig\.[4](https://arxiv.org/html/2611.24977#S2.F4)\(a\) illustrates CANDID’s loop\-recovery pipeline\. At each step, the agent predicts its next action from the current screenshot and recent history\. When the proposed action is identified as a potential repetitive loop and confirmed as stagnant, CANDID samples alternative candidates, filters out those matching the detected loop pattern, and executes one of the remaining candidates\.
##### Loop detection\.
Judging whether the agent is stuck in a loop is itself unreliable: the local CUA often cannot assess its own progress, while pure rule\-based heuristics risk flagging legitimate repetition as a loop\. We therefore use a two\-stage gate that separates cheap detection from reliable confirmation\. A lightweight rule\-based detector first compares the proposed action against a sliding window of recent actions \(governed by a window size, a minimum repeat count, and a target\-coordinate tolerance\) and flags repetitive patterns—for example, repeated scrolling is flagged asrepeated\_targetless\_action\_type\. Only for flagged cases does the local model then confirm whether the trajectory is genuinely stuck in a loop\. This confines the model’s harder judgment to the rare cases where it is needed, while leaving the initial action untouched otherwise\. Further details and sensitivity analysis are in Appendix[A\.1](https://arxiv.org/html/2611.24977#A1.SS1)and Appendix[B\.2](https://arxiv.org/html/2611.24977#A2.SS2), respectively\.
##### Recovery candidate generation\.
Simply requesting a single new action from the same context after a loop is confirmed often reproduces the same failed action\. Instead, once a loop is confirmed, the local agent samples a small set of executable candidates \(a hyperparameter set to four in our main experiments\) from the current environment state, leveraging its demonstrated ability to produce diverse actions under repeated sampling\. These candidates may include both repeated and alternative actions, and are passed to the filtering stage\.
##### Filtering and selection\.
Before the agent selects among candidates, CANDID first narrows the pool to reduce selection overhead: malformed or non\-executable actions are removed,DONEis disallowed during loop recovery, candidates matching the detected loop pattern are excluded, and duplicates are merged\. If no valid candidate remains, CANDID falls back to the original proposal; if exactly one remains, it is executed directly; otherwise, the agent selects among the filtered candidates based on the task instruction and recent trajectory\.
### 3\.2Unanimous DONE Gating
Local CUAs frequently misjudge task completion, often terminating prematurely \(Section[2\.2](https://arxiv.org/html/2611.24977#S2.SS2)\), motivating a conservative approach to acceptingDONE\. Fig\.[4](https://arxiv.org/html/2611.24977#S2.F4)\(b\) illustrates CANDID’s unanimousDONEgating module, which is triggered only when the agent proposesDONE\. Rather than accepting a single termination prediction, CANDID samples additional candidates from the same context and acceptsDONEonly when all valid candidates agree \(Case 1\); even a single continuation candidate, such asWAITorclick, is treated as evidence that the task may still be unfinished, in which case termination is rejected and allDONEcandidates are removed \(Case 2\)\. This conservative policy uses the agent’s own candidate diversity to challenge unsupportedDONEdecisions without relying on an external verifier\. We further show in Section[4\.3](https://arxiv.org/html/2611.24977#S4.SS3)that this unanimous policy more effectively suppresses premature completions than a majority voting\.
##### Candidate selection\.
OnceDONEcandidates are removed, the remaining continuation candidates are processed for execution\. In Case 2, for example, the sampled set containsDONE,WAIT, andclick; after filtering, onlyWAITandclickremain as continuation candidates, and exact duplicates are merged\. If a single continuation candidate remains, it is executed directly; otherwise, the local agent selects one from the remaining continuation candidates based on the task instruction, current screenshot, and recent trajectory\. The selector is restricted to choosing among the generated candidates and cannot introduce a new action\.
### 3\.3Profiling\-Based Auto\-Configuration
Qwen3\-VL8BQwen3\-VL30B\-A3BOpenCUA\-7BExact\-actionloop rate32\.12%29\.79%7\.09%Unanimity rate88\.2%78\.2%40\.8%Avg\. unique alts\.1\.231\.452\.34Multi\-type events1\.1%4\.3%23\.4%Applied controlLoopLoop \+ DONEDONE
Table 2:Model\-specific diagnostics withk=4k\{=\}4resampled candidates and the resulting control configuration\. Exact\-action loop rate denotes the fraction of steps that repeat the preceding action\. Avg\. unique alts\. denotes the average number of unique non\-DONEalternatives in disagreement cases, while multi\-type events denotes the fraction of disagreements spanning multiple coarse action types\.The two control modules target distinct failure modes and may benefit local CUAs differently\. Rather than tuning configurations post hoc, we estimate each module’s potential effectiveness using rollout diagnostics that are independent of task success, and apply a module only when its target condition is clearly met\. As shown in Table[2](https://arxiv.org/html/2611.24977#S3.T2), loop recovery targets models with high exact\-action loop rates \(Section[2\.2](https://arxiv.org/html/2611.24977#S2.SS2)\): Qwen3\-VL\-8B and Qwen3\-VL\-30B\-A3B show loop rates roughly four times higher than OpenCUA\-7B \(32\.12% and 29\.79% vs\. 7\.09%\) and are assigned loop recovery\. UnanimousDONEgating instead requires diverse, semantically distinct alternatives on disagreement to have any effect; OpenCUA\-7B shows both lowerDONEunanimity \(40\.8% vs\. 78–88%\) and far more multi\-type disagreements \(23\.4% vs\. 1\.1–4\.3%\), and Qwen3\-VL\-30B\-A3B shows a smaller but non\-trivial version of the same pattern, while Qwen3\-VL\-8B shows almost none\. We therefore applyDONEgating to OpenCUA\-7B and Qwen3\-VL\-30B\-A3B, but not to Qwen3\-VL\-8B\. These diagnostics are computed once before the main evaluation and fixed as a single configuration per model\.
## 4Results
ModelMethodTrialsDomain Accuracy \(%\)Overall \(%\)Office\(117\)Daily\(78\)Prof\.\(49\)Workflow\(93\)OS\(24\)Qwen3\-VL\-30B\-A3B\(s=50\)Baseline537\.4348\.1541\.6318\.2536\.6735\.31±\\pm0\.65CANDID539\.4249\.7646\.9422\.5246\.6738\.80±\\pm1\.24\(\+3\.49\)Qwen3\-VL\-30B\-A3B\(s=100\)Baseline537\.5448\.2341\.2318\.7836\.1635\.42±\\pm1\.34CANDID540\.6350\.5948\.5723\.8242\.9339\.68±\\pm0\.77\(\+4\.26\)Qwen3\-VL\-8B\(s=50\)Baseline532\.7443\.3744\.4914\.7744\.5732\.82±\\pm0\.98CANDID532\.5846\.9747\.3517\.3248\.3334\.84±\\pm1\.27\(\+2\.02\)Qwen3\-VL\-8B\(s=100\)Baseline533\.7744\.1745\.7214\.8942\.5033\.41±\\pm1\.29CANDID534\.4548\.3943\.4517\.6350\.0035\.42±\\pm1\.76\(\+2\.01\)OpenCUA\-7B\(s=50\)Baseline523\.9236\.3144\.9011\.3939\.1327\.21±\\pm1\.70CANDID524\.9138\.3962\.0410\.6753\.4431\.05±\\pm0\.97\(\+3\.84\)OpenCUA\-7B\(s=100\)Baseline523\.4233\.4640\.8211\.2542\.2526\.06±\\pm2\.16CANDID525\.3640\.2056\.7310\.9347\.8330\.60±\\pm2\.22\(\+4\.54\)
Table 3:Main evaluation results on the OSWorld benchmark, averaged over multiple trials \(trial counts noted per row\)\. Values in parentheses next to domain names indicate the task count\. Overall accuracy is reported as mean±\\pmstandard deviation; values in gray parentheses represent the absolute improvement over each model’s baseline\.### 4\.1Setup
##### Benchmark and Evaluation Setting\.
We evaluate our framework primarily on OSWorld[Xie et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib11), excluding eight Google Drive\-related tasks and reporting results on 361 tasks across five domains\. We additionally assess cross\-benchmark generalization on WindowsAgentArena[Bonatti et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib24)with 154 tasks, with detailed settings and results provided in Appendix[B\.4](https://arxiv.org/html/2611.24977#A2.SS4)\. All agents use screenshot observations, generate PyAutoGUI\-style actions, and use four recent screenshots as history \(h=4h=4\) for the main experiments\.
##### Models\.
Our evaluation covers three local computer\-use agents: Qwen3\-VL\-30B\-A3B\-Instruct, Qwen3\-VL\-8B\-Instruct, and OpenCUA\-7B\. This model selection encompasses both general\-purpose vision\-language models adapted to desktop control and models specialized for computer\-use behavior, allowing us to test the proposed control mechanisms across different model scales and agent families\.
##### Candidate sampling configuration\.
For both loop recovery and unanimousDONEselection, we fix the number of additional sampled candidates to four\. For Qwen\-based agents, we use a temperature of 0\.7 for default action generation andDONEcandidate sampling, and a higher temperature of 1\.0 for loop\-recovery candidate generation to encourage greater action diversity under repetitive execution patterns\. For OpenCUA\-7B, the default action generation uses greedy decoding \(temperature 0\.0\), while candidate generation uses a temperature of 0\.7 for both loop recovery andDONEselection\.
##### Inference\-Time Control Baselines\.
We compare CANDID against several inference\-time control baselines\. For test\-time scaling, we apply GTA\-1[Yang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib10), using the local CUAs as the planner while retaining the original GTA\-1\-7B grounding model and sampling four plans\. We follow the publicly released GTA\-1 implementation, including its available prompting and inference procedure, without modifying the disclosed components other than replacing the planner with the evaluated local CUA\. We also implement step\-wiseSelf\-Guidance, which generates compact progress feedback after each step and appends it to the next action\-generation prompt\. Finally, we implement VLAA\-GUI\-style[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1)Loop BreakerandVerifiermodules by adapting their core intervention concepts for local CUA evaluation, rather than reproducing the full multi\-agent architecture\. Additional details are provided in Appendix[C](https://arxiv.org/html/2611.24977#A3)\.
### 4\.2Main Result
##### Overall performance\.
Table[3](https://arxiv.org/html/2611.24977#S4.T3)reports the main results on OSWorld\. We evaluate CANDID under different step budgets to examine its effectiveness across varying execution lengths\. Across all evaluated local CUAs, CANDID consistently improves task success over the corresponding baseline, yielding gains of 2\.01–4\.54 percentage points depending on the model and step budget\. The improvements are observed under both shorter \(s=50s=50\) and longer \(s=100s=100\) budgets, demonstrating that CANDID remains effective across different interaction horizons\.
MethodDomain Accuracy \(%\)Overall \(%\)Office\(117\)Daily\(78\)Professional\(49\)Workflow\(93\)OS\(24\)Baseline37\.4348\.1541\.6318\.2536\.6735\.31Test\-Time Scaling MethodGTA\-112\.7438\.4148\.989\.7754\.1725\.19Feedback\-Based MethodsSelf\-Guidance34\.9636\.3644\.9018\.2650\.0033\.30VLAA\-Loop Breaker37\.7546\.6546\.9421\.3642\.1236\.98VLAA\-Verifier36\.3950\.2240\.1414\.7434\.7234\.22VLAA\-Combined34\.9849\.7344\.2220\.3237\.5035\.84CANDID39\.4249\.7646\.9422\.5246\.6738\.80Table 4:Comparison with inference\-time control baselines using the Qwen3\-VL\-30B\-A3B\-Instruct\. We compare CANDID against a GTA\-1[Yang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib10)test\-time scaling method, step\-wise Self\-Guidance, and VLAA\-GUI\-style[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1)feedback methods, including Loop Breaker, Verifier, and their combined variant\. VLAA\-GUI Loop Breaker is averaged over four runs, Verifier and Combined over three runs, and Baseline and CANDID over five runs; GTA\-1 and Self\-Guidance are based on a single run\.
##### Comparison with Inference\-Time Control Methods\.
As shown in Table[4](https://arxiv.org/html/2611.24977#S4.T4), CANDID outperforms the baseline and all inference\-time control methods\. Among the feedback\-based methods,Self\-Guidancedecreases accuracy from 35\.31% to 33\.30%\. The VLAA\-GUI\-styleLoop Breakerprovides a modest improvement to 36\.98%, whereas theVerifierand the combinedLoop Breaker\+Verifiervariant achieve 34\.22% and 35\.84%, respectively\. GTA\-1 decreases overall accuracy to 25\.19%, suggesting limitations in the local CUA’s high\-level planning and structured plan generation\. Overall, these results suggest that verbal feedback or planning\-time sampling alone is insufficient for local CUAs, whereas CANDID improves execution stability by introducing alternative actions selectively at high\-risk states\.
### 4\.3Ablation and Design Analysis
##### Complementary effects of control modules\.
Fig\.[5](https://arxiv.org/html/2611.24977#S4.F5)shows that the two control modules affect distinct failure modes\. Loop\-only substantially reduces repetitive action loops but shifts some failures toward premature false success, while Done\-only suppresses premature false success at the cost of increased execution exhaustion\. Combining both modules mitigates this trade\-off and achieves the highest average score of 38\.8%, demonstrating that loop recovery and conservative termination control are most effective when applied jointly\.
Figure 5:Failure\-mode distribution and accuracy across component configurations on Qwen3\-VL\-30B\-A3B \(s=50s=50\)\. Loop Recovery primarily reduces repetitive loops, Unanimous DONE reduces premature false success, and their combination achieves the highest average score\.BaselineMajorityUnanimousAccuracy \(%\)26\.0627\.2430\.60Hit Max19\.3329\.3364\.67False Success241\.33232\.00171\.67Redirection \(%\)–5\.5349\.93
Table 5:Comparison of majority and unanimousDONEselection on OpenCUA\-7B with a 100\-step budget\.
##### Majority vs\. Unanimous DONE\.
Table[5](https://arxiv.org/html/2611.24977#S4.T5)compares majority voting with the proposed unanimous termination policy\. Majority voting provides only a marginal improvement over the baseline, with a redirection rate of 5\.53%\. In contrast, unanimous selection redirects 49\.93% of proposed terminations and reduces false success from 241\.33 to 171\.67, improving task accuracy from 26\.06% to 30\.60%\. Although this increases max\-step failures, the overall result indicates that majority agreement is too weak to counter the model’s termination bias\.
While unanimous gating more effectively suppresses premature termination, it does not constitute a reliable semantic completion verifier\. In the task\-level proxy analysis, it achieves 56\.7% precision and 26\.3% specificity, compared with 54\.3% and 15\.2% for the VLAA\-GUI\-style Verifier, respectively \(Appendix[B\.3\.2](https://arxiv.org/html/2611.24977#A2.SS3.SSS2)\)\. Thus, CANDID primarily acts as a conservative filter against unsupported termination rather than solving completion verification itself\.
BaselinePer\-Step SamplingCANDIDAccuracy \(%\)35\.3123\.4838\.80Hit Max135\.0239\.0116\.6False Success97\.444\.0105\.2Avg\. Steps25\.6136\.8924\.40
Table 6:Comparison of dense per\-step sampling and sparse intervention with CANDID on Qwen3\-VL\-30B\-A3B with a 50\-step budget\. Hit Max and False Success are non\-mutually\-exclusive trajectory statistics; a task may satisfy both conditions\.
##### Per\-Step vs\. Triggered Sampling\.
Table[6](https://arxiv.org/html/2611.24977#S4.T6)compares CANDID with candidate sampling at every step\. Per\-step sampling lowers false success but substantially increases max\-step failures and reduces accuracy to 23\.48%\. In contrast, CANDID achieves 38\.80% accuracy while maintaining a similar trajectory length to the baseline\. These results show that candidate sampling is effective when applied selectively at failure\-prone decision points rather than throughout execution\.
Figure 6:Effect of candidate count on task accuracy across the two local CUAs\. Results with four candidates are averaged over five runs, while the remaining settings are averaged over three runs\. Shaded regions denote±1\\pm 1standard deviation across repeated runs\.
##### Sensitivity to Candidate Count\.
Fig\.[6](https://arxiv.org/html/2611.24977#S4.F6)examines the effect of the number of sampled candidates\. Increasing the candidate count from one to four consistently improves task accuracy for both models\. Further increasing it to eight provides only marginal gains, indicating diminishing returns beyond four candidates\. We therefore use four candidates in the main experiments to balance performance and cost\.
StatisticLoop RecoveryDONE GatingTotal Steps93719371Triggered2496 \(26\.6%\)429 \(4\.6%\)Intervention Applied1426 \(57\.1%\)196 \(45\.7%\)Original Action Kept1070 \(42\.9%\)233 \(54\.3%\)Selector Invoked1043 \(41\.8%\)96 \(22\.4%\)Selector Bypassed1453 \(58\.2%\)333 \(77\.6%\)
Table 7:Operational statistics of the candidate\-based control framework on Qwen3\-VL\-30B\-A3B\. Percentages for “Triggered” are computed over all execution steps, while the remaining percentages are computed over the triggered steps\.
### 4\.4Efficiency Analysis
##### Operational statistics\.
Table[7](https://arxiv.org/html/2611.24977#S4.T7)shows that CANDID intervenes selectively rather than modifying the entire execution process\. Loop recovery is triggered in 26\.6% of steps, with 57\.1% resulting in trajectory changes, indicating that repetitive behaviors are frequent yet recoverable\.DONEselection is triggered more sparsely, in only 4\.6% of steps, but redirects 45\.7% of those cases to non\-DONEcontinuations\. The selector is skipped in most cases, 58\.2% for loop recovery and 77\.6% forDONEselection, showing that many high\-risk decisions can be resolved through structural filtering alone\.
Figure 7:Efficiency comparison of baseline and CANDID on Qwen3\-VL\-30B\-A3B\. Execution time includes model inference, action execution, and environment initialization, with average steps and token usage\.
##### Runtime and token overhead\.
Fig\.[7](https://arxiv.org/html/2611.24977#S4.F7)compares the computational cost of CANDID and the baseline using Qwen3\-VL\-30B\-A3B\. CANDID increases prompt\-token usage by only 19%, while completion tokens, despite a larger relative increase from candidate generation and selection, remain negligible compared with prompt tokens\. CANDID also shortens the average trajectory from 25\.6 to 24\.4 steps per task, partially offsetting the additional inference cost\. Consequently, total execution time increases by only 10% \(359\.0→\\rightarrow396\.1 s/task\), indicating that CANDID improves task performance with modest additional cost\.
### 4\.5Case Study
Qualitative examples further illustrate how the two controls intervene at distinct failure points\. Loop recovery redirects repeatedEnteractions on a license screen to an executable alternative,typewrite\(’n’\), while the DONE gating prevents premature termination before sending a prepared email by selecting a concrete continuation action\. These cases show that CANDID converts detected risks into constrained executable alternatives\. Additional details are provided in Appendix[D](https://arxiv.org/html/2611.24977#A4)\.
## 5Related Work
##### Models and Frameworks\.
Recent progress in multimodal GUI agents has been driven by realistic computer\-use benchmarks[Xie et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib11);[Deng et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib22);[Rawles et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib25)and CUA\-specialized models that improve GUI grounding, action prediction, and long\-horizon interaction[Wang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib4);[Qin et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib5);[Xue et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib6);[Lin et al\. \(2024\)](https://arxiv.org/html/2611.24977#bib.bib21);[Xu et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib20)\. Agentic frameworks further improve reliability through multi\-agent coordination[Song et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib14);[Yang et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib13), hierarchical planning[Agashe et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib8), and test\-time scaling[Yang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib10);[Gonzalez\-Pumariega et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib9)\. However, these approaches often rely on proprietary models or substantial inference\-time computation, leaving efficient execution control for local CUAs comparatively underexplored\.
##### Inference\-Time Exploration and Selection\.
Reasoning\-oriented LLM studies use parallel inference\-time scaling through candidate sampling and branching search[Wang et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib26);[Yao et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib27);[Zhu et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib30);[Lee et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib31), and related CUA frameworks adopt trajectory reranking or parallel exploration[Yang et al\. \(2025\)](https://arxiv.org/html/2611.24977#bib.bib10);[Gonzalez\-Pumariega et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib9)\. While related in spirit, our approach applies candidate diversification only at failure\-prone points to recover in local CUAs\.
##### Execution Reliability and Recovery\.
Agent reliability has been enhanced via feedback, reflection, and trajectory refinement[Madaan et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib29);[Shinn et al\. \(2023\)](https://arxiv.org/html/2611.24977#bib.bib2);[Lin et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib28);[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1)\. While VLAA\-GUI improves automation using verifier and loop\-breaker[Han et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib1), it depends on proprietary models[Anthropic \(2026a\)](https://arxiv.org/html/2611.24977#bib.bib15);[Anthropic \(2026b\)](https://arxiv.org/html/2611.24977#bib.bib16);[Google \(2026\)](https://arxiv.org/html/2611.24977#bib.bib17);[Singh et al\. \(2026\)](https://arxiv.org/html/2611.24977#bib.bib18)and multi\-agent systems\. Conversely, CANDID improves reliability using only the local agent’s own action candidates, without auxiliary agents or verifiers\.
## 6Conclusion
We presented CANDID, a candidate\-based execution\-control framework for local computer\-use agents that addresses two coupled failure modes: repetitive action loops and premature DONE decisions, both of which persist under naive execution scaling and textual feedback\. By leveraging alternative actions generated by the local agent and intervening only at failure\-prone decision points, CANDID recovers from repetitive behavior and challenges unsupported termination\. Experiments on OSWorld demonstrate consistent improvements across multiple local CUAs over feedback\-based alternatives, with minimal execution overhead\. These results suggest that selectively allocating inference\-time computation to high\-risk execution states can be more effective than uniformly scaling computation throughout the trajectory\.
## Acknowledgments
This work was supported by the National Research Foundation of Korea \(NRF\) grant funded by the Korea government \(MSIT\) \(No\. RS\-2025\-00561961\)\. It was also supported by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(No\. RS\-2025\-02214497, Development of low\-level optimization program API technology for AI semiconductors\), as well as by the Institute of Information & Communications Technology Planning & Evaluation \(IITP\) grant funded by the Korea government \(MSIT\) \(No\. RS\-2026\-25507427, Development of Efficient Architectures and Training Techniques for High\-Performance Lightweight AI Models\)\.
## Limitations
Our framework focuses on lightweight execution\-time control for local computer\-use agents and is evaluated primarily on OSWorld desktop environments\. While the proposed mechanisms consistently improve robustness across multiple local CUAs, the current study does not explore broader settings such as mobile environments, large\-scale cloud agents, or long\-horizon multi\-agent orchestration\. CANDID also depends on the base agent producing sufficiently diverse and useful candidates; when sampled candidates collapse to the same failed behavior, candidate\-based recovery provides limited benefit\. Moreover, the current loop detector targets relatively local repetitive patterns and may miss longer\-range or irregular failure cycles\. Our candidate selection strategy relies on lightweight heuristic filtering and local reranking, and we do not yet integrate CANDID with higher\-level planning or memory modules\. Extending the framework with adaptive triggering, stronger verification, and higher\-level agentic signals remains an important direction for future work\.
## References
- Agasheet al\.\(2024\)S\. Agashe, J\. Han, S\. Gan, J\. Yang, A\. Li, and X\. E\. WangAgent s: an open agentic framework that uses computers like a human\.External Links:2410\.08164,[Link](https://arxiv.org/abs/2410.08164)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1)\.
- Agasheet al\.\(2025\)S\. Agashe, K\. Wong, V\. Tu, J\. Yang, A\. Li, and X\. E\. WangAgent s2: a compositional generalist\-specialist framework for computer use agents\.External Links:2504\.00906,[Link](https://arxiv.org/abs/2504.00906)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Anthropic \(2026a\)AnthropicIntroducing claude opus 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-opus\-4\-6](https://www.anthropic.com/news/claude-opus-4-6)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p2.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Anthropic \(2026b\)AnthropicIntroducing claude sonnet 4\.6\.Note:[https://www\.anthropic\.com/news/claude\-sonnet\-4\-6](https://www.anthropic.com/news/claude-sonnet-4-6)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Baiet al\.\(2025\)S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. ZhuQwen3\-vl technical report\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p5.1)\.
- Belcaket al\.\(2025\)P\. Belcak, G\. Heinrich, S\. Diao, Y\. Fu, X\. Dong, S\. Muralidharan, Y\. C\. Lin, and P\. MolchanovSmall language models are the future of agentic ai\.External Links:2506\.02153,[Link](https://arxiv.org/abs/2506.02153)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p2.1)\.
- Bonattiet al\.\(2024\)R\. Bonatti, D\. Zhao, F\. Bonacci, D\. Dupont, S\. Abdali, Y\. Li, Y\. Lu, J\. Wagle, K\. Koishida, A\. Bucker, L\. Jang, and Z\. HuiWindows agent arena: evaluating multi\-modal os agents at scale\.External Links:2409\.08264,[Link](https://arxiv.org/abs/2409.08264)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.24977#S4.SS1.SSS0.Px1.p1.1)\.
- Denget al\.\(2023\)X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. SuMind2Web: towards a generalist agent for the web\.External Links:2306\.06070,[Link](https://arxiv.org/abs/2306.06070)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Gonzalez\-Pumariegaet al\.\(2026\)G\. Gonzalez\-Pumariega, V\. Tu, C\. Lee, J\. Yang, A\. Li, and X\. E\. WangScaling agents for computer use\.External Links:2510\.02250,[Link](https://arxiv.org/abs/2510.02250)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px2.p1.1)\.
- Google \(2026\)GoogleGemini 3\.1 pro model card\.Note:[https://storage\.googleapis\.com/deepmind\-media/Model\-Cards/Gemini\-3\-1\-Pro\-Model\-Card\.pdf](https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card.pdf)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Hanet al\.\(2026\)Q\. Han, H\. Tu, Z\. Wang, H\. Dai, Y\. Zhou, N\. Lau, A\. A\. Cardenas, Y\. Xu, R\. Xu, C\. Xiong, Z\. Zheng, H\. Yao, Y\. Zhou, and C\. XieVLAA\-gui: knowing when to stop, recover, and search, a modular framework for gui automation\.External Links:2604\.21375,[Link](https://arxiv.org/abs/2604.21375)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p2.1),[§1](https://arxiv.org/html/2611.24977#S1.p3.1),[§2\.3](https://arxiv.org/html/2611.24977#S2.SS3.p1.1),[§4\.1](https://arxiv.org/html/2611.24977#S4.SS1.SSS0.Px4.p1.1),[Table 4](https://arxiv.org/html/2611.24977#S4.T4),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Leeet al\.\(2026\)Y\. Lee, H\. Yen, X\. Ye, and D\. ChenAgentic aggregation for parallel scaling of long\-horizon agentic tasks\.External Links:2604\.11753,[Link](https://arxiv.org/abs/2604.11753)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px2.p1.1)\.
- Linet al\.\(2024\)K\. Q\. Lin, L\. Li, D\. Gao, Z\. Yang, S\. Wu, Z\. Bai, W\. Lei, L\. Wang, and M\. Z\. ShouShowUI: one vision\-language\-action model for gui visual agent\.External Links:2411\.17465,[Link](https://arxiv.org/abs/2411.17465)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Linet al\.\(2026\)Z\. Lin, F\. Liu, Y\. Yang, J\. Lyu, Y\. Gao, Y\. Liu, Z\. Lu, Y\. Yu, M\. Yang, J\. Li, D\. Ye, and J\. JiangUI\-voyager: a self\-evolving gui agent learning via failed experience\.External Links:2603\.24533,[Link](https://arxiv.org/abs/2603.24533)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2025\)Z\. Liu, J\. Li, W\. X\. Zhao, D\. Gao, Y\. Li, and J\. WenPAL\-ui: planning with active look\-back for vision\-based gui agents\.External Links:2510\.00413,[Link](https://arxiv.org/abs/2510.00413)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1)\.
- Madaanet al\.\(2023\)A\. Madaan, N\. Tandon, P\. Gupta, S\. Hallinan, L\. Gao, S\. Wiegreffe, U\. Alon, N\. Dziri, S\. Prabhumoye, Y\. Yang, S\. Gupta, B\. P\. Majumder, K\. Hermann, S\. Welleck, A\. Yazdanbakhsh, and P\. ClarkSelf\-refine: iterative refinement with self\-feedback\.External Links:2303\.17651,[Link](https://arxiv.org/abs/2303.17651)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Panet al\.\(2024\)J\. Pan, Y\. Zhang, N\. Tomlin, Y\. Zhou, S\. Levine, and A\. SuhrAutonomous evaluation and refinement of digital agents\.External Links:2404\.06474,[Link](https://arxiv.org/abs/2404.06474)Cited by:[§2\.3](https://arxiv.org/html/2611.24977#S2.SS3.p1.1)\.
- Qinet al\.\(2025\)Y\. Qin, Y\. Ye, J\. Fang, H\. Wang, S\. Liang, S\. Tian, J\. Zhang, J\. Li, Y\. Li, S\. Huang, W\. Zhong, K\. Li, J\. Yang, Y\. Miao, W\. Lin, L\. Liu, X\. Jiang, Q\. Ma, J\. Li, X\. Xiao, K\. Cai, C\. Li, Y\. Zheng, C\. Jin, C\. Li, X\. Zhou, M\. Wang, H\. Chen, Z\. Li, H\. Yang, H\. Liu, F\. Lin, T\. Peng, X\. Liu, and G\. ShiUI\-tars: pioneering automated gui interaction with native agents\.External Links:2501\.12326,[Link](https://arxiv.org/abs/2501.12326)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Rawleset al\.\(2025\)C\. Rawles, S\. Clinckemaillie, Y\. Chang, J\. Waltz, G\. Lau, M\. Fair, A\. Li, W\. Bishop, W\. Li, F\. Campbell\-Ajala, D\. Toyama, R\. Berry, D\. Tyamagundlu, T\. Lillicrap, and O\. RivaAndroidWorld: a dynamic benchmarking environment for autonomous agents\.External Links:2405\.14573,[Link](https://arxiv.org/abs/2405.14573)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Shinnet al\.\(2023\)N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. YaoReflexion: language agents with verbal reinforcement learning\.External Links:2303\.11366,[Link](https://arxiv.org/abs/2303.11366)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p3.1),[§2\.3](https://arxiv.org/html/2611.24977#S2.SS3.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Singhet al\.\(2026\)A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram, A\. Nathan, A\. Luo, A\. Helyar, A\. Madry, A\. Efremov, A\. Spyra, A\. Baker\-Whitcomb, A\. Beutel, A\. Karpenko, A\. Makelov, A\. Neitz, A\. Wei, A\. Barr, A\. Kirchmeyer, A\. Ivanov, A\. Christakis, A\. Gillespie, A\. Tam, A\. Bennett, A\. Wan, A\. Huang, A\. M\. Sandjideh, A\. Yang, A\. Kumar, A\. Saraiva, A\. Vallone, A\. Gheorghe, A\. G\. Garcia, A\. Braunstein, A\. Liu, A\. Schmidt, A\. Mereskin, A\. Mishchenko, A\. Applebaum, A\. Rogerson, A\. Rajan, A\. Wei, A\. Kotha, A\. Srivastava, A\. Agrawal, A\. Vijayvergiya, A\. Tyra, A\. Nair, A\. Nayak, B\. Eggers, B\. Ji, B\. Hoover, B\. Chen, B\. Chen, B\. Barak, B\. Minaiev, B\. Hao, B\. Baker, B\. Lightcap, B\. McKinzie, B\. Wang, B\. Quinn, B\. Fioca, B\. Hsu, B\. Yang, B\. Yu, B\. Zhang, B\. Brenner, C\. R\. Zetino, C\. Raymond, C\. Lugaresi, C\. Paz, C\. Hudson, C\. Whitney, C\. Li, C\. Chen, C\. Cole, C\. Voss, C\. Ding, C\. Shen, C\. Huang, C\. Colby, C\. Hallacy, C\. Koch, C\. Lu, C\. Kaplan, C\. Kim, C\. Minott\-Henriques, C\. Frey, C\. Yu, C\. Czarnecki, C\. Reid, C\. Wei, C\. Decareaux, C\. Scheau, C\. Zhang, C\. Forbes, D\. Tang, D\. Goldberg, D\. Roberts, D\. Palmie, D\. Kappler, D\. Levine, D\. Wright, D\. Leo, D\. Lin, D\. Robinson, D\. Grabb, D\. Chen, D\. Lim, D\. Salama, D\. Bhattacharjee, D\. Tsipras, D\. Li, D\. Yu, D\. Strouse, D\. Williams, D\. Hunn, E\. Bayes, E\. Arbus, E\. Akyurek, E\. Y\. Le, E\. Widmann, E\. Yani, E\. Proehl, E\. Sert, E\. Cheung, E\. Schwartz, E\. Han, E\. Jiang, E\. Mitchell, E\. Sigler, E\. Wallace, E\. Ritter, E\. Kavanaugh, E\. Mays, E\. Nikishin, F\. Li, F\. P\. Such, F\. de Avila Belbute Peres, F\. Raso, F\. Bekerman, F\. Tsimpourlas, F\. Chantzis, F\. Song, F\. Zhang, G\. Raila, G\. McGrath, G\. Briggs, G\. Yang, G\. Parascandolo, G\. Chabot, G\. Kim, G\. Zhao, G\. Valiant, G\. Leclerc, H\. Salman, H\. Wang, H\. Sheng, H\. Jiang, H\. Wang, H\. Jin, H\. Sikchi, H\. Schmidt, H\. Aspegren, H\. Chen, H\. Qiu, H\. Lightman, I\. Covert, I\. Kivlichan, I\. Silber, I\. Sohl, I\. Hammoud, I\. Clavera, I\. Lan, I\. Akkaya, I\. Kostrikov, I\. Kofman, I\. Etinger, I\. Singal, J\. Hehir, J\. Huh, J\. Pan, J\. Wilczynski, J\. Pachocki, J\. Lee, J\. Quinn, J\. Kiros, J\. Kalra, J\. Samaroo, J\. Wang, J\. Wolfe, J\. Chen, J\. Wang, J\. Harb, J\. Han, J\. Wang, J\. Zhao, J\. Chen, J\. Yang, J\. Tworek, J\. Chand, J\. Landon, J\. Liang, J\. Lin, J\. Liu, J\. Wang, J\. Tang, J\. Yin, J\. Jang, J\. Morris, J\. Flynn, J\. Ferstad, J\. Heidecke, J\. Fishbein, J\. Hallman, J\. Grant, J\. Chien, J\. Gordon, J\. Park, J\. Liss, J\. Kraaijeveld, J\. Guay, J\. Mo, J\. Lawson, J\. McGrath, J\. Vendrow, J\. Jiao, J\. Lee, J\. Steele, J\. Wang, J\. Mao, K\. Chen, K\. Hayashi, K\. Xiao, K\. Salahi, K\. Wu, K\. Sekhri, K\. Sharma, K\. Singhal, K\. Li, K\. Nguyen, K\. Gu\-Lemberg, K\. King, K\. Liu, K\. Stone, K\. Yu, K\. Ying, K\. Georgiev, K\. Lim, K\. Tirumala, K\. Miller, L\. Ahmad, L\. Lv, L\. Clare, L\. Fauconnet, L\. Itow, L\. Yang, L\. Romaniuk, L\. Anise, L\. Byron, L\. Pathak, L\. Maksin, L\. Lo, L\. Ho, L\. Jing, L\. Wu, L\. Xiong, L\. Mamitsuka, L\. Yang, L\. McCallum, L\. Held, L\. Bourgeois, L\. Engstrom, L\. Kuhn, L\. Feuvrier, L\. Zhang, L\. Switzer, L\. Kondraciuk, L\. Kaiser, M\. Joglekar, M\. Singh, M\. Shah, M\. Stratta, M\. Williams, M\. Chen, M\. Sun, M\. Cayton, M\. Li, M\. Zhang, M\. Aljubeh, M\. Nichols, M\. Haines, M\. Schwarzer, M\. Gupta, M\. Shah, M\. Y\. Guan, M\. Huang, M\. Dong, M\. Wang, M\. Glaese, M\. Carroll, M\. Lampe, M\. Malek, M\. Sharman, M\. Zhang, M\. Wang, M\. Pokrass, M\. Florian, M\. Pavlov, M\. Wang, M\. Chen, M\. Wang, M\. Feng, M\. Bavarian, M\. Lin, M\. Abdool, M\. Rohaninejad, N\. Soto, N\. Staudacher, N\. LaFontaine, N\. Marwell, N\. Liu, N\. Preston, N\. Turley, N\. Ansman, N\. Blades, N\. Pancha, N\. Mikhaylin, N\. Felix, N\. Handa, N\. Rai, N\. Keskar, N\. Brown, O\. Nachum, O\. Boiko, O\. Murk, O\. Watkins, O\. Gleeson, P\. Mishkin, P\. Lesiewicz, P\. Baltescu, P\. Belov, P\. Zhokhov, P\. Pronin, P\. Guo, P\. Thacker, Q\. Liu, Q\. Yuan, Q\. Liu, R\. Dias, R\. Puckett, R\. Arora, R\. T\. Mullapudi, R\. Gaon, R\. Miyara, R\. Song, R\. Aggarwal, R\. Marsan, R\. Yemiru, R\. Xiong, R\. Kshirsagar, R\. Nuttall, R\. Tsiupa, R\. Eldan, R\. Wang, R\. James, R\. Ziv, R\. Shu, R\. Nigmatullin, S\. Jain, S\. Talaie, S\. Altman, S\. Arnesen, S\. Toizer, S\. Toyer, S\. Miserendino, S\. Agarwal, S\. Yoo, S\. Heon, S\. Ethersmith, S\. Grove, S\. Taylor, S\. Bubeck, S\. Banesiu, S\. Amdo, S\. Zhao, S\. Wu, S\. Santurkar, S\. Zhao, S\. R\. Chaudhuri, S\. Krishnaswamy, Shuaiqi, Xia, S\. Cheng, S\. Anadkat, S\. P\. Fishman, S\. Tobin, S\. Fu, S\. Jain, S\. Mei, S\. Egoian, S\. Kim, S\. Golden, S\. Mah, S\. Lin, S\. Imm, S\. Sharpe, S\. Yadlowsky, S\. Choudhry, S\. Eum, S\. Sanjeev, T\. Khan, T\. Stramer, T\. Wang, T\. Xin, T\. Gogineni, T\. Christianson, T\. Sanders, T\. Patwardhan, T\. Degry, T\. Shadwell, T\. Fu, T\. Gao, T\. Garipov, T\. Sriskandarajah, T\. Sherbakov, T\. Korbak, T\. Kaftan, T\. Hiratsuka, T\. Wang, T\. Song, T\. Zhao, T\. Peterson, V\. Kharitonov, V\. Chernova, V\. Kosaraju, V\. Kuo, V\. Pong, V\. Verma, V\. Petrov, W\. Jiang, W\. Zhang, W\. Zhou, W\. Xie, W\. Zhan, W\. McCabe, W\. DePue, W\. Ellsworth, W\. Bain, W\. Thompson, X\. Chen, X\. Qi, X\. Xiang, X\. Shi, Y\. Dubois, Y\. Yu, Y\. Khakbaz, Y\. Wu, Y\. Qian, Y\. T\. Lee, Y\. Chen, Y\. Zhang, Y\. Xiong, Y\. Tian, Y\. Cha, Y\. Bai, Y\. Yang, Y\. Yuan, Y\. Li, Y\. Zhang, Y\. Yang, Y\. Jin, Y\. Jiang, Y\. Wang, Y\. Wang, Y\. Liu, Z\. Stubenvoll, Z\. Dou, Z\. Wu, and Z\. WangOpenAI gpt\-5 system card\.External Links:2601\.03267,[Link](https://arxiv.org/abs/2601.03267)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p2.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px3.p1.1)\.
- Songet al\.\(2026\)L\. Song, Y\. Dai, V\. Prabhu, J\. Zhang, T\. Shi, L\. Li, J\. Li, S\. Savarese, Z\. Chen, J\. Zhao, R\. Xu, and C\. XiongCoAct\-1: computer\-using multi\-agent system with coding actions\.External Links:2508\.03923,[Link](https://arxiv.org/abs/2508.03923)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p2.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, B\. Wang, D\. Lu, J\. Yang, T\. Xie, J\. Wang, J\. Deng, X\. Guo, Y\. Xu, C\. H\. Wu, Z\. Shen, Z\. Li, R\. Li, X\. Li, J\. Chen, B\. Zheng, P\. Li, F\. Lei, R\. Cao, Y\. Fu, D\. Shin, M\. Shin, J\. Hu, Y\. Wang, J\. Chen, Y\. Ye, D\. Zhang, D\. Du, H\. Hu, H\. Chen, Z\. Zhou, H\. Yao, Z\. Chen, Q\. Gu, Y\. Wang, H\. Wang, D\. Yang, V\. Zhong, F\. Sung, Y\. Charles, Z\. Yang, and T\. YuOpenCUA: open foundations for computer\-use agents\.External Links:2508\.09123,[Link](https://arxiv.org/abs/2508.09123)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§1](https://arxiv.org/html/2611.24977#S1.p5.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2023\)X\. Wang, J\. Wei, D\. Schuurmans, Q\. Le, E\. Chi, S\. Narang, A\. Chowdhery, and D\. ZhouSelf\-consistency improves chain of thought reasoning in language models\.External Links:2203\.11171,[Link](https://arxiv.org/abs/2203.11171)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px2.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. YuOSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.External Links:2404\.07972,[Link](https://arxiv.org/abs/2404.07972)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§1](https://arxiv.org/html/2611.24977#S1.p5.1),[§4\.1](https://arxiv.org/html/2611.24977#S4.SS1.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Xuet al\.\(2025\)Y\. Xu, Z\. Wang, J\. Wang, D\. Lu, T\. Xie, A\. Saha, D\. Sahoo, T\. Yu, and C\. XiongAguvis: unified pure vision agents for autonomous gui interaction\.External Links:2412\.04454,[Link](https://arxiv.org/abs/2412.04454)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Xueet al\.\(2026\)T\. Xue, C\. Peng, M\. Huang, L\. Guo, T\. Han, H\. Wang, J\. Wang, X\. Zhang, X\. Yang, D\. Zhao, J\. Ding, X\. Ma, Y\. Xie, P\. Pei, X\. Cai, and X\. QiuEvoCUA: evolving computer use agents via learning from scalable synthetic experience\.External Links:2601\.15876,[Link](https://arxiv.org/abs/2601.15876)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2026\)B\. Yang, K\. Jin, Z\. Wu, Z\. Liu, Q\. Sun, Z\. Li, J\. Xie, Z\. Liu, F\. Xu, K\. Cheng, Q\. Li, Y\. Wang, Y\. Qiao, Z\. Wang, and Z\. DingOS\-symphony: a holistic framework for robust and generalist computer\-using agent\.External Links:2601\.07779,[Link](https://arxiv.org/abs/2601.07779)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p2.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025\)Y\. Yang, D\. Li, Y\. Dai, Y\. Yang, Z\. Luo, Z\. Zhao, Z\. Hu, J\. Huang, A\. Saha, Z\. Chen, R\. Xu, L\. Pan, S\. Savarese, C\. Xiong, and J\. LiGTA1: gui test\-time scaling agent\.External Links:2507\.05791,[Link](https://arxiv.org/abs/2507.05791)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1),[§4\.1](https://arxiv.org/html/2611.24977#S4.SS1.SSS0.Px4.p1.1),[Table 4](https://arxiv.org/html/2611.24977#S4.T4),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px2.p1.1)\.
- Yaoet al\.\(2023\)S\. Yao, D\. Yu, J\. Zhao, I\. Shafran, T\. L\. Griffiths, Y\. Cao, and K\. NarasimhanTree of thoughts: deliberate problem solving with large language models\.External Links:2305\.10601,[Link](https://arxiv.org/abs/2305.10601)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px2.p1.1)\.
- Zhouet al\.\(2024\)S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. NeubigWebArena: a realistic web environment for building autonomous agents\.External Links:2307\.13854,[Link](https://arxiv.org/abs/2307.13854)Cited by:[§1](https://arxiv.org/html/2611.24977#S1.p1.1)\.
- Zhuet al\.\(2025\)K\. Zhu, H\. Li, S\. Wu, T\. Xing, D\. Ma, X\. Tang, M\. Liu, J\. Yang, J\. Liu, Y\. E\. Jiang, C\. Zhang, C\. Lin, J\. Wang, G\. Zhang, and W\. ZhouScaling test\-time compute for llm agents\.External Links:2506\.12928,[Link](https://arxiv.org/abs/2506.12928)Cited by:[§5](https://arxiv.org/html/2611.24977#S5.SS0.SSS0.Px2.p1.1)\.
## Appendix ACANDID Implementation
### A\.1Loop Detection Details
This section details the loop detection procedure used by the loop\-recovery module in Section[3\.1](https://arxiv.org/html/2611.24977#S3.SS1)\. The implementation follows a two\-stage detection\. First, a deterministic rule detector checks whether the newly proposed action would extend a recent repetitive pattern\. Second, the local model confirms that the trajectory is stagnant before recovery candidates are generated\.
##### Action signatures\.
For each proposed action, the detector constructs a coarse signature consisting of an action type, an optional target coordinate, and a normalized action string\. Action types are extracted from parsed PyAutoGUI calls when available: click\-like calls are mapped to click variants,typewrite/writetotype,hotkey/presstokey,scroll/hscrolltoscroll,moveTotomouse\_move, anddragTotodrag\. If the parsed code is unavailable, the detector falls back to coarse markers in the natural language action string\. Target coordinates are extracted from the first\(x, y\)pair when present\.
Trigger reasonConditionIntuitionrepeated\_action\_type\_and\_targetThe proposed action has a target coordinate, and at leastr−1r\-1recent actions within the window have the same action type and a target within toleranceτ\\tauof the proposed target\.The agent is repeatedly operating on the same local screen region\.repeated\_targetless\_action\_typeThe proposed action type isscrollorwait, and at leastr−1r\-1recent actions in the window have the same action type\.The agent is repeatedly scrolling or waiting without a concrete target\.repeated\_exact\_text\_or\_key\_actionThe proposed action type istypeorkey, and the normalized full action string appears at leastr−1r\-1times in the recent window\.Text entry and keyboard commands require exact matching because the payload matters\.repeated\_response\_and\_action\_typeThe previous two assistant responses are identical after normalization, and there is at least one recent action with the same action type as the proposed action\.The model is reproducing the same reasoning or action description while continuing the same type of operation\.repeated\_cyclic\_action\_patternThe recent action\-signature sequence plus the proposed action ends with a repeated pattern of length 2–4\.The agent alternates among a small set of actions rather than repeating one literal action\.Table 8:Rule\-based loop trigger conditions\. Hererrisloop\_recovery\_min\_repeats,τ\\tauisloop\_recovery\_target\_tolerance, and the recent action window isloop\_recovery\_window\. In our main results, these arer=3r=3,τ=80\\tau=80, and a window of 5 actions\.
##### Rule\-based trigger\.
Letata\_\{t\}be the action proposed for the current step\. If the parsed action is empty or is a termination action, loop recovery is not triggered\. Otherwise, the detector comparesata\_\{t\}against the lastwwexecuted actions, wherew=loop\_recovery\_windoww=\\texttt\{loop\\\_recovery\\\_window\}\. The detector requires at leastr−1r\-1historical actions before testing repetition, because the current proposal itself supplies therr\-th repeated item\. The concrete trigger conditions are listed in Table[8](https://arxiv.org/html/2611.24977#A1.T8)\.
For cyclic patterns, the detector uses a slightly longer window,max\(w,8\)\\max\(w,8\), appends the current proposal, and converts each action into a compact sequence item\. Coordinate\-based actions are bucketed by the target toleranceτ\\tau;typeandkeyactions retain their normalized full action string; all other actions use only the coarse action type\. The detector then checks whether the sequence suffix repeats with a pattern length from 2 to 4\. This catches cases such as alternating between two nearby clicks or repeatedly cycling through scroll and click operations\.
##### Model confirmation gate\.
The rule detector is intentionally high\-recall: it identifies points where repetition is plausible, but it does not independently decide that the trajectory is unproductive\. A model confirmation call therefore follows a positive rule trigger\. The confirmation prompt contains the task, current screenshot, recent trajectory, and the current proposed action, and asks the same local model to return a JSON object with progress\_signal, loop\_detected, loop\_evidence, strategy\_signal, avoid\_actions, and escape\_direction\. Recovery proceeds only when eitherloop\_detected=trueorstrategy\_signal=SWITCH\. Otherwise, the original action is left unchanged and candidate generation is skipped\.
##### Candidate blocking after detection\.
The detected rule also defines what recovery candidates are forbidden to repeat\. Malformed candidates are removed, andDONEtermination is disallowed during loop recovery\. If a candidate has a different action type from the blocked action type, it remains eligible\. If it has the same action type, it is blocked when it lacks a distinguishable target or when its target lies within the same toleranceτ\\tauof the blocked target\. This ensures that recovery does not simply reselect the same local operation under a different surface form\.
### A\.2CANDID Prompts
Below we provide the prompt skeletons used by CANDID\. Runtime fields are shown in angle brackets\. The normal action\-generation prompt includes the computer\-use tool schema and requires oneActionline followed by one<tool\_call\>block\.
#### A\.2\.1Loop\-Recovery Candidate Selection
##### Loop\-confirmation prompt\.
After the rule\-based detector flags a potential repetitive loop, CANDID asks the local model to independently confirm that the proposed action would continue a no\-progress loop\. Rule\-detector evidence is deliberately omitted from this prompt to avoid biasing the confirmation\.
System PromptYou are a loop\-confirmation agent for a desktop automation task\. Analyze the recent trajectory and current screenshot for repetitive no\-progress behavior\. Focus only on loop recovery\. Do not judge whether the whole task is complete\. Return only a valid JSON object\.User PromptTask description:<TASK INSTRUCTION\>Recent trajectory before the current proposed action: <RECENT TRAJECTORY\>Current proposed action before execution: Assistant response:<MODEL RESPONSE\> Parsed action:<PARSED ACTION\> PyAutoGUI code:<PYAUTOGUI CODE\>Assess whether executing the current proposed action would continue a repetitive no\-progress loop\. Focus only on loop recovery\. Do not judge whether the whole task is complete\. Return JSON exactly with keys:\{"progress\_signal": "what visibly changed, or no visible progress", "loop\_detected": true/false, "loop\_evidence": "short evidence", "strategy\_signal": "KEEP or SWITCH", "avoid\_actions": \["short action/target patterns to avoid"\], "escape\_direction": "short direction for changing approach"\}Multimodal input: current screenshot\.
Recovery proceeds only if the returnedloop\_detectedvalue is true or thestrategy\_signalisSWITCH\.
##### Loop\-recovery candidate\-generation prompt\.
For each recovery sample, CANDID reuses the normal action\-generation system prompt, tool schema, recent multimodal history, and current screenshot\. It appends the following control block to the current user message:
Control Block Appended to the User PromptLoop recovery note: The recent trajectory status:<PROGRESS SIGNAL\>Avoid: •<AVOID ACTIONS\>Try instead: • choose a different target, menu path, keyboard shortcut, or navigation strategy•<ESCAPE DIRECTION\>Evidence: <LOOP EVIDENCE\>Generate the next action using the normal action format\. Do not terminate or answerDONEduring loop recovery\.
Each sampled output retains the base action format:
Action: <SHORT IMPERATIVE ACTION\> <tool\_call\> \{"name": "computer\_use", "arguments": \{<ACTION ARGUMENTS\>\}\} </tool\_call\>
##### Loop\-recovery candidate\-selection prompt\.
Malformed, duplicate, and blocked candidates are removed before selection\. The selector receives only the remaining candidates\.
System PromptYou select the best loop\-recovery action for a desktop automation agent\. The normal action model appears stuck in a repeated local pattern\. Choose exactly one provided candidate\. Do not invent a new action\. Prefer a candidate that changes target, action type, menu path, or modality while still making task progress\. Do not choose an action just because it is different; it must be useful for the current screen\. Return only a valid JSON object\.User PromptMultimodal input: current screenshot\.Task instruction:<TASK INSTRUCTION\>Recent trajectory:<RECENT TRAJECTORY\>Loop evidence: • reason:<TRIGGER REASON\> • proposed repeated action:<PROPOSED ACTION\> • blocked action type:<BLOCKED ACTION TYPE\> • blocked target:<BLOCKED TARGET\>Candidate recovery actions:Candidate<ID\>: Parsed action:<PARSED ACTION\> PyAutoGUI code:<PYAUTOGUI CODE\> Action type:<ACTION TYPE\> Target:<TARGET\> Model response:<MODEL RESPONSE\><ADDITIONAL CANDIDATES\>Return JSON exactly with keys:\{"selected\_candidate\_id": <integer\>, "selected\_action": "candidate parsed action", "reason": "short reason"\}
#### A\.2\.2UnanimousDONEGating
##### Candidate\-generation prompt\.
When the initial action isDONE, additional candidates are sampled from the base action\-generation prompt\. Thus, there is no separate DONE\-specific generation instruction\. The prompt contains the computer\-use system prompt and tool schema, recent screenshot–response history, the current screenshot, and the following current\-step instruction:
Current\-Step User PromptPlease generate the next move according to the UI screenshot, instruction and previous actions\.Instruction:<TASK INSTRUCTION\>Previous actions: <PREVIOUS ACTIONS\>Multimodal input: current screenshot\.
##### Candidate\-selection prompt\.
The originalDONEproposal and the additional samples are first deduplicated by executable action\. Under the unanimous policy, if any valid non\-DONEsample exists, allDONEcandidates are excluded\. The following selector prompt is used only when more than one candidate remains after this filtering:
System PromptYou select the best next action for a desktop automation agent\. Compare the candidates against the task instruction, current screenshot, and recent trajectory\. Choose exactly one provided candidate\. Do not invent a new action\. Choose the candidate that is most correct for the current state and most likely to complete the task successfully\. If a candidate claims the task is complete, select it only when the visible state and trajectory clearly support that claim\. Return only a valid JSON object\.User PromptMultimodal input: current screenshot\.Task instruction:<TASK INSTRUCTION\>Recent trajectory:<RECENT TRAJECTORY\>Candidate actions:Candidate<ID\>: Parsed action:<PARSED ACTION\> PyAutoGUI code:<PYAUTOGUI CODE\> IsDONE:<TRUE OR FALSE\> Model response:<MODEL RESPONSE\><ADDITIONAL CANDIDATES\>Return JSON exactly with keys:\{"selected\_candidate\_id": <integer\>, "selected\_action": "candidate parsed action", "reason": "short reason"\}
If filtering leaves a single valid non\-DONEcandidate, it is selected directly without an additional model call\. If all samples agree onDONE, termination is retained\.
### A\.3Computing Infrastructure
All experiments were conducted on a Linux \(Ubuntu\) server with an NVIDIA A100 80GB GPU\. Model serving was performed using vLLM \(v0\.12\.0\) on top of the Transformers library \(v5\.2\.0\)\.
## Appendix BAdditional Analysis
### B\.1Candidate Diversity
We analyze candidate diversity at triggered control points by counting the number of unique exact actions and unique action types before filtering\. As shown in Table[9](https://arxiv.org/html/2611.24977#A2.T9), Loop Recovery exhibits substantial diversity at the exact\-action level: among 2,496 triggered events, all four sampled candidates are distinct in 1,190 cases \(47\.7%\), while only 209 cases \(8\.4%\) produce identical actions\. In contrast, action\-type diversity is much lower, with 2,095 events \(83\.9%\) sharing a single action type across all candidates\. Similarly, DONE Gating frequently generates multiple distinct exact actions, with only 238 of 429 triggered events \(55\.5%\) producing identical candidates\. These results suggest that candidate diversity primarily arises from differences in action parameters, such as click locations or typed text, rather than coarse GUI action types\.
ModuleDiversity Metric1234Loop RecoveryExact action2094356621,190Loop RecoveryAction type2,095389120DONE GatingExact action238967322
Table 9:Candidate diversity at triggered control points\.
### B\.2Loop Detector Hyperparameter Sensitivity
wwτ\\taurrTrialsOverall \(%\)5803534\.84±\\pm1\.273803336\.05±\\pm0\.717803335\.94±\\pm0\.85503336\.88±\\pm0\.445403335\.02±\\pm1\.2851203334\.96±\\pm0\.675802333\.04±\\pm1\.035804334\.85±\\pm0\.535805335\.01±\\pm1\.24Table 10:Sensitivity analysis of the loop detector hyperparameters using Qwen3\-VL\-8B with a step budget ofs=50s=50\. Results are reported as mean task accuracy \(%\)±\\pmstandard deviation over multiple trials\.Table[10](https://arxiv.org/html/2611.24977#A2.T10)varies the loop\-detection window \(ww\), target tolerance \(τ\\tau\), and minimum repeat count \(rr\)\. Although several three\-trial settings yield numerically higher means than the five\-trial main result, these gains are not consistent across neighboring values and may reflect stochastic variation in agent execution\. In contrast, settingr=2r=2reduces accuracy to 33\.04%, nearly eliminating the gain over the 32\.82% baseline, suggesting that overly sensitive loop detection can interrupt legitimate repeated actions\. Overall, performance remains broadly stable across moderate changes in the window and tolerance\.
### B\.3Task\-Level Analysis of Control Decisions
This section provides a task\-level analysis of how CANDID’s control modules behave after being triggered, with particular attention to whether interventions alter the original decision and how effectively the DONE gate distinguishes successful from premature termination\.
OSWorld does not provide ground\-truth labels indicating whether an individual recovery or termination decision is correct at each execution step\. We therefore analyze control decisions using final task success as a task\-level proxy\. These results should not be interpreted as action\-level correctness; rather, they characterize whether trajectories eventually succeed after a control module is triggered\.
#### B\.3\.1Intervention Outcomes
We first examine whether each triggered control decision changes the agent’s original action\. For Loop Recovery, a positive decision indicates that the original action is replaced by a recovery action\. For Unanimous DONE Gating, a positive decision indicates that the originalDONEproposal is redirected to a non\-DONEcontinuation\. We define TP as a changed decision followed by task success, FP as a changed decision followed by task failure, TN as an unchanged decision followed by task success, and FN as an unchanged decision followed by task failure\. Table[11](https://arxiv.org/html/2611.24977#A2.T11)summarizes the resulting task\-level outcomes for both control modules\.
MethodTPFPTNFNLoop Recovery26148810Unanimous DONE Gating186711470Table 11:Task\-level outcomes of triggered control decisions\. Final task success is used as a proxy because OSWorld does not provide per\-step ground\-truth control labels\.Among the 34 successful tasks in which Loop Recovery was triggered, 26 tasks \(76\.5%\) contained at least one action\-changing intervention\. This suggests that recovery actions are frequently involved in successful triggered trajectories\. The relatively large FP count should be interpreted cautiously: Loop Recovery is activated primarily on difficult trajectories that already exhibit repetitive or stagnant behavior, and such tasks may fail eventually even when the local recovery action is reasonable\.
For Unanimous DONE Gating, the large TN count indicates that the module frequently preserves the original termination decision in trajectories that ultimately succeed\. This behavior is consistent with its role as a conservative policy: it redirects termination only when candidate disagreement provides evidence for continued execution\.
#### B\.3\.2Termination\-Gate Evaluation
We further compare Unanimous DONE Gating with the VLAA\-GUI\-style Verifier as termination gates\. In this analysis, TP denotes an acceptedDONEdecision followed by task success, FP denotes an acceptedDONEdecision followed by task failure, TN denotes a rejectedDONEdecision followed by task failure, and FN denotes a rejectedDONEdecision followed by task success\. Table[12](https://arxiv.org/html/2611.24977#A2.T12)reports the corresponding decision counts, and Table[13](https://arxiv.org/html/2611.24977#A2.T13)presents the derived precision, recall, specificity, and accuracy\.
MethodTriggeredTPFPTNFNVLAA\-GUI Verifier22911395174Unanimous DONE Gating269132101360
Table 12:Task\-level comparison of termination\-gate decisions on Qwen3\-VL\-30B\-A3B with a 50\-step budget\.MethodPrecisionRecallSpecificityAccuracyVLAA\-GUI Verifier54\.3%96\.6%15\.2%56\.8%Unanimous DONE Gating56\.7%100\.0%26\.3%62\.5%
Table 13:Precision, recall, specificity, and accuracy of the termination gates\.Both methods achieve high recall, indicating that tasks that ultimately succeed are rarely prevented from terminating\. However, precision remains modest for both gates, meaning that a substantial fraction of acceptedDONEdecisions still occur in trajectories that fail according to the OSWorld evaluator\. This limitation reflects the difficulty of inferring semantic task completion from the local model’s own observations and predictions\.
The main difference appears in specificity\. Unanimous DONE Gating achieves higher specificity than the VLAA\-GUI Verifier \(26\.3% vs\. 15\.2%\), indicating that it rejects a larger fraction of premature termination proposals\. This improvement also leads to higher task\-level decision accuracy \(62\.5% vs\. 56\.8%\)\. Overall, these results suggest that the unanimous policy provides a more effective sampling\-based check against unsupported termination than textual verification\.
ModelMethodDomain Accuracy \(%\)Overall \(%\)Office\(43\)Web\(30\)Windows System\(24\)Coding\(24\)Media\(21\)Windows Utilities\(12\)Qwen3\-VL\-30B\-A3BBaseline2\.3336\.6733\.3333\.339\.8125\.0021\.47CANDID2\.3333\.3350\.0029\.1724\.0925\.0024\.71\(\+3\.24\)Qwen3\-VL\-8BBaseline0\.0020\.0016\.6712\.5014\.290\.0010\.39CANDID0\.0023\.3320\.8312\.5014\.290\.0011\.69\(\+1\.30\)
Table 14:Evaluation results on 154 WindowsAgentArena tasks with a step budget ofs=50s=50and history lengthh=4h=4\. Values in parentheses next to domain names indicate the task count\. Results are based on a single run; values in gray parentheses represent the absolute improvement over each model’s baseline\.
### B\.4Generalization to WindowsAgentArena
We further evaluate CANDID on 154 WindowsAgentArena tasks to assess cross\-benchmark generalization\. As shown in Table[14](https://arxiv.org/html/2611.24977#A2.T14), CANDID improves overall task accuracy from 21\.47% to 24\.71% for Qwen3\-VL\-30B\-A3B and from 10\.39% to 11\.69% for Qwen3\-VL\-8B\. For Qwen3\-VL\-30B\-A3B, the largest gains occur in Windows System and Media & Video, although performance varies across domains\. As the results are based on a single run, they provide preliminary evidence of cross\-benchmark generalization\.
## Appendix CBaseline Implementations
This section describes three feedback\-style baselines implemented for the comparison\. Self\-Guidance is our own lightweight prompt\-based intervention\. The Loop Breaker and Verifier areVLAA\-GUI\-styleimplementations: they are inspired by the STOP and RECOVER components described in VLAA\-GUI, but they are adapted to our PyAutoGUI action space and local\-model setting rather than reproduced exactly\.
### C\.1Self\-Guidance
Self\-Guidance runs a separate diagnostic prompt before normal action generation and injects the resulting text into the next action prompt asGuidance\. The guidance model discusses progress, local loops, what to avoid, the next direction, and whether task completion looks unsafe\.
##### Prompt skeleton\.
The system prompt asks the model to diagnose the desktop trajectory:
System PromptYou are diagnosing progress in a desktop\-computer task\. Analyze the instruction, recent screenshot\-action trajectory, and current screenshot\. Produce short free\-form guidance for the next action\. Be concrete\. Mention if the agent appears stuck in a local loop\. Mention what should not be repeated\. Mention whether task completion looks unsafe and should be avoided\. Do not output tool calls, XML tags, code, or executable UI actions\.
The user message contains the task instruction, recent screenshots and actions, and the current screenshot\. The requested output format is:
Requested Output Format\- progress so far \- suspected issue \- avoid repeating \- next direction \- completion note
##### Output use\.
The output is not parsed as a formal decision\. If non\-empty, it is prepended to the next normal action\-generation prompt as:
Control Block Appended to the User PromptGuidance: <free\-form guidance text\>
The action model may ignore or follow the advice; no hard candidate filtering or action constraint is applied\.
### C\.2VLAA\-GUI\-Style Loop Breaker
The VLAA\-GUI\-style Loop Breaker is a reflection module that runs before action generation when previous responses exist\. It follows the RECOVER idea from VLAA\-GUI: inspect recent trajectory evidence, decide whether the agent is stuck in repetitive no\-progress behavior, and emit a strategy\-change signal\. In our implementation, the module does not directly choose an action\. Instead, when it returns a loop signal, it writes a textual loop\-recovery constraint into the next action prompt\.
##### Prompt skeleton\.
The system prompt is:
System PromptYou are a reflection agent for a desktop automation task\. Analyze the recent trajectory and current screenshot for repetitive no\-progress behavior\. Focus only on loop recovery\. Do not judge whether the whole task is complete\. Return only a valid JSON object\.
At each step, the user message includes the latest assistant response, the parsed action, and optionally the current screenshot\. The requested schema is:
Requested Output Schema\{"progress\_signal": "what visibly changed, or no visible progress", "loop\_detected": true/false, "loop\_evidence": "short evidence", "strategy\_signal": "KEEP or SWITCH", "avoid\_actions": \["short action/target patterns to avoid"\], "escape\_direction": "short direction for changing approach"\}
##### Output use\.
Ifloop\_detected=trueorstrategy\_signal=SWITCH, the module formats the result into a constraint block:
Control Block Appended to the User PromptLoop recovery constraint: \- strategy\_signal: SWITCH \- loop\_evidence: <evidence\> \- avoid\_actions: <patterns\> \- escape\_direction: <direction\> The next action must change approach and must not match avoid\_actions\.
This block is injected into the next action\-generation prompt\. The enforcement mode is prompt\-only: unlike our candidate\-based loop recovery, the VLAA\-GUI\-style Loop Breaker does not sample multiple executable candidates or remove repeated actions through a hard filter\.
### C\.3VLAA\-GUI\-Style Completion Verifier
The VLAA\-GUI\-style Verifier is activated only when the normal action model proposesDONE\. It follows the STOP idea from VLAA\-GUI: accept task termination only when the current screenshot or recent trajectory provides direct evidence that every task requirement is satisfied\. Our implementation is intentionally conservative and rejects ambiguous completion claims\.
##### Prompt skeleton\.
The system prompt is:
System PromptYou are an extremely strict task completion verifier for desktop automation\. Decide whether the task is already fully completed right now\. Mark completion only if every requirement has direct, unambiguous evidence in the current screenshot or recent trajectory text\. Do not assume hidden state or intent\. For side effects such as saving, exporting, sending, creating, deleting, or changing settings, require visible or trajectory evidence that the side effect was committed\. Prefer false negatives over false positives\. Return only a valid JSON object\.
The user message includes the task instruction, recent trajectory text, the candidate final response, and the parsed termination action\. The requested schema is:
Requested Output Schema\{"complete": true/false, "reason": "short evidence\-based reason", "missing\_steps": "what evidence/action is still needed, or empty if complete"\}
##### Output use\.
If the Verifier returnscomplete=false, or if it returnscomplete=truewhile still mentioning uncertainty or missing steps, the agent rejects the termination\. The current action is converted to a wait\-style continuation, and the next prompt receives feedback of the form:
Control Block Appended to the User PromptCompletion verifier rejected the previous terminate action: \- reason: <reason\> \- missing\_steps: <missing evidence or action\> Continue the task\. Do not terminate again until the missing evidence is visible\.
Thus, the VLAA\-GUI\-style Verifier is a feedback mechanism rather than a candidate\-selection mechanism: it can block aDONEaction and provide textual correction, but it does not itself select among concrete continuation actions\.
## Appendix DCase Study
### D\.1Loop\-Recovery Candidate Selection
Fig\.[8](https://arxiv.org/html/2611.24977#A4.F8)shows a concrete loop\-recovery event from a Qwen3\-VL 8B trajectory\. The agent repeatedly pressedEnterwhile reading the Miniconda license text; the loop detector identified this as an exact repeated key action, the confirmation gate validated the no\-progress pattern, and candidate filtering selected a different continuation action\.
Loop\-recovery example from a Qwen3\-VL 8B trajectory\.The task is a multi\-application setup task involving Miniconda installation\. At step 20, the baseline action proposal would have continued pressingEnterwhile the terminal was displaying the license text\.Recent executed actions\.press\(’enter’\)→\\rightarrowpress\(’enter’\)→\\rightarrowpress\(’enter’\)→\\rightarrowpress\(’enter’\)→\\rightarrowpress\(’enter’\)Raw proposed action\.“Press the Enter key to continue reading the Miniconda license agreement\.” Parsed code:pyautogui\.press\(’enter’\)\.Rule trigger\.The detector fires withrepeated\_exact\_text\_or\_key\_action: the proposed action type iskey, the normalized action is exactlypyautogui\.press\(’enter’\), and the same action appears 5 times in the recent window\. The cyclic detector also observes a repeated key pattern with repeat count 3\.Model confirmation\.The confirmation gate marks the trajectory as stagnant:“License text scrolling, no installation progress”; it setsloop\_detected=trueandstrategy\_signal=SWITCH, with the escape direction:“Press ’n’ or ’no’ to skip license and proceed to installation\.”Candidate filtering and selected recovery action\.Four recovery candidates are sampled\. Three candidates are blocked because they are stillkeyactions without a distinguishable target under the blocked action type\. The only selectable candidate changes the action type totypeand executespyautogui\.typewrite\(’n’\)\. Since it is the sole valid non\-blocked candidate, the selector is skipped and this action is executed directly\.
Figure 8:Concrete loop\-recovery event of a Qwen3\-VL 8B trajectory\. The original action would have repeatedEnter; loop recovery instead executespyautogui\.typewrite\(’n’\)after rule detection, model confirmation, and candidate filtering\.
### D\.2Unanimous Done Gating
Fig\.[9](https://arxiv.org/html/2611.24977#A4.F9)shows a concrete DONE\-selection event from an OpenCUA\-7B trajectory\. The initial action proposed successful termination even though the email composition window was still open\. Additional samples disagreed with termination and proposed clicking the visible Send button, so the unanimous policy rejectedDONEand forced a concrete continuation action\.
DONE\-selection example from an OpenCUA\-7B trajectory\.The task is a LibreOffice Writer sharing task\. At step 10, the document is attached to an email composition window, but the message has not yet been sent\.Initial termination proposal\.The baseline proposal is parsed asDONE: it claims that the document has been shared successfully and callscomputer\.terminate\(status=’success’\)\. However, the visible state still contains the email composition window and the Send button, indicating that a final send action is still required\.Candidate disagreement\.The DONE gating module samples four additional candidates from the same context\. The resulting candidate set contains five candidates in total: the originalDONE, one additionalDONE, and three valid non\-DONEcontinuations\. Because valid continuation actions exist, the unanimous policy rejects termination and excludes all DONE candidates before selection\.Selector decision\.The remaining candidates all attempt the missing final operation: clicking the Send button in the email composition toolbar\. The selector chooses candidate 1 with the reason that prior Send\-click attempts likely missed the correct coordinates and that clicking the visible Send button is the final required step\. The selected continuation executespyautogui\.click\(152, 144\), redirecting the original DONE decision to a concrete non\-DONE action\.
Figure 9:Concrete unanimous DONE\-gating event of an OpenCUA\-7B trajectory\. The initial proposal terminates with success, but sampled candidates reveal a valid non\-DONE continuation, so DONE candidates are excluded and the agent clicks the Send button instead\.相似文章
保障计算机使用代理的安全:面向部署落地可靠性的统一架构-生命周期框架
这篇学术论文提出了一种统一的架构-生命周期框架,旨在保障计算机使用代理(CUA)在从基准测试向真实软件环境过渡过程中的安全性。文章分析了感知层、决策层和执行层以及创建、部署、运行和维护等各个阶段中面临的可靠性挑战。
@dair_ai: 关于计算机使用智能体的杰出论文。(收藏)计算机使用智能体通过屏幕操控真实软件,……
PreAct 将成功的智能体运行编译成小型状态机程序,在重复任务上实现 8.5-13 倍更快的重放,无需逐步骤的语言模型调用,并通过运行时屏幕检查确保正确性。
CASEL:面向可靠长程工具使用智能体的后果感知选择性执行
论文提出 CASEL(后果感知选择性执行层),一种模型无关的运行时分配策略,通过综合校准失败概率、后果严重性、可逆性、歧义性和证据新鲜度来为工具调用选择执行、验证、沙箱、澄清、审批或放弃等模式,仿真结果显示其将有害副作用轨迹比例从 14.75% 降至 7.02%。
我构建了一个本地控制系统,用于处理代理故障、修复、评估和门控,以使类似自动研究的自我改进循环在实际代理代码库中生效
构建了一个本地控制系统来管理代理改进循环,捕获跟踪,发现重复故障,使用Codex/Claude Code起草修复方案,并仅在通过检查和评估后应用更改。
论计算机使用智能体的可靠性
一篇预印本论文,分析为何计算机使用智能体首次成功却在重复执行时失败,将不可靠性归因于执行随机性、任务模糊性和行为变异性,并倡导重复评估与稳定策略。