AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions

# Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions
Source: [https://arxiv.org/html/2605.21082](https://arxiv.org/html/2605.21082)
###### Abstract

Large Language Model \(LLM\) based agents have demonstrated proficiency in multi\-step interactions with graphical user interfaces \(GUIs\)\. While most research focuses on improving single\-task performance, practical scenarios often involve repetitive GUI tasks for which invoking LLM reasoning repeatedly, i\.e\., the ReAct paradigm, is inefficient\. Prior to LLMs, traditional Robotic Process Automation \(RPA\) offers runtime efficiency but demands significant manual effort to develop and maintain\. To bridge this gap, we proposeAutoRPA, a framework that automatically distills the decision logic of ReAct\-style agents into robust RPA functions\. AutoRPA introduces two core innovations: \(1\) Atranslator\-builder pipelinewhere a translator agent converts hard\-coded ReAct actions into soft\-coded procedures, and a builder agent synthesizes robust RPA functions via retrieval\-augmented generation over multiple trajectories; \(2\) Ahybrid repair strategyduring code verification, combining RPA execution with ReAct\-based fallback for iterative refinement\. Experiments across multiple GUI environments demonstrate that RPA functions generated by AutoRPA successfully solve similar tasks while reducing token usage by 82%~96%, significantly improving runtime efficiency and reusability\.

Machine Learning, ICML

## 1Introduction

Graphical User Interfaces \(GUIs\) are the primary medium for human interaction with computers and mobile devices\. Recently, Large Language Model \(LLM\) agents\(Hong et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib12); Xi et al\.,[2025](https://arxiv.org/html/2605.21082#bib.bib38)\)have demonstrated remarkable versatility in handling various GUI tasks\. Most current research focuses on improving the performance of LLM on novel, individual tasks, such as by designing more sophisticated workflows\(Wang et al\.,[2024c](https://arxiv.org/html/2605.21082#bib.bib35),[b](https://arxiv.org/html/2605.21082#bib.bib34)\)or by introducing end\-to\-end reinforcement learning to enhance reasoning capabilities\(Wang et al\.,[2025a](https://arxiv.org/html/2605.21082#bib.bib33)\)\. However, in practical deployment scenarios, a substantial portion of GUI tasks arerepetitive in nature: the same user may perform identical activity daily \(e\.g\., filing reports\), or different users may share common needs \(e\.g\., booking flights\)\. For such recurring task types, rather than invoking expensive LLM reasoning for each instance, a more desirable solution is to generatereusable, low\-cost automation functionsthat can be efficiently executed across similar tasks\.

Prior to LLMs, traditional Robotic Process Automation \(RPA\)\(Ivančić et al\.,[2019](https://arxiv.org/html/2605.21082#bib.bib14)\)addressed repetitive GUI tasks by executing manually crafted scripts that simulate user actions such as clicking and typing\. While efficient at runtime, RPA systems require significant human effort to develop and maintain, and are brittle to GUI layout changes\(Hellmann & Maurer,[2011](https://arxiv.org/html/2605.21082#bib.bib11); Qian et al\.,[2020](https://arxiv.org/html/2605.21082#bib.bib25)\)\. Therefore,the aim of this paperis to investigate how to automatically synthesize token\-efficient RPA functions for a specified task type with the assistance of LLMs\. Crucially, the generated RPA code should handle variations in both the environment and task instructions\.

![Refer to caption](https://arxiv.org/html/2605.21082v1/x1.png)Figure 1:Comparison of GUI automation paradigms\. \(a\) ReAct\-style LLM agents achieve high flexibility but incur substantial per\-instance costs, unsuitable for repetitive tasks\. \(b\) Traditional RPA offers efficiency for repetitive tasks but requires manual scripting\. \(c\) AutoRPA automatically synthesizes robust, low\-cost RPA functions for arbitrary task types from LLM agent interactions\.One direct solution is to have the LLM generate executable code directly based on the task description\(Kim et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib17)\)\. This method can be grounded in the Plan\-and\-Execute paradigm\(Kim et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib17); Sun et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib30)\), where the LLM agent generates the plan or code and then subprocesses execute it\. However, LLMs often struggle to produce robust, long\-horizon GUI code without curated expert knowledge of the environment\(Valmeekam et al\.,[2022](https://arxiv.org/html/2605.21082#bib.bib31)\)\. An alternative is skill learning\(Chen et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib4); Qian et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib24); Zhao et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib44)\), which stores the success trajectory of the LLM agent as exemplars for subsequent similar tasks\. However, these methods often struggled with generalizing skills to cope with different scenarios\(Chen et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib4)\), and relied heavily on the LLM’s intrinsic abilities to handle new tasks\.

To this end, we proposeAutoRPA, a general framework that utilizes LLM agents to generate high\-quality RPA code for GUIs\. Our framework “distills” the decision logic of ReAct agents into executable code: AutoRPA first utilizes a ReAct\-style agent to explore and collect successful trajectories\. To facilitate RPA code synthesis, we introduce a translator agent that transforms each hard\-coded ReAct step into a soft\-coded action, enabling dynamic execution in varying environments\. A builder agent then synthesizes robust RPA functions from these soft\-coded trajectories, incorporating necessary logic and conditionals\.

Moreover, the completion process of different specific instructions may vary in different situations\. Therefore, to derive a robust RPA code, the builder is augmented with a retrieval mechanism that indexes a tree\-structured trajectory database, enabling access to multiple past trajectories\. During code verification, rather than simply refining and retrying, we employ a hybrid repair strategy: if the execution of synthesized code fails, the ReAct agent resumes execution from the breakpoint, producing a corrective demonstration that the builder can incorporate for refinement\.

Our main contributions are as follows:

- •We introduceAutoRPAto address a practical problem: generating robust and efficient RPA functions for task types, targeting repetitive GUI automation scenarios\.
- •We propose atranslator\-builder pipelinethat transforms stepwise ReAct actions into soft\-coded actions, and synthesizes robust RPA functions via a tree\-structured trajectory retrieval\.
- •We develop ahybrid repair strategythat combines direct code verification with ReAct\-based fallback, enabling iterative improvement of synthesized code\.
- •Experiments on three GUI benchmarks demonstrate that AutoRPA can perform token\- and time\-efficient GUI automation\. Especially, AutoRPA reduces token usage by up to 96% while maintaining or exceeding the success rate of LLM\-based GUI agents\.

## 2Related Works

### 2\.1GUI Automation via Traditional Methods

Robotic Process Automation \(RPA\)\(Ivančić et al\.,[2019](https://arxiv.org/html/2605.21082#bib.bib14)\)has long been used to enhance productivity by automating repetitive tasks within Graphical User Interfaces \(GUIs\)\. Traditional RPA systems automate routine tasks by mimicking human actions, primarily relying on manually crafted scripts or predefined rules\(Hellmann & Maurer,[2011](https://arxiv.org/html/2605.21082#bib.bib11); Qian et al\.,[2020](https://arxiv.org/html/2605.21082#bib.bib25)\)\. However, developing RPA demands significant domain expertise and requires frequent manual updates to accommodate changes in GUI layouts and task requirements, thereby limiting its scalability and flexibility in diverse and dynamic scenarios\(Zhang et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib42)\)\. To overcome the limitations of traditional RPA, subsequent research has explored learning\-based approaches\(Humphreys et al\.,[2022](https://arxiv.org/html/2605.21082#bib.bib13); Liu et al\.,[2018](https://arxiv.org/html/2605.21082#bib.bib18)\)\. These methods train deep learning models to predict and execute actions based on GUI observations\. While demonstrating greater potential, these learning\-based techniques often necessitate substantial amounts of GUI interaction data for training and still face challenges regarding stability and generalization when deployed in real\-world applications\.

### 2\.2LLM Agents for GUI Automation

Within the field of GUI automation, LLM agents leverage the emergent reasoning capabilities of large language models\(OpenAI,[2023](https://arxiv.org/html/2605.21082#bib.bib20); Ouyang et al\.,[2022](https://arxiv.org/html/2605.21082#bib.bib22); Wei et al\.,[2022](https://arxiv.org/html/2605.21082#bib.bib37)\)and have been developed for applications across web, mobile, and desktop platforms\. GUI agents such as SeeAct\(Zheng et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib45)\)and WebVoyager\(He et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib10)\)leverage the multimodal capabilities of GPT\-4V\(OpenAI,[2023](https://arxiv.org/html/2605.21082#bib.bib20)\), which interact with dynamic web pages by processing both screenshots and HTML inputs\. AppAgent\(Zhang et al\.,[2025](https://arxiv.org/html/2605.21082#bib.bib43)\)also utilizes GPT\-4V to handle screenshots and XML data, allowing for the completion of various tasks within mobile applications\.

Beyond basic interaction, advanced studies aim to improve LLM agents through experience\-based optimization\. Reflexion\(Shinn et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib28)\)enables the LLM agent to reflect on failed trajectories and revise its behavior in future attempts\. ICE\(Qian et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib24)\)and ExpeL\(Zhao et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib44)\)build a skill library from successful execution traces to guide subsequent similar tasks\. AutoManual\(Chen et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib4)\)and Mobile\-Agent\-E\(Wang et al\.,[2025b](https://arxiv.org/html/2605.21082#bib.bib36)\)further allow the LLM to induce environmental rules from interactions\. Compared to our methods, these approaches focus on the self\-evolution of LLM agents, and the decision\-making process still relies on the LLM during testing\.

### 2\.3LLMs for Code Generation

LLMs have demonstrated impressive performance in natural\-language\-to\-code \(NL2Code\) tasks\(Jiang et al\.,[2026](https://arxiv.org/html/2605.21082#bib.bib15); Du et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib6)\), rivaling expert human programmers on benchmarks like HumanEval\(Du et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib6)\)and MBPP\(Austin et al\.,[2021](https://arxiv.org/html/2605.21082#bib.bib2)\)\. However, in novel or intricate environments like GUIs or games, generating the complete planning code upfront poses significant challenges for LLMs\(Valmeekam et al\.,[2022](https://arxiv.org/html/2605.21082#bib.bib31); Sun et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib30)\)\. As a result, for these agentic environments, enabling LLMs to generate reliable code often requires thorough domain\-specific knowledge and examples\(Wang et al\.,[2024a](https://arxiv.org/html/2605.21082#bib.bib32)\)\. In contrast, our framework utilizes the iterative, step\-by\-step paradigm of ReAct agents\(Yao et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib40)\)to explore and collect successful trajectories and then distills these into verified RPA code via translation and synthesis, without requiring curated knowledge or examples\.

## 3Methods

![Refer to caption](https://arxiv.org/html/2605.21082v1/x2.png)Figure 2:AutoRPA Overview:For a task in the target task type, AutoRPA explores and repairs bugs using the ReAct agent, while a translator agent converts the resulting actions into soft\-coded actions\. A builder then generates the RPA function based on the simplified trajectory from the trajectory bank\. The newly generated code will be verified on the seen tasks\. If it fails, it will be analyzed and repaired; if successful, the new task will be examined\.Overview\.We propose the AutoRPA framework to build RPA code tailored to user\-specified GUI task types\. We first formulate the problem in Section[3\.1](https://arxiv.org/html/2605.21082#S3.SS1)\. AutoRPA comprises three main phases, as illustrated in Fig\.[2](https://arxiv.org/html/2605.21082#S3.F2)\.Exploration phase:A ReAct\-style agent iteratively interacts with the GUI environment, while a translator agent converts its hard\-coded actions into soft\-coded actions, detailed in Section[3\.2](https://arxiv.org/html/2605.21082#S3.SS2)\.RPA Generation phase:Based on the translated trajectory, a builder agent synthesizes the RPA code, while actively retrieving relevant interaction information from a trajectory database, detailed in Section[3\.3](https://arxiv.org/html/2605.21082#S3.SS3)\.RPA Verification phase:The newly generated code will be validated on the seen tasks\. For the failed execution, an analyzer agent will analyze the breakpoint and use the ReAct agent to continue exploring for repairs, followed by code refinement, detailed in Section[3\.4](https://arxiv.org/html/2605.21082#S3.SS4)\. Finally, we evaluate the obtained RPA code at the testing stage\.

### 3\.1Problem Formulation

Formally, we model the GUI environment as a Partially Observable Markov Decision Process \(POMDP\):\(𝒮,𝒜,𝒯,𝒢,𝒪\)\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\mathcal\{G\},\\mathcal\{O\}\)\. The task set𝒢\\mathcal\{G\}comprises totallyKKtask types:𝒢=⋃k=1K𝒢k\\mathcal\{G\}=\\bigcup\_\{k=1\}^\{K\}\\mathcal\{G\}^\{k\}\. At the start of each episode, the environment samples a taskg∈𝒢kg\\in\\mathcal\{G\}^\{k\}described in natural language from one of the task types, and initializes a scenes0∈𝒮s\_\{0\}\\in\\mathcal\{S\}\. The environment will provide an initial observationo0∈𝒪o\_\{0\}\\in\\mathcal\{O\}, which is typically a screenshot of the current interface and textual information derived from the Document Object Model \(DOM\) or accessibility tree\. The agent can interact with the GUI environment through the permissible action set𝒜\\mathcal\{A\}\(e\.g\., click, type\)\. After executing an actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}at time steptt, the environment transitions to a new statest\+1s\_\{t\+1\}according to the dynamicsT​\(st\+1\|st,at\)T\(s\_\{t\+1\}\|s\_\{t\},a\_\{t\}\)and subsequently emits a new observationot\+1∼O​\(ot\+1\|st\+1\)o\_\{t\+1\}\\sim O\(o\_\{t\+1\}\|s\_\{t\+1\}\)\. At the end of the episode, a binary rewardr​\(τ\)∈\{0,1\}r\(\\tau\)\\in\\\{0,1\\\}is assigned based on the final trajectoryτ=\(g,o0,a0,o1,…,aT,oT\+1\)\\tau=\(g,o\_\{0\},a\_\{0\},o\_\{1\},\.\.\.,a\_\{T\},o\_\{T\+1\}\), indicating whether the taskggwas completed or not\.

Our objective is to generate an RPA functionFkF\_\{k\}for a user\-specified task type𝒢k\\mathcal\{G\}^\{k\}\. For any task instanceg∈𝒢kg\\in\\mathcal\{G\}^\{k\}, this functionFkF\_\{k\}should produce an action trajectoryτ=Fk​\(gk\)\\tau=F\_\{k\}\(g^\{k\}\)that successfully completes the task, i\.e\.,r​\(τ\)=1r\(\\tau\)=1\. Given that the functionFkF\_\{k\}might internally invoke LLMs to handle complex scenarios, we aim to construct and optimizeFkF\_\{k\}to minimize the token consumption while guaranteeing task success\. This objective is formulated as:

minFk⁡𝔼g∼𝒢k,s0∼𝒮​\[1−r​\(Fk​\(g\)\)\+λ⋅cost​\(Fk​\(g\)\)\],\\begin\{split\}\\min\_\{F\_\{k\}\}\\mathbb\{E\}\_\{g\\sim\\mathcal\{G\}^\{k\},s\_\{0\}\\sim\\mathcal\{S\}\}\[1\-r\(F\_\{k\}\(g\)\)\+\\lambda\\cdot\\text\{cost\}\(F\_\{k\}\(g\)\)\],\\end\{split\}\(1\)wherecost​\(Fk​\(g\)\)\\text\{cost\}\(F\_\{k\}\(g\)\)represents the token cost incurred during the process ofFkF\_\{k\}for taskgg;λ\\lambdais the tradeoff between reward and cost\. The prior approaches following the ReAct paradigm\(Yao et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib40)\)typically perform step\-by\-step inference withinFk​\(g\)F\_\{k\}\(g\), leading to substantial token costs\. In contrast, existing methods that attempt to directly generate the entire code forFk​\(g\)F\_\{k\}\(g\)upfront\(Sun et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib30)\)often struggle to consistently guarantee the success of the task\. In our AutoRPA, we sampleNNtasksgn∈𝒢k,1≤n≤Ng\_\{n\}\\in\\mathcal\{G\}^\{k\},1\\leq n\\leq Nto build a code styleFkF\_\{k\}and address both limitations by combining ReAct exploration with iterative code synthesis and verification\.

### 3\.2Exploration Phase

Starting with the first building taskg1g\_\{1\}, we generate its ReAct trajectory, and then save it into a trajectory bank\. This section details the process of generating trajectories following the ReAct paradigm and subsequently translating hard\-coded ReAct actions into reusable soft\-coded equivalents\.

ReAct Trajectory\.Our ReAct agent follows the implementations in prior works\(Putta et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib23); Rawles et al\.,[2025](https://arxiv.org/html/2605.21082#bib.bib26)\)\. At each time steptt, the ReAct agent receives the task instructiongg, history of its outputs \(including reasoning and actions\), the execution summaries of previous actions \(ρ0,…,ρt−1\\rho\_\{0\},\.\.\.,\\rho\_\{t\-1\}\), and the current multimodal observationoto\_\{t\}, which consists of a screenshot annotated using the Set\-of\-Mark \(SoM\) technique\(Yang et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib39)\)and a list of GUI elements extracted from the DOM or accessibility tree\. It outputs:\(1\) Analysisof the current observationoto\_\{t\};\(2\) Checklistof subtasks, indicating the steps completed and planned towards the goal;\(3\) Next actionat∈𝒜a\_\{t\}\\in\\mathcal\{A\}represented in JSON format, along with the reasoning for that action\. Following M3A\(Rawles et al\.,[2025](https://arxiv.org/html/2605.21082#bib.bib26)\), a separate summarizer agent is employed to summarize the effect of each executed actionata\_\{t\}based on the multimodal observations recorded before and after execution, yieldingρt\\rho\_\{t\}\. At the end of the episode, a concluder agent generates a conclusionCCbased on the action history and the environmental rewardrr\. Therefore, a complete ReAct trajectory for the taskggcan be represented as:

τReAct​\(g\)=\(g,o0,a0,ρ0,o1,…,aT,ρT,oT\+1,C\)\.\\tau\_\{\\mathrm\{ReAct\}\}\(g\)=\(g,o\_\{0\},a\_\{0\},\\rho\_\{0\},o\_\{1\},\.\.\.,a\_\{T\},\\rho\_\{T\},o\_\{T\+1\},C\)\.\(2\)To improve the success rate of the ReAct agent during the building stage, we adopt the reflection mechanism\(Chen et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib4); Shinn et al\.,[2023](https://arxiv.org/html/2605.21082#bib.bib28)\)\. For a failed episode, the conclusionCCwill also include a reflection, and the ReAct agent retries based on the reflection \(up toNrefN\_\{\\mathrm\{ref\}\}times\)\.

Noticeably, the specific ReAct agent used here is modular and can be substituted with other capable GUI agents, such as Computer\-Using Agent \(CUA\) from Anthropic\(Anthropic,[2024](https://arxiv.org/html/2605.21082#bib.bib1)\)or OpenAI\(OpenAI,[2024](https://arxiv.org/html/2605.21082#bib.bib21)\), as long as they can select the next action based on the GUI observation\. These GUI agents typically specify actions using hard\-coded identifiers \(e\.g\.,click\(index=2\)orclick\(position=\(144\.5, 278\.0\)\)\) for versatility and performance\(Zheng et al\.,[2024](https://arxiv.org/html/2605.21082#bib.bib45)\)\. However, these actions lack robustness, as they may fail if the GUI layout changes even slightly\. To address this, we introduce a translator agent that converts hard\-coded actions into soft\-coded equivalents using element attributes\.

Translator Agent\.The translator agent processes each effective action \(i\.e\., actions causing screen changes\) to generate robust, reusable procedures for the subsequent RPA building\. The inputs to the translator agent are similar to the summarizer agent: the output of the ReAct agent at stepttand the before\-and\-after multimodal observations\. The translator then outputs:\(1\) Robustness analysis: Evaluates the environmental dependency and failure modes of actionata\_\{t\}\.\(2\) Action translation: Converts the hard\-coded actionata\_\{t\}into a soft\-coded equivalentat′a^\{\\prime\}\_\{t\}\. For example:

`Python This translation involves generating a Python code snippet that: \(1\) locates the target GUI element based on its semantic attributes \(e\.g\., ‘text content’, ‘element type’, ‘available actions’\) rather than its index or absolute position\. We provide an additional find\_element function that can locate target elements based on matching input attribute values\. Furthermore, for accuracy in positioning, when multiple elements that meet the criteria are found, the GUI grounding model \(e\.g\., GPT\-4o\) will be employed for localization based on the ‘target\_description’; \(2\) optionally includes assertion statements within the code to verify that the action achieves the expected outcome in the subsequent observation, as well as fallback operations\. This translation enables dynamic element identification while maintaining operational equivalence, significantly improving action trajectory portability across interface variations and easing the subsequent RPA building\. Finally, we get the translated ReAct trajectory: τReAct′​\(g\)=\(g,o0,a0′,ρ0,o1,…,aT′,ρT,oT\+1,C\)\.\\tau\_\{\\mathrm\{ReAct\}\}^\{\\prime\}\(g\)=\(g,o\_\{0\},a\_\{0\}^\{\\prime\},\\rho\_\{0\},o\_\{1\},\.\.\.,a\_\{T\}^\{\\prime\},\\rho\_\{T\},o\_\{T\+1\},C\)\. \(3\)`

`Figure 3: Illustration of Code Generation and Refinement: Based on the initial exploration trajectory, the generated RPA code failed to generalize across scenarios\. Through the verification and hybrid repair processes, the builder agent can improve the robustness of the RPA code\.``3\.3 RPA Generation Phase with RAG Builder Agent\. After completing a building task g1g\_\{1\} with the ReAct agent, we employ a builder agent to generate a robust RPA function FkF\_\{k\} for task type 𝒢k\\mathcal\{G\}^\{k\}\. The input for the builder agent includes: the task text g1g\_\{1\}, the variables in the task text g1g\_\{1\} \(specified by the user or determined by the builder\), and the translated ReAct trajectory of the task\. Since the complete ReAct trajectory τReAct′​\(g\)\\tau\_\{\\mathrm\{ReAct\}\}^\{\\prime\}\(g\) contains excessively long observational information, we only provide the simplified version, i\.e\., ψ\(τReAct′\(g\)\)=\(g,a0′,ρ0,\.\.,aT′,ρT,C\)\\psi\(\\tau\_\{\\mathrm\{ReAct\}\}^\{\\prime\}\(g\)\)=\(g,a\_\{0\}^\{\\prime\},\\rho\_\{0\},\.\.,a\_\{T\}^\{\\prime\},\\rho\_\{T\},C\), with the original observations removed, while the complete version τReAct′​\(g\)\\tau\_\{\\mathrm\{ReAct\}\}^\{\\prime\}\(g\) is stored in a trajectory bank 𝒟τ\\mathcal\{D\}\_\{\\tau\}\. For the refinement of previously generated code, the input of the builder agent will additionally include: \(1\) the current function code FkF\_\{k\}, \(2\) its execution and repair trajectory τhybrid′​\(Fk,g∗\)\\tau\_\{\\mathrm\{hybrid\}\}^\{\\prime\}\(F\_\{k\},g\_\{\*\}\) \(described in Section 3\.4\), \(3\) conclusions CC from the trajectories of prior explorations or repairs\. Considering that the code may have been refined multiple times, we only provide the simplified trajectory of the last execution for brevity\. Builder with RAG\. However, we found that when exact observations were not provided to the builder agent, it occasionally generated erroneous assumptions about the actual interface state or misunderstood the rationale behind the executed actions, leading to redundant or inefficient code\. To address this, we adopt the Retrieval\-Augmented Generation \(RAG\) mechanism \(Gao et al\., 2023\) to allow the builder agent to proactively retrieve relevant observations from the database 𝒟τ\\mathcal\{D\}\_\{\\tau\}\. Specifically, we adapted a Tree\-organized Retrieval paradigm \(Chen et al\., 2023; Sarthi et al\., 2024\), treating each interaction step \(ot,at′,ρt,ot\+1\)\(o\_\{t\},a\_\{t\}^\{\\prime\},\\rho\_\{t\},o\_\{t\+1\}\) as a document block\. Notably, ot\+1o\_\{t\+1\} is the overlapping content in two adjacent document blocks\. Our tree\-organized database structure is as follows: 1\. The bottom layer preserves each interaction block \(ot,at′,ρt,ot\+1\)\(o\_\{t\},a\_\{t\}^\{\\prime\},\\rho\_\{t\},o\_\{t\+1\}\) of the trajectory\. 2\. The middle layer contains simplified trajectory ψ​\(τReAct′​\(g\)\)\\psi\(\\tau\_\{\\mathrm\{ReAct\}\}^\{\\prime\}\(g\)\) where action results ρt\\rho\_\{t\} serves as a summary of the corresponding interaction block\. 3\. The top layer stores the final conclusion CC, which serves as the summary of each trajectory\. Based on the tree\-organized database, we provide the builder agent with a tool function: fetch\_info\(traj, step=None\)\. When invoked with a trajectory identifier and \(optional\) time step tt, this function returns details from a lower layer: simplified trajectory ψ​\(τReAct′​\(g\)\)\\psi\(\\tau\_\{\\mathrm\{ReAct\}\}^\{\\prime\}\(g\)\) or multimodal interaction block \(ot,at′,ρt,ot\+1\)\(o\_\{t\},a\_\{t\}^\{\\prime\},\\rho\_\{t\},o\_\{t\+1\}\)\. The output process of the builder agent begins by describing its own reasoning process\. If it determines that more information is needed for coding, it can iteratively call the fetch\_info tool until no further context is required\. The builder agent then generates the RPA function FkF\_\{k\}, which includes the function name and its description, input parameter description, function body, and function usage example, as shown in Fig\. 3\. The generated code prunes redundant steps from the translated ReAct trajectory and incorporates logical statements \(conditions and loops\) to enhance robustness against task variations and environmental changes\. Additionally, considering that certain page information can only be extracted using MLLM, the builder agent can, following Adaplanner \(Sun et al\., 2023\), include a special action function env\_op\.ask\_mllm\(question, respond\_format\) in the code\. 3\.4 Verification Phase with Hybrid Repair RPA Execution\. During validation and testing, it will be checked whether there is an RPA function for the current task\. If available, the LLM needs to fill in the appropriate function parameters based on the task instruction and the RPA function, and then execute\. Hybrid Repair\. After the builder agent generates a new RPA function FkF\_\{k\}, we validate it through a rigorous process\. We execute FkF\_\{k\} on the seen tasks \{g1,…,gn\}\\\{g\_\{1\},\.\.\.,g\_\{n\}\\\} until encountering the first failure task g∗g\_\{\*\}\. Rather than directly requesting the builder to debug, we introduce an analyzer agent to analyze the breakpoint, as shown in Fig\. 3\. This agent needs to analyze the action trajectory that the code has executed, i\.e\., Fk​\(g∗\)F\_\{k\}\(g\_\{\*\}\), and the observation ot⁣∗o\_\{t\*\} at the breakpoint, subsequently producing the failure cause, completed subtasks, the feasible continuation plans, and whether the task can be completed at the breakpoint or needs to be restarted\. Based on the judgment of the analyzer, the ReAct agent then proceeds to complete the task g∗g\_\{\*\} \(as described in Section 3\.2\), either resuming from the breakpoint or restarting from the initial state s0s\_\{0\} \(and t∗t\* will be set to 0\)\. In the end, the hybrid trajectory τhybrid′\\tau\_\{\\mathrm\{hybrid\}\}^\{\\prime\} that combines the trajectory of code execution and the ReAct trajectory will be obtained: τhybrid′​\(Fk,g∗\)=Fk​\(g∗\)⊕\(A,ot⁣∗,at⁣∗′,ρt⁣∗,…,oT\+1,C\),\\displaystyle\\tau\_\{\\mathrm\{hybrid\}\}^\{\\prime\}\(F\_\{k\},g\_\{\*\}\)=F\_\{k\}\(g\_\{\*\}\)\\oplus\(A,o\_\{t\*\},a\_\{t\*\}^\{\\prime\},\\rho\_\{t\*\},\.\.\.,o\_\{T\+1\},C\), ψ​\(τhybrid′​\(Fk,g∗\)\)=ψ​\(Fk​\(g∗\)\)⊕\(A,at⁣∗′,ρt⁣∗,…,C\),\\displaystyle\\psi\(\\tau\_\{\\mathrm\{hybrid\}\}^\{\\prime\}\(F\_\{k\},g\_\{\*\}\)\)=\\psi\(F\_\{k\}\(g\_\{\*\}\)\)\\oplus\(A,a\_\{t\*\}^\{\\prime\},\\rho\_\{t\*\},\.\.\.,C\), where ⊕\\oplus is the concatenation operation and AA is the analyzer output\. We then provide the simplified hybrid trajectory ψ​\(τhybrid′\)\\psi\(\\tau\_\{\\mathrm\{hybrid\}\}^\{\\prime\}\) to the builder agent for refinement as described in Section 3\.3\. If the current RPA code has passed all the seen tasks, the next task gn\+1g\_\{n\+1\} will be taken from the building tasks for verification\. For each task, we allow the builder agent to modify the RPA code MM times to pass the validations on all seen tasks\. However, if the code still cannot pass all validations after MM modifications, we consider it difficult to automate 𝒢k\\mathcal\{G\}^\{k\} with RPA code\. Therefore, we adopt the ReAct agent to complete these tasks during testing\. For comparison, we also evaluate a variant of AutoRPA that disables ReAct during testing, referred to as AutoRPA \(code only\)\. Table 1: Testing success rate, average execution time, and token consumption of LLM agent methods on AndroidWorld\. 4 Experiments Benchmarks\. To facilitate experiments on repetitive tasks, we need that the task type in the benchmark can be instantiated into multiple specific tasks\. Therefore, we conduct the experiments on these three GUI environments: \(1\) AndroidWorld \(Rawles et al\., 2025\) is a realistic Android environment featuring 116116 task types across 2020 real\-world Android applications\. This characteristic significantly supports our code generalization testing\. AndroidWorld provides the screenshot and the Android accessibility tree for observations\. The action space mimics human interactions, including operations such as tapping, long\-pressing, and swiping\. \(2\) WebArena \(Zhou et al\., 2024\): A realistic web automation benchmark featuring self\-hosted websites that simulate real\-world applications\. Our experiments focus on the Reddit domain containing 1919 task types\. \(3\) MiniWoB\+\+ \(Liu et al\., 2018\) is a simulated web environment where agents perform various tasks on websites using keyboard and mouse operations\. Following prior research \(Chen et al\., 2024; Kim et al\., 2023; Sun et al\., 2023\), we selected a subset of MiniWoB\+\+ tasks comprising 99 task types with environmental feedback and 4444 types without feedback\. We designate the 99 types with feedback as ‘hard’ tasks and the remaining 4444 as ‘simple’ tasks\. Compared Methods\. On AndroidWorld and MiniWoB\+\+, we compare AutoRPA with these LLM\-based GUI agents: ReAct† \(Yao et al\., 2023\) denotes the ReAct agent as implemented in Section 3\.2\. SeeAct \(Zheng et al\., 2024\), M3A \(Rawles et al\., 2025\) and SteP \(Sodhi et al\., 2024\) also follow the ReAct paradigm but with different implementations\. RCI \(Kim et al\., 2023\) is ground in Plan\-and\-Execute paradigm\. AdaPlanner \(Sun et al\., 2023\) belongs to the skill learning paradigm of Plan\-and\-Execute agents\. AutoGuide \(Fu et al\., 2024\) and AutoManual \(Chen et al\., 2024\) belong to the skill learning paradigm of ReAct agents and additionally optimize a set of environmental rules\. Figure 4: Testing success rate and token consumption of different methods with GPT\-5 on WebArena \(Reddit\)\. Implementation Details\. We employ GPT\-4o, GPT\-4\.1, or GPT\-5 as the LLM backbone of our agents\. For AndroidWorld, we additionally evaluate with Claude\-4\.5\-sonnet as the backbone to demonstrate that our method can benefit from better backbones \(results are provided in the Appendix\)\. During the building stage, we sample NN=33 tasks for each task type, the ReAct agent can reflect and retry up to NrefN\_\{\\mathrm\{ref\}\}=22 rounds, and the builder agent is allowed to regenerate the code MM=33 times to pass all validations\. Table 2: Testing success rate and token consumption of LLM agent methods on 99 hard and all 5353 task types in MiniWoB\+\+ with GPT\-4\.1\. “AdaPlanner \(one demo\)” restricts human examples to one\. 4\.1 Main Results Main Results on AndroidWorld\. As shown in Tab\. 1, AutoRPA outperforms the existing ReAct\-style agents while consuming less than half of the time and 18\.6% of the tokens that ReAct† requires\. One might wonder: If AutoRPA uses the trajectories of ReAct to build RPA, why is the success rate of the tests higher than that of ReAct itself? The reason is that AutoRPA uses ReAct with Reflection during the building stage, which ensures successful trajectories; furthermore, due to the randomness of LLM outputs, tasks that were previously successful may not necessarily succeed again\. In contrast, the RPAs generated by AutoRPA exhibit high stability and robustness to testing tasks\. Notably, “AutoRPA \(code only\)” achieves a comparable success rate to ReAct† during testing by executing only the generated RPA code, while consuming minimal tokens \(used by grounder in find\_element and ask\_mllm\)\. This indicates that AutoRPA can generate efficient and robust RPA scripts for most tasks that ReAct can accomplish\. Main Results on WebArena\. The real\-world web tasks in WebArena are highly challenging, and different tasks within the same task type often require distinct strategies\. This necessitates the integration of various strategy logics into a single robust function\. As shown in Fig\. 4, AutoRPA and “AutoRPA \(Code\-only\)” achieve performance on par with previous methods, while demonstrating a significant reduction in token consumption\. Table 3: Ablation study on key components of AutoRPA on AndroidWorld\. Main Results on MiniWoB\+\+\. MiniWoB\+\+ has greater randomness in environmental states and task instructions, especially in the 99 types of hard tasks\. As shown in Tab\. 2, AutoRPA exceeds the previous methods based on ReAct and skill learning, while only consuming less than 10% tokens\. Meanwhile, “AutoRPA \(code only\)” achieves a comparable success rate to ReAct†\. It is worth noting that both AdaPlanner \(Sun et al\., 2023\) and AutoManual \(Chen et al\., 2024\) require more than one carefully crafted demonstration, indicating that they rely on prior knowledge of the environment to generate reasonable code\-style plans or actions\. In contrast, thanks to the ReAct exploration phase, AutoRPA can generate robust RPA code for all task types with only one demonstration on a simple task \(“click\-button”\)\. Figure 5: The success rate curve of varying building task numbers with GPT\-4\.1 on AndroidWorld\. 4\.2 Ablation Study More experimental results on token costs during building, comparisons with more advanced agents, more analysis, and case studies are provided in the Appendix\. Key components of AutoRPA\. As shown in Tab\. 3, when building without the ReAct agent, i\.e\., the builder agent directly generates the RPA code, the success rate drops dramatically\. The results also show that the translator agent is important for code generation\. The result of “Code Repair w/o ReAct” indicates the importance of our hybrid repair strategy\. The effect of the RAG mechanism provided to the builder is evidenced by the drop in “Builder w/o RAG”\. The number of building tasks NN\. As shown in Fig\. 5, with the increase in the number of building tasks, the test success rates of both AutoRPA and “AutoRPA \(code only\)” continue to improve\. Notably, the performance of “AutoRPA \(code only\)” gradually approaches that of ReAct†, indicating that a larger set of building tasks helps AutoRPA to generate and verify more robust RPA\. 5 Conclusion We proposed AutoRPA, a robust framework for automating RPA script synthesis from GUI agent interactions\. By introducing translator and builder agents, our system can transform the hard\-coded ReAct trajectory into adaptable RPA scripts\. Additionally, our hierarchical RAG and hybrid repair strategy enable reliable code refinement\. Experimental results across three GUI environments show that AutoRPA achieves competitive success rates compared to SOTA agents, while significantly reducing token usage\. Impact Statement This paper presents work whose goal is to advance the field of machine learning\. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here\. Acknowledgement This work was supported in part by the National Natural Science Foundation of China under Grants \(No\. 62502135, 62422204\), the Zhejiang Provincial Natural Science Foundation of China under Grants \(No\. LQN25F030014, LRG26F020001\), the Key Research and Development Program of Zhejiang Province \(No\. 2025C01026\), the Scientific Research Innovation Capability Support Project for Young Faculty\. References Anthropic \(2024\) Anthropic\. Introducing computer use, a new claude 3\.5 sonnet, and claude 3\.5 haiku, 2024\. \[Online\]\. https://www\.anthropic\.com/news/3\-5\-models\-and\-computer\-use\. Austin et al\. \(2021\) Austin, J\., Odena, A\., Nye, M\., Bosma, M\., Michalewski, H\., Dohan, D\., Jiang, E\., Cai, C\., Terry, M\., Le, Q\., et al\. Program synthesis with large language models\. arXiv:2108\.07732, 2021\. Chen et al\. \(2023\) Chen, H\., Pasunuru, R\., Weston, J\., and Celikyilmaz, A\. Walking down the memory maze: Beyond context limit through interactive reading\. arXiv:2310\.05029, 2023\. Chen et al\. \(2024\) Chen, M\., Li, Y\., Yang, Y\., Yu, S\., Lin, B\., and He, X\. Automanual: Constructing instruction manuals by llm agents via interactive environmental learning\. In Neural Information Processing Systems \(NeurIPS\), 2024\. DeepMind \(2025\) DeepMind, G\. Introducing the gemini 2\.5 computer use model, Oct 07 2025\. Du et al\. \(2024\) Du, X\., Liu, M\., Wang, K\., Wang, H\., Liu, J\., Chen, Y\., Feng, J\., Sha, C\., Peng, X\., and Lou, Y\. Evaluating large language models in class\-level code generation\. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering \(ICSE\)\. Association for Computing Machinery, 2024\. Fu et al\. \(2024\) Fu, Y\., Kim, D\.\-K\., Kim, J\., Sohn, S\., Logeswaran, L\., Bae, K\., and Lee, H\. Autoguide: Automated generation and selection of context\-aware guidelines for large language model agents\. In The Thirty\-eighth Annual Conference on Neural Information Processing Systems \(NeurIPS\), 2024\. Gao et al\. \(2023\) Gao, Y\., Xiong, Y\., Gao, X\., Jia, K\., Pan, J\., Bi, Y\., Dai, Y\., Sun, J\., Wang, H\., and Wang, H\. Retrieval\-augmented generation for large language models: A survey\. arXiv:2312\.10997, 2023\. Gonzalez\-Pumariega et al\. \(2025\) Gonzalez\-Pumariega, G\., Tu, V\., Lee, C\.\-L\., Yang, J\., Li, A\., and Wang, X\. E\. The unreasonable effectiveness of scaling agents for computer use\. ArXiv, abs/2510\.02250, 2025\. He et al\. \(2024\) He, H\., Yao, W\., Ma, K\., Yu, W\., Dai, Y\., Zhang, H\., Lan, Z\., and Yu, D\. Webvoyager: Building an end\-to\-end web agent with large multimodal models\. arXiv:2401\.13919, 2024\. Hellmann & Maurer \(2011\) Hellmann, T\. D\. and Maurer, F\. Rule\-based exploratory testing of graphical user interfaces\. In Agile Conference\. IEEE, 2011\. Hong et al\. \(2024\) Hong, W\., Wang, W\., Lv, Q\., Xu, J\., Yu, W\., Ji, J\., Wang, Y\., Wang, Z\., Dong, Y\., Ding, M\., et al\. Cogagent: A visual language model for gui agents\. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\), 2024\. Humphreys et al\. \(2022\) Humphreys, P\. C\., Raposo, D\., Pohlen, T\., Thornton, G\., Chhaparia, R\., Muldal, A\., Abramson, J\., Georgiev, P\., Santoro, A\., and Lillicrap, T\. A data\-driven approach for learning to control computers\. In International Conference on Machine Learning \(ICML\), 2022\. Ivančić et al\. \(2019\) Ivančić, L\., Vugec, D\. S\., and Bosilj\-Vuksic, V\. Robotic process automation: Systematic literature review\. In International Conference on Business Process Management \(BPM\), 2019\. Jiang et al\. \(2026\) Jiang, J\., Wang, F\., Shen, J\., Kim, S\., and Kim, S\. A survey on large language models for code generation\. ACM Trans\. Softw\. Eng\. Methodol\., 2026\. Kim et al\. \(2023\) Kim, G\., Baldi, P\., and McAleer, S\. M\. Language models can solve computer tasks\. In Neural Information Processing Systems \(NeurIPS\), 2023\. Kim et al\. \(2024\) Kim, S\., Moon, S\., Tabrizi, R\., Lee, N\., Mahoney, M\. W\., Keutzer, K\., and Gholami, A\. An llm compiler for parallel function calling\. In Forty\-first International Conference on Machine Learning \(ICML\), 2024\. Liu et al\. \(2018\) Liu, E\. Z\., Guu, K\., Pasupat, P\., Shi, T\., and Liang, P\. Reinforcement learning on web interfaces using workflow\-guided exploration\. In International Conference on Learning Representations \(ICLR\), 2018\. Lu et al\. \(2024\) Lu, Y\., Yang, J\., Shen, Y\., and Awadallah, A\. Omniparser for pure vision based gui agent\. arXiv:2408\.00203, 2024\. OpenAI \(2023\) OpenAI\. Gpt\-4 technical report\. arXiv:2303\.08774, 2023\. OpenAI \(2024\) OpenAI\. Computer\-using agent: Introducing a universal interface for ai to interact with the digital world, 2024\. \[Online\]\. https://openai\.com/blog/computer\-using\-agent\. Ouyang et al\. \(2022\) Ouyang, L\., Wu, J\., Jiang, X\., Almeida, D\., Wainwright, C\. L\., Mishkin, P\., Zhang, C\., Agarwal, S\., Slama, K\., Ray, A\., Schulman, J\., Hilton, J\., Kelton, F\., Miller, L\. E\., Simens, M\., Askell, A\., Welinder, P\., Christiano, P\. F\., Leike, J\., and Lowe, R\. J\. Training language models to follow instructions with human feedback\. In Neural Information Processing Systems \(NeurIPS\), 2022\. Putta et al\. \(2024\) Putta, P\., Mills, E\., Garg, N\., Motwani, S\., Finn, C\., Garg, D\., and Rafailov, R\. Agent q: Advanced reasoning and learning for autonomous ai agents\. arXiv:2408\.07199, 2024\. Qian et al\. \(2024\) Qian, C\., Liang, S\., Qin, Y\., Ye, Y\., Cong, X\., Lin, Y\., Wu, Y\., Liu, Z\., and Sun, M\. Investigate\-consolidate\-exploit: A general strategy for inter\-task agent self\-evolution\. arXiv:2401\.13996, 2024\. Qian et al\. \(2020\) Qian, J\., Shang, Z\., Yan, S\., Wang, Y\., and Chen, L\. Roscript: a visual script driven truly non\-intrusive robotic testing system for touch screen applications\. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering \(ICSE\), 2020\. Rawles et al\. \(2025\) Rawles, C\., Clinckemaillie, S\., Chang, Y\., Waltz, J\., Lau, G\., Fair, M\., Li, A\., Bishop, W\. E\., Li, W\., Campbell\-Ajala, F\., Toyama, D\. K\., Berry, R\. J\., Tyamagundlu, D\., Lillicrap, T\. P\., and Riva, O\. Androidworld: A dynamic benchmarking environment for autonomous agents\. In The Thirteenth International Conference on Learning Representations \(ICLR\), 2025\. Sarthi et al\. \(2024\) Sarthi, P\., Abdullah, S\., Tuli, A\., Khanna, S\., Goldie, A\., and Manning, C\. D\. Raptor: Recursive abstractive processing for tree\-organized retrieval\. In The Twelfth International Conference on Learning Representations \(ICLR\), 2024\. Shinn et al\. \(2023\) Shinn, N\., Cassano, F\., Labash, B\., Gopinath, A\., Narasimhan, K\., and Yao, S\. Reflexion: language agents with verbal reinforcement learning\. In Neural Information Processing Systems \(NeurIPS\), 2023\. Sodhi et al\. \(2024\) Sodhi, P\., Branavan, S\., Artzi, Y\., and McDonald, R\. Step: Stacked llm policies for web actions\. In First Conference on Language Modeling, 2024\. Sun et al\. \(2023\) Sun, H\., Zhuang, Y\., Kong, L\., Dai, B\., and Zhang, C\. Adaplanner: Adaptive planning from feedback with language models\. In Neural Information Processing Systems \(NeurIPS\), 2023\. Valmeekam et al\. \(2022\) Valmeekam, K\., Olmo, A\., Sreedharan, S\., and Kambhampati, S\. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change\. In Neural Information Processing Systems \(NeurIPS\), 2022\. Wang et al\. \(2024a\) Wang, G\., Xie, Y\., Jiang, Y\., Mandlekar, A\., Xiao, C\., Zhu, Y\., Fan, L\., and Anandkumar, A\. Voyager: An open\-ended embodied agent with large language models\. Transactions on Machine Learning Research \(TMLR\), 2024a\. Wang et al\. \(2025a\) Wang, H\., Zou, H\., Song, H\., Feng, J\., Fang, J\., Lu, J\., Liu, L\., Luo, Q\., Liang, S\., Huang, S\., Zhong, W\., Ye, Y\., Qin, Y\., Xiong, Y\., Song, Y\., Wu, Z\., Li, B\., Dun, C\., Liu, C\., Leng, F\., rui Wang, H\., Yu, H\., Chen, H\., and et al\. Ui\-tars\-2 technical report: Advancing gui agent with multi\-turn reinforcement learning\. arXiv:2509\.02544, 2025a\. Wang et al\. \(2024b\) Wang, J\., Xu, H\., Jia, H\., Zhang, X\., Yan, M\., Shen, W\., Zhang, J\., Huang, F\., and Sang, J\. Mobile\-agent\-v2: Mobile device operation assistant with effective navigation via multi\-agent collaboration\. In Neural Information Processing Systems \(NeurIPS\), 2024b\. Wang et al\. \(2024c\) Wang, J\., Xu, H\., Ye, J\., Yan, M\., Shen, W\., Zhang, J\., Huang, F\., and Sang, J\. Mobile\-agent: Autonomous multi\-modal mobile device agent with visual perception\. In ICLR 2024 Workshop on Large Language Model Agents, 2024c\. Wang et al\. \(2025b\) Wang, Z\., Xu, H\., Wang, J\., Zhang, X\., Yan, M\., Zhang, J\., Huang, F\., and Ji, H\. Mobile\-agent\-e: Self\-evolving mobile assistant for complex tasks\. In Workshop on Scaling Environments for Agents, 2025b\. Wei et al\. \(2022\) Wei, J\., Wang, X\., Schuurmans, D\., Bosma, M\., hsin Chi, E\. H\., Xia, F\., Le, Q\., and Zhou, D\. Chain of thought prompting elicits reasoning in large language models\. In Neural Information Processing Systems \(NeurIPS\), 2022\. Xi et al\. \(2025\) Xi, Z\., Chen, W\., Guo, X\., He, W\., Ding, Y\., Hong, B\., Zhang, M\., Wang, J\., Jin, S\., Zhou, E\., et al\. The rise and potential of large language model based agents: A survey\. Science China Information Sciences \(SCIS\), 68\(2\):121101, 2025\. Yang et al\. \(2023\) Yang, J\., Zhang, H\., Li, F\., Zou, X\., Li, C\., and Gao, J\. Set\-of\-mark prompting unleashes extraordinary visual grounding in gpt\-4v\. arXiv:2310\.11441, 2023\. Yao et al\. \(2023\) Yao, S\., Zhao, J\., Yu, D\., Du, N\., Shafran, I\., Narasimhan, K\., and Cao, Y\. ReAct: Synergizing reasoning and acting in language models\. In International Conference on Learning Representations \(ICLR\), 2023\. Ye et al\. \(2025\) Ye, J\., Zhang, X\., Xu, H\., Liu, H\., Wang, J\., Zhu, Z\., Zheng, Z\., Gao, F\., Cao, J\., Lu, Z\., Liao, J\., Zheng, Q\., Huang, F\., Zhou, J\., and Yan, M\. Mobile\-agent\-v3: Fundamental agents for gui automation\. ArXiv, abs/2508\.15144, 2025\. URL https://api\.semanticscholar\.org/CorpusID:280699844\. Zhang et al\. \(2024\) Zhang, C\., He, S\., Qian, J\., Li, B\., Li, L\., Qin, S\., Kang, Y\., Ma, M\., Liu, G\., Lin, Q\., et al\. Large language model\-brained gui agents: A survey\. arXiv:2411\.18279, 2024\. Zhang et al\. \(2025\) Zhang, C\., Yang, Z\., Liu, J\., Li, Y\., Han, Y\., Chen, X\., Huang, Z\., Fu, B\., and Yu, G\. Appagent: Multimodal agents as smartphone users\. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems \(CHI ’25\), 2025\. Zhao et al\. \(2024\) Zhao, A\., Huang, D\., Xu, Q\., Lin, M\., Liu, Y\., and Huang, G\. Expel: Llm agents are experiential learners\. In AAAI Conference on Artificial Intelligence \(AAAI\), 2024\. Zheng et al\. \(2024\) Zheng, B\., Gou, B\., Kil, J\., Sun, H\., and Su, Y\. Gpt\-4v\(ision\) is a generalist web agent, if grounded\. In Proceedings of the 41st International Conference on Machine Learning\. JMLR\.org, 2024\. Zhou et al\. \(2024\) Zhou, S\., Xu, F\. F\., Zhu, H\., Zhou, X\., Lo, R\., Sridhar, A\., Cheng, X\., Ou, T\., Bisk, Y\., Fried, D\., Alon, U\., and Neubig, G\. Webarena: A realistic web environment for building autonomous agents\. In The Twelfth International Conference on Learning Representations \(ICLR\), 2024\. Appendix A Limitations Although the AutoRPA framework has made significant contributions, there are still several limitations worth further discussion\. Firstly, our approach largely relies on the capabilities of advanced MLLMs such as GPT\-4\.1 to generate ReAct trajectories and RPA code, which may limit the applicability of this framework to less advanced models\. Secondly, our method has certain requirements for the GUI environment: 1\. In addition to screenshots, the environment needs to provide the DOM or accessibility tree, so that we can dynamically locate elements based on their attributes\. However, this limitation may be alleviated by current advanced screenshot parsing technologies, such as OmniParser\-V2 \(Lu et al\., 2024\)\. 2\. During the RPA building stage, users need to provide more than one task that belongs to one task type and the final reward rr to obtain an ideal RPA\. We plan to address this in future work by having LLM agents generate rewards and autonomously explore tasks, similar to what Voyager \(Wang et al\., 2024a\) has done\. Thirdly, the ReAct agent exhibits limited capability for complex and challenging tasks during the exploration stage\. It may be combined with more sophisticated methods, such as tree search algorithms \(Putta et al\., 2024\), to enhance the exploration capability of the ReAct agent for difficult tasks\. Finally, the Builder agent still shows a significant gap compared to humans in terms of tool usage and code generation\. For example, it sometimes fails to use the fetch\_info tool when it should do so to generate code more cautiously; and at other times, it overuses the fetch\_info tool when it’s not necessary\. Additionally, it opts for unnecessarily complicated solutions in certain situations where it should use the ask\_mllm action to obtain information directly\. This will be an obstacle to generating RPAs for more complex tasks\. Appendix B More Implementation Details Following the standard test setting on AndroidWorld \(Rawles et al\., 2025\), we test on task0 for each task type\. During the testing on Miniwob\+\+, we randomly sample 55 distinct tasks \(different from NN building tasks\)\. For WebArena \(Zhou et al\., 2024\), we select task types with ≥4 instances to enable a building phase \(3 instances\) and testing phase \(1 instance\), resulting in 19 task types\. For all benchmarks, we only provide the agents with a simple demonstration to illustrate the output format\. The maximum number of steps for a ReAct agent to complete a task in AndroidWorld is 10 times the task complexity \(capped at 50\), while in MiniWoB\+\+ it is 20 steps, and in WebArena 25 steps\. For building and testing tasks of a certain task type, we can automatically generate different instances in the environment by controlling the random number seed\. For example, if there are 33 building tasks, we use the tasks generated with seed=1,2,31,2,3 for building\. In AndroidWorld, the task generated with seed=0 is used for testing, while in MiniWoB\+\+, the tasks generated with seed=0,11,12,13,140,11,12,13,14 are used for testing\. During the testing stage, when evaluating "AutoRPA", we first check whether there exists a fully verified RPA function for the given task type; if not, we then check whether the task type can be accomplished using ReAct\. When evaluating "AutoRPA \(code only\)", we check whether there exists a fully verified RPA function for the given task type; if not, we choose the RPA function that has passed the most verification tasks during the RPA building process\. For SeeAct and M3A, we use their implementation by AndroidWorld \(Rawles et al\., 2025\)\. For RCI, AdaPlanner, and AutoManual, we use their official implementations on MiniWoB\+\+\. As described in Section 3\.4, under the default settings of AutoRPA, for tasks that cannot be verified by refining RPA code, the ReAct agent will be executed during testing\. To demonstrate the effectiveness of the generated RPA, AutoRPA \(code only\) will only execute the RPA code that has been successfully validated during testing\. In MiniWoB\+\+, for RCI \(Kim et al\., 2023\), AdaPlanner \(Sun et al\., 2023\), and AutoManual \(Chen et al\., 2024\), the agents are provided with 104104, 3838, and 44 expert demonstrations, respectively, according to their official code on GitHub\. Figure 6: The task type\-level performance of ReAct†, AutoRPA \(code only\), and AutoRPA with GPT\-4\.1 in MiniWoB\+\+ Table 4: Token usage \(in thousands\) per task type of each module during the building process in AndroidWorld and MiniWoB\+\+, using GPT\-4\.1\. Appendix C More Experiments and Results C\.1 Full Experimental Results Testing Success Rate of Each Task Type\. The task success rate of each task type in MiniWoB\+\+ is presented in Fig\. 6\. AutoRPA outperforms ReAct† on almost all task types, except for the multi\-layouts category, where the code generated by AutoRPA successfully passes the verification of the building tasks but fails to generalize to all testing tasks\. Token Cost for Building RPA\. In Tab\. 4, we list the tokens consumed by AutoRPA during the building stage\. We find that the main source of consumption comes from the exploration process of ReAct, especially in high\-difficulty tasks\. It is important to note that the overhead for the building process is a one\-time cost per task type; once the RPA code is built, it can be used to reduce expenses\. Furthermore, ‘online building’ is feasible for new tasks: the RPA code is generated immediately after the task is successfully completed using ReAct\. The initial building overhead is typically amortized after the deployed RPA code executes approximately four tasks\. Comparison with More Advanced Methods\. In Tab\. 5, we compare our results against state\-of\-the\-art GUI Agents, demonstrating that our method remains competitive\. It is crucial to note that AutoRPA is ReAct\-agnostic\. As detailed in the main text, our framework does not rely on any specific ReAct agent\. In fact, many powerful agents, such as LLMs fine\-tuned for ’Computer Use’ scenarios \(DeepMind, 2025; Wang et al\., 2025a\), are compatible with the ReAct paradigm\. To further demonstrate this, we experimented by replacing the LLM backbone of both ReAct and AutoRPA with a more powerful reasoning model\. As shown in the table, utilizing the powerful Claude\-sonnet\-4\.5 boosts the success rates for both methods, while AutoRPA consistently retains its advantage in token efficiency\. Furthermore, we experimented with the ReAct Agent that operates solely on screenshots, outputting actions via screen coordinates\. The Translator in our method successfully converts these hard\-coded actions into UI element decision logic, ultimately generating high\-quality RPA code\. This further underscores that our AutoRPA framework is orthogonal to the choice of ReAct Agent\. Statistical Results of Generated RPA\. In Tab\. 6, we present the number of verified RPAs generated by AutoRPA for each benchmark\. We find that the generated RPA has a high success rate on the testing tasks, indicating its robustness to task instructions\. Moreover, generating RPA typically requires about one round of refinement\. Table 5: Testing success rate and token consumption of LLM agent methods on AndroidWorld\. Appendix D More Ablation Studies RPA with Affordable GUI Grounder The find\_element function relies on a GUI grounding model to locate elements based on descriptions when element attributes alone are insufficient to uniquely identify a GUI element\. To further reduce the costs during testing, we also tried replacing the GUI grounding model used in find\_element and ask\_mllm during testing with a more affordable MLLM model\. In Tab\. 8, we replace the default grounding model, i\.e\., GPT\-4\.1, with GPT\-4o\-mini or Qwen2\.5\-VL\-72B\-Instruct when evaluating the generated RPA\. We find that using a weaker GUI grounding model results in a decrease in success rate\. The maximum rounds of code refinements MM and reflection NrefN\_\{\\mathrm\{ref\}\}\. As shown in Tab\. 7, with more refinement turns, the builder can have more chances to generate desirable RPA to pass all verification\. With more reflection rounds during the building, the ReAct agent can produce more successful trajectories, thereby strengthening the subsequent RPA generation\. Unify ReAct and Translator Agents In AutoRPA, after obtaining a hard\-coded ReAct trajectory from the ReAct agent, a Translator agent is employed to convert each action into a soft\-coded format that aligns with the current UI elements on the page\. We refer to this setup as the Separate Mode\. We also explore an alternative configuration, where the ReAct and Translator functionalities are unified\. In this Unified Mode, the ReAct Agent directly outputs both the hard\-coded action for immediate execution and the corresponding soft\-coded action, thereby eliminating the need for a separate translator agent\. Our experiments \(see Tab\. 9\) reveal that generating soft\-coded actions directly from the planner increases the input context length, which can slightly degrade the quality of the generated actions\. Table 6: The number of generated RPA, testing Success Rate \(SR\) of generating RPA, and average refinement turn in AndroidWorld and MiniWoB\+\+ using GPT\-4\.1\. Table 7: Ablation study on the number of code refinement turns MM and reflection rounds NrefN\_\{\\mathrm\{ref\}\} during the building stage of AutoRPA\. We report the success rate \(%\) on 99 hard and all 5353 task types in MiniWoB\+\+ with GPT\-4\.1\. \* denotes the default setting\. Table 8: Performance comparison of different GUI Grounders on RPA Quantity and Success Rate \(SR\) in AndroidWorld and MiniWoB\+\+\. Task seed = 0\. Table 9: Comparison of Separate and Unified Planner\-Translator Architectures in AndroidWorld and MiniWoB\+\+\. \* denotes the default setting in AutoRPA\. Appendix E Case study of AutoRPA E\.1 Generated Code: AutoRPA vs\. Prior Methods We use the tic\-tac\-toe task type in MiniWoB\+\+ as a case study to compare the skill code produced by prior methods, i\.e\., AdaPlanner \(Sun et al\., 2023\) and AutoManual \(Chen et al\., 2024\), and the RPA code generated by AutoRPA\. The task objective is: Playing as ’X’, win a game of tic\-tac\-toe\. The opponent is a computer program, and the actions of that computer program have a certain degree of randomness\. Therefore, using the exact same steps does not always guarantee success\. Figure 7: Code generated by different agent frameworks for tic\-tac\-toe\. To facilitate a clear comparison of the core differences, we present a streamlined version of the code, generated by AutoRPA, that preserves all essential decision logic\. AdaPlanner \(Sun et al\., 2023\)\. See Fig\. 7\. The code adopts a static three\-step plan: it always clicks the top\-left cell \(ttt\-0\), the centre \(ttt\-4\), and the bottom\-right \(ttt\-8\) in sequence, aiming to form the main diagonal X–X–X\. After each action, the code immediately issues an assert to verify that the expected number of X marks is present\. This design showcases AdaPlanner’s Plan\-then\-Execute with Skill Learning paradigm: the code is extremely concise \(no loops, no board parsing\) and provided the opponent does not block the diagonal win in the minimum three moves\. The fixed skill code produced by AdaPlanner is likely obtained when a particular attempt happens to succeed\. However, the policy is brittle: it neither reacts to any opponent move that interferes with the pre\-planned line\. So the first unexpected state will trigger an assertion failure and abort the run\. Then Adaplanner will need to call LLM to fix it in a closed\-loop manner\. Consequently, while highly efficient in favorable conditions, the code lacks the adaptability of the dynamic parsers and defensive heuristics\. AutoManual \(Chen et al\., 2024\)\. See Fig\. 7\. AutoManual resolves the Path Dependency issues found in Adaplanner\. It takes into account that the generated skill code serves as a prompt for the LLM, and thus, after a successful attempt by ReAct \(with code\-style actions\), it adds “\*Pause here\*” to remind future attempts\. However, it is unable to escape the constraints of Skill Learning for ReAct paradigm, resulting in that the generated skill code cannot be executed directly and relies on the LLM as the decision maker\. AutoRPA \(ours\)\. See Fig\. 7\. Unlike the prior methods, the code generated by our AutoRPA is more complicated and can automatically cope with various scenarios\. The function repeatedly parses the DOM via regular expressions that recognise both class attributes \(mark\-X/mark\-O\) and literal text content, offering strong robustness to UI variations\. Within the loop for player X, the script follows a clear priority queue: \(1\) Win: take any move that completes three in a row\. \(2\) Block: intercept any opponent’s winning move\. \(3\) Centre: occupy the middle cell if free\. \(4\) Corner: choose an empty corner\. \(5\) Edge: fall back to a side cell\. Each action is executed through env\_op\.find\_element followed by env\_op\.click\_xpath\(\), and then waits briefly for the UI to refresh before re\-evaluating the board\. Compared with other code, this RPA offers \(1\) greater HTML robustness via multi\-pattern parsing and retries, \(2\) a complete self\-contained game loop capable of finishing an entire match, and \(3\) parameterised symbols and abstracted DOM access, making it easier to port to other MiniWoB\+\+ tasks or different player orders\. E\.2 Examples of RPA Building Process We present an example of the RPA building process using the MarkorCreateNote task from the AndroidWorld benchmark\. The task instances of the task type are generated from the template: Create a new note in Markor named \{file\_name\} with the following text: \{text\}\. Exploration Phase: As illustrated in Fig\. 8, through the ReAct agent and the translator agent, we can obtain a hard\-coded trajectory and a soft\-coded trajectory\. Figure 8: ReAct Trajectory in AndroidWorld – MarkorCreateNote RPA Generation and Refinement Phase: After completing a building task with the ReAct agent, we employ a builder agent to generate a robust RPA function for the task type\. See RPA Function Info in Code Block \- Initial RPA Code on MarkorCreateNote\. In this example, the program was interrupted during the testing of the generated RPA function \(See Fig\. 9\)\. According to the analyzer’s diagnosis, the function failed to extract the file name during execution\. Furthermore, since the file suffix was changed from the default \.md to \.txt, it also became necessary to explicitly set the file suffix\. Based on the analyzer’s recommendation, the ReAct agent resumes execution from the current screen and performs actions such as entering the file name and specifying the file suffix\. After that, the Builder will generate the refined code based on the hybrid trajectory\. See refined code in Code Block \- Refined RPA Code on MarkorCreateNote\. Figure 9: Hybrid Repair Process in AndroidWorld – MarkorCreateNote code\]code:rpa\_initialPython code : Initial RPA Code on MarkorCreateNote code\]code:rpa\_refinedPython code : Refined RPA Code on MarkorCreateNote Appendix F Prompts F\.1 Prompts for Agents Generating ReAct Trajectory Our GUI agent framework builds upon the ReAct paradigm, with customized prompt engineering to better support GUI\-based tasks\. The system consists of three specialized components: \(1\) a ReAct agent for step\-by\-step planning and next\-action generation, \(2\) a summarizer agent for reporting the execution outcomes of individual actions, and \(3\) a concluder agent for summarizing the entire task trajectory\. We list their system prompts in Lst\. 1, Lst\. 2, and Lst\. 3 respectively\. Listing 1: System Prompts for ReAct Agent \[Role\] You are an intelligent agent that operates a GUI for a user\. Your role is to break down complex requests into optimal, step\-by\-step actions and execute them while adapting to changes\. \[ADMISSIBLE ACTIONS\] \{Env Operations\} \[Output Format\] Your output must strictly follow the structure below\. Headings must match exactly, including \`\#\#\#\`, spacing, and colons\. \#\#\# Observations: Summarize key input details and immediate screen observations\. Include all obvious insights\. \#\#\# Completed Tasks: List completed tasks, each starting with a ✓\. \#\#\# Plan Justification: Briefly explain the rationale behind your plan\. \#\#\# Plan List: List tasks to achieve the goal, each starting with a ❑; if the goal is achieved, output "goal completed\." \#\#\# Next Action Justification: Explain why the next action is chosen\. \#\#\# Action: Output the next action between '\`\`\`python' and '\`\`\`' \(one action, one line\)\. Use only \[ADMISSIBLE ACTIONS\]\. \[Your Workflow\] 1\. Analyze Input: Carefully examine all input\. Extract all goal\-relevant info — don’t miss anything useful\. Directly infer and record any obvious conclusions\. 2\. Evaluate Progress: Check if the goal is achieved; if so, stop\. Otherwise, update completed tasks\. 3\. Devise Plan: Break the goal into efficient, non\-redundant steps\. Identify and obtain any missing information\. 4\. Execute & Adjust: Analyze the UI info to decide actions\. Adjust the plan if elements are missing or inaccessible\. 5\. Error Handling: Retry once on failure; if it still fails, choose an alternative\. 6\. Generate Next Action: Choose the next logical action that advances the goal\. \[Guidelines\] Follow these guidelines: \- After you output the action, the action will be executed\. The results of each action and the new observations will be printed to you at next step\. \- Maintain a holistic view by identifying the specific steps required to complete the task using the current input\. \- Fully leverage provided input to reduce unnecessary follow\-up actions\. \- Interact only with verified UI elements\. \- Adapt dynamically – adjust to screen changes and execution failures\. Operation Guidelines: \- Choose the simplest method\. \- Ensure the index for click, long\_press, and input\_text is visible in both the screenshot and UI list\. \- Swipe to reveal hidden UI elements\. \- Confirm that a UI element supports the intended action before interacting\. \- Every element is clickable by default\. \- Use standard text selection methods \(long\-press and selection bar\) when needed\. Listing 2: Prompt for Summarizer Agent \[Role\] You are an ActionSummarizer agent\. Evaluate the success of actions executed on an Android device by comparing 'before' and 'after' screenshots\. \[Output Format\] Your output should consist of the following things: \#\#\# Screen Changes: Summarize the primary differences between the before and after screens \(max 30 words\)\. If none, respond with "Nothing Happens\." \#\#\# Execution Summary: In one concise line \(max 50 words\), state whether the page changed as expected, including the intent, outcome, and key insights\. \[Your Workflow\] 1\. Compare Screenshots: Focus on differences related to the highlighted element in the 'before' screenshot and the executed code\. 2\. Verify Purpose: Check if the executed code aligns with its intended purpose \(reason for code\) and if the highlighted element meets expectations\. 3\. Compare Code: Confirm that the expected code matches the executed code; if not, identify discrepancies\. 4\. Assess Outcome: Determine if the executed code met the intended goal\. 5\. Highlight Findings: Note key insights for future actions\. \[Guidelines\] \- If actions like \`answer\` or \`wait\` do not change the screen, assume success\. \- If no change occurs, clearly state the failure and possible reasons\. \- Rely primarily on screenshot analysis\. \- Focus on actionable insights; avoid redundant details\. \- For file\-related operations, make sure to operate only on the exact target file; do not interact with similar files\. You must locate and use the precise file required\. \- When naming files, ensure proper file extensions\. Listing 3: Prompt for Concluder Agent \[Role\] You are a concluder agent tasked with summarizing a failed screen operation trajectory and performing a critical reflection to prevent future errors\. \[Formatting Guidelines\] Your output must strictly follow the structure below\. Headings must match exactly, including \`\#\#\#\`, spacing, and colons\. Required sections \(in order\): \#\#\# Episode Conclusion: \#\#\# Reflection: \[Output Format\] \#\#\# Episode Conclusion: Provide a summary of the trajectory, clearly indicating where the failure occurred\. \#\#\# Reflection: Identify and describe the key actions that failed, including specific reasoning for each failure \(e\.g\., incorrect actions, UI misunderstandings, planner's incorrect judgment\)\. State the primary cause\(s\) of failure\. Provide clear, actionable steps to address these issues and achieve the task\. \[Your Workflow\] 1\. Analyze Trajectory: \- Identify and pinpoint exactly which step in the trajectory led to failure\. \- Clearly explain why this specific step failed \(e\.g\., incorrect actions, misinterpretation of UI, planner’s inaccurate decision\-making\)\. \- Highlight key decision points and provide specific reasoning behind each critical action\. 2\. Root Cause Analysis \(RCA\): \- Clearly state the underlying cause\(s\) of the failure\. \- Highlight any misjudgments or missed opportunities for correction\. 3\. Formulate Corrective Guidelines: \- Propose clear, actionable guidelines or improvements for avoiding similar failures in future attempts\. 4\. Summary Generation: \- Focus on key actions that directly contributed to the goal, showing how each step led to the next\. \- Highlight the reasoning behind critical decisions and their role in the task's success\. \- Write a single coherent paragraph in natural language, emphasizing the causal relationships between actions\. \[GUIDELINES\] \- Avoid generic or unrelated descriptions; strictly avoid irrelevant or overly broad content\. F\.2 Prompts for Translator Agent Listing 4: Prompt for Translator Agent \[System\] Generate a Soft\-coded Action by dynamically replacing the hardcoded index in the given action with an element\-matching strategy\. Before that, determine from the observation and action justification whether the original action can be decided by mechanical element matching; if not, consider using \`ask\_mllm\(\)\` for combined visual and language reasoning\. MLLMs are inherently random, so \`ask\_mllm\` may not always return expected results\. To ensure stability, use strict prompt constraints or minimize their usage\. If using \`find\_element\(\)\`, ensure that: 1\. The revised logic maintains the same intended behavior as the original hardcoded action; 2\. If indexing is not required, do not use the find\_element method\. \[Index Replacement\] You need to use this function to replace the hardcoded \`index\` value with the index variable generated by the \`find\_element\(\)\`\. \#\#\# Get Element Index env\_op\.find\_element\(\*\*kwargs\) \-\> int \# Use this function to find an element in the UI list using filtering criteria and return its index\. If no matching element is found, the index will be \-1\. \- Rules for generating \`kwargs\`: \- Use find\_element only when an index is required \(e\.g\., not for env\_op\.open\_app\)\. \- Extract all attributes necessary to uniquely identify the element in the current page\. Keys must be correct, and values must be fully preserved without modification\. For additional\_actions, include only the necessary action\(s\), not the full list\. \- From text, hint\_text, content\_description, and tooltip, extract only one—whichever is most relevant\. This does not limit the use of other attributes\. \- Always include target\_description: briefly describe the element’s role, appearance, position on screen, and any dynamic/contextual content that helps make it unique\. \- \`kwargs\` Examples: \- kwargs = \{"content\_description": "Save Changes", "target\_description": "rectangle button to save"\} \[ADMISSIBLE ACTIONS\] \{Env Operations\} \# You may additionally consider the following smart action patterns if relevant: \#\#\# Ask MLLM to answer one question Don’t rely too much on ask\_mllm\. env\_op\.ask\_mllm\(question: str, output\_format: str\) \-\> str \# Will return the answer in plain text\. \[Output Format\] Your output must include two sections: \#\#\# Thought: Provide a brief explanation \(under 30 words\) for how you constructed the dynamic UI element matching parameters \(i\.e\. the \`kwargs\`\)\. \#\#\# Soft\-coded Action: \- If not using \`find\_element\(\)\`: write new code in proper way\. \- If the original action requires a UI element index: Replace the hardcoded index with a dynamic lookup using \`env\_op\.find\_element\`\. You must output \*exactly\* these three lines of code: \`\`\`python kwargs = \{\.\.\.\} \# A dictionary describing the target UI element index = env\_op\.find\_element\(\*\*kwargs\) \# Use env\_op to locate the element dynamically env\_op\.xxx\(\.\.\.\) \# Replace xxx with the correct action using the index \`\`\` \- If the original action does NOT require an index: Simply output the soft\-coded action without calling env\_op\.find\_element\. F\.3 Prompts for Builder Agent Listing 5: System Prompts for Builder Agent \[Role\] You are an expert AI coding assistant specializing in generating Skill Code for RPA \(Robotic Process Automation\)\. Your goal is to analyze multiple execution trajectories and initial Skill Codes to produce robust, flexible, and reusable code that reliably completes similar tasks of the provided Task Template\. \[ADMISSIBLE ACTIONS\] \{Env Operations\} \#\#\# Get Element Index env\_op\.find\_element\(\*\*kwargs\) \-\> int \# Use this function to find an element in the UI list using filtering criteria and return its index\. If no matching element is found, the index will be \-1\. \- Rules for generating \`kwargs\`: \- Use find\_element only when an index is required \(e\.g\., not for env\_op\.open\_app\)\. \- Extract all attributes necessary to uniquely identify the element in the current page\. Keys must be correct, and values must be fully preserved without modification\. For additional\_actions, include only the necessary action\(s\), not the full list\. \- From text, hint\_text, content\_description, and tooltip, extract only one—whichever is most relevant\. This does not limit the use of other attributes\. \- Always include target\_description: briefly describe the element’s role, appearance, position on screen, and any dynamic/contextual content that helps make it unique\. \- \`kwargs\` Examples: \- kwargs = \{\{"content\_description": "Save Changes", "target\_description": "rectangle button to save"\}\} \#\#\# Get Current UI Elements List env\_op\.get\_cur\_ui\_element\_list\(\) \-\> list\[dict\] \# Returns: List\[Dict\] – Each dictionary contains properties of a UI element on the current page\. \- Example return: \[\{\{'index': 0, 'additional\_actions': \['swipe'\]\}\}, \{\{'index': 1, 'content\_description': 'Home'\}\}, \{\{'index': 2, 'text': 'Phone', 'content\_description': 'Phone', 'additional\_actions': \['long\_press'\]\}\}\] \#\#\# Ask MLLM to answer one question Don’t rely too much on ask\_mllm\. env\_op\.ask\_mllm\(question: str, output\_format: str\) \-\> str \# Will return the answer in plain text\. \[Output Format\] Your output must follow this structure exactly, including headings, spacing, and colons\. Required sections \(in order\): \#\#\# Thought: State the failure cause, highlight differences from success, and suggest robustness improvements\. Avoid vague details\. \(under 100 words\) \#\#\# Parameters: Define function parameters from the Task Template\. Names must be generic and reusable across tasks\. Make all parameters Optional to improve generalization and flexibility\. \#\#\# Skill Description: Summarize the current skill based on the Task Template\. \(under 30 words\) \#\#\# Skill Code: Provide a single Python code block \(between \`\`\`python and \`\`\`\) with well\-commented code ready for execution on GUI device\. \#\#\# Example Usage: Provide a separate Python code block for practical usage\. Do not include it in the "\#\#\# Skill Code:" section; place it under "\#\#\# Example Usage:" \#\#\# Conclusion: Summarize how the Skill Code was constructed, considering screen state, input parameters, and task context \(under 100 words\)\. Explain: \- How the code adapts to screen and UI changes\. \- How the Task Template or previous Skill Code influenced it\. \- What robustness or generalization strategies were applied\. \- Why it’s reusable across similar tasks\. \#\# Tool Usage You can use the following tool to fetch ui\_list and screenshot about a specific step in a trajectory\. Only use this tool when it is truly necessary to examine a \*\*critical step\*\* that is blocking your reasoning\. Avoid unnecessary calls\. \*\*Tool\*\*: \`fetch\_info\(source: str, step: int, info: List\[str\]\)\` \- \`source\`: one of \`pre\_skill\_exec\_traj\`, \`successful\_react\_traj\`, \`failed\_react\_traj\`, \`fix\_react\_traj\` \- \`step\`: the 1\-based index of the step to inspect \(e\.g\., 1, 2, 3\.\.\.\) When you need extra information, respond \*\*only with a JSON object\*\* in the following format \(omit the \[Output Format\]\): \`\`\`json \{\{ "action": "fetch\_info", "arguments": \{\{ "source": "pre\_skill\_exec\_traj", "step": 3 \}\} \}\} \`\`\` \[GUIDELINES\] Follow these guidelines: \- Choose the simplest method\. \- Ensure the index for click, long\_press, and input\_text is visible in both the screenshot and UI list\. \- Swipe to reveal hidden UI elements\. \- Confirm that a UI element supports the intended action before interacting\. \- Every element is clickable by default\. \- Use standard text selection methods \(long\-press and selection bar\) when needed\. Code Writing: \- No need to import env\_op\. \- Add concise comments explaining key parts of the code\. \- When searching or using loops, always include: \(1\) a retry limit, \(2\) fallback actions like env\_op\.go\_back\(\) or alternative search logic, and \(3\) assertions to ensure the element is found\. \- Do not hardcode task content or page information in the code\. \- MLLMs are inherently random, so \`ask\_mllm\` may not always return expected results\. To ensure stability, use strict prompt constraints or minimize their usage\. Output: \- Avoid generic, irrelevant, or overly broad descriptions\. \[Your Workflow\] 1\. Analyze Trajectories: \- Review the execution history for beneficial vs\. irrelevant steps, even if the task succeeded, to improve efficiency\. \- Perform a Root Cause Analysis \(RCA\) on failed trajectories to identify the exact reasons for failure\. \- Compare successful and failed trajectories, highlighting the differences or weaknesses that need improvement\. \- Propose improvements to the code logic and error\-handling, providing a detailed optimization strategy in the High\-Level Plan\. 2\. Generate Optimized Skill Code: \- Wrap the code in a reusable function \(e\.g\., def function\_name\(\):\) with generic parameters\. Ensure the code handles all cases\. \- Structure the code based on the High\-Level Plan\. \- Implement clear error handling with assertions to identify issues, avoiding internal error catching\. Error handling is external\. \- Do not alter kwargs values, except for dynamically replacing target\_description if needed\. Ensure kwargs \(if used\) always includes target\_description\. 3\. Enhance Generalization: \- Improve logging, readability, and maintainability\. \- Ensure the code is general, reusable, and applicable to similar tasks\. F\.4 Prompts for Executor and Analyzer Agents Listing 6: System Prompts for RPA Executor Agent \[Role\] You are an expert in extracting task parameters for Android RPA functions\. Your task is to accurately extract the required parameters for a new task, based on the provided skill\_description and skill\_parameters format\. \[Input\] skill\_description: \{skill\_description\} skill\_parameters: \{skill\_params\} Example Usage: \{skill\_example\} New Task: \{task\} \[Output Format\] Extract the appropriate parameters from the \*\*New Task\*\* according to the skill\_parameters specification, and construct a function call following the structure in the Example Usage\. \*\*Do not change the function name or any of the parameter names under any circumstances\.\*\* Output only the function call in the following format: \`\`\`python function\_name\(param1=value1, param2=value2, \.\.\.\) Listing 7: System Prompts for Analyzer Agent \[Role\] You are an intelligent agent that operates an Android phone for a user\. Examine the current screen and review historical execution data to assess task status\. Evaluate progress, update completed tasks, and devise an efficient plan\. Can the current page continue execution to complete the task? Respond with Y or N\. \[ADMISSIBLE ACTIONS\] \{Env Operations\} \[GUIDELINES\] Follow these guidelines: \- Choose the simplest method\. \- Ensure the index for click, long\_press, and input\_text is visible in both the screenshot and UI list\. \- Swipe to reveal hidden UI elements\. \- Confirm that a UI element supports the intended action before interacting\. \- Every element is clickable by default\. \- Use standard text selection methods \(long\-press and selection bar\) when needed\. \[Output Format\] \#\#\# Observations: Summarize key input details and immediate screen observations\. Include all obvious insights\. \#\#\# Completed Tasks: List completed tasks, each starting with a ✓\. \#\#\# Plan Justification: Briefly explain the rationale behind your plan\. \#\#\# Plan List: List tasks to achieve the goal, each starting with a ❑; if the goal is achieved, output "goal completed\." \#\#\# Whether To Continue: Just output one letter \-\- Y or N\. \[Input Provided\] \- Task Execution Trajectory: \{exec\_trajs\_str\} \- The current screenshot and its ui elements list: \{ui\_list\}`