KnowAct-GUIClaw: Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill

# Know Deeply, Act Perfectly, Personal GUI Assistant with Self-Evolving Memory and Skill
Source: [https://arxiv.org/html/2607.12625](https://arxiv.org/html/2607.12625)
Lychee Team, Harbin Institute of Technology, Shenzhen AI Training Platform, Shenzhen Loop Area Institute [![[Uncaptioned image]](https://arxiv.org/html/2607.12625v1/figures/github-mark.png)Codes](https://github.com/HITsz-TMG/KnowAct)[![[Uncaptioned image]](https://arxiv.org/html/2607.12625v1/figures/website-logo.png)Website](https://shibosusu.github.io/KnowAct-GUIClaw/)[![[Uncaptioned image]](https://arxiv.org/html/2607.12625v1/figures/logs-logo.png)Experimental Logs](https://github.com/HITsz-TMG/KnowAct/releases/tag/Result)

###### Abstract

OpenClaw has emerged as a leading agent framework for complex task automation, yet its existing variants face two core bottlenecks: insufficient cross\-platform GUI interaction support and no built\-in self\-evolution mechanism\. These flaws limit its adaptation to heterogeneous device ecosystems and prevent performance improvements through continuous learning from execution experience\. To resolve these issues, we propose the “Know Deeply, Act Perfectly” paradigm for personal assistants, which holds that accumulated human\-machine interaction and task\-running experience directly improve execution accuracy and efficiency, unifying cognitive comprehension and operational execution\. Based on this paradigm, we introduceKnowAct\-GUIClaw, a novel Know\-Route\-Act\-Reflect framework designed to address OpenClaw’s GUI manipulation deficits and break through its cross\-platform and recursive self\-improvement constraints\. First, the host agent leverages accumulated interaction experience and task\-relevant knowledge for long\-horizon task decomposition and allocation \(Know\)\. Second, a pluggable GUI subagent with an experience\-attributable memory system \(Know\) and self\-evolving skill library \(Act\), enabling seamless cross\-platform migration and fast\-path integration\. Especially, this framework continuously stores user profiles and feedback to improve the accuracy of task decomposition and tool calls\. Extensive experiments across Android, iOS, HarmonyOS and Windows show thatKnowAct\-GUIClawachieves superior UI manipulation efficiency, accuracy and cross\-platform adaptability\. Especially, the GUIClaw with open\-source Kimi\-2\.6 models achieves the best performance \(64\.1%\) on the long\-horizon MobileWorld benchmark, beating all agential frameworks and closed\-source agentical models, e\.g\., Seed\-2\.0\-Pro and GPT\-5\.5\. Additionally, the knowledgeable memory and execution skills supported by our framework are transferable across diverse base models, improving by 8\.5% with Kimi\-2\.6 and 16\.2% with Qwen3\.5\-35B\-A3B\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/front.png)Figure 1:The success rate \(SR\) comparison on MobileWorld GUI\-Only tasks\. The bars summarize Table[1](https://arxiv.org/html/2607.12625#S5.T1)together with the additional Kimi\-basedKnowAct\-GUIClawruns; gray bars denote specialized GUI models, colored external bars denote general model families, and highlighted bars denoteKnowAct\-GUIClawvariants with memory and skills\. The experimental results show thatKnowAct\-GUIClawachieves SOTA performance and that the memory and skill are effective for different base models\.###### Contents

1. [1Introduction](https://arxiv.org/html/2607.12625#S1)
2. [2Related Work](https://arxiv.org/html/2607.12625#S2)1. [2\.1Personal Assistant Agent](https://arxiv.org/html/2607.12625#S2.SS1) 2. [2\.2GUI Agents](https://arxiv.org/html/2607.12625#S2.SS2) 3. [2\.3Memory and Skill Reuse](https://arxiv.org/html/2607.12625#S2.SS3)
3. [3Preliminaries](https://arxiv.org/html/2607.12625#S3)1. [3\.1GUI Automation as a POMDP](https://arxiv.org/html/2607.12625#S3.SS1) 2. [3\.2Host\-Centric Multi\-Agent Systems](https://arxiv.org/html/2607.12625#S3.SS2)
4. [4KnowAct\-GUIClaw](https://arxiv.org/html/2607.12625#S4)1. [4\.1Overview: A Know–Route–Act–Reflect Stack](https://arxiv.org/html/2607.12625#S4.SS1) 2. [4\.2Know: Context Gathering and Host Control](https://arxiv.org/html/2607.12625#S4.SS2) 3. [4\.3Route: Task Decomposition and Information Contracts](https://arxiv.org/html/2607.12625#S4.SS3) 4. [4\.4Act: Hybrid GUI–Fast Path Execution](https://arxiv.org/html/2607.12625#S4.SS4) 5. [4\.5Reflect: Trajectory Distillation and Skill Evolution](https://arxiv.org/html/2607.12625#S4.SS5)
5. [5Experiments](https://arxiv.org/html/2607.12625#S5)1. [5\.1Benchmarks and Metrics](https://arxiv.org/html/2607.12625#S5.SS1) 2. [5\.2Main Results of MobileWorld](https://arxiv.org/html/2607.12625#S5.SS2) 3. [5\.3Ablation and Efficiency Analysis](https://arxiv.org/html/2607.12625#S5.SS3) 4. [5\.4AndroidDaily Results](https://arxiv.org/html/2607.12625#S5.SS4) 5. [5\.5Case Study](https://arxiv.org/html/2607.12625#S5.SS5) 6. [5\.6Cross\-Platform Checks](https://arxiv.org/html/2607.12625#S5.SS6) 7. [5\.7Reproducibility](https://arxiv.org/html/2607.12625#S5.SS7)
6. [6Conclusion](https://arxiv.org/html/2607.12625#S6)
7. [7Contributors](https://arxiv.org/html/2607.12625#S7)
8. [References](https://arxiv.org/html/2607.12625#bib)
9. [ASupplementary Case Studies](https://arxiv.org/html/2607.12625#A1)
10. [BAction Space](https://arxiv.org/html/2607.12625#A2)
11. [CSkill Extraction and Prompt Contracts](https://arxiv.org/html/2607.12625#A3)1. [DSkill and Shortcut Examples](https://arxiv.org/html/2607.12625#A4)

## 1Introduction

Large language model \(LLM\) agents are transitioning from one\-off dialogue bots to stateful, long\-running personal assistants\. They process user requests from diverse input channels, preserve persistent workspace state, invoke external tools, coordinate auxiliary subagents, and resume interrupted long\-duration workflows\. Two dominant architectural lines guide their development: ReAct\-based agents unify logical reasoning and environmental action execution\(yao2023reactsynergizingreasoningacting\), while general autonomous agent frameworks modularize planning, memory, tool utilization, and multi\-agent dialogue as interchangeable runtime building blocks\(zhou2023agentsopensourceframeworkautonomous\)\. Practical deployed systems–OpenClaw, Nanobot, and Hermes–integrate these foundational designs into local\-first assistant platforms with built\-in channel routing, tool extensions, configurable skills, session persistence, and structured memory storage\(openclaw2026;nanobot2026;hermesagent2026\)\. As capable autonomous personal assistants, these platforms autonomously decompose and fulfil complex user objectives through targeted external tool calls\.

However, numerous real\-world user tasks demand interaction with graphical user interfaces \(GUIs\), rather than well\-structured, standardized application programming interfaces \(APIs\)\. For instance, a personal assistant may be required to inspect mobile applications, migrate data across different apps, handle permission pop\-up dialogues, or execute multi\-step workflows within login\-protected environments\. Recent benchmarks spanning web, mobile, and desktop platforms collectively indicate that GUI manipulation poses unique challenges that mandate visual grounding, sequential decision\-making, and resilient execution amid dynamically shifting interface states\(deng2023mind2webgeneralistagentweb;zhou2024webarenarealisticwebenvironment;koh2024visualwebarenaevaluatingmultimodalagents;xie2024osworldbenchmarkingmultimodalagents;rawles2025androidworlddynamicbenchmarkingenvironment;li2026windowsworld\)\. A suite of dedicated GUI agents—including AppAgent, Mobile\-Agent, CogAgent, OS\-Atlas, ShowUI, UI\-TARS, and Aguvis—have delivered remarkable advances in multimodal GUI action generation grounded solely on screen snapshots\(zhang2023appagentmultimodalagentssmartphone;wang2024mobileagentautonomousmultimodalmobile;hong2024cogagentvisuallanguagemodel;wu2024osatlasfoundationactionmodel;lin2024showuivisionlanguageactionmodelgui;qin2025uitarspioneeringautomatedgui;xu2025aguvisunifiedpurevision\)\. Hence,a natural research question arises: how can we endow OpenClaw\-style agents with the ability to efficiently interact with visual graphical environments?

A straightforward approach is to integrate a standalone GUI agent into the OpenClaw framework\. When deployed for long\-horizon personal assistant tasks, this approach suffers from severe inefficiency and fragility due to four drawbacks:Firstly, high\-level user instructions frequently span multiple disjoint applications\. Concise free\-text summaries often fail to retain intermediate data values extracted from one app for subsequent cross\-app operations\. Conversely, rigidly enforcing explicit target application labels for vague tasks without accounting for device environments often leads to spurious app\-allocation hallucinations\.Secondly, GUI observations are inherently incomplete\. Each individual data modality—screen captures, accessibility trees, foreground app IDs, historical action trajectories, and internal model reasoning logs—only reveals fragments of the full underlying device state\. This necessitates the host agent to record historical trajectories and provide actionable guidance for the lightweight GUI Agent\.Thirdly, successful and failed trajectories are typically discarded once a task terminates\. When re\-running analogous tasks, the agent is forced to re\-launch target applications, redo redundant navigation steps, and re\-learn known shortcuts and recurring failure patterns from scratch\.Finally, most GUI workflows do not integrate faster non\-visual shortcuts, such as web search tools, Android deep links, system intents, and reusable predefined action sequences\. And such shortcuts cannot be safely repurposed as persistent long\-term skills without validation against the real\-time interface page\.

This work presentsKnowAct\-GUIClaw, an agent framework augmented with structured knowledge and executable skills, built upon an OpenClaw\-style host runtime and GUI\-centric agent execution engine\. The framework comprises two core functional components: an attribution policy\-enhanced personalized memory system and a self\-evolving skill library that supports rapid skill invocation and iterative optimization\. The design follows a simple principle:*Knowing Deeply, Acting Perfectly*\. Before acting, the agent retrieves task\-relevant app candidates, tools, policies, and GUI hints from all agents’ memories\. During action, it treats GUI control as a partially observable decision process and records structured trajectory evidence or skills\. After action, it converts useful traces into a skill library or generalized memory, so future runs can use operational knowledge rather than repeating exploratory behaviour\. Specifically,KnowAct\-GUIClawtreats GUI automation as a collaboration between a capable host agent and a lightweight GUI executor, not as a single monolithic GUI agent\. The host owns the user\-facing context—conversation, workspace memory, the user profile, external tools, and orchestration—and decides*what*to do and which*cli tool*could be used; the GUI executor owns screenshot perception, action normalization, device backends, skill validation, and trajectory recording, and decides*how*to carry it out on screen\. Framing the problem this way makes efficiency a first\-class objective:the system invokes the GUI subagent only when visual interaction is genuinely required, reuses validated operational skills and fast CLI tools whenever possible\.

On this foundation,KnowAct\-GUIClawcontributes four connected mechanisms\.

- •Two\-tier host–executor collaboration\.We design a structured, resumable GUI task interface\. The host can assign limited GUI tasks to the executor and retrieve standardized outputs \(fully completed, partially completed, or blocked\), along with progress updates and resumption cues\. Incomplete or stalled task runs act as reusable checkpoints: rather than sending one fuzzy command, the host either resumes unfinished execution or rearranges subsequent workflows\. We also adopt an adaptive host involvement rule, where the host directly handles qualified information\-gathering subtasks with tools instead of forwarding all subtasks to the GUI executor\. This boosts task success rates and cuts computational overhead\.
- •Memory\-grounded routing and information transfer\.A built\-in routing mechanism classifies user requests as single\-app tasks or cross\-app workflows spanning multiple applications\. For cross\-app sequences, every subtask explicitly defines its input and output data\. A temporary shared data board transmits these structured values across subtasks, while persistent routing memory provides fixed candidate applications and auxiliary reference context\.
- •Knowledge\- and skill\-augmented GUI execution\.KnowAct\-GUIClawdistills trajectories into parameterized, state\-validated skills, letting the executor commit to a reusable action prefix as a single decision\. The same abstraction covers frequent click\-then\-type patterns and Android deeplink and intent shortcuts, which are taken only when their launch behavior and target page state satisfy the skill’s contract\.
- •Trajectory\-derived memory and skill evolution\.Post\-run reflection summarizes traces, distills success and failure lessons into retrievable experience memory, and refines the skill set as new evidence accumulates\. This knowledge feeds back to the memory\-grounded router before task decomposition and to the executor during execution, closing the loop from past runs to future decisions\.

Extensive experimental results demonstrate that our model attains SOTA performance on the challenging MobileWorld benchmark, while enabling cross\-platform deployment as well as memory and skill transfer\.

## 2Related Work

### 2\.1Personal Assistant Agent

LLM\-agent runtimes pair language models with action interfaces, memory, and control loops, as in ReAct\(yao2023reactsynergizingreasoningacting\), the Agents framework\(zhou2023agentsopensourceframeworkautonomous\), and OS\-Copilot\(wu2024oscopilotgeneralistcomputeragents\)\.Recently Two salient design philosophies recur when this pattern is packaged into local\-first assistants\. OpenClaw and Nanobot are configuration\-centric: the host exposes message channels, external tools and Model Context Protocol \(MCP\) servers, conversation history and session state, long\-lived workspace memory, and user\-authored skills as first\-class, declarable components, so that most behavior is specified around the model rather than left to it\(openclaw2026;nanobot2026\)\. Hermes instead foregrounds adaptation, growing its competence over time by turning prior interactions into reusable routines that the assistant can later invoke\(hermesagent2026\)\. Both directions show that a capable personal assistant benefits from treating control logic, memory, tools, and history as managed state around the model\.

### 2\.2GUI Agents

GUI and computer\-use agents study how language models act through visual interfaces rather than clean APIs\. On the web, Mind2Web and WebArena expose long\-horizon navigation across realistic sites through HTML and accessibility\-tree observations\(deng2023mind2webgeneralistagentweb;zhou2024webarenarealisticwebenvironment\), while VisualWebArena adds visually grounded decision making over rendered pages\(koh2024visualwebarenaevaluatingmultimodalagents\), and SeeAct and WebVoyager explore screenshot\- and page\-structure\-based control\(zheng2024gpt4visiongeneralistwebagent;he2024webvoyagerbuildingendtoendweb\)\. Beyond the browser, Android in the Wild and AndroidWorld provide large\-scale demonstrations and dynamic, programmatically checked tasks\(rawles2023androidwildlargescaledataset;rawles2025androidworlddynamicbenchmarkingenvironment\), OSWorld and OmniACT cover broader desktop and web computer\-use settings\(xie2024osworldbenchmarkingmultimodalagents;kapoor2024omniactdatasetbenchmarkenabling\), and AppAgent and Mobile\-Agent solve smartphone tasks with human\-like actions and multimodal perception\(zhang2023appagentmultimodalagentssmartphone;wang2024mobileagentautonomousmultimodalmobile\)\. Model\-centric systems such as CogAgent, OS\-Atlas, ShowUI, UI\-TARS, and Aguvis further improve perception, grounding, and visual action generation\(hong2024cogagentvisuallanguagemodel;wu2024osatlasfoundationactionmodel;lin2024showuivisionlanguageactionmodelgui;qin2025uitarspioneeringautomatedgui;xu2025aguvisunifiedpurevision\)\.These works have substantially improved GUI task execution, their primary focus is often on task completion within benchmarked interaction environments\.

### 2\.3Memory and Skill Reuse

Agent memory and skill reuse let agents improve without weight updates\(vu2018sentence\)in both language\-only and embodied settings\. Reflexion stores verbal feedback to guide later attempts\(shinn2023reflexionlanguageagentsverbal\), Generative Agents maintain a reflective memory stream for coherent long\-term behavior\(park2023generativeagentsinteractivesimulacra\), Voyager builds an executable skill library for open\-ended embodied learning\(wang2023voyageropenendedembodiedagent\), and KnowAgent uses an action knowledge base to curb planning hallucination\(zhu2025knowagentknowledgeaugmentedplanningllmbased\)\. ReasoningBank distills reusable success and failure rationales into retrievable memory for self\-evolving agents\(ouyang2026reasoningbankscalingagentselfevolving\), GUI\-specific work such as LearnAct and CUA\-Skill highlights the value of demonstrations, retrieved knowledge, and engineered skills for computer\-use agents\(liu2025learnactfewshotmobilegui;chen2026cuaskilldevelopskillscomputer\), and a recent survey frames agent memory as a write–manage–read loop coupled with perception and action\(du2026memoryautonomousllmagentsmechanisms\)\. Across these lines, textual memory and executable skills address related but distinct reuse problems: memory preserves guidance, rationales, and context, whereas skills compress procedures into reusable behavior\.

## 3Preliminaries

### 3\.1GUI Automation as a POMDP

GUI automation is naturally modeled as a partially observable Markov decision process \(POMDP\)\(kaelbling1998planning\):

ℳ=\(𝒮,𝒜,𝒪,T,Ω,R,γ\),\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{O\},T,\\Omega,R,\\gamma\),\(1\)where𝒮\\mathcal\{S\}is the hidden device state,𝒜\\mathcal\{A\}the action space,𝒪\\mathcal\{O\}the observation space,TTandΩ\\Omegathe transition and observation functions,RRthe task reward, andγ\\gammathe discount factor\. The defining property of GUI control is partial observability: the agent never observes𝒮\\mathcal\{S\}directly but only an observationoto\_\{t\}—typically a screenshot with optional screen metadata, the foreground app, and a bounded action history—that exposes a projection of hidden state such as navigation stacks, login and permission status, asynchronous loading, and off\-screen form values\. A GUI agent must therefore act and recover from errors under this uncertainty rather than assume a fully known interface\.

### 3\.2Host\-Centric Multi\-Agent Systems

Multi\-agent systems distribute problem solving across agents with distinct roles, capabilities, and local information, coordinating through communication and shared environments rather than a single monolithic policy\(wooldridge2009introduction\)\. Recent LLM agent frameworks adopt this view, pairing a central orchestrator that maintains conversation state, memory, and tool access with specialized environment\- or tool\-facing agents to which it delegates subtasks\(zhou2023agentsopensourceframeworkautonomous\)\. For GUI automation, this host\-orchestration pattern motivates separating long\-horizon task management from low\-level interaction with a partially observed device; Section[4](https://arxiv.org/html/2607.12625#S4)instantiates this as a concrete execution stack for mobile GUI tasks\.

## 4KnowAct\-GUIClaw

### 4\.1Overview: A Know–Route–Act–Reflect Stack

Recent mobile\-agent systems and benchmarks point to three requirements: hybrid GUI–shortcut action spaces improve efficiency when shortcuts are available\(zhao2026masbenchunifiedbenchmarkshortcutaugmented\); perception, memory, and action work as one execution stack\(ren2026xomniclawtechnicalreportunified\); and a GUI\-agent framework should learn from the execution trajectories\(tang2026clawgui\)\.KnowAct\-GUIClaworganizes new long\-horizon task execution as a four\-stage loop:*Know*,*Route*,*Act*, and*Reflect*\. Figure[2](https://arxiv.org/html/2607.12625#S4.F2)summarizes this loop and the persistent stores that feed every stage\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/know-route-act-reflect.png)Figure 2:Overview of theKnowAct\-GUIClawexecution loop\. Two persistent stores—a memory and history store and a skill and shortcut store—supply advisory context to every stage\.*Know*gathers evidence and assembles a reasoning context;*Route*ranks app candidates and turns the request into either a single GUI task or an ordered multi\-app workflow whose subtasks exchange typed values through a blackboard;*Act*runs GUIClaw’s observe–reason–act loop over the hybrid action space of GUI primitives, skills, deeplink/intent shortcuts, and intervention actions; and*Reflect*distills each trajectory into updated skills and experience memory that feed back into the stores\.GUI execution should serve as a subagent within the personal user assistant framework, specifically to address scenarios where no standardized API endpoints are available and graphical interfaces are dynamically rendered in real time\. Accordingly, the GUI task interface acts as the formal boundary between the host agent and the UI subagent\. Each GUI task returns a structured output consisting of three components: its completion status, a concise execution summary, and the final screen state upon task termination\. The host agent retains responsibility for user\-facing task orchestration and long\-horizon context maintenance, while the GUI subagent operates as a self\-contained engine dedicated to low\-level device control\.

### 4\.2Know: Context Gathering and Host Control

Active retrieval and policy injection\.Before any GUI action, the Know stage gathers most of its context actively: prior GUI memories and candidate skills are retrieved by semantic similarity and kept advisory, never overriding the current instruction, while policy memory is injected directly rather than ranked\. Figure[3](https://arxiv.org/html/2607.12625#S4.F3)illustrates this advisory role on a Mastodon task: the retrieved lesson redirects the task from unsupported mobile\-app settings to the web administration panel without replaying an old trajectory\.

Host\-held context and active recall\.The host agent keeps the running session context and forwards only what a GUI task needs when it issues that task, and it recalls its own stored memories, such as session history, agent memory, and the user profile, only when it judges them relevant\. When the inputting instruction is fuzzy, it draws on these memories to propose defaults as explicit, overridable assumptions\.This capability is critical for an agentic assistant to achieve continuous capability improvement when deployed across mobile and desktop platforms\.

Host\-centric Control\.The host agent delegates selectively\. It answers a subtask itself when conversation context, recalled memory, or non\-GUI tools suffice and no device\-local GUI state must be observed or changed, and it issues a GUI task only when the subtask needs a live app session, visual grounding, cross\-app manipulation, text entry, or on\-device verification\. Resolving eligible subtasks directly reduces hand\-offs and token\-heavy GUI traces\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/gui_memory_case.png)Figure 3:Experience memory improves a GUI task by changing the task context before low\-level control begins\. Without the retrieved memory \(Top\), GUIClawinvites continues through Mastodon’s mobile settings and reaches a nonproductive path for invite\-link creation\. With the retrieved memory \(bottom\), the Know stage supplies an advisory lesson that invite links with advanced settings that require the web administration panel; GUIClaw then opens the web interface, navigates to account settings, and reaches the invite\-people page\. The example shows that experience memory guides app choice, decomposition, and recovery while live screen observations still ground each action\.
### 4\.3Route: Task Decomposition and Information Contracts

Task decomposition\.The routing policy emits either a single GUI task or an ordered multi\-app workflow\. In the multi\-app case, each subtask is a goal\-level tuple\(gi,hi,Ii,Oi\)\(g\_\{i\},h\_\{i\},I\_\{i\},O\_\{i\}\):gig\_\{i\}is the app\-scoped goal,hih\_\{i\}optionally narrows the app,IiI\_\{i\}names the inputs it requires, andOiO\_\{i\}names the values it should return\. The router does not predict screen sequences\. Because each subtask has its corresponding app, the relevant memory and skills are retrieved afresh for that subtask rather than inherited from the top\-level route\.

Information transfer\.The blackboard makes cross\-app data flow explicit\. LetGGdenote GUIClaw, which executes a subtask over the POMDP, letEEdenote the evidence\-to\-value mapping, and letBiB\_\{i\}denote the blackboard after subtaskii\. A subtask sees only the declared inputs already onBi−1B\_\{i\-1\}, and only its declared outputs are written back from the trajectory evidence:

G​\(gi,hi,Bi−1​\[Ii\]\)→τi,E​\(τi,Oi\)→Bi​\[Oi\]\.G\(g\_\{i\},h\_\{i\},B\_\{i\-1\}\[I\_\{i\}\]\)\\rightarrow\\tau\_\{i\},\\qquad E\(\\tau\_\{i\},O\_\{i\}\)\\rightarrow B\_\{i\}\[O\_\{i\}\]\.\(2\)If a required input or a declared output is missing, the workflow fails closed rather than running a subtask on incomplete state or letting a later one infer or fabricate the value\. This rule matters for cross\-app transfer, comparison, and fact\-collection tasks\. Figure[4](https://arxiv.org/html/2607.12625#S4.F4)traces the typed tuple contract, while Figure[5](https://arxiv.org/html/2607.12625#S5.F5)\(Section[5](https://arxiv.org/html/2607.12625#S5)\) shows the same mechanism in a real cross\-app trajectory\.

### 4\.4Act: Hybrid GUI–Fast Path Execution

Hybrid action space\.GUIClaw executes each routed subtask over a hybrid action space:

𝒜=𝒜gui∪𝒜skill∪𝒜shortcut∪𝒜ask\.\\mathcal\{A\}=\\mathcal\{A\}\_\{\\mathrm\{gui\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{skill\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{shortcut\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{ask\}\}\.\(3\)𝒜gui\\mathcal\{A\}\_\{\\mathrm\{gui\}\}contains human\-like primitives such as tap, swipe, scroll, text input, navigation, app open/close, and wait;𝒜skill\\mathcal\{A\}\_\{\\mathrm\{skill\}\}contains skills distilled from reusable GUI behavior\.𝒜shortcut\\mathcal\{A\}\_\{\\mathrm\{shortcut\}\}contains Android deeplinks and intents\(androiddeeplinks2026;androidintents2026\)which bypass lengthy navigation workflows when verified as valid on the target device\. Predefined action sequences with historically calibrated interaction parameters can also be stored in this set to eliminate the visual grounding overhead of the GUI subagent, though such rigid shortcut schemes suffer from limited generalization capability\.𝒜ask\\mathcal\{A\}\_\{\\mathrm\{ask\}\}contains intervention actions for cases that require user input or authorization\. These types trade efficiency against reliability, so the executor validates skills and shortcuts against the current state before using them\. Appendix[B](https://arxiv.org/html/2607.12625#A2)gives the unified action space, by platform, that realizes𝒜gui∪𝒜ask\\mathcal\{A\}\_\{\\mathrm\{gui\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{ask\}\}\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/blackboard-subgoal.png)Figure 4:Blackboard\-mediated execution in the Route stage\. The short\-lived blackboardBBstores typed inputs and outputs known so far\. Each subtask\(gi,hi,Ii,Oi\)\(g\_\{i\},h\_\{i\},I\_\{i\},O\_\{i\}\)checks its declared inputsIiI\_\{i\}, reads their values fromBB, runs GUIClaw’s observe–reason–act loop to produce a trajectoryτi\\tau\_\{i\}, and writes only its declared outputsOiO\_\{i\}back toBB\([2](https://arxiv.org/html/2607.12625#S4.E2)\)\. A missing required input or output makes the workflow fail closed, so later subtasks consume observed typed values rather than free\-form summaries\.Observe–Reason–Act loop\.Within this action space, GUIClaw runs a grounded control loop:

pt→ot→at→et→ot\+1,p\_\{t\}\\rightarrow o\_\{t\}\\rightarrow a\_\{t\}\\rightarrow e\_\{t\}\\rightarrow o\_\{t\+1\},\(4\)whereptp\_\{t\}is the prompt,oto\_\{t\}is the current observation,ata\_\{t\}is the action, andete\_\{t\}its result on the device\. The prompt carries the subtask, task’s blackboard, policy memory, interface hints, and available skills, and a normalization step maps each model’s output into a common action format\. Appendix[C](https://arxiv.org/html/2607.12625#A3)gives the prompt contracts used to expose, induce, and validate these skills\.

Skills\.Skills turn repeated interaction into reusable procedures\. Each stored skill contains an identifier, app and platform scope, description, parameters, reliability counters, and an ordered sequence of steps\. Stable fields such as app package names, coordinates, components, and intent payloads are fixed in advance, while task\-dependent values are represented as placeholders and grounded at run time\. Before each subtask, the GUI subagent retrieves app\-scoped candidates by lexical and embedding signals, then uses a lightweight applicability prompt to select one useful prefix or reject all candidates\. Retrieved skills \(Top\-5\) are advisory:KnowAct\-GUIClawmay apply one or fall back to ordinary actions\. Before each step, it checks the expected state through a deterministic state contract when available and through a visual valid\-state check otherwise; on a mismatch, it either runs a bounded recovery subgoal, skips an optional obstacle step, or returns control to ordinary GUI execution\. Appendix[D](https://arxiv.org/html/2607.12625#A4)gives concrete skill records, including a multi\-step skill with per\-step state validation\.

Deeplink and intent shortcuts\.𝒜shortcut\\mathcal\{A\}\_\{\\mathrm\{shortcut\}\}contains page\-validated Android deeplinks and intents that bypass long GUI navigation\. During execution, GUIClaw treats them as one\-step skills withopen\_deeplinkoropen\_intent, and uses them only after their target page, required parameters, and app state have been validated\. This prevents broad Android constants such asSEARCH,SEND, orGET\_CONTENTfrom being executed merely because a manifest exposes them\. Appendix[D](https://arxiv.org/html/2607.12625#A4)shows representative skill and shortcut entries\.

### 4\.5Reflect: Trajectory Distillation and Skill Evolution

Post\-run summarization\.After each GUI task, reflection condenses the trajectory into a short note of where execution ended and what remains, so a partial or blocked task becomes a checkpoint the host can resume without redoing finished work\. The post\-run processor summarizes the trace and, when enabled, evaluates the outcome before learning from it\. Runs with no useful signal, e\.g\., empty, cancelled, timed out before progress, or completed entirely by an already reused skill, are not written to the long\-term stores\.

Skill extraction\.For trajectories selected for skill learning, reflection extracts a reusable procedure rather than storing raw actions for replay\. It rewrites GUI events into structured evidence: the task, platform, observed apps, action sequence, action parameters, screenshots, target\-control hints, and inferred state contracts\. A vision LLM receives this evidence under a restricted prompt that permits only declarative action sequences\. The candidate skill is normalized and checked for supported actions, declared parameters, executable fixed fields, reusable app scope, and valid\-state coverage\. Accepted steps inherit the state contracts inferred from the trajectory, so later execution validates screen state instead of replaying an ungrounded script\. Offline extraction uses the same agent after filtering short or abnormal traces, keeping bounded successful prefixes, clustering structurally identical skills, and recording success counts for retrieval priority\.

Skill evolution\.KnowAct\-GUIClawseparates repair from new extraction\. When a trajectory shows a reused skill failing, reflection records the failed step, screen evidence, error, original skill, and prior feedback, then asks an evolution prompt to update that same skill in place: it may narrow the description, add guarded optional obstacle handling, or refresh stale targets and state contracts, but it may not replace the skill with an unrelated workflow or add destructive terminal actions\. Repair therefore takes precedence over new extraction; only when no failed reused skill is present does the system mine a new skill from reusable behavior\.

Shortcut validation\.Shortcut candidates are mined from app manifests as discovery evidence, not trusted actions\. A validation run launches candidate variants, records the foreground app, ADB output, UI tree, and screenshot, and asks a verifier to return usability, page status, payload preservation, parameters, and a natural\-language capability description\. Only page\-validated records are promoted into one\-step skills; merely launchable records remain candidates unless explicitly allowed by the validation configuration\. Appendix[C](https://arxiv.org/html/2607.12625#A3)gives the validation prompt\.

Experience memory\.Experience memory stores textual policies derived from running trajectories\. Inspired by ReasoningBank\(ouyang2026reasoningbankscalingagentselfevolving\),KnowAct\-GUIClawformats each trace into a concise action\-and\-UI summary and uses separate success and failure prompts to induce at most a few actionable memory items\. The memory inducer skips short or abnormal traces, resolves outcome and app at the per\-trace level, caps the number of retained items per task, and drops near\-duplicates within the same app\. These lessons are distinct from executable skills and feed later tasks at both levels: routing draws on them to choose apps and decomposition patterns, while the GUI agent uses them as hints about layout, shortcut reliability, and recovery\. Appendix[C](https://arxiv.org/html/2607.12625#A3)gives the corresponding induction prompts and filtering rules\.

## 5Experiments

### 5\.1Benchmarks and Metrics

We evaluateKnowAct\-GUIClawon two mobile GUI benchmarks that include diverse real\-user tasks\.*MobileWorld*\(kong2025mobileworldbenchmarkingautonomousmobile\)is our primary benchmark: it emphasizes long\-horizon mobile tasks, many of which span multiple applications\. From its 201 tasks over 20 applications, we use the 117\-task GUI\-Only subset, scored by the benchmark’s native deterministic evaluators\.*AndroidDaily*\(yan2025stepguitechnicalreport\)is a complementary end\-to\-end benchmark that groups tasks by type, complexity, and ambiguity; many of its tasks require returning an explicit answer rather than only manipulating the UI\. Lacking a native evaluator, we score its GUI\-only tasks withqwen3\.5\-flashas an LLM judge and its answer\-returning tasks with two human experts, with correct tasks scoring1\.01\.0and partial tasks0\.50\.5\.

Setup\.Unless noted,KnowAct\-GUIClawpairs a Qwen3\.5\-397B\-A17B host with a Qwen3\.5\-35B\-A3B GUI executor\. In the Kimi\-K2\.6 configurations of Table[1](https://arxiv.org/html/2607.12625#S5.T1), Kimi\-K2\.6 serves as both the host and GUI executor\. The Kimi\-to\-Qwen transfer configuration retains the default Qwen host–executor pairing and instead uses a joint set of experience memory and skills distilled from trajectories generated by the Kimi\-K2\.6 host–executor configuration\. All MobileWorld configurations use the same 117\-task GUI\-Only subset, native deterministic evaluators, and 50\-step cap\. Following the public leaderboard, our primary MobileWorld metric is the single\-run success rate \(SR\), i\.e\. pass@1; we report pass@3 only as a repeated\-attempt upper bound, in a separate table \(Table[5](https://arxiv.org/html/2607.12625#S5.T5)\)\. For AndroidDaily we report a*resolved*setting over the 194 available tasks and an*all*setting that scores the 41 unavailable entries as0across all 235 tasks\. The AndroidDaily evaluation uses iOS devices\.

Efficiency metrics\.We report SR together with execution cost: GUI steps, the number of GUI task invocations, executed GUI\-trace tokens, and host tokens\. The*total*column measures the executed GUI traces and is comparable across systems;*host total*measures the additional generation that the host spends to route, coordinate, call external tools such as web search, and resolve eligible subtasks\.

### 5\.2Main Results of MobileWorld

Table 1:MobileWorld GUI\-Only success rate \(pass@1\)\. External rows are grouped by the public MobileWorld leaderboard categories\(kong2025mobileworldbenchmarkingautonomousmobile\)\. AllKnowAct\-GUIClawrows are our own runs; the Kimi\-to\-Qwen transfer row equips the 35B executor with experience memory and skills distilled from Kimi\-K2\.6 trajectories\. Rows matching configurations B, C, and F are analyzed in Table[2](https://arxiv.org/html/2607.12625#S5.T2), and Table[5](https://arxiv.org/html/2607.12625#S5.T5)reports the repeated\-attempt upper bound\.ModelSR \(pass@1, %\)General modelsSeed\-2\.0\-Pro\(seed2026seed20modelcardintelligence\)63\.2GPT\-5\.5\(openai2026gpt55\)62\.4Gemini\-3\.1\-Pro\-Preview\(google2026geminipro\)58\.1Claude\-Opus\-4\.7\(anthropic2026claudeopus47\)56\.4Kimi\-K2\.6\(moonshot2026kimik26\)55\.6Kimi\-K2\.5\(moonshot2026kimik25\)49\.6Claude\-Sonnet\-4\.5\(anthropic2025claudesonnet45\)47\.8Qwen3\.5\-397B\-A17B\(qwen2026qwen35\)42\.7Qwen3\.5\-122B\-A10B\(qwen2026qwen35\)35\.0Qwen3\.5\-35B\-A3B\(qwen2026qwen35\)24\.8Qwen3\-VL\-235B\-A22B\(bai2025qwen3vltechnicalreport\)12\.8Agentic systemsGemini\-3\-Pro \+ UI\-Ins\-7B\(google2026geminipro;chen2025uiinsenhancinggui\)55\.6GPT\-5 \+ UI\-Ins\-7B\(singh2026openaigpt5card;chen2025uiinsenhancinggui\)54\.0Claude\-4\.5\-Sonnet \+ UI\-Ins\-7B\(anthropic2025claudesonnet45;chen2025uiinsenhancinggui\)47\.8Specialized GUI modelsGUI\-Owl\-1\.5\-32B\(xu2026mobileagentv35multiplatformfundamentalgui\)43\.9ForgeOwl\-8B\(liu2026mobileforgeannotationfreeadaptationmobile\)41\.0MAI\-UI\-235B\-A22B\(zhou2025maiuitechnicalreportrealworld\)39\.7GUI\-Owl\-1\.5\-8B\(xu2026mobileagentv35multiplatformfundamentalgui\)38\.2MAI\-UI\-32B\(zhou2025maiuitechnicalreportrealworld\)36\.2Doubao\-1\.5\-UI\-TARS\(qin2025uitarspioneeringautomatedgui\)26\.3UI\-Venus\-72B\(gu2025uivenustechnicalreportbuilding\)16\.4GUI\-Owl\-7B\(ye2025mobileagentv3fundamentalagentsgui\)7\.7Ours \(Open\-source Qwen3\.5\)35B \+ host & memory34\.535B \+ host, memory & skills37\.935B \+ host, Kimi\-derived memory & skills41\.0397B host acts directly46\.2Ours \(Open\-source Kimi\-K2\.6\)\+ host & memory61\.5\+ host, memory & skills64\.1Table[1](https://arxiv.org/html/2607.12625#S5.T1)comparesKnowAct\-GUIClawwith the public MobileWorld leaderboard\. The Qwen3\.5\-35B\-A3B GUI executor reaches 24\.8 SR on its own; running it insideKnowAct\-GUIClawwith the host and experience memory raises pass@1 to 34\.5, and enabling skills on top raises it to 37\.9\. Equipping the same 35B executor with experience memory and skills distilled from Kimi\-K2\.6 trajectories further raises SR to 41\.0, a 16\.2\-point gain over the base executor and 3\.1 points above the standard 35B host–memory–skills configuration\. With Kimi\-K2\.6 as the base model,KnowAct\-GUIClawreaches 61\.5 with the host and experience memory and 64\.1 after enabling skills, the highest SR in the table\. Letting the 397B Qwen host act directly reaches 46\.2; using only open Qwen3\.5 models, this configuration remains above the public Qwen3\.5\-397B\-A17B plain\-GUI score \(42\.7\) and every specialized end\-to\-end GUI model listed\. We isolate the host–memory and skills increments next\.

### 5\.3Ablation and Efficiency Analysis

Table 2:MobileWorld GUI\-Only ablation and efficiency \(our runs\)\. Rows A–C use the Qwen3\.5\-35B\-A3B GUI executor and rows D–F the Qwen3\.5\-397B\-A17B;\+ host & memadds theKnowAct\-GUIClawhost, router, and experience memory, and\+ skillsenables skills on top of that setting\. In E–F the 397B host resolves eligible subtasks directly rather than delegating each to the 35B executor \(Section[4\.2](https://arxiv.org/html/2607.12625#S4.SS2)\)\. Row D is our plain\-GUI 397B run, distinct from its 42\.7 public leaderboard score\.*Total*is GUI\-trace tokens per task;*host total*is the host’s own generation, including external tool calls, and is the extra token costKnowAct\-GUIClawadds on top\. Highlighted rows mark the host\- and skill\-augmented configurations\.ConfigSettingSR \(%\)GUI stepsGUI tasksTotalHost TotalGUI executor: Qwen3\.5\-35B\-A3BAexec\. only24\.826\.71\.0281,266–B\+ host & mem34\.526\.82\.3279,21165,224C\+ skills37\.925\.12\.5278,28963,792GUI executor: Qwen3\.5\-397B\-A17BDexec\. only40\.726\.11\.0254,459–E\+ host & mem43\.326\.81\.4273,35210,096F\+ skills46\.223\.71\.7260,51610,982

Table[2](https://arxiv.org/html/2607.12625#S5.T2)reports a system\-level ablation along two executor scales\.

Host and memory raise accuracy at a modest comparable cost\.Adding the host, router, and experience memory raises SR at both executor scales \(A→\\toB,24\.8→34\.524\.8\\to 34\.5; D→\\toE,40\.7→43\.340\.7\\to 43\.3\)\. Total tokens stay flat for the 35B executor \(281281k vs\.279279k\) but rise about7%7\\%for the 397B executor \(254254k to273273k\), where the host splits a task into more GUI subtrajectories\. Because total tokens are dominated by visual observation, the accuracy gain reflects better task organization rather than a larger GUI\-executor budget; the host total is65,22465\{,\}224tokens per task when the host must coordinate the weaker 35B executor across2\.32\.3GUI task invocations \(about23%23\\%of total\), but only10,09610\{,\}096with the 397B host at1\.41\.4invocations \(under4%4\\%\)\.

Skills cut steps and total tokens \(B→\\toC, E→\\toF\)\.Enabling skills on top of the host–memory setting lowers GUI steps and total tokens—modestly for the 35B executor, and more clearly for the 397B host, where steps fall from26\.826\.8to23\.723\.7and total tokens by about5%5\\%—while SR rises rather than falls\. The saving comes from shorter trajectories and fewer screenshot\-heavy observations\. Table[3](https://arxiv.org/html/2607.12625#S5.T3)isolates this effect on the tasks that actually invoke a skill\.

Experience memory and skills transfer across base models\.Table[1](https://arxiv.org/html/2607.12625#S5.T1)includes a cross\-model transfer test in which the Qwen3\.5\-35B\-A3B executor uses experience memory and executable skills distilled from Kimi\-K2\.6 trajectories\. This transferred configuration reaches 41\.0 SR, compared with 37\.9 for the standard 35B host–memory–skills configuration and 24\.8 for the base executor\. The result shows that both textual experience rationales and state\-validated executable behaviors can carry task knowledge across model families rather than remaining tied to the model that generated the source trajectories\. Together with the Kimi\-K2\.6 results \(61\.5 with host and memory; 64\.1 with skills\), this establishes transferability in the tested Kimi\-to\-Qwen direction without assuming universal model\-independent transfer\.

A capable host should not route everything through the GUI \(F\)\.Letting the 397B host resolve eligible information\-query subtasks directly, rather than delegating each to the 35B executor, yields the best SR \(46\.246\.2\) while lowering both cost components relative to the delegated configuration C: the host answers eligible subtasks itself instead of spawning a GUI task for each \(1\.71\.7GUI task invocations versus2\.52\.5for C\), so total tokens fall to260,516260\{,\}516\(from278,289278\{,\}289\) and host total to10,98210\{,\}982tokens per task \(from63,79263\{,\}792\)\. F thus delivers the best accuracy at the lowest GUI\-trace execution cost among the Qwen ablations, while keeping host overhead minimal\. For external context, Qwen3\.7\-Plus reaches43\.643\.6SR with30\.030\.0steps, while the public Kimi\-K2\.6 baseline reaches55\.655\.6; our Kimi host–memory and skill\-augmented configurations reach61\.561\.5and64\.164\.1, respectively \(Table[1](https://arxiv.org/html/2607.12625#S5.T1)\)\.

Table 3:Skill\-reuse effect on the MobileWorld tasks that invoke at least one skill \(8383with the 35B executor,8787with the 397B host\), each compared against the same tasks run without skills\. SR and pass@3 deltas are in percentage points\.Qwen3\.5\-35B\-A3B executorQwen3\.5\-397B\-A17B executorMetric \(skill\-using tasks\)\+ skillsno skillsΔ\\Delta\+ skillsno skillsΔ\\DeltaGUI steps / task25\.729\.0−3\.3\-3\.322\.325\.6−3\.3\-3\.3total tokens / task284,279303,014−6\.2%\-6\.2\\%242,930258,084−5\.9%\-5\.9\\%single\-run SR \(%\)40\.635\.7\+4\.9\+4\.950\.248\.3\+1\.9\+1\.9pass@3 \(%\)54\.253\.0\+1\.2\+1\.263\.263\.2\+0\.0\+0\.0

Skill reuse, measured where it applies\.Restricting to the tasks that invoke a skill concentrates the same effect \(Table[3](https://arxiv.org/html/2607.12625#S5.T3)\): reuse removes about three GUI steps per task \(−3\.3\-3\.3\), cuts total and prompt tokens by roughly6%6\\%, and improves SR \(\+4\.8\+4\.8and\+1\.9\+1\.9points\)\.

Section[5\.5](https://arxiv.org/html/2607.12625#S5.SS5)complements these aggregate comparisons with qualitative evidence for routing and information transfer, experience\-memory correction, navigation compression through skills and shortcuts, and host recovery and direct participation\.

### 5\.4AndroidDaily Results

Table 4:AndroidDaily \(end\-to\-end\) performance by task type, complexity, and ambiguity\. Our AndroidDaily evaluation is conducted on iOS devices\. ForKnowAct\-GUIClaw,*resolved*excludes the 41 unavailable entries and rates the remaining 194 tasks, while*all*scores those entries as0over all 235 tasks\.Task TypeComplexityAmbiguityModel / SettingFilterQueryAnalyzeAtomicComp\.Cond\.LowMidHighTotalUI\-TARS\-1\.557\.6465\.9736\.7161\.4113\.6460\.3857\.0554\.9057\.8956\.64Step\-GUI\-4B44\.7764\.2933\.7254\.0319\.6142\.8651\.2138\.3259\.5249\.06Step\-GUI\-8B52\.5063\.8232\.9559\.0914\.0042\.8654\.0844\.5561\.5452\.50Ours \(Resolved\)80\.5676\.8877\.0881\.2562\.5075\.0078\.8977\.5078\.9578\.61Ours \(All\)68\.4066\.1351\.3970\.9841\.6753\.2364\.9465\.9662\.5064\.89

We conduct the AndroidDaily evaluation on iOS devices\. AndroidDaily contains many long\-horizon tasks that require cross\-application analysis and an explicit answer rather than a single visual goal\. Table[4](https://arxiv.org/html/2607.12625#S5.T4)shows thatKnowAct\-GUIClaw’s largest margins over GUI\-only baselines fall on*Analyze*and*complex*\(Comp\.\) tasks, where explicit subtask decomposition, blackboard information transfer, and retrieved memory reduce repeated trajectory solving\. The difference between the resolved and all settings reflects app availability and environment mismatch: these factors, rather than policy quality alone, account for a substantial fraction of end\-to\-end failure\.

### 5\.5Case Study

This case study uses representative workflows to illustrate four characteristics ofKnowAct\-GUIClaw: routing and information transfer, experience\-memory correction, navigation compression through skills and shortcuts, and host recovery and direct participation\. The screenshots come from actual executions; host cards, step labels, per\-step thoughts, and blackboard pills are overlays added for readability\.

Routing and information transfer\.Long\-horizon, cross\-app tasks amplify memory decay and goal drift when one GUI trajectory must retain every intermediate summary\.KnowAct\-GUIClawinstead decomposes the request into app\-scoped GUI tasks and passes only declared outputs through the blackboard\. Figure[6](https://arxiv.org/html/2607.12625#S5.F6)\(a\) isolates this handoff in an Email\-to\-Clock workflow, while Figure[7](https://arxiv.org/html/2607.12625#S5.F7)shows a longer Email\-to\-Messages\-to\-Maps workflow in which the resolved address becomes an explicit input to both downstream GUI tasks\. The cart\-to\-SMS case in Appendix[A](https://arxiv.org/html/2607.12625#A1)similarly transfers product names, an order number, and a recipient phone number from shopping to messaging \(Figure[8](https://arxiv.org/html/2607.12625#A1.F8)\)\. These bounded handoffs reduce the history each GUI task must preserve and keep later goals tied to explicit inputs\.

Experience\-memory correction\.Experience memory supplies prior failure lessons as advisory task context before low\-level control begins\. Figure[3](https://arxiv.org/html/2607.12625#S4.F3)contrasts an invite\-link task with and without retrieved memory\. Without memory, GUIClaw remains in Mastodon’s mobile settings and follows a nonproductive path; with memory, it recalls that advanced invite links require the web administration panel and switches to the productive interface\. The retrieved lesson corrects app and route selection without replacing live screen evidence\.

Navigation compression through skills and shortcuts\.Skills compress repeated procedures, while validated shortcuts eliminate navigation when they can reach a task\-relevant page directly\. In Figure[5](https://arxiv.org/html/2607.12625#S5.F5), one validated JD search shortcut lands on the product\-results page, whereas the corresponding Taobao branch requires five ordinary GUI steps\. In Figure[8](https://arxiv.org/html/2607.12625#A1.F8), a validated messaging shortcut opens the SMS compose view with the recipient phone number and message body already populated, leaving GUIClaw to verify and send the message\. These cases visualize the navigation savings measured more broadly on skill\-using tasks in Table[3](https://arxiv.org/html/2607.12625#S5.T3)\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/demo/case1_trajectory.png)Figure 5:KnowAct\-GUIClaw execution of a cross\-app price comparison\. The host supplies the product model from the user profile and routes two GUI tasks\. A validated JD search shortcut lands directly on the results page, whereas Taobao requires five ordinary GUI steps\. The blackboard carries both observed prices into the host’s recommendation, illustrating the step and token savings in Table[3](https://arxiv.org/html/2607.12625#S5.T3)\.Host recovery and direct participation\.The host intervenes when GUI execution fails or returns a value that does not satisfy a downstream contract\. Figure[6](https://arxiv.org/html/2607.12625#S5.F6)\(b\) shows failure\-driven re\-planning: after a message\-by\-name attempt fails because the contact is absent, the host records the failed lookup, recovers the phone number from a resume, and delegates a new Messages GUI task\. Figure[7](https://arxiv.org/html/2607.12625#S5.F7)shows direct tool participation after the Email GUI task returns only the ambiguous hotel name “Harvard Square Hotel\.” The host resolves the full street address through web search before delegating the SMS and Maps GUI tasks, avoiding a longer GUI\-only address lookup; the final Maps task reports a1313\-minute walk\. The two trajectories distinguish host\-side re\-planning from tool\-assisted information resolution while preserving typed subtask boundaries\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/demo/appendix_cases.png)Figure 6:Cases of our workflow with attribution experience\.\(a\)Email\-to\-alarm transfers the observed party time through the blackboard; the host subtracts one hour before the GUI subagent sets the alarm\.\(b\)Failure\-driven re\-planning records a failed contact lookup, recovers the phone number from a resume, and delegates the final Messages GUI task\.![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/case_multi-task_tool_and_gui.png)Figure 7:Host\-mediated recovery in a conference\-location task\. The Email GUI task returns only the hotel name, so the host resolves the full address through web search before delegating the Messages and Maps GUI tasks\. Maps reports a1313\-minute walk\. The workflow combines partial GUI evidence with external tools while preserving typed subtask boundaries\.Table 5:MobileWorld repeated\-attempt upper bound for the A–F configurations of Table[2](https://arxiv.org/html/2607.12625#S5.T2)\. “Any of 3” counts a task solved if at least one of three runs succeeds; “all 3” requires all three\. SR \(pass@1\) is repeated for reference\. This shows our model steadily boosts accuracy without large random volatility\.MetricABCDEFsingle\-run SR \(%\)24\.834\.537\.940\.743\.346\.2pass@3, any of 3 \(%\)34\.247\.951\.351\.359\.059\.8pass@3, all 3 \(%\)15\.418\.826\.530\.829\.132\.5
### 5\.6Cross\-Platform Checks

We further test whether the same KnowAct\-GUIClaw interface remains usable outside the Android evaluation stack\. On HarmonyOS, we run MobileWorld\-derived tasks that are not tied, or only weakly tied, to Android\-specific mirrored app states; weakly tied cases are manually initialized before evaluation, and outcomes are judged withqwen3\.5\-flash\. On Windows, we use a manually designed desktop set to check basic usability across browser, file, office, terminal, and system\-control workflows\.Under this protocol,KnowAct\-GUIClawsolves48/6348/63HarmonyOS tasks \(76\.2%76\.2\\%\) and21/3021/30Windows tasks \(70\.0%70\.0\\%\)\.TableLABEL:tab:cross\-platform\-casesreports ten source\-matched cases from each platform\.

Table 6:Representative cross\-platform usability cases\. HarmonyOS instructions are translated into English when necessary while preserving their source requirements\. Windows instructions name one concrete application wherever the source offered alternatives or referred to a generic application\.InstructionSuccessHarmonyOS mobile checksNext Saturday from 10:00 a\.m\. to 12:30 p\.m\., I will travel to Shanghai Hongqiao Railway Station\. Add a Calendar event named “Business Trip\.” Find attractions within 10 km of the station that I can visit before work on Monday, and place them in the event description as “attraction name: address,” separated by commas\.YesCheck my calendar and send a WeChat message to Su with the dates of my arrival in Shanghai\. The message must contain only the two dates inMM/DD/YYYYformat, separated by a comma\.YesI received a coffee invitation for 3:00 p\.m\. tomorrow\. Check my calendar; if I am available, reply “OK” to Ping An Xi Le in QQ and create the corresponding calendar event\. Otherwise, reply “Not available in this time slot\.”YesPlan exactly one shortest taxi route in Chengdu from Chengdu Shuangliu International Airport Terminal 2 to my hotel at No\. 8, South Section of Chunxi Road, Jinjiang District, visiting exactly Kuanzhai Alley and Jinli Ancient Street in either order\. Send Ping An Xi Le in QQ the names and coordinates of all four locations as “name: longitude, latitude,” the shortest visit order, the three driving distances in meters, and the total driving distance in meters; separate each ordered list with commas\.YesAn error message on my phone contains the keywordTurnOffWifi\. Search thegoogle\-research/android\_worldGitHub repository for related issues\. If similar issues exist, send every issue link, separated by commas, to Ping An Xi Le in QQ; otherwise send “no turnoffwifi issues\.”YesFind the resume file downloaded most recently within the past month in Downloads, and send it to my HR colleague with the subjectcandiaditaes\_cv\.YesTake a selfie and share it with Jimmy via email\.YesIncrease the font size and icons on my phone to the maximum setting\.YesSet a weekend alarm for 8:25 a\.m\. with the ringtone “beebeep” and vibration off\.NoHalloween is approaching\. Use TaoBao to place an order for a set of temporary tattoos, and hand control back to me when the payment page appears\.NoWindows desktop checksOpen the Ctrip website and search for a one\-way flight from Beijing to Tokyo on the Friday after next\. Filter for nonstop flights, sort by price in ascending order, and select the cheapest option\.YesOpen Baidu Maps in Google Chrome and search for “Starbucks near Tiananmen Square\.” Select the highest\-rated result from the left\-hand list, open its details, copy its building or street address, and save the address toaddress\.txton the desktop using Notepad\.YesOpenloop\.pyin Visual Studio Code, add a breakpoint on line 3, open Run and Debug, and start debugging\.YesUse Google Chrome to find the current Apple \(AAPL\) share price, calculate the value of 100 shares, and save the total tostock\.txton the desktop using Notepad\.YesOpen Windows Settings, retrieve the detailed operating\-system version, and save it in a new draft email in Microsoft Outlook\.YesOpen the PDF report in Documents with Microsoft Edge, translate the first paragraph of Chapter 1 into English using Google Translate, and save the translation as a plain\-text file on the desktop using Notepad\.YesOpen GitHub in Google Chrome, search for thenanobotproject, copy its latest commit hash, and save it tocommit\_record\.txton the desktop using Notepad\.YesOpen Microsoft PowerPoint, create a blank slide, insert a local image, and apply a “Fly In” animation to the image\.YesI will send a test message in Slack within the next minute\. Keep the desktop idle; when the Slack notification appears, click it immediately, open the conversation, and reply “Received\.”NoOpen the locally installed WeChat, open File Transfer Assistant, send the text “Test emoji,” and then use the built\-in emoji panel to send a “laughing” emoji\.NoHarmonyOS failure modes\.The failed HarmonyOS cases expose both low\-level control difficulties and state\-semantic mismatches\. In the weekend\-alarm case, the minute field uses an inertial wheel picker\. GUIClaw repeatedly swipes within a nearby range but cannot settle on 25 minutes, so it exhausts the step budget without setting 8:25 a\.m\. In the temporary\-tattoo case, the agent stops on the order\-confirmation screen before submitting the order, rather than handing control back after reaching the subsequent payment page\. This premature termination reflects application\-level UI diversity: after a payment method is selected, TaoBao presents a “Pay Now” button, which the model misinterprets as evidence that the current screen is already the payment page; in fact, the payment page appears only after that button is pressed\. The full set also shows system\-UI mismatches: several quick\-setting tasks repeatedly open the left notification pane even though the relevant HarmonyOS controls require a swipe from the right\.

Windows failure modes\.Windows failures expose three distinct grounding bottlenecks\. First, the WeChat case reaches File Transfer Assistant and sends the text correctly, but selects a different face from the built\-in panel instead of the requested laughing emoji\. This error reveals a limitation in mapping a natural\-language affect label to the corresponding visually similar, unlabeled icon\. Second, the notification case requires sustained observation followed by an immediate click on a short\-lived toast, exposing weak temporal grounding\. The remaining failures primarily involve continuous geometric control, deep application\-specific controls, and targets outside the current viewport, as seen in free\-form canvas manipulation, window arrangement, and bottom\-of\-page actions\. Thus, desktop reliability depends not only on pointer precision and timing, but also on semantic grounding between language and graphical symbols\.

### 5\.7Reproducibility

MobileWorld results use the benchmark’s deterministic evaluators on the 117 GUI\-Only tasks at a 50\-step cap\. We log GUI\-executor traces and host calls separately, so that the GUI\-trace total and host total columns of Table[2](https://arxiv.org/html/2607.12625#S5.T2)can be reproduced independently\. AndroidDaily scoring \(qwen3\.5\-flashfor GUI\-only tasks, two human experts under the1\.0/0\.5/01\.0/0\.5/0rule for answer\-returning tasks\) and the resolved/all splits follow the protocol described above\. You can see the experimental logs at[https://github\.com/HITsz\-TMG/KnowAct/releases/tag/Result](https://github.com/HITsz-TMG/KnowAct/releases/tag/Result)\.

## 6Conclusion

This work proposes KnowAct\-GUIClaw, a cross\-platform GUI agent framework built on the “Know Deeply, Act Perfectly" paradigm to address two core limitations of mainstream OpenClaw\-style agent systems: insufficient cross\-device GUI interaction capacity and the absence of native self\-evolution mechanisms\. The framework instantiates a four\-stage Know–Route–Act–Reflect closed\-loop pipeline that decouples high\-level task orchestration \(host agent\) from low\-level visual device manipulation \(GUI subagent\)\. It introduces two core reusable storage modules: attribution\-aware persistent experience memory and state\-validated self\-evolving skill libraries, alongside a typed blackboard information transfer protocol to standardize cross\-application data flow and eliminate fuzzy free\-text context loss\. Future work targets tighter native integration of the Knowledge module, external general\-purpose tools, and the GUI subagent to eliminate rigid sequential pipeline handoffs\. We will design a unified joint planner to concurrently weigh knowledge retrieval, non\-visual tool calls and GUI actions within one decision cycle and cut host\-subagent communication overhead\.

## 7Contributors

Core Contributors

Yunxin Li, Jinchao Li, Baotian Hu, Min Zhang

Contributors

Shibo Su, Zhenran Xu, Chenrui Zhao, Tongshu Bian, Xiaoman Liang, Meishan Zhang

Corresponding Author

Baotian Hu

Harbin Institute of Technology, Shenzhen; Shenzhen Loop Area Institute

Email: hubaotian@hit\.edu\.cn

## References

## Appendix ASupplementary Case Studies

Figure[8](https://arxiv.org/html/2607.12625#A1.F8)supplements the routing and navigation\-compression cases in Section[5\.5](https://arxiv.org/html/2607.12625#S5.SS5)\. The first GUI task extracts the product names, order number, and recipient phone number from TaoDian as explicit outputs\. A validated messaging shortcut then opens the SMS compose view with the recipient and message body populated, leaving GUIClaw to verify and send the message\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/case_multi-task_cart_sms.png)Figure 8:Cart\-to\-SMS cross\-app execution\. The TaoDian GUI task extracts the product names, order number, and recipient phone number and transfers them to the downstream messaging task\. A validated messaging shortcut opens the SMS compose view with the recipient and message body already populated, illustrating both blackboard information transfer and navigation compression\.Figure[9](https://arxiv.org/html/2607.12625#A1.F9)provides an annotated Chinese companion for the host\-mediated recovery case in Figure[7](https://arxiv.org/html/2607.12625#S5.F7)\. It shows the same control flow in which an underspecified GUI output is routed back to the host, grounded through web search, and then consumed by downstream Messages and Maps GUI tasks\.

![Refer to caption](https://arxiv.org/html/2607.12625v1/figures/case_multi-task_tool_and_gui_cn.png)Figure 9:Chinese annotated companion for the host\-mediated recovery case in Figure[7](https://arxiv.org/html/2607.12625#S5.F7)\. The panel highlights the same control flow in which an underspecified GUI output is routed back to the host, grounded through web search, and then consumed by downstream Messages and Maps GUI tasks\.
## Appendix BAction Space

Table[7](https://arxiv.org/html/2607.12625#A2.T7)gives the unified action space GUIClaw exposes to the GUI executor, realizing𝒜gui∪𝒜ask\\mathcal\{A\}\_\{\\mathrm\{gui\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{ask\}\}from Section[4\.4](https://arxiv.org/html/2607.12625#S4.SS4)\. The two platforms share a common core of pointer, text, navigation, and task\-control actions; desktop adds key combinations, an explicit pointer swipe, and application launch and close, while mobile adds a dedicated enter key\. This largely shared action surface is what lets a single host drive both platforms through the same GUI task interface\.

Table 7:Unified action space of GUIClaw across platforms\.Sharedactions are available on both mobile and desktop;DesktopandMobilelist the platform\-specific additions\. Together they realize𝒜gui∪𝒜ask\\mathcal\{A\}\_\{\\mathrm\{gui\}\}\\cup\\mathcal\{A\}\_\{\\mathrm\{ask\}\}\(Section[4\.4](https://arxiv.org/html/2607.12625#S4.SS4)\); skills \(𝒜skill\\mathcal\{A\}\_\{\\mathrm\{skill\}\}\), Android deeplink/intent shortcuts \(𝒜shortcut\\mathcal\{A\}\_\{\\mathrm\{shortcut\}\}\), and external Model Context Protocol \(MCP\) tool calls extend this set through the runtime\.EnvironmentActionDefinitionSharedClick\(x,yx,y\)Clicks or taps at coordinates\(x,y\)\(x,y\)\.DoubleTap\(x,yx,y\)Double\-clicks or double\-taps at\(x,y\)\(x,y\)\.LongPress\(x,yx,y\)Long\-presses at\(x,y\)\(x,y\)\.Drag\(x1,y1,x2,y2x\_\{1\},y\_\{1\},x\_\{2\},y\_\{2\}\)Drags from\(x1,y1\)\(x\_\{1\},y\_\{1\}\)to\(x2,y2\)\(x\_\{2\},y\_\{2\}\)\.Scroll\(x,yx,y, direction\)Scrolls at\(x,y\)\(x,y\)in the given direction\.Type\(content\)Types the specified content\.Back\(\)Returns to the previous screen\.Home\(\)Returns to the home screen\.Wait\(\)Pauses for a brief moment\.Finished\(\)Returns the final answer or marks the task complete or infeasible\.CallUser\(\)Requests user intervention\.DesktopHotkey\(key\)Presses the specified key combination\.Swipe\(x1,y1,x2,y2x\_\{1\},y\_\{1\},x\_\{2\},y\_\{2\}\)Swipes the pointer from\(x1,y1\)\(x\_\{1\},y\_\{1\}\)to\(x2,y2\)\(x\_\{2\},y\_\{2\}\)\.OpenApp\(name\)Launches the specified application\.CloseApp\(name\)Closes the specified application\.MobilePressEnter\(\)Presses the “enter” key\.
## Appendix CSkill Extraction and Prompt Contracts

KnowAct\-GUIClawuses prompt contracts at both execution time and reflection time\. The execution prompt below is the runtime MobileWorld\-style template used for prompt\-side skill selection\(kong2025mobileworldbenchmarkingautonomousmobile\): GUIClaw retrieves top\-kkrelevant skills, formats them as theCompact skillscatalog, and lets the GUI executor choose a listed skill or continue with ordinary actions\. Reflection then turns GUI experience into reusable knowledge through additional contracts rather than unrestricted trace replay\. First, trajectory evidence is normalized into a task, an app scope, ordered actions, screenshots, target hints, and state contracts\. Second, a skill\-extraction agent compresses this evidence into one reusable, parameterized procedure and rejects outputs that are not executable or not state guarded\. Third, a separate evolution agent repairs a previously reused skill only when a failure trace identifies that skill as the failure source\. Finally, a memory\-induction agent writes textual success and failure lessons that guide future routing and execution without becoming executable actions\. The boxes below reproduce the runtime source prompts without shortening; braces denote runtime placeholders and structured inputs supplied by the caller\.

`\(a\) Execution Prompt \(b\) Skill extraction prompt \(c\) Skill evolution prompt \(d\) Shortcut validation prompt \(e\) Experience\-memory induction prompt`

`Appendix D Skill and Shortcut Examples The three boxes below show excerpts from KnowAct\-GUIClaw’s Android skill and shortcut store, abbreviated for readability: timestamps and some keyword arguments are dropped, 64\-character state\-contract hashes and long component names are truncated, and one Chinese accessibility label is glossed in ASCII\. They illustrate the three forms covered by the single skill abstraction \(Section 4\.4\): \(a\) a parameterized click\-then\-type pattern in 𝒜skill\\mathcal\{A\}\_\{\\mathrm\{skill\}\}; \(b\) a multi\-step skill whose steps carry an expected state and, where available, a structural state contract checked before each step; and \(c\) validated deeplink and intent shortcuts in 𝒜shortcut\\mathcal\{A\}\_\{\\mathrm\{shortcut\}\}\. Stable values, such as package names, button coordinates, and intent components, are fixed in advance, while request\-dependent values, such as the query string, are supplied at run time\. The validated tag marks shortcut candidates that passed on\-device validation rather than static manifest discovery alone\. \(a\) Parameterized pattern: click\-then\-type \(b\) Multi\-step skill with per\-step state validation \(Pinduoduo\) \(c\) Validated deeplink and intent shortcuts \(JD\) These entries populate the skill and shortcut store that Know retrieves and Act applies during GUI task execution\.`