A Task-State Representation for Long-Horizon Mobile GUI Agents

# A Task-State Representation for Long-Horizon Mobile GUI Agents
Source: [https://arxiv.org/html/2607.00502](https://arxiv.org/html/2607.00502)
Yujie Zheng2∗,Zikang Liu1∗,Xin Zhao1†,Ji\-Rong Wen1 1Gaoling School of Artificial Intelligence, Renmin University of China 2School of Software, Beihang University \{janeyujie555,jasonlaw8121,batmanfly\}@gmail\.com, jrwen@ruc\.edu\.cn

###### Abstract

While long\-horizon mobile GUI agents typically rely on thought\-action\-observation loops, they struggle to separate persistent task states from transient screen observations\. As execution histories grow, this entanglement imposes a severe context burden, causing agents to forget initial requirements, hallucinate progress, or repeatedly interact with stale interfaces\. To address this, we introduce Task\-State Representation \(TSR\)—a training\-free framework that explicitly decouples task state from sensory input\. Acting as a lightweight external wrapper, TSR maintains three structured components: a global instruction summary, a dynamic progress tracker for subgoals, and a transition\-aware action verifier\. By continuously updating through pre\- and post\-action visual comparisons, TSR effectively guides the agent’s reasoning without requiring architectural modifications\. Experiments across four mobile GUI benchmarks validate TSR’s effectiveness, yielding up to a 12 absolute point increase in success rate on complex cross\-application and memory\-intensive tasks\.

A Task\-State Representation for Long\-Horizon Mobile GUI Agents

Yujie Zheng2∗, Zikang Liu1∗, Xin Zhao1†, Ji\-Rong Wen11Gaoling School of Artificial Intelligence, Renmin University of China2School of Software, Beihang University\{janeyujie555,jasonlaw8121,batmanfly\}@gmail\.com, jrwen@ruc\.edu\.cn

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\.## 1Introduction

Automating mobile tasks via graphical user interfaces \(GUIs\) remains a long\-standing goal in the development of intelligent agents\. Recent multimodal large language models \(MLLMs\)Liuet al\.\([2023](https://arxiv.org/html/2607.00502#bib.bib23)\)have facilitated prompt\-based GUI actors that observe screenshots, reason about the current state, and generate executable actions, such as tapping, typing, or scrolling\(Zhanget al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib10); Honget al\.,[2024](https://arxiv.org/html/2607.00502#bib.bib11)\)\. The dominant paradigm for handling long\-horizon tasks adopts a thought\-action\-observationYaoet al\.\([2022](https://arxiv.org/html/2607.00502#bib.bib1)\)loop: at each step, the actor receives the task instruction, a window of recent screenshots, and a history of previous reasoning and actions, and subsequently generates a new reasoning trace followed by an action\(Zhanget al\.,[2024](https://arxiv.org/html/2607.00502#bib.bib17); Rawleset al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib4)\)\. This append\-all design relies on an implicit assumption—that the actor can reliably maintain awareness of the overall task goal and the cumulative progress from an ever\-growing raw trajectory\.

In practice, this assumption breaks down as the steps increases\. We identify three recurring failure modes in long\-horizon mobile benchmarks: \(1\)*goal drift*, where the actor gradually loses sight of the original task after observing numerous intermediate screens; \(2\)*progress hallucination*, where the actor lose access to earlier visual observations and fabricates past states when reasoning about cumulative progress; and \(3\)*stale\-screen repetition*, where the actor misinterprets a delayed update in the user interface as a failed action and enters a localized loop\.

These failures are not solely attributable to limitations in the model’s capacity\. Rather, they arise from a structural deficiency in how the input of the actor is organized: the standard prompt conflates two fundamentally different categories of information—*persistent task state*\(the request of the user, the accomplished subgoals, and the remaining steps\) and*transient observation state*\(the content displayed on the current screen\)\. Without an explicit mechanism to separate and maintain the former, the actor must re\-derive the task progress from the raw history at every step—a burden that scales linearly with the length of the trajectory\.

We propose atask\-state representationthat addresses this separation\. The representation is maintained externally to a fixed GUI actor and is updated at each step by a training\-free state updater that compares the pre\-action and post\-action screenshots\. It comprises three functional views: a*global task\-state summary*that preserves the original instruction, a*progress tracker*that records the completed and remaining subgoals, and a*transition\-aware focus*that verifies the effectiveness of the preceding action and guides the next decision\. The resulting state block is serialized and injected into the prompt of the actor, requiring no model retraining and architectural modification\.

We evaluate two base models across four mobile GUI benchmarks: MobileWorldKonget al\.\([2025](https://arxiv.org/html/2607.00502#bib.bib5)\), AndroidWorldRawleset al\.\([2025](https://arxiv.org/html/2607.00502#bib.bib4)\), MemGUI\-BenchLiuet al\.\([2026](https://arxiv.org/html/2607.00502#bib.bib6)\), and VenusBench\-MobileGonget al\.\([2026](https://arxiv.org/html/2607.00502#bib.bib7)\)\. Our approach improves success rates in most configurations, with gains of up to 12% on long\-horizon tasks\. Our contributions are:

- •A task\-state representation that separates persistent task state from transient observations for long\-horizon mobile GUI agents\.
- •Empirical evaluation showing consistent improvements across four benchmarks and two base models, with ablations suggesting synergistic necessity of all three state views\.
- •Deeper analysis on the effect of structured task\-state, identifying task horizon and state\-tracking demand as key moderating factors\.

## 2Method

![Refer to caption](https://arxiv.org/html/2607.00502v1/x1.png)Figure 1:Overview of the task\-state representation\. At each step, the state updater compares pre\- and post\-action screenshots and updates a structured state block which is then injected into the actor’s input\.### 2\.1Problem Formulation

We formulate mobile GUI automation as a sequential decision\-making problem\. Given a task instructionIIand an initial screen observationo0o\_\{0\}, a GUI actorπθ\\pi\_\{\\theta\}repeatedly observes the current screen and generates a responsert=\(Tt,at\)r\_\{t\}=\(T\_\{t\},a\_\{t\}\), whereTtT\_\{t\}denotes a reasoning trace andata\_\{t\}represents an executable action drawn from a predefined action space𝒜\\mathcal\{A\}\(detailed in Appendix[B](https://arxiv.org/html/2607.00502#A2)\)\. Following the execution ofata\_\{t\}, the environment transitions to a new screen stateot\+1o\_\{t\+1\}, and the rollout continues until task completion or a predefined step limit\.

To reduce context overhead, existing systems retain only themmmost recent screenshots alongside the full text history\(Liet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib9)\)\. Formally, at steptt, the actor generates:

at=πθ​\(I,Ot,Ht−1\),a\_\{t\}=\\pi\_\{\\theta\}\(I,\\;O\_\{t\},\\;H\_\{t\-1\}\),whereOt=\{ot−m\+1,…,ot\}O\_\{t\}=\\\{o\_\{t\-m\+1\},\\ldots,o\_\{t\}\\\}denotes the recent observation window andHt−1=\{\(T1,a1\),…,\(Tt−1,at−1\)\}H\_\{t\-1\}=\\\{\(T\_\{1\},a\_\{1\}\),\\ldots,\(T\_\{t\-1\},a\_\{t\-1\}\)\\\}is the interaction history comprising all prior reasonings and actions\. This design preserves local visual continuity but discards earlier observations\. Consequently, the actor must infer task progress from past responses, a growing burden in long trajectories that leads to the potential failures\.

### 2\.2Task\-State Representation

To address these limitations, we introduce an externally maintained task\-state representationStS\_\{t\}, which is updated at each step and injected into the actor’s input\. The state comprises three functional views, which are detailed below\. Rather than operating as separate modules, these views are fields within a unified state object and are updated jointly at each step\.

#### Global Task\-State Summary\.

When early visual contexts are truncated, the actor is prone to losing sight of the initial task goal\. To counteract this intent decay, we design a summary module to record the persistent semantics of the task along with the cumulative progress\. The primary role of this summary is to maintain the visibility of the original task instruction, even when recent screenshots display only a narrow slice of the interface\. This mechanism prevents the actor from drifting away from the intent of the user during long sequences of interaction\.

#### Progress Tracker\.

During complex long\-horizon tasks, the agent often loses track of their overarching goals, creating severe ambiguity in evaluating current progress\. To resolve this ambiguity, we design a progress tracker that decomposes the task into atomic requirements and tracks the execution status of each\. It maintains four fields: task decomposition, completed milestones, current subgoal, and remaining requirements\. By making verified progress and unresolved steps explicit, the tracker mitigates the risk of the actor hallucinating earlier observations or terminating prematurely\.

#### Transition\-Aware Focus\.

Furthermore, sinceHt−1H\_\{t\-1\}reflects past intentions rather than environmental feedback, the actor risks repetitive loops during system delays or failures\. To close this loop, the transition view evaluates the previous action by comparing observationsot−1o\_\{t\-1\}andoto\_\{t\}\. If the outcome is uncertain, the representation prevents blind repetition by generating a*next\-action focus*—such as verifying the state or refreshing a list—to guide the actor’s next decision\.

### 2\.3State Update and Actor Injection

At the beginning of a task, the initial stateS0S\_\{0\}is derived solely from the task instructionII\. At each subsequent steptt, following the execution of actionat−1a\_\{t\-1\}by the actor and the generation of observationoto\_\{t\}by the environment, the state updater𝒰ϕ\\mathcal\{U\}\_\{\\phi\}receives the task instruction, the previous state, the previous response of the actor, and the screenshots captured before and after the action\. It then produces the updated state through a single function call:

St=𝒰ϕ​\(I,St−1,rt−1,ot−1,ot\)\.S\_\{t\}=\\mathcal\{U\}\_\{\\phi\}\(I,\\;S\_\{t\-1\},\\;r\_\{t\-1\},\\;o\_\{t\-1\},\\;o\_\{t\}\)\.The updater𝒰ϕ\\mathcal\{U\}\_\{\\phi\}is implemented via a prompted LLM that outputs a structured JSON object encompassing all three views\. Subsequently, the actor receives the standard context augmented with a serialized rendering of the task state:

at=πθ​\(I,Ot,Ht−1,ℐ​\(St\)\),a\_\{t\}=\\pi\_\{\\theta\}\(I,\\;O\_\{t\},\\;H\_\{t\-1\},\\;\\mathcal\{I\}\(S\_\{t\}\)\),whereℐ​\(St\)\\mathcal\{I\}\(S\_\{t\}\)denotes an injection function that formats the three state views into a text block appended to the prompt of the actor\. Crucially,πθ\\pi\_\{\\theta\}remains*fixed*throughout, so that the entire mechanism operates externally at inference time without any training or architectural modifications\.

## 3Experiments

### 3\.1Experimental Setup

We evaluate our approach on four online GUI benchmarks, including MobileWorld\(Konget al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib5)\), AndroidWorld\(Rawleset al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib4)\), MemGUI\-Bench\(Liuet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib6)\)and VenusBench\-Mobile\(Gonget al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib7)\)\. We implement the standard GUI actor based on previous studiesKonget al\.\([2025](https://arxiv.org/html/2607.00502#bib.bib5)\)as baseline\. More details are presented in Appendix[A](https://arxiv.org/html/2607.00502#A1)\.

### 3\.2Main Results

Table[1](https://arxiv.org/html/2607.00502#S3.T1)reports success rates and average steps across all settings\. We list our findings below\.

Table 1:Performance comparison of baseline and our proposed task\-state representation\.First, the most pronounced improvements appear on MobileWorld \(\+12% for Qwen3\.5\-plus, \+9% for Kimi\-k2\.5\), the benchmark with the longest average trajectories and cross\-application dependencies\. This is consistent with the hypothesis that explicit progress tracking is most beneficial when the agent navigate extended interaction sequences\.

Besides, on memory\-intensive and user\-centric benchmarks \(MemGUI\-Memory, VenusBench\-Mobile\), both models benefit \(\+3\.48% to \+5\.22%\)\. However, the representation is not universally positive: Qwen3\.5\-plus degrades by 3\.45% on AndroidWorld while Kimi\-k2\.5 improves by 3\.45% on the same benchmark, suggesting that the utility of structured task\-state tracking depends on the base model’s inherent planning capability and the task complexity distribution\.

Third, higher success rates do not always require more steps\. In several cases \(Kimi\-k2\.5 on VenusBench\-Mobile: 41\.2→\\to38\.3 steps; Qwen3\.5\-plus on MemGUI\-Memory: 65\.4→\\to64\.5 steps\), the representation enables more direct task completion\. Conversely, on MobileWorld the step count increases alongside SR, indicating that the representation helps agents persist through complex tasks they would otherwise abandon\.

### 3\.3Ablation Study

We ablate on Qwen3\.5\-plus by separately removing the task\-state summary, progress tracker, and transition\-aware focus from the representation while still updating the internal state\. Results are shown in Table[2](https://arxiv.org/html/2607.00502#S3.T2)\.

Table 2:Ablation on Qwen3\.5\-plus \(Success Rate, %\)\.Firstly, we discover that on MobileWorld, the full representation outperforms every ablated variant by 5–7%, suggesting that the three components function synergistically\. Among individual removals, dropping the transition\-aware focus causes the largest degradation \(55→\\to48\), suggesting that verifying action effectiveness is the most critical signal for long\-horizon tasks\. However, the pattern reverses on AndroidWorld\. Specifically, removing the transition module improves over the baseline \(63\.79 vs\. 61\.21\), while the full representation hurts\. This indicates that for shorter, single\-application tasks, transition verification introduces unnecessary decision noise\. The task\-state representation is therefore best viewed as conditionally beneficial—most valuable when tasks demand persistent state tracking across many steps\.

### 3\.4Case Study

![Refer to caption](https://arxiv.org/html/2607.00502v1/x2.png)Figure 2:Stale\-screen recovery\. The baseline repeats the same removal action against a delayed UI update\. The task\-state representation detects the uncertain transition and redirects the actor to verify before retrying\.![Refer to caption](https://arxiv.org/html/2607.00502v1/x3.png)Figure 3:Over\-decomposition failure\. The state updater decomposes the task too aggressively, causing the actor to inspect items individually rather than using global visual inference, exceeding the step budget\.Figure[2](https://arxiv.org/html/2607.00502#S3.F2)illustrates a bookmark\-management task where the baseline enters a repetition loop after a delayed UI update; the transition\-aware focus detects the uncertain effect and guides the actor to verify before retrying\. Figure[3](https://arxiv.org/html/2607.00502#S3.F3)shows a failure mode: when the updater decomposes a task too aggressively, the actor inspects items individually instead of leveraging global visual inference, exceeding the step budget\. The task\-state representation should therefore be treated as guidance rather than an override of current\-screen evidence\.

## 4Related Work

#### Mobile GUI Agents\.

Recent multimodal LLMsLiuet al\.\([2023](https://arxiv.org/html/2607.00502#bib.bib23),[2025b](https://arxiv.org/html/2607.00502#bib.bib24)\)have enabled prompt\-based GUI actors that translate instructions into touch actions on mobile screens\(Zhanget al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib10); Honget al\.,[2024](https://arxiv.org/html/2607.00502#bib.bib11)\)\. Subsequent work improves grounding accuracy\(Chenget al\.,[2024](https://arxiv.org/html/2607.00502#bib.bib12)\)and scales toward long\-horizon tasks via hierarchical planning\(Luet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib8)\), history compression\(Liet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib9)\)or active loadingLiuet al\.\([2025a](https://arxiv.org/html/2607.00502#bib.bib25)\)\. However, existing methods often require retraining, limiting their use as general wrappers\. To address this, we propose a training\-free task\-state representation that enriches prompts without altering the base agent\.

#### Long\-Horizon Context for LLM Agents\.

Managing growing interaction trajectories is a central challenge for LLM agents\. Existing approaches primarily address this through trajectory compression: HiAgent\(Huet al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib2)\)chunks working memory by subgoals, IterResearch\(Chenet al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib3)\)periodically reconstructs a condensed workspace, and summarization\-based methods learn to shorten history via RL\(Luet al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib15)\)\. These methods treat the trajectory as a monolithic stream to be shortened; in contrast, we maintain a separate structured representation alongside the actor’s unmodified observation history\.

## 5Conclusion

In this paper, we propose a task\-state representation that decouples persistent task state from transient screen observations to counter goal drift, progress hallucination, and stale\-screen repetition for long\-horizon mobile GUI agents\. Our approach augments a fixed actor with three externally maintained views: a global task\-state summary, a progress tracker, and a transition\-aware focus, without any architectural modification\. Evaluated across four benchmarks, our method improves success rates by up to 12 % on long\-horizon cross\-app tasks and yields consistent 3–5 % gains on memory\-intensive tasks, validating the effectiveness of our approach\.

## Limitations

The state updater can propagate errors: if it incorrectly marks a subgoal as completed or omits a remaining requirement, the actor conditions subsequent decisions on a flawed state, potentially compounding mistakes across steps\. Additionally, the updater introduces one extra LLM call per step \(1024 tokens\), increasing both latency and cost; for short tasks where the baseline already succeeds, this overhead provides no benefit and may degrade performance through over\-decomposition\. All experiments are conducted on Android\-based mobile benchmarks; whether the representation generalizes to desktop or web GUI environments remains untested\. Finally, the current design updates all three state views jointly—a more selective mechanism that activates views only when needed could reduce noise on simpler tasks\.

## References

- G\. Chen, Z\. Qiao, X\. Chen, D\. Yu, H\. Xu, W\. X\. Zhao, R\. Song, W\. Yin, H\. Yin, L\. Zhang,et al\.\(2025\)IterResearch: rethinking long\-horizon agents with interaction scaling\.arXiv preprint arXiv:2511\.07327\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px2.p1.1)\.
- Seeclick: harnessing gui grounding for advanced visual gui agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9313–9332\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Gong, Z\. Cai, S\. Dai, Y\. Zhou, Z\. Gu, C\. Meng, and S\. Shen \(2026\)VenusBench\-mobile: a challenging and user\-centric benchmark for mobile gui agents with capability diagnostics\.arXiv preprint arXiv:2604\.06182\.Cited by:[Appendix A](https://arxiv.org/html/2607.00502#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.00502#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.00502#S3.SS1.p1.1)\.
- W\. Hong, W\. Wang, Q\. Lv, J\. Xu, W\. Yu, J\. Ji, Y\. Wang, Z\. Wang, Y\. Dong, M\. Ding,et al\.\(2024\)Cogagent: a visual language model for gui agents\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 14281–14290\.Cited by:[§1](https://arxiv.org/html/2607.00502#S1.p1.1),[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- M\. Hu, T\. Chen, Q\. Chen, Y\. Mu, W\. Shao, and P\. Luo \(2025\)Hiagent: hierarchical working memory management for solving long\-horizon agent tasks with large language model\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 32779–32798\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px2.p1.1)\.
- Q\. Kong, X\. Zhang, Z\. Yang, N\. Gao, C\. Liu, P\. Tong, C\. Cai, H\. Zhou, J\. Zhang, L\. Chen,et al\.\(2025\)MobileWorld: benchmarking autonomous mobile agents in agent\-user interactive and mcp\-augmented environments\.arXiv preprint arXiv:2512\.19432\.Cited by:[Appendix A](https://arxiv.org/html/2607.00502#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.00502#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.00502#S3.SS1.p1.1)\.
- D\. Li, Z\. Pan, Z\. Zhang, R\. Chen, H\. Wang, H\. Chen, and H\. Jiang \(2026\)Rethinking token pruning for historical screenshots in gui visual agents: semantic, spatial, and temporal perspectives\.arXiv preprint arXiv:2603\.26041\.Cited by:[§2\.1](https://arxiv.org/html/2607.00502#S2.SS1.p2.2),[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- G\. Liu, P\. Zhao, Y\. Liang, Q\. Luo, S\. Tang, Y\. Chai, W\. Lin, H\. Xiao, W\. Wang, S\. Chen,et al\.\(2026\)MemGUI\-bench: benchmarking memory of mobile gui agents in dynamic environments\.arXiv preprint arXiv:2602\.06075\.Cited by:[Appendix A](https://arxiv.org/html/2607.00502#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.00502#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.00502#S3.SS1.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.Advances in neural information processing systems36,pp\. 34892–34916\.Cited by:[§1](https://arxiv.org/html/2607.00502#S1.p1.1),[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Liu, J\. Li, W\. X\. Zhao, D\. Gao, Y\. Li, and J\. Wen \(2025a\)PAL\-ui: planning with active look\-back for vision\-based gui agents\.arXiv preprint arXiv:2510\.00413\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- Z\. Liu, K\. Zhou, W\. X\. Zhao, D\. Gao, Y\. Li, and J\. Wen \(2025b\)Less is more: high\-value data selection for visual instruction tuning\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 3712–3721\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- M\. Lu, W\. Sun, W\. Du, Z\. Ling, X\. Yao, K\. Liu, and J\. Chen \(2025\)Scaling llm multi\-turn rl with end\-to\-end summarization\-based context management\.arXiv preprint arXiv:2510\.06727\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px2.p1.1)\.
- Z\. Lu, F\. Tang, G\. Liu, K\. Song, X\. Tan, J\. Ma, W\. Zhang, W\. Lu, J\. Xiao, Y\. Zhuang,et al\.\(2026\)UI\-copilot: advancing long\-horizon gui automation via tool\-integrated policy optimization\.arXiv preprint arXiv:2604\.13822\.Cited by:[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- C\. Rawles, S\. Clinckemaillie, Y\. Chang, J\. Waltz, G\. Lau, M\. Fair, A\. Li, W\. Bishop, W\. Li, F\. Campbell\-Ajala,et al\.\(2025\)Androidworld: a dynamic benchmarking environment for autonomous agents\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 406–441\.Cited by:[Appendix A](https://arxiv.org/html/2607.00502#A1.SS0.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2607.00502#S1.p1.1),[§1](https://arxiv.org/html/2607.00502#S1.p5.1),[§3\.1](https://arxiv.org/html/2607.00502#S3.SS1.p1.1)\.
- K\. Team, T\. Bai, Y\. Bai, Y\. Bao, S\. Cai, Y\. Cao, Y\. Charles, H\. Che, C\. Chen, G\. Chen,et al\.\(2026\)Kimi k2\. 5: visual agentic intelligence\.arXiv preprint arXiv:2602\.02276\.Cited by:[Appendix A](https://arxiv.org/html/2607.00502#A1.SS0.SSS0.Px3.p1.1),[Table 4](https://arxiv.org/html/2607.00502#A3.T4)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[Appendix A](https://arxiv.org/html/2607.00502#A1.SS0.SSS0.Px3.p1.1),[Table 4](https://arxiv.org/html/2607.00502#A3.T4)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2022\)React: synergizing reasoning and acting in language models\.arXiv preprint arXiv:2210\.03629\.Cited by:[§1](https://arxiv.org/html/2607.00502#S1.p1.1)\.
- C\. Zhang, Z\. Yang, J\. Liu, Y\. Li, Y\. Han, X\. Chen, Z\. Huang, B\. Fu, and G\. Yu \(2025\)Appagent: multimodal agents as smartphone users\.InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems,pp\. 1–20\.Cited by:[§1](https://arxiv.org/html/2607.00502#S1.p1.1),[§4](https://arxiv.org/html/2607.00502#S4.SS0.SSS0.Px1.p1.1)\.
- J\. Zhang, J\. Wu, T\. Yihua, M\. Liao, N\. Xu, X\. Xiao, Z\. Wei, and D\. Tang \(2024\)Android in the zoo: chain\-of\-action\-thought for gui agents\.InFindings of the Association for Computational Linguistics: EMNLP 2024,pp\. 12016–12031\.Cited by:[§1](https://arxiv.org/html/2607.00502#S1.p1.1)\.

## Appendix AExperimental Details

#### Datasets\.

We evaluate on four benchmarks spanning different difficulty axes\.MobileWorld\(Konget al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib5)\): long\-horizon, cross\-application workflows; we exclude agent\-user interaction and MCP\-augmented tasks, retaining 100 GUI\-only tasks\.AndroidWorld\(Rawleset al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib4)\): reproducible Android tasks with programmatic reward verification\.MemGUI\-Bench\(Liuet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib6)\): 115 memory\-intensive tasks requiring cross\-temporal and cross\-spatial recall \(denoted MemGUI\-Memory for the memory\-specific subset\)\.VenusBench\-Mobile\(Gonget al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib7)\): realistic user\-centric tasks with capability diagnostics; we exclude unsupported tasks, leaving 118 tasks\.

#### Baselines\.

Our baseline is the standard MobileWorld actor, which instantiatesπθ\\pi\_\{\\theta\}as defined in Section 2\.1\. We compare this baseline against the same actor augmented with the task\-state representationStS\_\{t\}\.

#### Implementation Details\.

We use two multimodal LLMs as the underlying model for both the actor and the state updater: Qwen3\.5\-plus\(Yanget al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib21)\)and Kimi\-k2\.5\(Teamet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib22)\), accessed via Bailian API with temperature 0\.

## Appendix BAction Space Definition

The action space𝒜\\mathcal\{A\}differs between benchmarks\. Table[3](https://arxiv.org/html/2607.00502#A2.T3)lists all available actions\.

Table 3:Action space𝒜\\mathcal\{A\}\. AW = AndroidWorld only\. Coordinates are normalized to\[0,S\]\[0,S\]whereSSis a model\-specific scale factor \(1000 for Qwen3\.5\-plus, 1 for Kimi\-k2\.5\)\.scrollmoves content;swipeperforms system\-level finger gestures\.answerandstatusterminate the episode\.
## Appendix CLLM Configuration

Table 4:Representation and inference configurations\. Both components use the same underlying model \(Qwen3\.5\-plus\(Yanget al\.,[2025](https://arxiv.org/html/2607.00502#bib.bib21)\)or Kimi\-k2\.5\(Teamet al\.,[2026](https://arxiv.org/html/2607.00502#bib.bib22)\)\) accessed via provider APIs\.The maximum number of interaction steps per task is set to 50 for MobileWorld; for AndroidWorld, MemGUI\-Bench, and VenusBench\-Mobile, we use the benchmark\-specific preset limits defined by each environment\. The state updater is invoked at every step regardless of the benchmark\.

## Appendix DActor Prompt Template

The actor receives a structured prompt comprising a role definition, action framework, execution principles, the task instruction, and the injected task\-state representationℐ​\(St\)\\mathcal\{I\}\(S\_\{t\}\)\. The template below shows the MobileWorld variant; AndroidWorld omitsopen\_appandswipe\. Coordinates are normalized to\[0,S\]\[0,S\]whereSSis the model\-specific scale factor\.

```
# Role: Android Phone Operator AI
You are an AI that controls an Android
phone to complete user requests.

# Action Framework
Respond with EXACT JSON format for one
of these actions:
| Action       | JSON Format             |
|--------------|-------------------------|
| click        | {"action_type":"click", |
|              |  "coordinate":[x,y]}   |
| long_press   | {"action_type":         |
|              |  "long_press",          |
|              |  "coordinate":[x,y]}   |
| drag         | {"action_type":"drag",  |
|              |  "start_coordinate":    |
|              |  [x1,y1],              |
|              |  "end_coordinate":      |
|              |  [x2,y2]}              |
| input_text   | {"action_type":         |
|              |  "input_text",          |
|              |  "text":"content"}     |
| answer       | {"action_type":"answer",|
|              |  "text":"response"}    |
| scroll       | {"action_type":"scroll",|
|              |  "direction":"down"}   |
| open_app     | {"action_type":         |
|              |  "open_app",            |
|              |  "app_name":"Calendar"} |
| status       | {"action_type":"status",|
|              |  "goal_status":         |
|              |  "complete"}            |

# Execution Principles
1. Communication: ALWAYS use ’answer’
   to reply to user questions.
2. Efficiency: Choose the simplest path.
3. Navigation: scroll = content move;
   swipe = system gesture.
4. Text input: Click input box first.

# Task Instruction
{I}

# Task-State Representation
## Global Summary
{cumulative task state}

## Progress Tracker
Task Decomposition:
- {decomposition items}
Completed Progress:
- {completed items}
Current Subgoal: {current subgoal}
Remaining Requirements:
- {remaining items}

## Transition-Aware Focus
Last Step Result:
- Effectiveness: {effective/uncertain}
- Outcome: {last_step_result}
Next Action Focus: {next_action_focus}

# Output Format
Thought: [Your analysis of current state]
Action: [Single JSON action]
```

## Appendix EState Updater Prompt Template

The state updater𝒰ϕ\\mathcal\{U\}\_\{\\phi\}receives the following prompt along with before/after screenshots \(ot−1o\_\{t\-1\},oto\_\{t\}\) as image inputs\.

#### System message\.

```
You maintain a structured task-state
representation for a mobile GUI actor.
Return strict JSON only.
```

#### User message\.

```
Task Instruction: {I}

Previous Task-State Representation (JSON):
{S_{t-1}}

Last Step:
- Actor reasoning: {T_{t-1}}
- Action executed: {a_{t-1}}
- Screenshot before action: [image o_{t-1}]
- Screenshot after action: [image o_t]

Instructions:
1. Compare the before/after screenshots
   to determine action effectiveness.
2. Update task decomposition if the task
   requires multiple steps.
3. Preserve all verified completed
   progress from previous state.
4. Update the current subgoal.
5. Align remaining requirements.
6. Write next_action_focus to guide
   the actor’s next decision.

Rules:
- Treat the previous state as persistent
  working memory across steps.
- Base judgments only on visible UI state;
  do not invent hidden information.
- Keep each field concrete and relevant.
- Return JSON only, no explanation.

Output JSON schema:
{
  "action_effective": true|false|null,
  "task_summary": "...",
  "task_decomposition": ["..."],
  "completed_progress": ["..."],
  "current_subgoal": "...",
  "remaining_requirements": ["..."],
  "last_step_result": "...",
  "next_action_focus": "..."
}
```