AndroidReality: How Far Are Mobile Agents from the Real World?

# AndroidReality: How Far Are Mobile Agents from the Real World?
Source: [https://arxiv.org/html/2608.07775](https://arxiv.org/html/2608.07775)
Xiaoou Liu1\*Longchao Da1\*Hanyang Chen1Yuan Ling2Hua Wei1† 1Arizona State University2Independent Researcher \{xiaoouli, longchao, hchen478, hua\.wei\}@asu\.edu ericalingyuan@gmail\.com

###### Abstract

Mobile agents have achieved promising results on clean online benchmarks such as AndroidWorld, yet their performance often degrades sharply in real\-world deployment due to environmental variations and imperfect interface conditions\. In this work, we introduceAndroidReality, a perturbation\-based framework for evaluating and improving the robustness of mobile agents\. Through a Markov Decision Process \(MDP\) perspective, we organize real\-world interface variability into a principled taxonomy of perturbations along three axes: state, transition, and action\. Guided by this taxonomy, we build a perturbed mobile benchmark on top of AndroidWorld with realistic and controllable perturbation injections, enabling systematic robustness evaluation of mobile agents\. Our evaluation reveals substantial robustness gaps and four recurring error categories, motivating a simple training\-free Test\-Time Introspective Recovery \(TTIR\) mechanism that mitigates these failures on both perturbed and clean settings\. Together, these results position robustness as a missing dimension in mobile agent evaluation and establish benchmark perturbation as an effective tool for both stress testing and surfacing latent weaknesses of mobile agents\. Benchmark and implementation are available[here](https://github.com/Xiao0o0o/AndroidReality)\.

11footnotetext:Equal contribution\.22footnotetext:Corresponding author\.## 1Introduction

Mobile agents\[cheng2025kairos,gu2026generalization,li2025mobileuse,qin2025uitars,ye2025mobileagentv3,xu2026mobileagentv35\]have recently achieved strong performance on online evaluation environments\[chen2024spa,kong2025mobileworld,rawlesandroidworld,xu2025androidlab\], demonstrating promising capabilities in multi\-step GUI grounding, planning, and action execution\. These benchmarks played an important role in accelerating progress by providing standardized interfaces and reproducible tasks\. However, despite encouraging results in such well\-defined and controlled settings, the performance of current agents often degrades substantially in real\-world deployment\[li2025mobileuse,kong2025mobileworld,wu2024foundations,yang2026gui\]\. As shown in Figure[1](https://arxiv.org/html/2608.07775#S1.F1), practical mobile environments expose agents to a wide range of conditions that are absent from existing benchmarks, including transient pop\-ups and notifications, delayed UI responses, varying display configurations, and unreliable action execution, which create a substantial gap between benchmark performance and real\-world robustness\[chen2026d,zhang2025hyperclick\]\.

This gap suggests that current benchmark success may overestimate agent reliability under the variability of real\-world deployment\. Clean benchmarks instantiate a simplified and stationary environment: the interface follows a canonical layout, the environment transitions predictably in response to actions, and action execution is assumed to be reliable\. Real\-world deployment, by contrast, deviates from each of these assumptions along well\-defined axes, namely interface presentation, environment dynamics, and action execution\[da2025survey,long2025survey\], and the same task can therefore unfold along many different trajectories\. As a result, failures in deployment often do not stem from a lack of task\-solving ability alone, but from brittleness to environment variations that current agents were never meaningfully exposed to during training or evaluation\[li2025mobileuse,gonzalez2026reliability,luo2026lost\]\.

![Refer to caption](https://arxiv.org/html/2608.07775v1/x1.png)Figure 1:Examples of perturbations introduced byAndroidReality, designed to emulate real\-world deployment conditions that mobile agents face but that existing benchmarks largely overlook\.State\-levelperturbations modify the observation surface, including font\-size changes, UI theme changes, display\-density and screen\-size changes, orientation shifts, locale/date\-format variations, and partial\-loading skeleton screens, etc\.Action\-levelperturbations alter the execution of the agent’s intended actions, including realistic action delays, randomly dropped actions, frozen or loading execution states, app hangs requiring restart, and resets to the home screen\.Transition\-levelperturbations insert unexpected intermediate states, including trial offers, interstitial ads, consent sheets, permission dialogs, security pop\-ups, rating prompts, notifications, and app/system update prompts\.In this work, we introduce a perturbation\-based frameworkAndroidRealityfor systematically studying the robustness of mobile agents in online mobile environments\. Our key idea is to augment a clean benchmark with controlled perturbations that emulate common real\-world variations, and to use these perturbations to systematically characterize where and why current mobile agents fail\. We adopt a Markov Decision Process \(MDP\) view of mobile interaction, where a task is modeled asℳ=\(𝒮,𝒜,𝒯,ℛ,γ\),\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\mathcal\{R\},\\gamma\),with𝒮\\mathcal\{S\}the space of UI states,𝒜\\mathcal\{A\}the GUI action space,𝒯\\mathcal\{T\}the transition dynamics induced by the operating system and target applications, andℛ\\mathcal\{R\}the task completion signal\. Real deployment can be naturally viewed as a family of perturbed MDPs that share task semantics with the canonical benchmark MDP but vary along specific components of the MDP tuple, and robustness, under this view, is the ability of an agent to maintain task performance across this family\.

This formulation suggests a natural taxonomy: perturbations can be organized by which component of the MDP they act upon\.State\-levelperturbations modify the surface form of state observations without altering the underlying transition structure \(e\.g\., font size changes, orientation shifts, etc\.\)\.Transition\-levelperturbations modify the environment dynamics𝒯\\mathcal\{T\}by inserting unexpected intermediate states or redirecting action outcomes \(e\.g\., pop\-up interruptions, system notifications, etc\.\)\.Action\-levelperturbations modify the mapping between intended and executed actions \(e\.g\., delays, dropped actions, and unintended actions\)\. This taxonomy provides a unified foundation for organizing the perturbation space and for attributing robustness failures to specific stages of the agent\-env interaction\.

Guided by this taxonomy, we constructAndroidReality, an online perturbed mobile benchmark that injects realistic and parameterizable variations along all three axes flexibly\. While prior robustness\-oriented benchmarks have focused on individual subsets of these axes, such as naturalistic pop\-ups\[yang2026gui,chen2026d\]or task\-template variation\[gu2026generalization\],AndroidRealityprovides more comprehensive coverage of mobile interface variability through a unified interface\. Because the perturbations are controlled, they enable fine\-grained robustness evaluation beyond aggregate task success rate, allowing us to diagnose which class of perturbation is responsible for the largest share of failures for a given agent\. Building on this evaluation, we conduct a systematic error analysis that reveals a set of recurring failure patterns across state\-of\-the\-art mobile agents, ranging from distraction by interrupting elements to rigidity in following outdated plans to the inability to recognize and recover from execution failures\.

Motivated by these patterns, we introduce a*Test\-Time Introspective Recovery*mechanism that monitors the agent’s interaction trajectory, detects when it has entered one of the identified failure modes, and triggers a targeted recovery action before continuing with the task\. The mechanism is training\-free and model\-agnostic and can be plugged into existing mobile agents without modifying their underlying policy\. We additionally find that applying this module can recover failure cases in the original clean setting as well, suggesting that the failure modes uncovered through perturbation\-based stress testing are not artifacts of perturbation alone but rather reflect broader weaknesses in current mobile agents\. Our contributions can be summarized as follows:

- •We constructAndroidReality, an online mobile benchmark that injects realistic and controllable perturbations into the agent–environment interaction loop, organized by a perturbation taxonomy grounded in the MDP formalism \(state, transition, or action\)\.
- •We conduct asystematic robustness evaluation and error analysisof SOTA mobile agents onAndroidReality, identifying which classes of perturbation drive the largest performance degradation and characterizing four recurring failure patterns behind them\.
- •Guided by this error analysis, we propose a test\-time introspective recovery module, a training\-free, plug\-and\-play intervention that can recover some failure cases on both perturbed and clean settings\.

## 2Related Work

### 2\.1GUI Agent Systems

Mobile GUI agents have evolved rapidly with the rise of multimodal foundation models\[steiner2024paligemma,xie2024show,bai2025qwen3,team2025kimi,wei2026deepseek\], progressing from rule\-based scripts to LLM/VLM\-driven autonomous systems\[rawlesandroidworld,wang2024mobile\]\. Representative agents such asCogAgent\[hong2024cogagent\]andAppAgent\[zhang2024appagent\]demonstrate strong planning and grounding capabilities on standard interfaces\. However, these agents are primarily developed and evaluated under*clean*interface conditions, where the environment is stationary, and execution is reliable\. Recent studies have shown that such agents can be easily distracted by deceptive content\[liu2025hijacking\]or fail under under\-specified task rules\[chen2026knowu\], raising concerns about their robustness in deployment\.Different from these works, we do not propose a new agent architecture; instead, we introduce a perturbation\-based framework that systematically evaluates existing agents through a principled MDP\-grounded taxonomy, attributing failures to specific stages of the ‘agent and environment’ interaction loop\.

### 2\.2Mobile Agent Benchmarks

Evaluation for mobile agents has shifted from static datasets to dynamic, online environments\. Early benchmarks such asAITW\[rawles2023aitw\]andAndroidControl\[li2024effects\]relied on offline human demonstrations and could not assess error recovery\.AndroidWorld\[rawlesandroidworld\]established the standard for online evaluation with reproducible state\-based verification, while follow\-up benchmarks such asMobileWorld\[kong2025mobileworld\]andSPA\-Bench\[chen2024spa\]extend this paradigm to longer horizons and more complex application workflows\. A more recent line of work begins to examine robustness under interface anomalies\.GUI\-Robust\[yang2026gui\]andD\-GARA\[chen2026d\]evaluate agents in the presence of system interruptions, pop\-ups, and network failures, whileMobileSafetyBench\[lee2026mobilesafetybench\]focuses on adversarial safety scenarios\.Different from these works, which treat anomalies as discrete high\-level events with task\-level success metrics, we ground our perturbations in the MDP formalism and inject controllable variations at the state, transition, and action levels\. This allows fine\-grained, MDP\-stage\-level error attribution rather than aggregate success rates, and converts a canonical benchmark into a family of perturbed MDPs for systematic robustness analysis\. Table[1](https://arxiv.org/html/2608.07775#S2.T1)summarizes howAndroidRealitycompares to existing benchmarks along these dimensions\.

BenchmarkPlat\.Env\.\#TasksPert\. TypeTimingAttr\.\#PMDP StageStandard mobile / GUI agent benchmarksAndroidWorld\[rawlesandroidworld\]MD116––Coarse0–AndroidLab\[xu2025androidlab\]MD138––Coarse0–SPA\-Bench\[chen2024spa\]MD340––Coarse0–MobileWorld\[kong2025mobileworld\]MD201––Coarse0–Robustness\-oriented benchmarksGUI\-Robust\[yang2026gui\]W\+DS5318NatPreCoarse7ObsD\-GARA\[chen2026d\]MD152NatRunCoarse∼\\sim5Obs\+TransMobileSafetyBench\[lee2026mobilesafetybench\]MD–AdvRunCoarse–ObsST\-WebAgentBench\[levy2024st\]WD–Adv\+NatPre\+RunCoarse–Obs\+ActAndroidRealityMD116NatPre\+RunFine \(MDP\)24Obs\+Act\+Trans

Table 1:Comparison of GUI/mobile agent benchmarks across robustness evaluation dimensions\.Plat\.: Mobile \(M\) / Web \(W\) / Desktop \(D\)\.Env\.: Static \(S\) / Dynamic online \(D\)\.Pert\. Type: Adversarial \(Adv\) / Naturalistic \(Nat\)\.Timing: Pre\-task \(Pre\) / Runtime \(Run\)\.Attr\.: Error attribution granularity\.\#P: number of distinct perturbation types\.MDP Stage: Observation \(Obs\) / Action \(Act\) / Transition \(Trans\)\.

## 3AndroidRealityBenchmark

### 3\.1MDP Perturbation Formulation

We augment AndroidWorld with a perturbation layer that operates inside the agent–environment interaction loop, converting the original benchmark into a family of perturbed MDPs:

ℳ0=\(𝒮,𝒜,𝒯,ℛ,γ\)→Pc,θℳ~c,θ=\(𝒮~,𝒜~,𝒯~,ℛ,γ\)\\mathcal\{M\}\_\{0\}=\(\\mathcal\{S\},\\mathcal\{A\},\\mathcal\{T\},\\mathcal\{R\},\\gamma\)\\xrightarrow\{\\;\\;P\_\{c,\\theta\}\\;\\;\}\\widetilde\{\\mathcal\{M\}\}\_\{c,\\theta\}=\(\\widetilde\{\\mathcal\{S\}\},\\widetilde\{\\mathcal\{A\}\},\\widetilde\{\\mathcal\{T\}\},\\mathcal\{R\},\\gamma\)\(1\)where𝒮\\mathcal\{S\}denotes the UI state space,𝒜\\mathcal\{A\}denotes the set of executable mobile actions,𝒯\\mathcal\{T\}denotes the environment transition dynamics,ℛ\\mathcal\{R\}denotes the task completion reward, andγ\\gammais the discount factor\. The perturbation operatorPc,θP\_\{c,\\theta\}wraps the clean interaction loop and transforms the canonical MDP into a perturbed MDPℳ~c,θ\\widetilde\{\\mathcal\{M\}\}\_\{c,\\theta\}, wherec∈\{State,Transition,Action\}c\\in\\\{\{\\color\[rgb\]\{0\.53515625,0\.65625,0\.83984375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.53515625,0\.65625,0\.83984375\}\\textbf\{State\}\},\{\\color\[rgb\]\{0\.61328125,0\.45703125,0\.68359375\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.61328125,0\.45703125,0\.68359375\}\\textbf\{Transition\}\},\{\\color\[rgb\]\{0\.66015625,0\.2265625,0\.140625\}\\definecolor\[named\]\{pgfstrokecolor\}\{rgb\}\{0\.66015625,0\.2265625,0\.140625\}\\textbf\{Action\}\}\\\}specifies the MDP component being perturbed, andθ\\thetadenotes the perturbation type and its configuration\.

In the clean setting, the agent interacts with a single canonical environment: it observes screenshots, UI trees, and available APIs, produces actions such as tapping, scrolling, typing, or selecting, and receives task\-completion feedback from the evaluator\. In contrast, real mobile environments do not correspond to a single fixed MDP\. A task with the same semantic goal may unfold under different display settings, unexpected intermediate screens, delayed responses, or imperfect action execution\. All perturbed MDPs preserve the original task semantics and reward definition, but modify the observation surface, transition trajectory, or action execution process\. This design allows us to stress\-test whether an agent has learned robust task\-solving behavior or instead relies on brittle assumptions tied to the clean benchmark\.

### 3\.2Perturbation Injection

The perturbation wrapper is implemented as an intermediate layer between the Android emulator and the agent\. The wrapper intercepts the standard interaction loop and selectively applies perturbation operators according to the chosen MDP componentccand configurationθ\\theta\. Formally, a clean interaction step followsst\+1∼𝒯​\(st,at\)s\_\{t\+1\}\\sim\\mathcal\{T\}\(s\_\{t\},a\_\{t\}\), while the perturbed step under configurationθ\\thetacan be written as:

st\+1∼𝒯~θ​\(st,ϕθA​\(at\)\),o~t\+1=ϕθS​\(st\+1\)s\_\{t\+1\}\\sim\\widetilde\{\\mathcal\{T\}\}\_\{\\theta\}\(s\_\{t\},\\;\\phi^\{A\}\_\{\\theta\}\(a\_\{t\}\)\),\\qquad\\tilde\{o\}\_\{t\+1\}=\\phi^\{S\}\_\{\\theta\}\(s\_\{t\+1\}\)\(2\)whereϕθA\\phi^\{A\}\_\{\\theta\}perturbs the agent’s action before it is executed,𝒯~θ\\widetilde\{\\mathcal\{T\}\}\_\{\\theta\}replaces the original transition with one that may lead to unexpected intermediate next states, andϕθS\\phi^\{S\}\_\{\\theta\}perturbs the observation returned to the agent\. The agent then conditions its next decision ono~t\+1\\tilde\{o\}\_\{t\+1\}instead ofst\+1s\_\{t\+1\}\. Each operator corresponds to one of the three perturbation classes defined in our taxonomy\. We provide a visual illustration of representative perturbations from each class in Figure[1](https://arxiv.org/html/2608.07775#S1.F1)and a full summary in Table[3](https://arxiv.org/html/2608.07775#A1.T3)\.

State\-levelperturbations\(ϕθS\\phi^\{S\}\_\{\\theta\}\) modify the state returned to the agent while preserving the underlying task progress\. These perturbations alter the visual or structural form of the observation, such as font size, UI theme, display density, screen orientation, and partially loaded UI content, so that the agent receives perturbed screenshots, UI trees, or API\-visible states, while the task goal and success verifier remain unchanged\.Transition\-levelperturbations\(𝒯~θ\\widetilde\{\\mathcal\{T\}\}\_\{\\theta\}\) modify the trajectory induced by the environment dynamics\. Instead of directly transitioning from the current state to the next targeted state, the environment may pass through an unexpected intermediate state, such as a trial\-offer advertisement, a message notification, a consent sheet, a rating prompt, or a security/permission pop\-up\. These perturbations simulate common mobile interruptions that require the agent to recognize off\-trajectory states and recover appropriately\.Action\-levelperturbations\(ϕθA\\phi^\{A\}\_\{\\theta\}\) modify the mapping from the agent’s intended action to the action actually realized in the environment\. For example, an intended action may be silently dropped, executed only after a noticeable delay, or followed by an app hang requiring a restart\. These perturbations expose whether the agent can detect failed execution and revise its plan, rather than assuming that every action was successfully executed\.

### 3\.3Evaluation Protocol

For each task, we evaluate mobile agents under both the clean AndroidWorld environment and our perturbedAndroidRealityenvironments\. The clean setting measures standard benchmark competence under the canonical MDPℳ0\\mathcal\{M\}\_\{0\}, while the perturbed setting measures robustness under the family of perturbed MDPsℳ~c,θ\\widetilde\{\\mathcal\{M\}\}\_\{c,\\theta\}\. Unless otherwise specified, we instantiate one perturbation type at a time so that the performance change can be attributed to a specific perturbation source\.

We report the task success rate under the clean setting and under each perturbation category\. To quantify robustness degradation, we define the category\-level performance drop as:

Δc=SR​\(ℳ0\)−SR​\(ℳ~c\)\\Delta\_\{c\}=\\mathrm\{SR\}\(\\mathcal\{M\}\_\{0\}\)\-\\mathrm\{SR\}\(\\widetilde\{\\mathcal\{M\}\}\_\{c\}\)\(3\)whereSR​\(ℳ0\)\\mathrm\{SR\}\(\\mathcal\{M\}\_\{0\}\)denotes the success rate in the clean environment andSR​\(ℳ~c\)\\mathrm\{SR\}\(\\widetilde\{\\mathcal\{M\}\}\_\{c\}\)denotes the success rate under perturbation categorycc\. A largerΔc\\Delta\_\{c\}indicates that the agent is more brittle to that class of perturbation\. When needed, we also compute perturbation\-specific degradationΔc,θ\\Delta\_\{c,\\theta\}to identify which concrete perturbation type causes the largest failure rate\.

Beyond binary task success, perturbations may also increase the number of steps an agent needs to complete the task, for example, by requiring the agent to dismiss a pop\-up, wait through a loading delay, or recover from a dropped action\. We therefore additionally report the step overhead:

Δstep,c=T¯​\(ℳ~c\)−T¯​\(ℳ0\)\\Delta\_\{\\text\{step\},c\}=\\overline\{T\}\(\\widetilde\{\\mathcal\{M\}\}\_\{c\}\)\-\\overline\{T\}\(\\mathcal\{M\}\_\{0\}\)\(4\)whereT¯​\(ℳ\)\\overline\{T\}\(\\mathcal\{M\}\)denotes the average number of steps used by the agent under environmentℳ\\mathcal\{M\}\.

## 4Experiment

### 4\.1Experiment Setup

Benchmark and task environment\.We evaluate mobile agents onAndroidReality, a perturbed benchmark built on AndroidWorld\[rawlesandroidworld\]\.AndroidRealitypreserves the original task definitions, Android emulator interface, action space, and programmatic success verifier, while inserting the perturbation wrapper described in Section[3](https://arxiv.org/html/2608.07775#S3)and illustrated in Figure[5](https://arxiv.org/html/2608.07775#A1.F5)\. Each task is paired with one fixed perturbation instance, and the same task\-perturbation pairs are shared across all models for reproducible and model\-agnostic comparison\. The perturbations coverstate,action, andtransitioncomponents of the agent\-environment loop; full perturbation configurations are provided in Appendix[B](https://arxiv.org/html/2608.07775#A2)and Table[4](https://arxiv.org/html/2608.07775#A1.T4)\.

Evaluated models\.We evaluate eight open\-source mobile agents from GUI\-OWL\[ye2025mobileagentv3\], GUI\-OWL\-1\.5\[xu2026mobileagentv35\], UI\-TARS\[qin2025uitars\], and UI\-TARS\-1\.5\[qin2025uitars\], covering model scales from22B to3232B\. All models use their released prompts and observe the latest screenshot together with the action history\. Due to the page limit, implementation, model serving, and logging details are reported in Appendix[B](https://arxiv.org/html/2608.07775#A2)\.

### 4\.2Robustness Evaluation

We evaluate eight open\-source mobile agents under both the clean and perturbed settings ofAndroidReality\. Beyond aggregate task success rates, our MDP\-grounded taxonomy enables us to break down robustness performance by perturbation class, revealing distinct failure profiles across the three components of the MDP interaction loop\. Table[2](https://arxiv.org/html/2608.07775#S4.T2)reports the per\-class results, while the detailed categorical perturbation analysis \(across full types\) is shown in Figure[6](https://arxiv.org/html/2608.07775#A3.F6)and Figure[7](https://arxiv.org/html/2608.07775#A3.F7)\.

Observation: All agents degrade under real\-world perturbation\.Every evaluated agent suffers substantial degradation under perturbation, with absolute drops of1313–3636points on overall success rate, uniformly across model families, parameter scales, and training paradigms\. Performance loss spans all three MDP components rather than concentrating on a single class, indicating that current mobile agents lack the robustness needed for reliable deployment despite their strong canonical\-benchmark performance\. Notably, drops are comparable across very different agents, such as GUI\-Owl\-32B and GUI\-Owl\-1\.5\-8B, which both lose roughly1515–2020points, suggesting that scaling and additional post\-training alone do not close the robustness gap\. The top\-three challenging types are in Table[5](https://arxiv.org/html/2608.07775#A3.T5)\.

∙\\bulletState perturbations affect all agents most uniformly\.Among the three perturbation classes,state\-levelperturbations cause the largest and most uniform degradation across agents\. By modifying the rendered observation through font scale, UI theme, or orientation, these perturbations push UI elements outside the spatial regions that agents have learned to associate with their target functions, and agents fail to re\-ground their coordinate predictions to the new layout\. As a result, agents often issue coordinates aligned with the unperturbed layout, miss the intended target, and repeatedly retry the same erroneous coordinate without re\-grounding against the current observation\.

∙\\bulletAction perturbations have the smallest effect\.Action\-levelperturbations, in contrast, lead to the smallest drops across most agents\. This relative resilience comes from a property that current agents already possess by default: when an action does not produce the expected screen change because it was delayed, dropped, or absorbed by a frozen state, the agents naturally retry the same action on the next step\. This implicit recovery behavior masks much of the underlying execution noise, particularly for short\-lived perturbations such asaction\_delayoraction\_drop\. Action\-level robustness is therefore better characterized as a side\-effect of the agents’ default retry behavior\.

∙\\bulletTransition perturbations show the largest cross\-agent variance\.Transition\-levelperturbations exhibit the most variable behavior across agents: drops range from−2\.27\-2\.27on GUI\-Owl\-1\.5\-4B to−29\.54\-29\.54on GUI\-Owl\-1\.5\-2B and−26\.14\-26\.14on GUI\-Owl\-32B\. This variance reflects qualitatively different agent strategies for handling unexpected intermediate states\. Some agents recognize pop\-ups, consent sheets, or notifications as off\-task and dismiss them cleanly with only a small step\-budget cost\. Others treat the interruption as part of the task and engage deeply with multi\-step dialogs, as shown in Figure[8](https://arxiv.org/html/2608.07775#A3.F8), consuming most of their budget before reaching the actual goal\. The unusually small drop on GUI\-Owl\-1\.5\-4B is a notable exception, as its low clean\-setting success rate \(43\.1043\.10\) leaves limited headroom for further degradation, reminding us that absolute robustness drop must be interpreted alongside clean\-setting capability\. We also observe a more subtle failure mode: even when an agent successfully dismisses a transition perturbation, a non\-trivial fraction of subsequent attempts fail to complete the original task that the agent had solved correctly on the clean run, suggesting that interruptions disrupt the agent’s goal memory and lead to wrong\-subgoal resumption or premature termination\. Transition perturbations, therefore, probe not only the agent’s ability to handle off\-trajectory states but also its capacity to maintain goal\-directed memory across distractions\.

Table 2:Evaluation results of open\-source GUI\-specialized agent models under clean and perturbed environments\. Success rates are reported in percentages\. Values in parentheses indicate absolute performance drops from the corresponding clean setting\.Δ\\DeltaStep reports the change in average step length from clean to perturbed settings, where red indicates increases and green indicates decreases\. Numbers†reported by the original paper\[qin2025uitars\]\.ModelOverallStateActionTransitionCleanPert\.Δ\\DeltaStepCleanPert\.Δ\\DeltaStepCleanPert\.Δ\\DeltaStepCleanPert\.Δ\\DeltaStepGUI\-Owl\-7B63\.7948\.28\(\-15\.51\)\+0\.9163\.6440\.91\(\-22\.73\)\+1\.7360\.7148\.21\(\-12\.50\)\-1\.9465\.9155\.68\(\-10\.23\)\+1\.77GUI\-Owl\-32B64\.6641\.81\(\-22\.85\)\+0\.5465\.9147\.73\(\-18\.18\)\+0\.5960\.7135\.71\(\-25\.00\)\+1\.3865\.9139\.77\(\-26\.14\)\-0\.14GUI\-Owl\-1\.5\-2B48\.7126\.29\(\-22\.42\)\+3\.4750\.0027\.27\(\-22\.73\)\+3\.6141\.0730\.36\(\-10\.71\)\+5\.6852\.2722\.73\(\-29\.54\)\+1\.78GUI\-Owl\-1\.5\-4B43\.1029\.74\(\-13\.36\)\+1\.7254\.5529\.55\(\-25\.00\)\+4\.0242\.8630\.36\(\-12\.50\)\-0\.1031\.8229\.55\(\-2\.27\)\-0\.25GUI\-Owl\-1\.5\-8B56\.9037\.07\(\-19\.83\)\+3\.7756\.8243\.18\(\-13\.64\)\+7\.1250\.0030\.36\(\-19\.64\)\-0\.4661\.3635\.23\(\-26\.13\)\+2\.29GUI\-Owl\-1\.5\-32B67\.6743\.10\(\-24\.57\)\+3\.5469\.3238\.64\(\-30\.68\)\+7\.1257\.1442\.86\(\-14\.28\)\+1\.2172\.7347\.73\(\-25\.00\)\+1\.05UI\-TARS\-7B\-SFT33\.00†25\.86\(\-7\.14\)––29\.55\(\-3\.45\)––28\.57\(\-4\.43\)––20\.45\(\-12\.55\)–UI\-TARS\-1\.5\-7B64\.20†27\.59\(\-36\.61\)––29\.55\(\-34\.65\)––25\.00\(\-39\.20\)––27\.27\(\-36\.93\)–

### 4\.3Error Analysis

To understand*why*agents fail under perturbation, we conducted a systematic error analysis over all failed trajectories from our main evaluation\. Failures were tagged by manually inspecting per\-step screenshots, action logs, and the agent’s reasoning traces\. We identify four recurring error categories that together account for most of the perturbation\-induced failures across all evaluated models\. Figure[2](https://arxiv.org/html/2608.07775#S4.F2)provides a schematic illustration of the four categories of recurring failures\.

![Refer to caption](https://arxiv.org/html/2608.07775v1/x2.png)Figure 2:Visual illustration of the four error categories surfaced byAndroidReality\.\(1\)Stale\-Coordinate Stuck\(𝒮\\mathcal\{S\}\): under state\-level perturbations such as orientation changes, the agent’s predicted coordinate misses the rendered target, and the agent repeatedly retries the same incorrect coordinate\. \(2\)Silent Action Failures\(𝒜\\mathcal\{A\}\): action\-level perturbations silently drop the issued action; the agent does not notice the missed click and proceeds as if it had succeeded\. \(3\)Off\-Task Engagement with Distractions\(𝒯\\mathcal\{T\}\): a transition\-level perturbation injects off\-trajectory states \(e\.g\., system pop\-ups\), and the agent treats them as part of the task, navigating deeper into the distraction until failure\. \(4\)Goal Memory Drift After Distraction\(𝒯\\mathcal\{T\}\): the agent correctly dismisses the distraction but loses track of the original task, returning to a menu or terminating without completing the intended workflow\. The symbols in parentheses denote the MDP component each error class is most strongly associated with, in correspondence to this work’s benchmark design\.①Stale\-Coordinate Stuck\.The most striking failure pattern occurs understate\-levelperturbations\. When display density, theme, or orientation alters the rendered layout, the agent often emits coordinates that miss the new position of the target element and land on empty space or an adjacent UI element\. After observing that the screen has not changed in the expected way, the agent does not re\-ground its prediction to the new layout; instead, it retries the same incorrect coordinate\-level actions repeatedly, in extreme cases up to dozens of identical clicks before the step budget is exhausted\. This pattern is observed uniformly across all four model sizes, suggesting that current agents lack an explicit re\-grounding behavior when an action fails to produce the expected state change\.

② Silent Action Failures\.Underaction\-levelperturbations such asaction\_drop\_random, the action generated by the agent is silently delayed, dropped, or absorbed by an unresponsive screen\. Although agents naturally retry the same action on the next step \(which masks much of the underlying noise on simpler tasks\), a non\-trivial fraction of failures occur when the agent does*not*register that the previous action had no effect, and proceeds with subsequent steps in its plan as if it had succeeded\. The result would be a trajectory that is internally coherent but desynchronized from the actual environment state, and that often ends with the agent declaring task completion on a screen that never advanced\. This failure mode is a direct consequence of agents lacking an explicit verification step between action generation and continuation\. There is an example shown in Figure[2](https://arxiv.org/html/2608.07775#S4.F2)\(②\)\.

③Off\-Task Engagement with Distractions\.Undertransition\-levelperturbations such as pop\-ups, consent sheets, advertisements, and notifications, the agent often fails to recognize the interruption as off\-task and instead treats it as part of the intended workflow\. This manifests in several ways: the agent may engage deeply with multi\-step dialogs, navigate through fake interaction flows until the step budget is exhausted, or treat the interruption’s dismissal as task completion and emitterminate\(success\)without ever attempting the original task\. Across these variants, the underlying issue is the same: the agent does not distinguish between off\-trajectory states introduced by the environment and on\-trajectory states relevant to the goal\. The original task may never even begin once the distraction takes hold of the agent’s attention\. The illustration is shown in Figure[2](https://arxiv.org/html/2608.07775#S4.F2)\(③\)\.

④Goal Memory Drift After Distraction\.A more subtle and consequential failure mode arises undertransition\-levelperturbations: the agent*successfully*dismisses an interruption, yet subsequently fails to complete the original task\. In a non\-trivial fraction of trajectories, the agent correctly identifies the pop\-up or notification as off\-task, dismisses it cleanly, and then either resumes on the wrong subgoal, restarts the task from scratch, or terminates prematurely as if the task were already complete\. These are tasks the same agent solves correctly on the clean run, indicating that the failure is not caused by an inability to handle the interruption itself, but by the disruption it imposes on the agent’s working memory of the original task plan\. This category reveals a robustness dimension that aggregate success rates conflate with raw distraction\-handling failures: even when the dismissal is correct, the cognitive cost of the interruption can derail the trajectory\. Similarly, please refer to Figure[2](https://arxiv.org/html/2608.07775#S4.F2)\(④\) for illustration\.

## 5Test\-Time Introspective Recovery

![Refer to caption](https://arxiv.org/html/2608.07775v1/x3.png)Figure 3:The TTIR framework\. TTIR diagnoses four failure signals from our error analysis: stuck\-loop behavior, unsatisfied action effects, off\-task interference, and task misalignment\. When a failure is detected, it invokes a limited recovery session before returning the recovered state to the agent–emulator loop\.The error analysis in Section[4\.3](https://arxiv.org/html/2608.07775#S4.SS3)surfaced four failure categories that share a common root cause: the agent does not check whether its current state remains consistent with successful task progress before continuing\. This observation suggests that robustness can be improved by introducing an explicit self\-monitoring mechanism at test time, without retraining the underlying agent\. We proposeTest\-Time Introspective Recovery \(TTIR\): a training\-free, model\-agnostic mechanism that augments the agent’s interaction loop with a*diagnose\-then\-recover*step before each action\. At every step, a lightweight diagnoser inspects the current screenshot together with the recent action history and asks four targeted questions, each designed to detect one of the failure categories identified in our analysis\. When a problem is detected, TTIR dispatches a recovery strategy specifically tailored to that category\. The diagnoser and recovery agent both reuse the same backbone VLM as the main agent, introducing no additional parameters or training data\.

### 5\.1TTIR Module Design

TTIR is organized around a one\-to\-one correspondence between the four error categories from the Section[4\.3](https://arxiv.org/html/2608.07775#S4.SS3)and four ‘diagnose\-then\-recover strategies’ \(dashed orange box on the right side of Figure[3](https://arxiv.org/html/2608.07775#S5.F3)\)\. At each step, the ‘TTIR Diagnosis Module’ \(i\.e\., diagnoser\) produces a structured JSON judgment of the current state along four dimensions, each derived from a targeted question\. If any dimension flags a problem, TTIR enters a bounded recovery session in which the recovery agent is invoked through a recovery\-specific prompt that declares the diagnosed problem, supplies the recovery target condition, and specifies the relevant constraints\. The recovery agent then generates several corrective actions tailored to that specific failure pattern, and control returns to the main agent once the diagnosed problem is resolved or the recovery budget is exhausted\. TTIR therefore provides a pure test\-time improvement to the agent’s interaction loop\. Figure[3](https://arxiv.org/html/2608.07775#S5.F3)illustrates the overall design of TTIR\. The four strategies are summarized as follows: we provide details on the diagnostic signals and recovery prompts for each strategy in Appendix[D\.1](https://arxiv.org/html/2608.07775#A4.SS1)\.

①Stuck\-loop detection⇒\\Rightarrowalternative\-action recovery\.TTIR detects stuck loops using both model\-based and deterministic signals\. The diagnoser checks whether recent actions form a repeated unresolved attempt and outputsrepeat\_failure\_detected\. TTIR also triggers this diagnosis when the lastNNsteps \(defaultN=3N\{=\}3\) have identical action, summary, and thought\. Once triggered, TTIR applies an*alternative\-action*recovery: the recovery agent is told that the previous action has repeatedly failed and is instructed to choose a different visible target or route toward the same sub\-goal instead of retrying the same coordinate\.

②Action\-effect verification⇒\\Rightarrowalternative\-action recovery\.The ‘TTIR Diagnosis’ module compares the agent’s previous intended effect against the current screenshot, and it will flag the cases where the action was executed, while its expected outcome is not visible\. After that, the ‘TTIR Recovery’ module was prompted to try another way to achieve the same intended effect\.

![Refer to caption](https://arxiv.org/html/2608.07775v1/fig/recovery_number.png)Figure 4:Comparison of failed task counts with and without TTIR \(Ours\)\. It shows a notable reduction in failures under the Perturbation condition, while also rectifying erroneous task outcomes in the Clean condition\.③Interference detection⇒\\Rightarrowdismiss\-interference recovery\.The ‘TTIR Diagnosis’ module inspects the screenshot for off\-task overlays such as pop\-ups, consent sheets, advertisements, and notifications\. Then, the ‘TTIR Recovery’ was prompted to close, decline, or navigate past the interfering element, while explicitly forbidding engagement with the interruption’s content\.

④Task\-alignment check⇒\\Rightarrowre\-orientation recovery\.The diagnoser checks whether the current screen still belongs to the original task path, particularly after a perturbation has been handled\. The recovery agent was prompted to find an action that restores the correct task context \(e\.g\., navigating back or re\-opening the relevant app\), without attempting to complete the task itself\.

### 5\.2Empirical Study

To evaluate the effectiveness of TTIR, we apply it to GUI\-Owl\-7B and compare its performance with and without TTIR on both the perturbed and clean settings\. TTIR is applied as a pure test\-time setting, with no fine\-tuning or reinforcement learning\. Figure[4](https://arxiv.org/html/2608.07775#S5.F4)reports the number of failed tasks before and after applying TTIR\.

In theperturbed setting, TTIR reduces the number of failed tasks from 60 to 50 \(a 17% relative reduction\)\. More notably, in theclean setting, TTIR still reduces failed tasks from 44 to 35 \(a 20% relative reduction\), even though no perturbation is injected\. This is consistent with our observation in Section[4\.3](https://arxiv.org/html/2608.07775#S4.SS3)that the four error categories are not artifacts of perturbation but latent weaknesses already present in current mobile agents, and indicates that the gains stem from genuine improvements in agent introspection rather than perturbation\-specific countermeasures\. We provide one qualitative case study for each of the four recovery strategies in Appendix[D\.2](https://arxiv.org/html/2608.07775#A4.SS2)\.

## 6Conclusion

In this work, we investigated robustness as a dimension of mobile agent evaluation that is currently underexplored\. We formalized real\-world interface variability through a Markov Decision Process \(MDP\) perspective, deriving a perturbation taxonomy organized by which MDP component each perturbation acts upon\. Based on this taxonomy, we constructedAndroidReality, it is an online mobile benchmark built on AndroidWorld that injects realistic and controllable perturbations along state, transition, and action axes, which can support fine\-grained robustness evaluation across mobile agents\. Our evaluation of eight mobile agents onAndroidRealityconsistently showed substantial performance degradation under categorized perturbation, and we conducted a comprehensive systematic error analysis that surfaced four recurring failure categories\. Motivated by this analysis, we further proposedTest\-Time Introspective Recovery \(TTIR\), a training\-free mechanism that pairs each error category with a targeted diagnose\-and\-recover routine, mitigating failures on both perturbed and clean settings\. We expect theAndroidRealitycan serve as a useful framework for the research community to study the robustness in mobile\-agent evaluation and to push for real\-world use by surfacing failure modes that are otherwise difficult to observe under clean benchmark conditions\.

## References

## Appendix AAndroidRealitySpecifications and Visualization

![Refer to caption](https://arxiv.org/html/2608.07775v1/x4.png)Figure 5:Overview of the proposedAndroidRealitybenchmark\. Starting from a clean AndroidWorld environment, we wrap the agent–environment interaction loop with an MDP perturbation layer that injects realistic and controllable variations along three axes:State\-levelperturbations that alter UI observations such as UI theme, orientation, font size and display dpi;Transition\-levelperturbations that introduce intermediate environment dynamics such as pop\-ups, notifications, update, rating and notification; andAction\-levelperturbations that corrupt the mapping from intended actions to executed actions through drops, delays, or frozen\. The agent receives perturbed screenshots, UI trees, and APIs, then performs thoughts, actions, and conclusions over the task set\. This design converts a single canonical benchmark MDP into a family of perturbed MDPs that preserve task semantics while exposing robustness failures under real\-world interface variability\. Beyond robustness evaluation, the benchmark supports analysis and improvement, such as failure cases collection, which is then grouped into common error types, and finally used to guide test\-time introspective recovery\.### A\.1Perturbation Taxonomy

In this section, we enumerate all 24 perturbations inAndroidReality, listing their MDP class, name, description, and tunable parameters in Table[3](https://arxiv.org/html/2608.07775#A1.T3)\.

Table 3:Perturbation taxonomy inAndroidReality\. Each perturbation preserves the original task goal but modifies a specific MDP component of the ‘agent and environment’ interaction loop\. All perturbations arefully parameterizedalong three orthogonal axes, which are:*magnitude*,*timing*, and*persistence*, and it enables controllable severity sweeps and reproducible evaluation\. Tunable parameters that determine perturbationstrengthare highlighted inbold\.MDP Comp\.PerturbationDescriptionTunable ParametersStatedisplay\_size\_dpi\_compactCompresses display density to test robustness to compact screen configurations\.Density scale,size scale, trigger step, settle time, restore\-on\-closedisplay\_size\_dpi\_zoomed\_outAlters display scale and DPI to test recognition under altered visual sizing\.Density scale,size scale, trigger step, settle time, restore\-on\-closefont\_size\_large/smallChanges text rendering, potentially shifting layout and element positions\.Font scale, trigger step, settle time, restore\-on\-closeui\_theme\_darkSwitches the interface to dark mode, altering colors and contrast\.Theme mode, trigger step, settle time, restore\-on\-closeorientation\_landscapeRotates the display to landscape, changing spatial layout\.Orientation sequence, trigger step, settle time, restore\-on\-closelocale\_date\_formatSwitches locale\-dependent date and time formats \(US12 / EU24\)\.Date format,time format, trigger step, settle time, restore\-on\-closepartial\_loading\_skeletonRenders skeleton placeholders to simulate partially loaded screens\.Duration steps, trigger step, block actions, content textActionaction\_delay\_realisticDelays action execution to simulate latency in real mobile interactions\.Delay probability,delay seconds, random seedaction\_drop\_randomRandomly drops issued actions, testing detection of failed operations\.Drop probability, random seedexecution\_delay\_loadingIntroduces post\-action loading delays, requiring waiting vs\. failure judgment\.Duration steps, trigger step, block actions, loading textexecution\_delay\_frozenFreezes execution after an action, simulating temporary unresponsiveness\.Duration steps, trigger step, block actions, frozen\-screen textapp\_hang\_restart\_requiredSimulates an application hang requiring restart or recovery\.Trigger step, dialog text, settle time, recovery optionsstate\_reset\_homeResets the UI to home, testing recovery after losing task context\.Trigger step, return\-home flag, settle timeTransitionpopup\_permissionPermission dialogs that alter the immediate action context\.Trigger stepSurface styleBlockingAuto\-dismiss durationContent titleContent messageContent buttonsPrimary/secondary actionsProfile poolpopup\_securitySecurity\-related pop\-ups that test interruption handling\.bottom\_sheet\_consentConsent\-style bottom sheets requiring dismissal or decision making\.interstitial\_adFull\-screen ad interruptions that block the original trajectory\.ad\_trial\_offerTrial\-offer advertisements as unexpected intermediate states\.rate\_dialogRating prompts, a common non\-task dialog in mobile apps\.notification\_messageMessage notifications that may distract the agent from its task\.notification\_deliveryNotification\-style interruptions related to delivery events\.update\_app\_sheetApp\-update prompts that redirect the interaction trajectory\.update\_systemSystem\-update prompts emulating OS\-level interruptions\.

### A\.2Experimental Parameter Configurations

For reproducibility, Table[4](https://arxiv.org/html/2608.07775#A1.T4)reports the default parameter values used for each tunable perturbation in our main experiments\. All evaluations reported in the main text use these default values unless otherwise specified\.

Table 4:Default parameter values used for the tunable perturbations in our main experiments\. These parameters jointly control each perturbation and remain fixed across all evaluated agents to ensure reproducible and model\-agnostic comparison\.TypeParameterDefault ValueFont scalingFONT\_SIZE\_LARGE\_SCALE1\.35Font scalingFONT\_SIZE\_SMALL\_SCALE0\.82Display scalingDISPLAY\_COMPACT\_DENSITY\_SCALE0\.85Display scalingDISPLAY\_COMPACT\_SIZE\_SCALE0\.94Display scalingDISPLAY\_ZOOMED\_OUT\_DENSITY\_SCALE0\.76Display scalingDISPLAY\_ZOOMED\_OUT\_SIZE\_SCALE0\.88OrientationORIENTATION\_PROFILE\_SEQUENCElandscapeApp responsivenessAPP\_HANG\_TRIGGER\_STEPS2Action delayACTION\_DELAY\_PROBABILITY0\.15Action delayACTION\_DELAY\_SECONDS1\.5Action delayACTION\_DELAY\_SEED0Action dropACTION\_DROP\_PROBABILITY0\.08Action dropACTION\_DROP\_SEED0

## Appendix BExperimental Details

This appendix expands the experimental setup summarized in Section[4\.1](https://arxiv.org/html/2608.07775#S4.SS1)\.

#### Benchmark construction\.

We conduct experiments onAndroidReality, which is built on top of AndroidWorld\[rawlesandroidworld\]\. The benchmark keeps the original AndroidWorld task definitions, Android emulator interface, action space, and programmatic success verifier unchanged\. The only modification is the insertion of the perturbation wrapper from Section[3](https://arxiv.org/html/2608.07775#S3), which intercepts the agent–environment loop and applies controlled perturbations to the state, action, or transition component of the MDP\. Figure[5](https://arxiv.org/html/2608.07775#A1.F5)illustrates the overall framework ofAndroidReality, including the perturbation wrapper, the AndroidWorld task environment, and the agent\-emulator interaction loop\. Thus, the task goal and reward definition remain fixed, while the interaction conditions are made more realistic\.

#### Perturbation instantiation\.

Each task is paired with exactly one perturbation instance\. The perturbation random seed is fixed, so the same task–perturbation pair is realized identically for all evaluated models\. Unless otherwise specified, each perturbation uses a fixed default severity\.State\-levelperturbations persist throughout the episode as global interface configurations, including font size, display density, UI theme, orientation, locale/date format, and partial\-loading skeleton screens\.Action\-levelperturbations modify the execution of intended actions under seed\-controlled probabilities, including action delays, dropped actions, loading or frozen states, app hangs, and home resets\.Transition\-levelperturbations are injected at task\-specific runtime steps and introduce unexpected intermediate states, including advertisements, consent sheets, permission dialogs, security pop\-ups, rating prompts, notifications, and app/system update prompts\. The full parameter list and default values are provided in Table[4](https://arxiv.org/html/2608.07775#A1.T4); the evaluation protocol is described in Section[3\.3](https://arxiv.org/html/2608.07775#S3.SS3)\.

#### Evaluated models\.

We evaluate eight open\-source mobile agents from GUI\-OWL\[ye2025mobileagentv3\], GUI\-OWL\-1\.5\[xu2026mobileagentv35\], UI\-TARS\[qin2025uitars\], and UI\-TARS\-1\.5\[qin2025uitars\]\. The selected models cover22B to3232B parameter scales and include both supervised fine\-tuning and reinforcement\-learning\-based training paradigms\. This model suite allows us to test whether robustness failures are model\-specific or shared across current mobile\-agent systems\.

#### Model serving\.

For all experiments, agents observe the most recent screenshot and their action history\. We use the original released prompts for each model\. All models are served with vLLM using sampling temperature0\.10\.1\. Models with≤8\\leq 8B parameters are served on a single NVIDIA A100 GPU, while3232B models are served on two NVIDIA A100 GPUs\. The Android emulator runs on a separate CPU host and communicates with the inference server through the perturbation wrapper\.

#### Execution protocol\.

For each model, we evaluate the full task set under both clean and perturbed settings\. The clean setting is the original AndroidWorld environment, and the perturbed setting isAndroidRealitywith one perturbation profile applied to each task\. Each task–perturbation pair is evaluated once under the fixed seed\. Because perturbations may introduce additional intermediate states, interruptions, or delayed responses, all clean and perturbed runs use a step budget of20×20\\timesthe human reference length\.

#### Logging\.

During evaluation, we record the full interaction trajectory for each run, including the task goal, perturbation configuration, per\-step screenshots, action history, issued actions, model reasoning traces, and final verification result\. For both clean AndroidWorld and perturbedAndroidRealitysettings, each run uses a step budget of20×20\\timesthe human reference length to account for perturbation\-induced intermediate states, delayed responses, and recovery attempts\. For failed runs, we retain the complete trajectory to support the error analysis in Section[4\.3](https://arxiv.org/html/2608.07775#S4.SS3)and the TTIR intervention study in Section[5\.1](https://arxiv.org/html/2608.07775#S5.SS1)\. These logs enable manual attribution of failures to visual grounding errors, silent execution failures, off\-task engagement with distractions, or goal\-memory drift after interruptions\.

## Appendix CError Category Summary

This section provides a detailed reference for the four error categories identified in our analysis \(Section[4\.3](https://arxiv.org/html/2608.07775#S4.SS3)\)\. We first present a visual illustration of all four categories, followed by one representative example drawn from real failure trajectories\.

### C\.1Representative Failure Example

To complement the schematic illustration in Figure[2](https://arxiv.org/html/2608.07775#S4.F2), we present a real failure trajectory fromAndroidRealitythat demonstrates how a perturbation can derail an otherwise solvable task\. Figure[8](https://arxiv.org/html/2608.07775#A3.F8)shows a representative case in which the agent encounters a transition\-level perturbation in the form of an unexpected pop\-up\. Although the agent initially follows the correct task plan, the appearance of the pop\-up diverts its attention away from the original goal: the agent engages with the pop\-up content, follows its prompts, and never resumes the intended workflow\. The trajectory ultimately ends in failure, despite the same agent successfully completing this task in the clean setting\. This example concretely illustrates the failure mode introduced in Section[4\.3](https://arxiv.org/html/2608.07775#S4.SS3)and shows how a single transition\-level perturbation can trigger task failure through distraction\-induced trajectory drift\.

![Refer to caption](https://arxiv.org/html/2608.07775v1/x5.png)Figure 6:‘Clean\-before’ \(before applying AndroidReality\) vs\. ‘Perturbed\-after’ \(After applying all reality perturbation profiles\) success rates of GUI\-Owl\-1\.5\-32B across individual AndroidReality perturbation profiles\. Each subplot reports the success rate on the same task subset before and after applying one perturbation type, where null or skipped perturbed runs are counted as failures\. The results show that robustness degradation is highly perturbation\-specific\. State\-level perturbations that substantially alter visual grounding, such as UI theme changes, display\-density/size changes, and landscape orientation, cause the sharpest drops, in several cases reducing success to zero\. In contrast, format\-preserving perturbations such as locale/date\-format changes or partial\-loading skeletons are less disruptive \(note, it also varies on the model’s ability\)\. Action\-level perturbations show mixed effects: some execution failures such as loading delays and app hangs reduce performance, whereas short\-lived delays or dropped actions can be partially masked by the agent’s default retry behavior\. Transition\-level perturbations also vary widely, with security pop\-ups, update sheets, and notification messages exposing failures in distraction handling and task continuity\. Overall, this fine\-grained breakdown shows that AndroidReality reveals not only aggregate robustness degradation, but also which concrete real\-world interface variations most strongly challenge current mobile agents\.Table 5:Most influential perturbation types, defined as perturbation profiles for which all six models show lower success rates after perturbation\. Success rates are reported in percentages\.Δ\\Deltadenotes the absolute change from clean to perturbed performance, i\.e\., Perturbed−\-Clean\.ModelNotification MessageDisplay Size DPI CompactExecution Delay FrozenClean→\\rightarrowPert\.Δ\\DeltaClean→\\rightarrowPert\.Δ\\DeltaClean→\\rightarrowPert\.Δ\\DeltaGUI\-Owl\-1\.5\-2B60\.00→20\.0060\.00\\rightarrow 20\.00\-40\.0025\.00→12\.5025\.00\\rightarrow 12\.50\-12\.5066\.67→25\.0066\.67\\rightarrow 25\.00\-41\.67GUI\-Owl\-1\.5\-4B40\.00→0\.0040\.00\\rightarrow 0\.00\-40\.0062\.50→12\.5062\.50\\rightarrow 12\.50\-50\.0050\.00→25\.0050\.00\\rightarrow 25\.00\-25\.00GUI\-Owl\-7B60\.00→20\.0060\.00\\rightarrow 20\.00\-40\.0075\.00→12\.5075\.00\\rightarrow 12\.50\-62\.5083\.33→58\.3383\.33\\rightarrow 58\.33\-25\.00GUI\-Owl\-1\.5\-8B100\.00→20\.00100\.00\\rightarrow 20\.00\-80\.0062\.50→12\.5062\.50\\rightarrow 12\.50\-50\.0050\.00→41\.6750\.00\\rightarrow 41\.67\-8\.33GUI\-Owl\-32B80\.00→20\.0080\.00\\rightarrow 20\.00\-60\.0075\.00→25\.0075\.00\\rightarrow 25\.00\-50\.0083\.33→66\.6783\.33\\rightarrow 66\.67\-16\.67GUI\-Owl\-1\.5\-32B100\.00→0\.00100\.00\\rightarrow 0\.00\-100\.0056\.25→0\.0056\.25\\rightarrow 0\.00\-56\.2583\.33→66\.6783\.33\\rightarrow 66\.67\-16\.67Average73\.33→13\.3373\.33\\rightarrow 13\.33\-60\.0059\.38→12\.5059\.38\\rightarrow 12\.50\-46\.8869\.44→47\.2269\.44\\rightarrow 47\.22\-22\.22

![Refer to caption](https://arxiv.org/html/2608.07775v1/x6.png)Figure 7:Clean\-before vs\. Perturbed\-after success rates of GUI\-Owl\-1\.5\-8B across individual AndroidReality perturbation profiles\. Following the GUI\-Owl\-1\.5\-32B analysis, this figure shows that the same perturbation\-induced robustness gap also appears in the smaller 8B model, but with a different sensitivity profile\. Several state\-level perturbations continue to cause substantial degradation, especially display\-density changes and landscape orientation, suggesting that the model remains highly dependent on canonical visual layouts and screen geometry\. Interestingly,ui\_theme\_darkimproves from the clean subset to the perturbed subset in this split, indicating that some perturbation\-specific results can be affected by the underlying task subset and the model’s baseline capability on those tasks, rather than reflecting uniformly harmful perturbations\. Action\-level perturbations show mixed behavior: execution loading, frozen execution, and partial loading are relatively stable, while app hangs and random action drops produce larger failures, suggesting that the model can tolerate some transient execution noise but struggles when the action effect is lost or the task context must be recovered\. Transition\-level perturbations again expose large variation: security pop\-ups, interstitial ads, ad\-trial offers, notification messages, and notification delivery lead to clear drops, while bottom\-sheet consent, update\-system, and update\-app\-sheet perturbations can be less disruptive\.![Refer to caption](https://arxiv.org/html/2608.07775v1/x7.png)Figure 8:The actual failure case analysis of GUI\-Owl\-32B under a clean environment and AndroidReality\. In the clean setting, the model successfully completes the task by opening the Audio Recorder app, recording an audio clip, and saving it with the correct name\. In AndroidReality, however, a realistic security pop\-up interrupts the interaction flow\. The model fails to properly handle this unexpected intermediate state: it clicks into the security center and gets trapped in the security settings, and forgets the actual task to accomplish, eventually leading to task failure\. This example illustrates how AndroidReality exposes robustness challenges that are hidden in clean environments, especially the difficulty of recovering from realistic mobile interruptions\.

## Appendix DTest\-Time Introspective Recovery

### D\.1Detailed TTIR Strategies

This section provides the complete diagnostic signals and recovery prompts for each of the four TTIR strategies summarized in Section[5\.1](https://arxiv.org/html/2608.07775#S5.SS1)\.

#### ①Stuck\-loop detection⇒\\Rightarrowalternative\-action recovery\.

Diagnosis\.Stuck\-loop detection combines two complementary signals\. First, the diagnoser inspects the recent action history and is asked to judge whether the agent is caught in a repeated unresolved attempt, returningrepeat\_failure\_detectedas a boolean\. Second, as a deterministic safety net independent of the diagnoser’s output, we additionally trigger this signal whenever the lastNNsteps \(defaultN=5N\{=\}5\) emit identical action, summary, and thought\. Either signal alone is sufficient to trigger recovery, which provides robustness against cases where the diagnoser fails to flag a clear loop\.

Recovery\.TTIR triggers an*alternative\-action*recovery: the recovery agent is explicitly informed that the previous coordinate has been repeatedly attempted without producing the intended effect, and is instructed to select a different visible target or take a different route to the same sub\-goal, rather than retrying the failed coordinate\. The recovery prompt carries the previously failed action together with the diagnosed reason, anchoring the recovery agent on the specific stuck pattern rather than re\-planning the full task\. This breaks the stale\-coordinate loop by redirecting the agent toward an alternative grounding for the same intent\.

#### ②Action\-effect verification⇒\\Rightarrowalternative\-action recovery\.

Diagnosis\.The diagnoser first infers the agent’s previous*intended effect*from the last action, its summary, and its thought\. It then checks whether this intended effect is reflected in the current screenshot and assignsprevious\_intended\_effect\_satisfiedone of three values:yes,no, oruncertain\. A value ofnomeans that the action was issued, but the expected outcome is not visible on the screen\. This indicates a likely silent execution failure, such as a dropped, delayed, or absorbed action\.

Recovery\.TTIR triggers an*alternative\-action*recovery: the recovery agent is prompted to attempt a different way to achieve the same intended effect, for example, instead of blindly reissuing the same action, it may select a different visible target, scroll to reveal an occluded element, or substitute an equivalent action\. The prompt explicitly carries the previous intended effect as the recovery target, so that the recovery agent stays focused on the original sub\-goal rather than re\-planning the full task\.

#### ③Interference Detection⇒\\RightarrowDismiss\-Interference Recovery\.

Diagnosis\.The diagnoser inspects the screenshot for off\-task overlays such as pop\-ups, consent sheets, advertisements, system updates, and notification banners, and setsinterference\_presentaccordingly\. A value oftrueindicates that the agent should resolve the off\-task element before continuing with the main task\.

Recovery\.TTIR triggers a*dismiss\-interference*recovery, with the explicit instruction to close, decline, or navigate past the interfering element\. The recovery scope is restricted to interference resolution: the recovery agent is told that its only goal in this session is to make the original task UI visible again, and is forbidden from engaging with the content of the interruption \(e\.g\., it does not click “Sign up” inside an ad\-trial offer or proceed through a fake security wizard\)\. Once the diagnoser confirms the screen is no longer blocked by the interference, the session terminates and the main agent resumes\.

#### ④Task\-Alignment Check⇒\\RightarrowRe\-Orientation Recovery\.

Diagnosis\.The diagnoser determines whether the current screen still belongs to the original task path, assigningtask\_alignmentone ofon\_track,off\_track, oruncertain\. This signal is particularly relevant after a successful interference dismissal, where the agent may have lost track of the original task and landed on a home screen, an app drawer, or an unrelated screen, indicating goal memory drift\.

Recovery\.TTIR triggers a*re\-orientation*recovery: the recovery agent is prompted with the original task goal and the most recent on\-task subgoal, and is asked to issue the shortest action that restores the correct task context, for example, navigating back, re\-opening the relevant app, or returning to the previously active screen\. The recovery is again scoped: the agent does not attempt to complete the task, only to restore an on\-task state from which the main agent can resume\.

### D\.2Recovery Case Studies

In this section, we present qualitative examples in which each of the four recovery strategies discussed in Section[5\.1](https://arxiv.org/html/2608.07775#S5.SS1)successfully repairs a failed trajectory\. As shown in Table[6](https://arxiv.org/html/2608.07775#A4.T6), we summarize the mapping between the four recurring failure categories and the corresponding TTIR diagnosis and recovery strategies\.

Table 6:Mapping between the four recurring failure categories and the corresponding TTIR diagnosis and recovery strategies\.Failure CategoryTTIR DiagnosisRecovery Strategy① Figure[9](https://arxiv.org/html/2608.07775#A4.F9): Stale\-Coordinate Stuck① Stuck\-loop detectionAlternative\-action recovery② Figure[10](https://arxiv.org/html/2608.07775#A4.F10): Silent Action Failures② Action\-effect verificationAlternative\-action recovery③ Figure[11](https://arxiv.org/html/2608.07775#A4.F11): Off\-Task Engagement with Distractions③ Interference detectionDismiss\-interference recovery④ Figure[12](https://arxiv.org/html/2608.07775#A4.F12): Goal Memory Drift After Distraction④ Task\-alignment checkRe\-orientation recovery

![Refer to caption](https://arxiv.org/html/2608.07775v1/x8.png)Figure 9:①Case study: stuck\-loop recovery via TTIR\.The task asks the agent to opentask\.htmland navigate an “X” marker to the bottom\-right cell of a grid using the on\-screen*Up*,*Down*,*Left*, and*Right*buttons\.\(Top, Failure trajectory without TTIR\.\)GUI\-Owl\-7B openstask\.htmlcorrectly and identifies the*Right*button as the appropriate move\. However, its predicted tap coordinate does not register on the actual button under the perturbed layout, and the X remains in place\. Without an explicit stuck\-loop check, the agent continues to issue the same tap on the*Right*button across consecutive steps, never re\-grounding to a different visible target\. The trajectory exhausts the step budget and the X never reaches the bottom\-right cell\.\(Bottom, Success trajectory with TTIR\.\)With TTIR enabled, the same agent enters the same loop, but after three identical taps the diagnoser’s hard\-stuck rule fires \(same\_issue\_streak = 3\)\. This triggers analternative\_actionrecovery: the recovery agent is instructed that the previous coordinate has been repeatedly attempted without effect, and is asked to choose a different visible target or a different route to the same sub\-goal\. Following the recovery action, the agent successfully advances the X and ultimately reaches the bottom\-right cell, completing the task\.![Refer to caption](https://arxiv.org/html/2608.07775v1/x9.png)Figure 10:②Case study: silent execution failure recovered by TTIR\.The task asks the agent to create a new contact for “Hugo Pereira” with phone number\+13920741751, under thefont\_size\_smallstate\-level perturbation that compresses the form layout\.\(Top, Failure trajectory without TTIR\.\)GUI\-Owl\-7B successfully fills in the*First name*and*Last name*fields, then attempts to type the phone number into the*Phone*field\. However, because the perturbed font size shifts the relative positions of the form fields, the typed digits are silently entered into the*Company*field instead, as highlighted by the red box\. Without an explicit verification step between issuing the action and continuing, the agent does not notice the misplacement and proceeds to save the contact, producing a final record where the phone number is stored under*Company*rather than*Phone*\. The task is judged as failed\.\(Bottom, Success trajectory with TTIR\.\)With TTIR enabled, the same misplacement occurs initially, but before the agent’s next action, the diagnoser compares the intended effect \(“phone number entered in the Phone field”\) against the current screenshot and setsprevious\_intended\_effect\_satisfied = no\. This triggers analternative\_actionrecovery: the recovery agent clears the misplaced text and re\-enters the phone number into the correct*Phone*field\. After the recovery completes, the main agent saves the contact with all fields correctly filled, and the task succeeds\.![Refer to caption](https://arxiv.org/html/2608.07775v1/x10.png)Figure 11:③Case study: off\-task engagement with distractions via TTIR\.The agent is asked to delete three recipes \(Zucchini, Garlic, Soup\) from the Broccoli recipe app, under thepopup\_securitytransition\-level perturbation that injects a fake “Verify this action” security pop\-up upon launching the app\.\(Top, Failure trajectory without TTIR\.\)GUI\-Owl\-7B encounters the security pop\-up but fails to recognize it as off\-task\. Instead of dismissing it, the agent clicks the*Continue*button, treating the pop\-up as part of the intended workflow\. This leads it into a multi\-step “Security Center / Protected action review” wizard injected by the perturbation, where the agent dutifully reviews the risk summary, checks all verification checkboxes across three pages, and proceeds to the approval\-scope screen\. By the time the wizard ends, the step budget is exhausted; the agent never reaches the Broccoli app, and not a single recipe is deleted\.\(Bottom, Success trajectory with TTIR\.\)With TTIR enabled, the diagnoser inspects the same screen and identifies the security pop\-up as off\-task, settinginterference\_present = true\. This triggers adismiss\-interferencerecovery: the recovery agent clicks the*Cancel*button \(red box\) and the pop\-up disappears, restoring the original task UI\. Control then returns to the main agent, which proceeds normally to open the Broccoli app, locate each target recipe, and delete them through the recipe\-detail menu, completing the task successfully\.![Refer to caption](https://arxiv.org/html/2608.07775v1/x11.png)Figure 12:④Case study: goal drift detection via TTIR\.The agent is asked to set the screen brightness to its maximum value, under thenotification\_messagetransition\-level perturbation that injects a “New message” notification banner partway through the trajectory\.\(Top, Failure trajectory without TTIR\.\)GUI\-Owl\-7B begins correctly by pulling down the Quick Settings panel to access the brightness control\. A new message notification then appears and is dismissed by the agent\. However, after the dismissal, the agent finds itself back on the home screen and, lacking any check that this state is still consistent with the original task, concludes that the task is complete and emitsterminate\(success\)without ever adjusting the brightness slider\. Although the agent successfully handled the distraction itself, the brief interruption was sufficient to disrupt its working memory of the original task plan\.\(Bottom, Success trajectory with TTIR\.\)The same dismissal occurs, but TTIR’s diagnoser then compares the current screen against the task goal and setstask\_alignment = off\-track, recognizing that returning to the home screen does not satisfy a brightness\-adjustment task\. This triggers are\-orientationrecovery: the recovery agent re\-opens the Quick Settings panel, locates the brightness slider, and drags it to the maximum value, restoring the trajectory to the original task path\. Control then returns to the main agent, which verifies the change and the task succeeds\.