AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees

# Visual Token Reduction for GUI Agents with Adaptive Quadtrees
Source: [https://arxiv.org/html/2605.19260](https://arxiv.org/html/2605.19260)
Yuankai Li UC Davis ykali@ucdavis\.edu &Tinghui Zhu UC Davis thuzhu@ucdavis\.edu &Ha Min Son UC Davis hmson@ucdavis\.edu &Zhe Zhao UC Davis zao@ucdavis\.edu &Xin Liu UC Davis xinliu@ucdavis\.edu &Muhao Chen UC Davis muhchen@ucdavis\.edu

###### Abstract

Large Multimodal Models \(LMMs\) have recently emerged as promising backbones for Graphical User Interfaces \(GUIs\) agent models, where high\-resolution GUI screenshots are introduced to the prompts at each iteration step\. However, these screenshots exhibit highly non\-uniform spatial information density: large regions may carry little information and are visually homogeneous, while key text and icons may require high visual fidelity\. Existing approaches to this problem either require additional training or rely on attention\-based token compression, ignoring the structured layout and spatial redundancy of GUI screenshots\. To fill the gap, this paper proposesAQuaUI, a training\-free inference\-time token reduction method for GUI agent models that utilizes the non\-uniform information density in screenshots\. AQuaUI constructs an adaptivequadtreeon each screenshot input and keeps one representative merged token per leaf of the quadtree\. AQuaUI preserves the spatial positions of retained tokens throughout the pipeline to ensure that all position\-encoding stages remain consistent\. To further improve temporal consistency across multi\-step GUI interactions, we propose aconditional quadtreealgorithm that leverages the continuity between consecutive screenshots within a single request\. Specifically, it refines the current quadtree using previous quadtrees as references, helping preserve fine\-grained regions across static or mildly shifted GUI states\. We implement AQuaUI on state\-of\-the\-art GUI agent models and conduct experiments on standard grounding and navigational benchmarks\. AQuaUI consistently shows improved accuracy–efficiency trade\-offs over prior baselines\. Notably, on GUI\-Owl\-1\.5\-32B\-Instruct, AQuaUI achieves up to13\.22%speedup and29\.52%fewer visual tokens while retaining99\.06%of full\-token performance, suggesting that the spatial redundancy of GUI screenshots can be exploited at inference time without retraining\.

## 1Introduction

With the rapid development of Large Multimodal Models \(LMMs\), current state\-of\-the\-art models have gained the ability to understand and navigate through complex UI elements in Graphical User Interfaces \(GUIs\)\. This, combined with research progress within autonomous agents, has led to various GUI agents designed to interpret user instructions and autonomously perform complex tasks\. As a result, GUI agents have emerged as a new research focus of agentic AI\.

Earlier GUI agents often adopt a hybrid way that combines textual representations, such as accessibility trees or HTML, with visual information\[[8](https://arxiv.org/html/2605.19260#bib.bib14),[38](https://arxiv.org/html/2605.19260#bib.bib15),[27](https://arxiv.org/html/2605.19260#bib.bib26),[19](https://arxiv.org/html/2605.19260#bib.bib30)\]\. Such systems, however, often face cross\-platform adaptability and require frequent task\-specific optimization\. In response to these limitations, native GUI agent models that directly take screenshots as input\[[33](https://arxiv.org/html/2605.19260#bib.bib32),[1](https://arxiv.org/html/2605.19260#bib.bib27),[16](https://arxiv.org/html/2605.19260#bib.bib29)\]have become increasingly popular as they demonstrate competent performance on trending GUI agent benchmarks\[[31](https://arxiv.org/html/2605.19260#bib.bib12),[25](https://arxiv.org/html/2605.19260#bib.bib11)\]\. Screenshots provide a natural perspective of understanding GUIs and a unified solution in cases where accessibility trees are unavailable\. The end\-to\-end design also offers a data\-centric view that allows model performance to improve through data scaling and iterative feedback\.

Such GUI agent models typically needs to process high\-resolution screenshots at every interaction step, resulting in a massive number of visual tokens even with merging and patching techniques\. For example, a typical Pixel 6 phone screenshot with a resolution of 2400×\\times1080, when processed with Qwen3\-VL\[[2](https://arxiv.org/html/2605.19260#bib.bib33)\], can produce around 3000 tokens\. This substantially increases computational cost and limits the amount of conversation history that can be retained within the context window\. Token compression, therefore, has become increasingly important for GUI agents, as it can reduce inference cost, improve throughput, and enable longer interaction histories\.

Recent research observes that GUI screenshots contain substantially more low\-information regions than natural images, while also exhibiting simpler and more regular spatial structures\[[10](https://arxiv.org/html/2605.19260#bib.bib5),[16](https://arxiv.org/html/2605.19260#bib.bib29)\]\. Previous works of token compression in Vision Language Models \(VLMs\) target natural images and are therefore not well suited to GUI images\[[5](https://arxiv.org/html/2605.19260#bib.bib38),[12](https://arxiv.org/html/2605.19260#bib.bib39),[36](https://arxiv.org/html/2605.19260#bib.bib8)\]\. Other works that focus on GUI images develop token reduction methods as part of GUI agent model training\[[16](https://arxiv.org/html/2605.19260#bib.bib29),[22](https://arxiv.org/html/2605.19260#bib.bib40)\], leaving open whether they can be directly applied to arbitrary GUI agent models without training\.

To fill this gap, we propose AQuaUI, a training\-free framework that can be easily applied to any downstream LMMs\. We usequadtree\[[11](https://arxiv.org/html/2605.19260#bib.bib35)\]to select certain tokens before they are send to the text transformer\. The pipeline is illusatrated in[Figure˜1](https://arxiv.org/html/2605.19260#S1.F1)\.

We evaluate AQuaUI on grounding benchmarks including UI\-Vision\[[21](https://arxiv.org/html/2605.19260#bib.bib31)\], ScreenSpot\-Pro, ScreenSpot\-V2\[[13](https://arxiv.org/html/2605.19260#bib.bib13)\], OSWorld\-G\[[31](https://arxiv.org/html/2605.19260#bib.bib12)\]and MMBench\-GUI\[[29](https://arxiv.org/html/2605.19260#bib.bib19)\]\. For navigational evalution, we adopt AndroidWorld\[[25](https://arxiv.org/html/2605.19260#bib.bib11)\]and AndroidControl\[[14](https://arxiv.org/html/2605.19260#bib.bib3)\]\. AQuaUI can outperform prior baselines with better accuracy\-latency trade\-off, especially with larger models\. On Qwen3\-VL models, AQuaUI can compress the visual tokens by 30% without hurting overall performance\.

In summary, the main contribution of our paper can be summarized as follows:

- •We propose AQuaUI, a training\-free inference\-time visual\-token reduction method that exploits the structural layout of GUI screenshots using an adaptive quadtree\.
- •We introduce a conditional quadtree algorithm that uses previous screenshots within the same request to preserve fine\-grained partitions across static or mildly shifted screenshots\.
- •We implement AQuaUI on various state\-of\-the\-art GUI agent models and evaluate it on standard GUI grounding and navigation benchmarks, showing improved accuracy–efficiency trade\-offs over prior baselines\.

![Refer to caption](https://arxiv.org/html/2605.19260v1/x1.png)

Figure 1:An overview of the entire pipeline of AQuaUI\. Given a GUI screenshot, AQuaUI will first build an adaptive quadtree to capture the UI layout\. For each leaf node in that quadtree, all but one representative tokens are discarded\. These representative tokens are sent to the languge model with their position information for the final output\. Additionally, we design a conditional quadtree refinement algorithm that reuses previous screenshots when building the quadtree to remain token consistency\.
## 2Related Work

### 2\.1GUI Agents

Recent work on GUI agents has rapidly expanded both the scale of backbone models and the range of supported platforms\. UI\-TARS\[[24](https://arxiv.org/html/2605.19260#bib.bib7)\]and GUI\-Owl\[[35](https://arxiv.org/html/2605.19260#bib.bib1)\]train LMMs on screen\-grounded trajectories, while UI\-Voyager\[[17](https://arxiv.org/html/2605.19260#bib.bib2)\], MAI UI\[[37](https://arxiv.org/html/2605.19260#bib.bib10)\], and ClawGUI\[[26](https://arxiv.org/html/2605.19260#bib.bib44)\]explore complementary directions in planning, multi\-agent coordination, and tool use\. Despite various differences, these systems share a common serving bottleneck: each interaction step introduces high\-resolution screenshots whose visual tokens can dominate prefill latency, attention cost, and KV\-cache memory\.

Several recent methods recognize that UI screenshots require specialized treatment\. ShowUI\[[16](https://arxiv.org/html/2605.19260#bib.bib29)\]reduces visual tokens by leveraging UI structural regularity, while FocusUI\[[22](https://arxiv.org/html/2605.19260#bib.bib40)\]selects instruction\-relevant UI patches and introduces a position\-aware strategy to mitigate the loss of positional continuity\. These methods highlight the importance of spatial structure for UI grounding, but their main results are obtained in model\-specific training settings\. In contrast, AQuaUI targets training\-free deployment on off\-the\-shelf models\.

### 2\.2Compression Methods for LMMs

Token Pruning\.A broad line of work reduces the number of visual tokens by estimating token importance from attention and similarity\. FastV\[[5](https://arxiv.org/html/2605.19260#bib.bib38)\]prunes visual tokens in deep layers using attention from the language stream; G\-Prune\[[12](https://arxiv.org/html/2605.19260#bib.bib39)\]models token importance through graph propagation; and SimIgnore\[[36](https://arxiv.org/html/2605.19260#bib.bib8)\]uses cross\-modal similarity to discard redundant patches\. Recent generic pruners such as HiPrune\[[18](https://arxiv.org/html/2605.19260#bib.bib21)\]and VisionZip\[[34](https://arxiv.org/html/2605.19260#bib.bib22)\]further improve training\-free visual\-token reduction\. These methods primarily frame reduction as token\-importance estimation\. GUI screenshots introduce a different situation where low\-attention tokens may still be important because they define labels or coordinate context\. Equally importantly, such methods rely on explicit calculation of attention maps, which could add extra complexity to FlashAttention\[[7](https://arxiv.org/html/2605.19260#bib.bib23)\]or KV Cache management\.

KV\-cache compression\.Another line of work compresses the KV cache after tokens have already entered the transformer\. SnapKV\[[15](https://arxiv.org/html/2605.19260#bib.bib34)\]selects KV entries using attention from an observation window, while PyramidKV\[[4](https://arxiv.org/html/2605.19260#bib.bib4)\]allocates different cache budgets across different layers\. Subsequent methods improve selection with diversity terms, pre\-RoPE scores, or mixed policies\[[3](https://arxiv.org/html/2605.19260#bib.bib17),[20](https://arxiv.org/html/2605.19260#bib.bib16)\]\. GUI KV\[[10](https://arxiv.org/html/2605.19260#bib.bib5)\]specifically calculates the spatial and temporal scoring of KV cache based on GUI screenshots\. These techniques are largely orthogonal to AQuaUI: they reduce memory and decoding cost after visual tokens have been encoded, whereas AQuaUI reduces the visual token sequence itself before KV\-cache is computed\.

## 3Method

In this section, we formalize the problem in[Section˜3\.1](https://arxiv.org/html/2605.19260#S3.SS1), then present the basic token reduction method for AQuaUI in[Section˜3\.2](https://arxiv.org/html/2605.19260#S3.SS2)\. Building a conditional quadtree using previous images is discussed in[Section˜3\.3](https://arxiv.org/html/2605.19260#S3.SS3)\.

### 3\.1Task Definition

At each steptt, a GUI agent receives a screenshotsts\_\{t\}and a text promptptp\_\{t\}containing the user instruction and interaction history\. A native GUI agent model produces a response and action\(rt,at\)=ℳ​\(pt,\{si\}i=0t\)\(r\_\{t\},a\_\{t\}\)=\\mathcal\{M\}\(p\_\{t\},\\\{s\_\{i\}\\\}\_\{i=0\}^\{t\}\)\. Each screenshotsts\_\{t\}is encoded as a sequence of visual tokenstokvis\\text\{tok\}\_\{\\text\{vis\}\}using a vision encoder, which are processed jointly with text tokenstoktext\\text\{tok\}\_\{\\text\{text\}\}\. The goal is to find a different set of tokenstokcomp\\text\{tok\}\_\{\\text\{comp\}\}whose length\|tokcomp\|<\|tokvis\|\|\\text\{tok\}\_\{\\text\{comp\}\}\|<\|\\text\{tok\}\_\{\\text\{vis\}\}\|while preserving task performance\.

### 3\.2Adaptive Quadtree Tokenization

For GUI screenshots, information is particularly non\-uniformly distributed\[[16](https://arxiv.org/html/2605.19260#bib.bib29)\], and we propose aquadtreealgorithm inspired by previous work\[[11](https://arxiv.org/html/2605.19260#bib.bib35),[6](https://arxiv.org/html/2605.19260#bib.bib37)\]\. A quadtree is a hierarchical data structure that recursively partitions a surface into four quadrants until a stopping criterion is satisfied\. In image processing, quadtrees have been widely used as compact image representations: homogeneous regions are represented by large leaves, while visually complex regions are recursively divided into smaller cells\. This property makes quadtrees particularly suitable for GUI screenshots with large background panels, margins as well as small concentrated regions with text and icons\. A quadtree can therefore adapt its partition to the information density of the screenshot, using coarse leaves for redundant regions and fine leaves for detail\-rich regions\.

Based on this observation, AQuaUI first build an adaptive quadtree, and then uses its leaves as adaptive visual\-token units, which we describe below\. The detailed algorithm and implementation is discussed in[Appendix˜A](https://arxiv.org/html/2605.19260#A1)\.

##### Grid Alignment and Boundary Handling

Since dimensions of GUI screenshots are not necessarily powers of two, which is a precondition of applying quadtree, AQuaUI therefore first decomposes the image into two parts: a centered region that can be tiled by square chunks of sizeC×CC\\times C, and boundary margins that cannot be included in this chunk layout, which are kept untouched\. AQuaUI then runs an adaptive quadtree independently inside each chunk\.

##### Splitting Criteria

For a quadtree nodennof sizewn×hnw\_\{n\}\\times h\_\{n\}, the default split criteria is the area\-weighted grayscale variances​\(n\)=wn​hn⋅Var⁡\(grey⁡\(n\)\)s\(n\)=w\_\{n\}h\_\{n\}\\cdot\\operatorname\{Var\}\(\\operatorname\{grey\}\(n\)\), and we recursively split untils​\(n\)≤1000⋅αs\(n\)\\leq\\ 1000\\cdot\\alpha\.

The implementation also supports a gradient\-based criterion, where the score functionssgives the maximum magnitude of the gradient of nodenn, which can be useful when the edges provide a stronger signal under certain circumstances\. We will discuss the choice of different criteria in[Section˜5\.2](https://arxiv.org/html/2605.19260#S5.SS2)

##### Representative Token Selection

For each final leaf, AQuaUI selects the representative token block\. By default, this is the center block\(x′=⌊x0\+x12⌋,y′=⌊y0\+y12⌋\)\(x^\{\\prime\}=\\lfloor\\frac\{x\_\{0\}\+x\_\{1\}\}\{2\}\\rfloor,\\quad y^\{\\prime\}=\\lfloor\\frac\{y\_\{0\}\+y\_\{1\}\}\{2\}\\rfloor\)if the leaf spans block coordinates fromx0x\_\{0\}tox1x\_\{1\}andy0y\_\{0\}toy1y\_\{1\}\.

### 3\.3Conditional Quadtree Building

GUI trajectories often contain adjacent screenshots that are nearly identical\. A page may remain static, a menu may pop up, or a scroll operation may shift content by a few token blocks\. If each screenshot is compressed independently, a later frame may discard details that were preserved in an earlier frame, which can weaken the reasoning and grounding ability\.

AQuaUI introduces a conditional quadtree mechanism that infers history from the ordered images in the same request\. It is a heuristic algorithm that builds quadtree conditioned on the state change of each chunk in one of three modes: static, shifted, or replaced\.

##### Static mode

We begin by identifying patterns that most pixels remain the same, which we call the Static mode\. We define a similarity value between two chunks\. Consider the chunk AQuaUI produced previously, we summarize it into a condensed grid signatureSSof size\(Hc,Wc\)\(H\_\{c\},W\_\{c\}\)by computing the greyscale of each block of size\(b,b\)\(b,b\)so that each cell ofSSstores the rounded mean grayscale of the correspondingb×bb\\times bblock\.\(Hc,Wc\)\(H\_\{c\},W\_\{c\}\)denotes the number of blocks within a chunk\.

We define similarity of two adjacent signaturesStS\_\{t\}andSt−1S\_\{t\-1\}as following:

similarity​\(St,St−1\)=1−1255​Hc​Wc×∑i,j\|St​\(i,j\)−St−1​\(i,j\)\|\\text\{similarity\}\(S\_\{t\},S\_\{t\-1\}\)=1\-\\frac\{1\}\{255H\_\{c\}W\_\{c\}\}\\times\\sum\_\{i,j\}\|S\_\{t\}\(i,j\)\-S\_\{t\-1\}\(i,j\)\|A region is static ifsimilarity≥τs​t​a​t​i​c\\text\{similarity\}\\geq\\tau\_\{static\}\.

##### Shifted mode

If two chunks are not static, we then consider if a scroll operation happens\. We call this the Shifted mode\. To capture the scrolling feature, we add a block\-level shiftδ=\(Δ​i,Δ​j\)\\delta=\(\\Delta i,\\Delta j\)in the previous definition and change to average over the overlapping area\. We search over\|Δ​i\|,\|Δ​j\|\|\\Delta i\|,\|\\Delta j\|in certain range and classify the region as shifted if the best valid shiftδ⋆\\delta^\{\\star\}satisfies

sim⁡\(δ⋆\)≥τshiftandsim⁡\(δ⋆\)≥sim⁡\(St,St−1\)\+γ\.\\operatorname\{sim\}\(\\delta^\{\\star\}\)\\geq\\tau\_\{\\mathrm\{shift\}\}\\quad\\text\{and\}\\quad\\operatorname\{sim\}\(\\delta^\{\\star\}\)\\geq\\operatorname\{sim\}\(S\_\{t\},S\_\{t\-1\}\)\+\\gamma\.

##### Replaced mode

Regions satisfying neither condition are treated as replaced\. A new quadtree is constructed where no priors are used\.

We set these parameters heuristically, whereτs​t​a​t​i​c=0\.97,τs​h​i​f​t=0\.94,γ=0\.03\\tau\_\{static\}=0\.97,\\tau\_\{shift\}=0\.94,\\gamma=0\.03and the shift search range is\|Δ​i\|,\|Δ​j\|≤4\|\\Delta i\|,\|\\Delta j\|\\leq 4blocks\. The full algorithm can be seen in[Algorithm˜1](https://arxiv.org/html/2605.19260#alg1)\([Section˜A\.4](https://arxiv.org/html/2605.19260#A1.SS4)\)\.

## 4Experiment and Results

We introduce the basic setup of the experiment in[Section˜4\.1](https://arxiv.org/html/2605.19260#S4.SS1), and then compare AQuaUI with the baseline in[Section˜4\.2](https://arxiv.org/html/2605.19260#S4.SS2)\. We continue to explore the generalization of AQuaUI on different models with Qwen3\-VL architecture\. Navigational experiments using AndroidControl and AndroidWorld are discussed in[Section˜4\.3](https://arxiv.org/html/2605.19260#S4.SS3)\.

### 4\.1Experiment Setup

Benchmarks\.To evaluate the performance of AQuaUI, we evaluate it on multiple benchmarks\. Thanks to ClawGUI’s\[[26](https://arxiv.org/html/2605.19260#bib.bib44)\]effort on reproducability of benchmarking GUI agent models, we reuse the majority of its experiment setting, which consists of various grounding benchmarks including ScreenSpot\-Pro\[[13](https://arxiv.org/html/2605.19260#bib.bib13)\], ScreenSpot\-V2\[[30](https://arxiv.org/html/2605.19260#bib.bib9)\], OSWorld\-g\[[31](https://arxiv.org/html/2605.19260#bib.bib12)\], UI\-Vision\[[21](https://arxiv.org/html/2605.19260#bib.bib31)\], MMBench\-GUI\[[29](https://arxiv.org/html/2605.19260#bib.bib19)\], and one offline navigational benchmark, AndroidControl\[[14](https://arxiv.org/html/2605.19260#bib.bib3)\]\. For online navigational benchmark, we choose AndroidWorld\[[25](https://arxiv.org/html/2605.19260#bib.bib11)\]\. We use grounding benchmarks as the main evaluation because they provide controlled, reproducible measurement, and report navigation benchmarks as complementary interactive evaluations\.

Models\.We evaluate AQuaUI on the Qwen2\-VL\[[28](https://arxiv.org/html/2605.19260#bib.bib20)\]series and the ShowUI\[[16](https://arxiv.org/html/2605.19260#bib.bib29)\]series for a direct comparison withShowUI\. Additionally, to examine the generalization ability, we evaluate several series of SOTA GUI agent model such as the GUI\-Owl\-1\.5\[[32](https://arxiv.org/html/2605.19260#bib.bib41)\], the MAI\-UI\[[37](https://arxiv.org/html/2605.19260#bib.bib10)\]and the Qwen3\-VL\[[2](https://arxiv.org/html/2605.19260#bib.bib33)\]series\. These models share the same backbone model as Qwen3\-VL but differs with different training strategy\.

Metrics\.For grounding benchmarks and AndroidControl, we report click accuracy \(Acc\.\)\. For AndroidWorld, we report success rate \(SR\)\.

Baseline\.Our primary baseline is ShowUI’s UI\-guided token selection, the closest GUI\-specific visual\-token reduction method with available implementation\. We also include a random reduction baseline and FastV in[Section˜5\.1](https://arxiv.org/html/2605.19260#S5.SS1)to test whether AQuaUI actually benefits from quadtree structure\. The model choice is limited byShowUI’s implementation, as no recent implementation on Qwen3\-VL series is provided\.

### 4\.2Main Results

[Table˜1](https://arxiv.org/html/2605.19260#S4.T1)reports the performance and latency differences between AQuaUI andShowUIfor five grounding benchmarks\.[Table˜2](https://arxiv.org/html/2605.19260#S4.T2)reports the result of AQuaUI on other SOTA GUI agent models\.[Figure˜2](https://arxiv.org/html/2605.19260#S4.F2)displays the performance breakdown on UI\-Vision and ScreenSpot\-V2\. The rest of breakdown performance is reported in detail in[Appendix˜B](https://arxiv.org/html/2605.19260#A2)\. We make the following observations:

AQuaUI preserves task accuracy better at comparable compression rates\.At similar compression rates, AQuaUI consistently produces smaller average accuracy degradation thanShowUI’s UI\-guided visual token selection\. On Qwen2\-VL\-7B\-Instruct, AQuaUI reduces tokens by 26\.82% and merely drops accuracy by 3\.29 points on average compared toShowUI’s 5\.95\-point drop at 27\.55% compression\. A similar trend is observed on ShowUI\-2B with a slightly higher compression rate and a smaller average accuracy drop\.

Token reduction introduces noticable overhead under Qwen2\-VL implementations\.BothShowUIand AQuaUI introduce additional latency in this setting, despite reducing the number of visual tokens\. This suggests that the overhead of token\-selection logic on Qwen2\-VL can not be neglected, especially for smaller models where the full\-token inference cost is already low\. Nevertheless, AQuaUI still introduces substantially less overhead on Qwen2\-VL\-7B\-Instruct thanShowUI, with \+0\.02s comparing to \+0\.31s, indicating AQuaUI is more efficient when serving\.

Spatially structured tasks benefit more from AQuaUI\.[Figure˜2](https://arxiv.org/html/2605.19260#S4.F2)suggests that AQuaUI is particularly effective on tasks requiring preserving structured GUI evidence\. On ScreenSpot\-V2, AQuaUI introduce ssmaller degradation than ShowUI on Desktop and Web for Qwen2\-VL\-7B\-Instruct, suggesting that AQuaUI better preserves layout\-sensitive visual context\.

Table 1:Accuracy and latency across GUI benchmarks compared withShowUI’s UI Guided Token Selection\. UI\_mask\_ratio is set to default 0\.5\. Accuracy and compression are reported in percentages; latency is reported in seconds\. SS\-Pro: ScreenSpot\-Pro; SS\-v2: ScreenSpot\-V2; OSW\-G: OSWorld\-G; UI\-V: UI\-Vision; MMB\-GUI: MMBench\-GUI\.BackendSS\-ProSS\-V2OS\-GUI\-VISIONMMB\-GUIComp\.Avg\.Δ\\DeltaAcc\.Lat\.Acc\.Lat\.Acc\.Lat\.Acc\.Lat\.Acc\.Lat\.Acc\.Lat\.Qwen2\-VL\-7B\-InstructTransformers10\.311\.1067\.301\.0321\.141\.253\.411\.1739\.571\.320––\+ShowUI8\.601\.4855\.111\.2816\.871\.612\.721\.4228\.691\.6227\.55\-5\.95\+0\.31vLLM10\.441\.8767\.141\.2919\.361\.483\.341\.3839\.901\.570––\+AQuaUI9\.872\.0160\.061\.3017\.941\.503\.101\.3032\.801\.5926\.82\-3\.29\+0\.02ShowUI\-2BTransformers7\.530\.7476\.650\.6118\.290\.645\.970\.6543\.490\.660––\+ShowUI5\.950\.7865\.720\.7116\.340\.725\.040\.6633\.670\.7627\.78\-5\.04\+0\.07vLLM7\.591\.1876\.490\.5318\.120\.456\.060\.4444\.050\.640––\+AQuaUI6\.581\.3564\.070\.6216\.340\.525\.840\.5536\.000\.7028\.24\-4\.70\+0\.10![Refer to caption](https://arxiv.org/html/2605.19260v1/x2.png)Figure 2:Performance breakdown on UI\-Vision and ScreenSpot\-V2\. More detailed results can be seen in[Appendix˜B](https://arxiv.org/html/2605.19260#A2)\.AQuaUI preserves accuracy consistently on Qwen3\-VL\.Across the Qwen3\-VL series, AQuaUI reduces around 30% of visual tokens while incurring less than one point of average accuracy degradation\. The average drops are only 0\.61, 0\.48, and 0\.85 points for Qwen3\-VL\-2B, 8B, and 32B, respectively\. This suggests that AQuaUI is more robust with Qwen3\-VL’s positional embedding design\.

Latency gains emerge as model size increases\.Although token reduction introduces additional overhead, the savings become visible for larger backbones\. For Qwen3\-VL\-8B and Qwen3\-VL\-32B, AQuaUI reduces average latency by 0\.09s and 0\.23s\. Similar trends appear on other larger models\. In contrast, the 2B models show little latency changes\. This suggests that inference\-time visual\-token reduction is especially valuable for larger GUI agents, where the language model computation dominates the reduction overhead\.

Table 2:Accuracy and latency across GUI benchmarks for various GUI agent models\. Accuracy and compression are reported in percentages; latency is reported in seconds\. SS\-Pro: ScreenSpot\-Pro; SS\-V2: ScreenSpot\-v2; OSW\-G: OSWorld\-G; UI\-V: UI\-Vision; MMB\-GUI: MMBench\-GUI\.BackendSS\-ProSS\-V2OS\-GUI\-VISIONMMB\-GUIComp\.Avg\.Δ\\DeltaAcc\.Lat\.Acc\.Lat\.Acc\.Lat\.Acc\.Lat\.Acc\.Lat\.Acc\.Lat\.Qwen3\-VL\-2B\-InstructvLLM43\.961\.1788\.840\.4348\.490\.3115\.240\.3873\.340\.500––\+AQuaUI42\.881\.1788\.840\.4648\.490\.3115\.060\.4171\.560\.5229\.52\-0\.61\+0\.02Qwen3\-VL\-8B\-InstructvLLM56\.231\.9594\.260\.7459\.680\.5227\.780\.6384\.310\.860––\+AQuaUI55\.281\.6794\.260\.7159\.330\.5227\.720\.5883\.280\.7729\.72\-0\.48\-0\.09Qwen3\-VL\-32B\-InstructvLLM58\.573\.2494\.891\.2966\.250\.9736\.171\.3587\.151\.510––\+AQuaUI56\.172\.4495\.051\.1764\.301\.0335\.961\.2387\.311\.3329\.52\-0\.85\-0\.23MAI\-UI\-2BvLLM58\.891\.2592\.450\.5055\.240\.3729\.990\.4182\.640\.570––\+AQuaUI57\.501\.2591\.430\.5155\.240\.3629\.570\.4381\.440\.5829\.52\-0\.81\+0\.01MAI\-UI\-8BvLLM64\.522\.1594\.811\.0464\.120\.8640\.450\.9488\.621\.200––\+AQuaUI59\.141\.8894\.180\.9763\.590\.8539\.820\.8987\.511\.0929\.25\-1\.66\-0\.10GUI\-Owl\-1\.5\-8B\-InstructvLLM68\.882\.6993\.400\.8867\.140\.6537\.270\.9282\.611\.080––\+AQuaUI67\.682\.2193\.320\.8564\.650\.6636\.720\.8381\.860\.9729\.52\-1\.01\-0\.10GUI\-Owl\-1\.5\-32B\-InstructvLLM70\.973\.3094\.421\.2866\.430\.9537\.451\.2987\.151\.500––\+AQuaUI69\.892\.7794\.031\.1665\.900\.8936\.721\.1186\.531\.3029\.52\-0\.67\-0\.22
### 4\.3Navigational Results

Table 3:Accuracy for various GUI agent models on AndroidControl\.BackendHighLowComp\.Avg\.Δ\\DeltaAcc\.Lat\.Acc\.Lat\.Acc\.Lat\.Qwen3\-VL\-2B\-InstructvLLM54\.800\.6773\.830\.660––\+AQuaUI54\.140\.7873\.370\.7329\.33\-0\.56\+0\.09Qwen3\-VL\-8B\-InstructvLLM56\.971\.2866\.261\.230––\+AQuaUI56\.431\.2166\.321\.1829\.33\-0\.24\-0\.06Table 4:Success rate on AndroidWorld\. Random seed is fixed to3030\. Accuracy and compression are reported in percentages\.BackendSR\.Comp\.UI\-VoyagervLLM70\.69–\+AQuaUI68\.1031\.90\- Conditional Quadtree65\.5232\.06MAI\-UI\-8BvLLM57\.76–\+AQuaUI56\.0331\.89\- Conditional Quadtree53\.4432\.06[Tables˜3](https://arxiv.org/html/2605.19260#S4.T3)and[4](https://arxiv.org/html/2605.19260#S4.T4)evaluate AQuaUI on AndroidControl and AndroidWorld\. For AndroidWorld, random seed is fixed to3030\. For MAI\-UI\-8B’s result on AndroidWorld, we report our reproduced dense vLLM baseline under the same evaluation protocol used for AQuaUI\.

AQuaUI remains stable on AndroidControl\.On AndroidControl, AQuaUI preserves performance under roughly 29% visual token compression\. For Qwen3\-VL\-2B\-Instruct, the average accuracy drop across high\- and low\-level instructions is only 0\.56 points\. This indicates that the quadtree reduction remains effective beyond static grounding benchmarks and transfers to navigational settings\.

Interactive long\-horizon evaluation is more sensitive\.On AndroidWorld, AQuaUI also maintains most of the dense model performance, but the degradation is larger than on AndroidControl\. UI\-Voyager drops from 70\.69% to 68\.10%, while MAI\-UI\-8B drops from 57\.76% to 56\.03%\. This is expected because AndroidWorld evaluates multi\-step interaction, where small grounding or perception errors may accumulate over the trajectory\. Nevertheless, the results suggest that AQuaUI can reduce visual token load while retaining competitive navigation performance\.

Conditional quadtree building is essential\.The conditional quadtree mechanism provides a clear benefit on AndroidWorld\. Removing it causes a consistent success\-rate drop under nearly identical compression rates\. For UI\-Voyager, success rate decreases from 68\.10% to 65\.52%; For MAI\-UI\-8B, success rate decreases from 56\.03% to 53\.44%\. This indicates that the gain is not simply due to retaining more tokens\. Rather, conditional quadtree refinement helps preserve fine\-grained partitions across adjacent screenshots\.

## 5Ablation Studies

### 5\.1AQuaUI Preserves UI Layout

In this section, we discuss whether AQuaUI’s quadtree design is genuinely useful by comparing it with randomly dropping visual tokens\[[23](https://arxiv.org/html/2605.19260#bib.bib25)\]and FastV\[[5](https://arxiv.org/html/2605.19260#bib.bib38)\]\. For FastV, we retain 65% of tokens from the fourth transformer layer for a fair comparison, using the implementation from PACT\[[9](https://arxiv.org/html/2605.19260#bib.bib24)\]and the migration to Qwen3\-VL\. We point out that FastV prunes inside the transformer and still incurs full prefill cost through early layers, whereas AQuaUI reduces tokens before the language transformer entirely\.[Table˜5](https://arxiv.org/html/2605.19260#S5.T5)shows that random dropping produces large accuracy drops across all benchmarks, confirming that the adaptive quadtree is the key source of AQuaUI’s effectiveness\. Especially, random reduction drops ScreenSpot\-V2 accuracy from 67\.14% to 57\.47% on Qwen2\-VL\-7B\-Instruct, whereas AQuaUI achieves 60\.06%, recovering 3\.77 points over random\. FastV performs more steadily than AQuaUI on Qwen2\-VL\-7B\-Instruct as the model can still get full image in early layers\. However, it is worth noting that the gap between AQuaUI and FastV is model\-dependent\. Although FastV performs well on Qwen2\-VL\-7B\-Instruct, it degrades severely on Qwen3\-VL\-8B\-Instruct, falling even below random dropping on ScreenSpot\-Pro\.

Table 5:Accuracy comparison with random token dropping with 30% visual tokens removed across benchmarks\. Accuracy is reported in percentages\.SS\-ProSS\-V2OS\-GUI\-VisionMMB\-GUIQwen2\-VL\-7B\-InstructvLLM10\.4467\.1419\.363\.3439\.90\+Random Reduction5\.4457\.4715\.632\.6529\.60\+FastV8\.7367\.4514\.743\.3036\.03\+AQuaUI9\.8760\.0617\.943\.1032\.80Qwen3\-VL\-8B\-InstructvLLM58\.5794\.8966\.2536\.1787\.15\+Random Reduction39\.4786\.9546\.1821\.1574\.21\+FastV25\.1185\.8540\.3212\.8162\.91\+AQuaUI55\.2894\.2659\.3327\.7283\.28
### 5\.2Choice of Compression Parameter and Score Function

In this section, we will discuss how different parameters and score functions affect compression quality\. We make the following observations\.

![Refer to caption](https://arxiv.org/html/2605.19260v1/x3.png)Figure 3:Performance change with respect to different parameter and score function\. Left: ScreenSpot\-V2\. Right: UI\-Vision\. Each dot represent a choice ofα\\alpha\. From left\-to\-right, for variance\-based score function,α=\{1,4,8,16,32,64\}\\alpha=\\\{1,4,8,16,32,64\\\}and for gradient\-based score function,α=\{10,15,30,60,120\}\\alpha=\\\{10,15,30,60,120\\\}\.Variance\-based compression is more stable, while gradient\-based compression is more aggressive\.Across ScreenSpot\-V2 and UI\-Vision, the variance\-based strategy yields a relatively narrow compression range and preserves performance consistently\. In contrast, the gradient\-based strategy reaches substantially higher compression rates, but its performance degrades more noticeably under aggressive compression\. This suggests a clear stability–compression trade\-off between the two strategies\.

Appropriate compression can preserve performance with fewer visual tokens\.The results show that a properly chosenα\\alphacan reduce visual tokens while maintaining comparable performance\. In particular, moderate compression often incurs only a small accuracy change\. This indicates that the model tolerates a meaningful amount of visual\-token reduction, but excessive compression removes task\-relevant visual details\.

## 6Conclusion

We presented AQuaUI, a training\-free inference\-time visual\-token reduction method for GUI agent models\. Instead of treating visual\-token reduction as a generic token\-importance estimation problem, AQuaUI exploits the spatial structure of GUI screenshots through adaptive quadtree partitioning\. Homogeneous regions are represented with coarse leaves, while others areas retain finer, with each leaf representing by one visual token\. We further introduced a conditional quadtree refinement mechanism that leverages temporal continuity of screenshots, reusing previous partitions only when the current screen region is static or mildly shifted and falling back to reconstruction of a quadtree when the content changes\.

Across GUI grounding and navigation benchmarks, AQuaUI consistently reduces visual tokens while preserving accuracy\. The results show that spatial redundancy in GUI screenshots can be exploited effectively at inference time, especially for larger GUI agent backbones\. More broadly, our findings suggest that efficient GUI perception should not merely ask which tokens are important, but also where the tokens are located\.

## References

- \[1\]\(2024\)Developing a computer use model\.External Links:[Link](https://www.anthropic.com/news/developing-computer-use)Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1)\.
- \[2\]S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p3.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p2.1)\.
- \[3\]Z\. Cai, W\. Xiao, H\. Sun, C\. Luo, Y\. Zhang, K\. Wan, Y\. Li, Y\. Zhou, L\. Chang, J\. Gu,et al\.\(2025\)R\-kv: redundancy\-aware kv cache compression for reasoning models\.arXiv preprint arXiv:2505\.24133\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p2.1)\.
- \[4\]Z\. Cai, Y\. Zhang, B\. Gao, Y\. Liu, Y\. Li, T\. Liu, K\. Lu, W\. Xiong, Y\. Dong, J\. Hu,et al\.\(2024\)Pyramidkv: dynamic kv cache compression based on pyramidal information funneling\.arXiv preprint arXiv:2406\.02069\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p2.1)\.
- \[5\]L\. Chen, H\. Zhao, T\. Liu, S\. Bai, J\. Lin, C\. Zhou, and B\. Chang\(2024\)An image is worth 1/2 tokens after layer 2: plug\-and\-play inference acceleration for large vision\-language models\.InEuropean Conference on Computer Vision,pp\. 19–35\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p1.1),[§5\.1](https://arxiv.org/html/2605.19260#S5.SS1.p1.1)\.
- \[6\]K\. R\. Chickering, B\. Li, and M\. Chen\(2025\)QLIP: a dynamic quadtree vision prior enhances mllm performance without retraining\.arXiv preprint arXiv:2505\.23004\.Cited by:[§3\.2](https://arxiv.org/html/2605.19260#S3.SS2.p1.1)\.
- \[7\]T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré\(2022\)Flashattention: fast and memory\-efficient exact attention with io\-awareness\.Advances in neural information processing systems35,pp\. 16344–16359\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p1.1)\.
- \[8\]X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. Su\(2023\)Mind2web: towards a generalist agent for the web\.Advances in Neural Information Processing Systems36,pp\. 28091–28114\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1)\.
- \[9\]M\. Dhouib, D\. Buscaldi, S\. Vanier, and A\. Shabou\(2025\)Pact: pruning and clustering\-based token reduction for faster visual language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 14582–14592\.Cited by:[§5\.1](https://arxiv.org/html/2605.19260#S5.SS1.p1.1)\.
- \[10\]K\. Huang, H\. Qiu, Y\. Dai, C\. Xiong, and C\. Wu\(2025\)Gui\-kv: efficient gui agents via kv cache with spatio\-temporal awareness\.arXiv preprint arXiv:2510\.00536\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p2.1)\.
- \[11\]G\. M\. Hunter and K\. Steiglitz\(1979\)Operations on images using quad trees\.IEEE Transactions on Pattern Analysis and Machine IntelligencePAMI\-1\(2\),pp\. 145–153\.External Links:[Document](https://dx.doi.org/10.1109/TPAMI.1979.4766900)Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p5.1),[§3\.2](https://arxiv.org/html/2605.19260#S3.SS2.p1.1)\.
- \[12\]Y\. Jiang, Q\. Wu, W\. Lin, W\. Yu, and Y\. Zhou\(2025\)What kind of visual tokens do we need? training\-free visual token pruning for multi\-modal large language models from the perspective of graph\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 4075–4083\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p1.1)\.
- \[13\]K\. Li, Z\. Meng, H\. Lin, Z\. Luo, Y\. Tian, J\. Ma, Z\. Huang, and T\. Chua\(2025\)Screenspot\-pro: gui grounding for professional high\-resolution computer use\.InProceedings of the 33rd ACM International Conference on Multimedia,pp\. 8778–8786\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[14\]W\. Li, W\. Bishop, A\. Li, C\. Rawles, F\. Campbell\-Ajala, D\. Tyamagundlu, and O\. Riva\(2024\)On the effects of data scale on ui control agents\.Advances in Neural Information Processing Systems37,pp\. 92130–92154\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[15\]Y\. Li, Y\. Huang, B\. Yang, B\. Venkitesh, A\. Locatelli, H\. Ye, T\. Cai, P\. Lewis, and D\. Chen\(2024\)Snapkv: llm knows what you are looking for before generation\.Advances in Neural Information Processing Systems37,pp\. 22947–22970\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p2.1)\.
- \[16\]K\. Q\. Lin, L\. Li, D\. Gao, Z\. Yang, S\. Wu, Z\. Bai, S\. W\. Lei, L\. Wang, and M\. Z\. Shou\(2025\)Showui: one vision\-language\-action model for gui visual agent\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 19498–19508\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1),[§1](https://arxiv.org/html/2605.19260#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2605.19260#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p2.1)\.
- \[17\]Z\. Lin, F\. Liu, Y\. Yang, J\. Lyu, Y\. Gao, Y\. Liu, Z\. Lu, Y\. Yu, M\. Yang, J\. Li,et al\.\(2026\)UI\-voyager: a self\-evolving gui agent learning via failed experience\.arXiv preprint arXiv:2603\.24533\.Cited by:[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p1.1)\.
- \[18\]J\. Liu, G\. Zhu, and F\. Du\(2026\)HiPrune: training\-free visual token pruning via hierarchical attention in vision\-language models \(student abstract\)\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 41275–41277\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p1.1)\.
- \[19\]Y\. Lu, J\. Yang, Y\. Shen, and A\. Awadallah\(2024\)Omniparser for pure vision based gui agent\.arXiv preprint arXiv:2408\.00203\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1)\.
- \[20\]W\. Mao, X\. Lin, W\. Huang, Y\. Xie, T\. Fu, B\. Zhuang, S\. Han, and Y\. Chen\(2026\)TriAttention: efficient long reasoning with trigonometric kv compression\.arXiv preprint arXiv:2604\.04921\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p2.1)\.
- \[21\]S\. Nayak, X\. Jian, K\. Q\. Lin, J\. A\. Rodriguez, M\. Kalsi, R\. Awal, N\. Chapados, M\. T\. Özsu, A\. Agrawal, D\. Vazquez,et al\.\(2025\)Ui\-vision: a desktop\-centric gui benchmark for visual perception and interaction\.arXiv preprint arXiv:2503\.15661\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[22\]M\. Ouyang, K\. Q\. Lin, M\. Z\. Shou, and H\. T\. Ng\(2026\)FocusUI: efficient ui grounding via position\-preserving visual token selection\.arXiv preprint arXiv:2601\.03928\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p4.1),[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p2.1)\.
- \[23\]T\. Peng, Y\. Du, P\. Ji, S\. Dong, K\. Jiang, M\. Ma, Y\. Tian, J\. Bi, Q\. Li, W\. Du,et al\.\(2025\)Can visual input be compressed? a visual token compression benchmark for large multimodal models\.arXiv preprint arXiv:2511\.02650\.Cited by:[§5\.1](https://arxiv.org/html/2605.19260#S5.SS1.p1.1)\.
- \[24\]Y\. Qin, Y\. Ye, J\. Fang, H\. Wang, S\. Liang, S\. Tian, J\. Zhang, J\. Li, Y\. Li, S\. Huang,et al\.\(2025\)Ui\-tars: pioneering automated gui interaction with native agents\.arXiv preprint arXiv:2501\.12326\.Cited by:[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p1.1)\.
- \[25\]C\. Rawles, S\. Clinckemaillie, Y\. Chang, J\. Waltz, G\. Lau, M\. Fair, A\. Li, W\. Bishop, W\. Li, F\. Campbell\-Ajala,et al\.\(2024\)Androidworld: a dynamic benchmarking environment for autonomous agents\.arXiv preprint arXiv:2405\.14573\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1),[§1](https://arxiv.org/html/2605.19260#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[26\]F\. Tang, Z\. Lu, B\. Zhang, W\. Lu, J\. Xiao, Y\. Zhuang, and Y\. Shen\(2026\)ClawGUI: a unified framework for training, evaluating, and deploying gui agents\.arXiv preprint arXiv:2604\.11784\.Cited by:[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[27\]J\. Wang, H\. Xu, J\. Ye, M\. Yan, W\. Shen, J\. Zhang, F\. Huang, and J\. Sang\(2024\)Mobile\-agent: autonomous multi\-modal mobile device agent with visual perception\.arXiv preprint arXiv:2401\.16158\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1)\.
- \[28\]P\. Wang, S\. Bai, S\. Tan, S\. Wang, Z\. Fan, J\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge,et al\.\(2024\)Qwen2\-vl: enhancing vision\-language model’s perception of the world at any resolution\.arXiv preprint arXiv:2409\.12191\.Cited by:[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p2.1)\.
- \[29\]X\. Wang, Z\. Wu, J\. Xie, Z\. Ding, B\. Yang, Z\. Li, Z\. Liu, Q\. Li, X\. Dong, Z\. Chen,et al\.\(2025\)Mmbench\-gui: hierarchical multi\-platform evaluation framework for gui agents\.arXiv preprint arXiv:2507\.19478\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[30\]Z\. Wu, Z\. Wu, F\. Xu, Y\. Wang, Q\. Sun, C\. Jia, K\. Cheng, Z\. Ding, L\. Chen, P\. P\. Liang,et al\.\(2024\)Os\-atlas: a foundation action model for generalist gui agents\.arXiv preprint arXiv:2410\.23218\.Cited by:[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[31\]T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.\(2024\)Osworld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1),[§1](https://arxiv.org/html/2605.19260#S1.p6.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p1.1)\.
- \[32\]H\. Xu, X\. Zhang, H\. Liu, J\. Wang, Z\. Zhu, S\. Zhou, X\. Hu, F\. Gao, J\. Cao, Z\. Wang,et al\.\(2026\)Mobile\-agent\-v3\. 5: multi\-platform fundamental gui agents\.arXiv preprint arXiv:2602\.16855\.Cited by:[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p2.1)\.
- \[33\]Y\. Xu, Z\. Wang, J\. Wang, D\. Lu, T\. Xie, A\. Saha, D\. Sahoo, T\. Yu, and C\. Xiong\(2024\)Aguvis: unified pure vision agents for autonomous gui interaction\.arXiv preprint arXiv:2412\.04454\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1)\.
- \[34\]S\. Yang, Y\. Chen, Z\. Tian, C\. Wang, J\. Li, B\. Yu, and J\. Jia\(2025\)Visionzip: longer is better but not necessary in vision language models\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19792–19802\.Cited by:[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p1.1)\.
- \[35\]J\. Ye, X\. Zhang, H\. Xu, H\. Liu, J\. Wang, Z\. Zhu, Z\. Zheng, F\. Gao, J\. Cao, Z\. Lu,et al\.\(2025\)Mobile\-agent\-v3: fundamental agents for gui automation\.arXiv preprint arXiv:2508\.15144\.Cited by:[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p1.1)\.
- \[36\]X\. Zhang, F\. Zeng, Y\. Quan, Z\. Hui, and J\. Yao\(2025\)Enhancing multimodal large language models complex reason via similarity computation\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 10203–10211\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.19260#S2.SS2.p1.1)\.
- \[37\]H\. Zhou, X\. Zhang, P\. Tong, J\. Zhang, L\. Chen, Q\. Kong, C\. Cai, C\. Liu, Y\. Wang, J\. Zhou,et al\.\(2025\)MAI\-ui technical report: real\-world centric foundation gui agents\.arXiv preprint arXiv:2512\.22047\.Cited by:[§2\.1](https://arxiv.org/html/2605.19260#S2.SS1.p1.1),[§4\.1](https://arxiv.org/html/2605.19260#S4.SS1.p2.1)\.
- \[38\]S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2023\)Webarena: a realistic web environment for building autonomous agents\.arXiv preprint arXiv:2307\.13854\.Cited by:[§1](https://arxiv.org/html/2605.19260#S1.p2.1)\.

## Appendix ADetailed Algorithm

### A\.1Grid Alignment and Boundary Handling

Given a resized screenshot of sizeW×HW\\times H, AQuaUI constructs the adaptive partition on the model’s merged\-token grid\. Letppdenote the ViT patch size andmmdenote the spatial merge size\. Each LLM\-side visual token corresponds to a block of sizeb=p⋅mb=p\\cdot m\. For Qwen2\-VL and Qwen3\-VL,p=14p=14andm=2m=2, sob=28b=28pixels\.

Since dimensions of GUI screenshots are not necessarily powers of two, directly applying a single quadtree to the whole image won’t work\. AQuaUI therefore first decomposes the image into two parts: a centered region that can be tiled by square chunks of sizeC×CC\\times C, and boundary margins that cannot be included in this chunk layout, which are kept asb×bb\\times btoken\-sized leaves\.

On the centered region, AQuaUI tiles square chunks and then runs an adaptive quadtree independently inside each chunk\. This design keeps all split boundaries aligned to the merged\-token grid and avoids malformed edge regions\.

To be more specific, the chunk side length is computed from the maximum depthddsupported by the image dimensions:

dh=⌊log2⁡\(H/b\)⌋,dw=⌊log2⁡\(W/b\)⌋\\displaystyle d\_\{h\}=\\lfloor\\log\_\{2\}\(H/b\)\\rfloor,\\quad d\_\{w\}=\\lfloor\\log\_\{2\}\(W/b\)\\rfloord=max⁡\(1,min⁡\(dh,dw\)\),C=b×2\(d−1\)\\displaystyle d=\\max\(1,\\min\(d\_\{h\},d\_\{w\}\)\),\\quad C=b\\times 2^\{\(d\-1\)\}

### A\.2Splitting Criteria

For a quadtree nodenn, we will describe the splitting criteria as follows\. The default score functionssis the area weigth greyscale variance:

s​\(n\)=wn​hn⋅Var⁡\(gray⁡\(n\)\)\.s\(n\)=w\_\{n\}h\_\{n\}\\cdot\\operatorname\{Var\}\(\\operatorname\{gray\}\(n\)\)\.A region is recursively split whens​\(n\)\>1000⋅αs\(n\)\>1000\\cdot\\alpha\. Area weighting makes the threshold meaningful across different region sizes\. Large non\-uniform regions are split even when their per\-pixel variance is moderate, while large uniform regions can collapse to one token\. Increasing alpha therefore increases compression\.

The implementation also supports a gradient\-based criterion, where the score functionssgives the max magnitude of gradient of nodenn, which can be useful when edges provide a stronger signal\.

### A\.3Representative Token Selection

For each final leaf, AQuaUI selects the center merged\-token block\. If the leaf spans block coordinates fromx0x\_\{0\}tox1x\_\{1\}andy0y\_\{0\}toy1y\_\{1\}, the representative block isx′=⌊x0\+x12⌋,y′=⌊y0\+y12⌋x^\{\\prime\}=\\lfloor\\frac\{x\_\{0\}\+x\_\{1\}\}\{2\}\\rfloor,\\quad y^\{\\prime\}=\\lfloor\\frac\{y\_\{0\}\+y\_\{1\}\}\{2\}\\rfloor\. The representative merged\-token coordinate is therefore\(x′,y′\)\(x^\{\\prime\},y^\{\\prime\}\)

### A\.4Conditional Quadtree Building

GUI trajectories often contain adjacent screenshots that are nearly identical\. A page may remain static, a menu may open, or a scroll operation may shift content by a few token blocks\. If each screenshot is compressed independently, a later frame may merge away details that were preserved in an earlier frame, weakening temporal reasoning and action grounding\.

AQuaUI introduces a conditional quadtree mechanism that infers history from the ordered images in the same request\. We design a heuristic algorithm for this\.

#### A\.4\.1Static, Shifted, and Replaced Modes

##### Static mode

We begin by identify patterns that most pixels remain the same, which we called the Static mode\. For this purpose, we define a similarity value between two chunks\. Consider the chunks we produced in[Section˜3\.2](https://arxiv.org/html/2605.19260#S3.SS2), we compress it into a condensed grid signartureSSof size\(Hc,Wc\)\(H\_\{c\},W\_\{c\}\)by computing the greyscale of each block of size\(b,b\)\(b,b\)so that each cell ofSSstores the rounded mean grayscale of the correspondingb×bb\\times bblock withS∈\[0,255\]Hc×WCS\\in\[0,255\]^\{H\_\{c\}\\times W\_\{C\}\}\.

For the similarity of two adjacent signaturesStS\_\{t\}andSt−1S\_\{t\-1\}, we define similarity as following:

similarity​\(St,St−1\)=1−1255​Hc⋅Wc×∑i,j\|St​\(i,j\)−St−1​\(i,j\)\|\\text\{similarity\}\(S\_\{t\},S\_\{t\-1\}\)=1\-\\frac\{1\}\{255H\_\{c\}\\cdot W\_\{c\}\}\\times\\sum\_\{i,j\}\|S\_\{t\}\(i,j\)\-S\_\{t\-1\}\(i,j\)\|
A region is static ifsimilarity≥τs​t​a​t​i​c\\text\{similarity\}\\geq\\tau\_\{static\}\.

##### Shifted mode

If two chunks are not static, we then consider if a scroll operation happens\. We call this the Shifted mode\. To capture the scrolling feature, we add a block\-level shiftδ\\deltain previous definition\. More specifically, given current and previous chunk signaturesSt,St−1∈\[0,255\]Hc×WcS\_\{t\},S\_\{t\-1\}\\in\[0,255\]^\{H\_\{c\}\\times W\_\{c\}\}, we consider a candidate block\-level shiftδ=\(Δ​x,Δ​y\)\\delta=\(\\Delta x,\\Delta y\)\. We continue to calculate the similarity on the overlapping parts of the two signaturesΩ​\(δ\)\\Omega\(\\delta\)\. We defineΩ​\(δ\)\\Omega\(\\delta\)as the following:

Ω​\(δ\)=\{\(i,j\)∣0≤i<Hc,0≤j<Wc,0≤i−Δ​x<Hc,0≤j−Δ​y<Wc\}\.\\Omega\(\\delta\)=\\\{\(i,j\)\\mid 0\\leq i<H\_\{c\},\\;0\\leq j<W\_\{c\},\\;0\\leq i\-\\Delta x<H\_\{c\},\\;0\\leq j\-\\Delta y<W\_\{c\}\\\}\.The overlap ratio isρ​\(δ\)=\|Ω​\(δ\)\|Hc​Wc\.\\rho\(\\delta\)=\\frac\{\|\\Omega\(\\delta\)\|\}\{H\_\{c\}W\_\{c\}\}\.We only evaluate shifts satisfyingρ​\(δ\)≥ρmin\\rho\(\\delta\)\\geq\\rho\_\{\\min\}\. Therefore, similarity becomes

similarity​\(δ\)=1−1255​\|Ω​\(δ\)\|​∑\(i,j\)∈Ω​\(δ\)\|St​\(i,j\)−St−1​\(i−Δ​x,j−Δ​y\)\|\.\\text\{similarity\}\(\\delta\)=1\-\\frac\{1\}\{255\|\\Omega\(\\delta\)\|\}\\sum\_\{\(i,j\)\\in\\Omega\(\\delta\)\}\\left\|S\_\{t\}\(i,j\)\-S\_\{t\-1\}\(i\-\\Delta x,j\-\\Delta y\)\\right\|\.We search over\|Δ​x\|,\|Δ​y\|≤dmax\|\\Delta x\|,\|\\Delta y\|\\leq d\_\{\\max\}and classify the region as shifted if the best valid shiftδ⋆\\delta^\{\\star\}satisfies

sim⁡\(δ⋆\)≥τshiftandsim⁡\(δ⋆\)≥sim⁡\(St,St−1\)\+γ\.\\operatorname\{sim\}\(\\delta^\{\\star\}\)\\geq\\tau\_\{\\mathrm\{shift\}\}\\quad\\text\{and\}\\quad\\operatorname\{sim\}\(\\delta^\{\\star\}\)\\geq\\operatorname\{sim\}\(S\_\{t\},S\_\{t\-1\}\)\+\\gamma\.In shifted mode, only coarse previous leaves are translated and reused:

max⁡\(w,h\)≥2×b\\max\(w,h\)\\geq 2\\times bFine leaves are regenerated because small alignment errors around text and icons can be more harmful than independent recompression\.

##### Replaced mode

Regions satisfying neither condition are treated as replaced\. A new quadtree is constructed where no priors are used\.

We set these parameters heuristically, whereτs​t​a​t​i​c=0\.97,τs​h​i​f​t=0\.94,γ=0\.03,ρm​i​n=0\.5\\tau\_\{static\}=0\.97,\\tau\_\{shift\}=0\.94,\\gamma=0\.03,\\rho\_\{min\}=0\.5and the shift search range is\|Δ​x\|,\|Δ​y\|≤4\|\\Delta x\|,\|\\Delta y\|\\leq 4blocks\. The full algorithm can be seen in[Algorithm˜1](https://arxiv.org/html/2605.19260#alg1)

### A\.5Implementation Details

We implement AQuaUI on both Qwen2\-VL and Qwen3\-VL with vLLM as the serving backend\. Although the high\-level quadtree construction is shared across models, the two model families require different integration paths because their visual tokenization and position\-encoding pipelines differ\. We describe the model\-specific details below\.

#### A\.5\.1Qwen2\-VL Details

Rather than reimplementing Qwen2\-VL preprocessing, AQuaUI first runs the vanilla fast processor and then indexes into the resulting dense patch tensor\. This preserves Qwen’s original resizing, normalization, and patch extraction behavior\.

Letmmdenote the spatial merge size and letwwdenote the width of the dense merged\-token grid\. For a representative merged\-token coordinate\(xrep,yrep\)\(x\_\{\\mathrm\{rep\}\},y\_\{\\mathrm\{rep\}\}\), the corresponding offset in the dense patch tensor is

k=\(yrep⋅w\+xrep\)⋅m2\.k=\(y\_\{\\mathrm\{rep\}\}\\cdot w\+x\_\{\\mathrm\{rep\}\}\)\\cdot m^\{2\}\.The selected patch rows are then

ℐrep=\{k,k\+1,…,k\+m2−1\}\.\\mathcal\{I\}\_\{\\mathrm\{rep\}\}=\\\{k,k\+1,\\ldots,k\+m^\{2\}\-1\\\}\.For the standard Qwen2\-VL setting,m=2m=2, so each retained merged token corresponds to four ViT patch rows\. This indexing keeps the selected token compatible with Qwen2\-VL’s original merge\-block ordering\.

In our default Qwen2\-VL implementation, reduction is applied after the vision encoder\. The image is first processed by the dense Qwen2\-VL ViT, and AQuaUI then selects the visual embeddings corresponding to the representative merged\-token coordinates\. This conservative path does not reduce ViT computation, but it preserves dense vision encoding quality while reducing the visual sequence passed to the LLM\. The reduced visual embeddings are accompanied by their original merged\-token coordinates, which are injected into the LLM\-side M\-RoPE position construction\. This avoids treating the sparse selected tokens as a compact dense image grid\.

We also implement a pre\-ViT Qwen2\-VL ablation, where the selected patch rows are passed directly into the vision encoder\. This path can reduce both ViT\-side and LLM\-side computation, but it is more aggressive because the vision encoder receives a sparse visual sequence\. We therefore use post\-ViT reduction as the default Qwen2\-VL setting and treat pre\-ViT reduction as a speed\-oriented ablation\.

#### A\.5\.2Qwen3\-VL Details

For Qwen3\-VL, AQuaUI applies reduction before the vision encoder\. After the vanilla fast processor produces the dense patch tensor, we select the patch rows corresponding to the representative quadtree tokens and construct a reduced visual input\. Unlike Qwen2\-VL post\-ViT reduction, this path reduces the number of tokens processed by both the vision encoder and the LLM, allowing larger end\-to\-end latency gains on Qwen3\-VL backbones\.

For pre\-ViT reduction, the reduced visual tokens still need to be represented by a valid rectangular grid for vLLM’s multimodal batching and placeholder expansion\. We therefore use aspect\-aware padded grid packing: the selected tokens are packed into a near\-aspect rectangular grid, and when padding is necessary, the final selected token is duplicated\. These duplicate tokens are used only to satisfy shape constraints; their true positions are still tracked through the quadtree metadata\.

We patch Qwen3\-VL’s ViT\-side rotary position construction to use true quadtree positions rather than compact packed\-grid positions\. These coordinates follow Qwen’s original merge\-block\-interleaved ordering\.

Qwen3\-VL also includes an absolute position interpolation path\. This path must use the original dense grid extent rather than the maximum retained coordinate\.

Finally, we patch the LLM\-side M\-RoPE construction\. For image tokens, the height and width rotary positions are replaced with the true quadtree coordinates\. After injection, the M\-RoPE delta is recomputed so that subsequent text tokens remain correctly aligned with the multimodal position sequence\. This step is essential: without it, the model would receive fewer visual tokens but would interpret them as occupying compacted positions, which corrupts the geometry needed for GUI grounding\.

Qwen3\-VL is the more natural target for AQuaUI’s pre\-ViT reduction, because both vision\-side and LLM\-side position encodings can be patched to remain consistent with the sparse quadtree layout\.

Algorithm 1Conditional quadtree building for screenshotsts\_\{t\}givenst−1s\_\{t\-1\}1:Current and previous chunks

\{Ctk\},\{Ct−1k\}\\\{C\_\{t\}^\{k\}\\\},\\\{C\_\{t\-1\}^\{k\}\\\}; thresholds

τstatic,τshift,γ,ρmin,dmax\\tau\_\{\\mathrm\{static\}\},\\tau\_\{\\mathrm\{shift\}\},\\gamma,\\rho\_\{\\min\},d\_\{\\max\}
2:Refined partition

ℒt\\mathcal\{L\}\_\{t\}for

sts\_\{t\}
3:foreach chunk region

kkdo

4:

ℒtk←\\mathcal\{L\}\_\{t\}^\{k\}\\leftarrowindependent quadtree of

CtkC\_\{t\}^\{k\}
5:Compute signatures

Stk,St−1kS\_\{t\}^\{k\},S\_\{t\-1\}^\{k\}
6:if

sim​\(𝟎\)≥τstatic\\mathrm\{sim\}\(\\mathbf\{0\}\)\\geq\\tau\_\{\\mathrm\{static\}\}then

7:

𝒫k←\\mathcal\{P\}^\{k\}\\leftarrowleaves of

Ct−1kC\_\{t\-1\}^\{k\}⊳\\trianglerightstatic

8:else

9:

δ⋆←arg⁡maxδ:ρ​\(δ\)≥ρmin⁡sim​\(δ\)\\delta^\{\\star\}\\leftarrow\\arg\\max\_\{\\delta:\\,\\rho\(\\delta\)\\geq\\rho\_\{\\min\}\}\\mathrm\{sim\}\(\\delta\)
10:if

sim​\(δ⋆\)≥τshift\\mathrm\{sim\}\(\\delta^\{\\star\}\)\\geq\\tau\_\{\\mathrm\{shift\}\}and

sim​\(δ⋆\)≥sim​\(𝟎\)\+γ\\mathrm\{sim\}\(\\delta^\{\\star\}\)\\geq\\mathrm\{sim\}\(\\mathbf\{0\}\)\+\\gammathen

11:

𝒫k←\\mathcal\{P\}^\{k\}\\leftarrowcoarse leaves of

Ct−1kC\_\{t\-1\}^\{k\}translated by

δ⋆\\delta^\{\\star\}⊳\\trianglerightshifted

12:else

13:

𝒫k←∅\\mathcal\{P\}^\{k\}\\leftarrow\\emptyset⊳\\trianglerightreplaced

14:endif

15:endif

16:foreach leaf

ℓ∈ℒtk\\ell\\in\\mathcal\{L\}\_\{t\}^\{k\}do

17:

𝒫k​\(ℓ\)←\{ℓ′∈𝒫k∣ℓ′⊆ℓ\}\\mathcal\{P\}^\{k\}\(\\ell\)\\leftarrow\\\{\\ell^\{\\prime\}\\in\\mathcal\{P\}^\{k\}\\mid\\ell^\{\\prime\}\\subseteq\\ell\\\}
18:if

𝒫k​\(ℓ\)≠∅\\mathcal\{P\}^\{k\}\(\\ell\)\\neq\\emptysetand

∑ℓ′∈𝒫k​\(ℓ\)area​\(ℓ′\)=area​\(ℓ\)\\sum\_\{\\ell^\{\\prime\}\\in\\mathcal\{P\}^\{k\}\(\\ell\)\}\\mathrm\{area\}\(\\ell^\{\\prime\}\)=\\mathrm\{area\}\(\\ell\)then

19:Replace

ℓ\\ellwith

𝒫k​\(ℓ\)\\mathcal\{P\}^\{k\}\(\\ell\)in

ℒtk\\mathcal\{L\}\_\{t\}^\{k\}
20:endif

21:endfor

22:endfor

23:return

ℒt←⋃kℒtk\\mathcal\{L\}\_\{t\}\\leftarrow\\bigcup\_\{k\}\\mathcal\{L\}\_\{t\}^\{k\}

## Appendix BBreakdown Statisics on Grounding Benchmarks

[Table˜6](https://arxiv.org/html/2605.19260#A2.T6),[Table˜7](https://arxiv.org/html/2605.19260#A2.T7),[Table˜8](https://arxiv.org/html/2605.19260#A2.T8),LABEL:tab:uivision\_breakdown,LABEL:tab:mmb\_breakdowndisplay the performance breakdown of different benchmarks\. We can observe that AQuaUI is particularly effective on tasks requiring preserving structured GUI evidence\.

Table 6:ScreenSpot\-Pro breakdown\. Accuracy is reported in percentages\.BackendTextIconOverallQwen2\-VL\-7B\-InstructTransformers15\.052\.6510\.31\+ShowUI12\.282\.658\.60vLLM15\.352\.4810\.44\+ AQuaUI14\.432\.489\.87ShowUI\-2BTransformers10\.243\.157\.53\+ShowUI8\.092\.485\.95vLLM10\.852\.327\.59\+ AQuaUI9\.312\.156\.58Table 7:ScreenSpot\-V2 breakdown\. Accuracy is reported in percentages\.BackendMobileDesktopWebAvg\.TextIconAvg\.TextIconAvg\.TextIconAvg\.Qwen2\-VL\-7B\-InstructTransformers75\.8665\.8871\.6674\.2351\.4364\.6769\.2358\.6264\.3067\.30\+ShowUI55\.1759\.7257\.0964\.4345\.7156\.5956\.4146\.3151\.7255\.11vLLM74\.1466\.3570\.8673\.7151\.4364\.3769\.6659\.6164\.9967\.14\+ AQuaUI59\.6648\.8255\.0972\.1650\.0062\.8768\.3858\.1363\.6260\.06ShowUI\-2BTransformers91\.7272\.9983\.8377\.3258\.5769\.4683\.7662\.5673\.9176\.65\+ShowUI80\.0064\.9373\.6567\.5355\.7162\.5767\.0949\.7559\.0465\.72vLLM91\.3874\.4184\.2376\.8057\.8668\.8682\.4863\.0573\.4676\.49\+ AQuaUI75\.5263\.0370\.2671\.6553\.5764\.0761\.1152\.2256\.9864\.07Table 8:OSWorld\-g breakdown\. Accuracy is reported in percentages\. Refusal samples are excluded from Overall accuracy\. TextMatch: Text Matching; ElemRecog: Element Recognition; LayoutUnd: Layout Understanding; FineManip: Fine\-grained Manipulation\.BackendTextMatchElemRecogLayoutUndFineManipOverallQwen2\-VL\-7B\-InstructTransformers31\.6725\.1622\.5916\.6721\.14\+ShowUI25\.8319\.6117\.9912\.1216\.87vLLM28\.7522\.5521\.3416\.6719\.36\+ AQuaUI27\.9221\.5718\.4113\.6417\.94ShowUI\-2BTransformers31\.6723\.5322\.1814\.3918\.29\+ShowUI26\.2522\.2219\.2511\.3616\.34vLLM31\.6723\.8621\.7612\.8818\.12\+ AQuaUI27\.5021\.2419\.6712\.1216\.34Table 9:UI\-Vision breakdown\. Accuracy is reported in percentages\.BackendBasicFunctionalSpatialOverallQwen2\-VL\-7B\-InstructTransformers4\.404\.061\.913\.41\+ShowUI4\.122\.821\.342\.27vLLM4\.634\.011\.553\.34\+ AQuaUI4\.293\.841\.343\.10ShowUI\-2BTransformers8\.138\.012\.125\.97\+ShowUI6\.666\.831\.915\.04vLLM8\.138\.352\.076\.06\+ AQuaUI8\.137\.672\.075\.84
Table 10:MMBench\-GUI \(L2 Grounding\) breakdown\. Accuracy is reported in percentages\.BackendWindowsMacOSLinuxiOSAndroidWebOverallQwen2\-VL\-7B\-InstructTransformers25\.4126\.6321\.7159\.0152\.1842\.8839\.57\+ShowUI19\.5214\.9118\.0941\.1539\.2433\.6628\.69vLLM26\.7027\.7920\.9358\.5451\.9043\.6939\.90\+ AQuaUI23\.5726\.6319\.6443\.6334\.8842\.3932\.80ShowUI\-2BTransformers28\.3630\.5419\.1263\.8256\.9649\.8443\.49\+ShowUI21\.9218\.3817\.0550\.6245\.8539\.8133\.67vLLM29\.2831\.2619\.3864\.2956\.6851\.1344\.05\+ AQuaUI24\.8627\.3510\.0851\.5547\.2642\.5636\.00

## Limitations

##### Model\- and serving\-stack\-specific implementation\.

AQuaUI is training\-free, but it is not implementation\-free\. Correctly preserving visual\-token positions requires modifying model\-specific position\-encoding paths and transporting quadtree metadata through the serving stack\. Our current implementation focuses on Qwen2\-VL and Qwen3\-VL in vLLM\. Adapting the same idea to other multimodal models may require additional engineering, especially when the model uses different visual token layouts, position encodings, placeholder expansion rules, or multimodal batching interfaces\.

##### Limited gains when reduction overhead dominates\.

Although AQuaUI reduces the number of visual tokens, it also introduces preprocessing overhead from quadtree construction, metadata transport, and position injection\. As a result, latency gains are not guaranteed for small models or settings where the vision encoder and LLM\-side computation are already inexpensive\. Our results show that the speedup becomes more visible on larger backbones, where the saved visual\-token computation can outweigh the extra reduction overhead\. More importantly, there are operations in our algorithm that can vectorized for speedup, but we don’t implement these improvements for now\.

##### Qwen2 and Qwen3 reduction paths differ\.

The method is more naturally supported on Qwen3\-VL, where pre\-ViT reduction can be combined with patched ViT\-side and LLM\-side position encodings\. For Qwen2\-VL, our conservative default applies reduction after the vision encoder, which preserves dense visual encoding quality but does not reduce ViT computation\. Therefore, the latency benefit on Qwen2\-VL is more limited\. This difference suggests that the effectiveness of serving\-time token reduction depends not only on the compression algorithm, but also on where the model architecture allows sparse visual tokens to be introduced safely\.

## Broader Impacts

This work aims to improve the efficiency of GUI agents by reducing redundant visual tokens in high\-resolution screenshots\. More efficient GUI\-agent can lower inference cost and reduce GPU memory usage\. This could make screen\-grounded agents more practical on edge devices, which may support broader access to mobile\-device control and cross\-platform user\-interface interaction\.