GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding

# GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
Source: [https://arxiv.org/html/2608.21832](https://arxiv.org/html/2608.21832)
Md Abrar Jahin Md Rizwan ParvezUniversity of Southern California USC Information Sciences InstituteQatar Computing Research Institute \(QCRI\)jahin@usc\.edumparvez@hbku\.edu\.qa

###### Abstract

Computer\-use agents ground natural\-language instructions in screenshots to locate interface elements, yet existing benchmarks do not isolate whether models bind relational language to the correct element\. We introduceGUI\-Primitives, a 994\-item benchmark of contrastive instruction pairs over seven spatial relations in graphical user interfaces \(left/right, above/below, containment, alignment, proximity, list ordinal, occlusion\)\. Each pair holds the screenshot and anchor fixed while changing the relation expression, so the correct target moves between two designated candidates\. Five annotators validate a 196\-item subset \(κ=0\.94\\kappa=0\.94well\-formedness;κ=0\.79\\kappa=0\.79target selection\)\. Nineteen vision\-language models reach at most 32% strict point\-in\-box accuracy\. Because models emit unconstrained coordinates, we classify each prediction by the candidate region it falls within\. Predictions fall outside both candidates on 60–92% of items\. Conditional on falling within a candidate region, target selection reaches 0\.82–0\.90 for horizontal position, vertical position, proximity, and list ordinal, but does not differ significantly from 0\.50 for containment and occlusion: most failures reflect candidate localization rather than relation understanding\. Across ten models, benchmark accuracy correlates with ScreenSpot\-Pro accuracy \(Spearmanρ=\+0\.74\\rho=\+0\.74\), an exploratory association at this sample size\. Marking the two designated candidates raises selection accuracy by 35–57 percentage points, an oracle diagnostic that supplies the candidate set rather than a deployable method\. We release the benchmark, predictions, and code\.

$\\dagger$$\\dagger$footnotetext:Corresponding Author\(s\)## 1Introduction

A new class of*computer\-use agents*now reads a screenshot, decides what to do, and emits a click coordinate or a keystroke\([27](https://arxiv.org/html/2608.21832#bib.bib1);[2](https://arxiv.org/html/2608.21832#bib.bib2);[18](https://arxiv.org/html/2608.21832#bib.bib12);[4](https://arxiv.org/html/2608.21832#bib.bib20);[24](https://arxiv.org/html/2608.21832#bib.bib21);[21](https://arxiv.org/html/2608.21832#bib.bib25)\)\. These agents are beginning to be deployed; their first job, before any agent loop, is the same job a person does when they open a window: read the screen, find the right control, and put the pointer in the right place\. We call this*GUI grounding*, and a growing body of literature shows that frontier vision\-language models still trail humans in this task\([10](https://arxiv.org/html/2608.21832#bib.bib3);[31](https://arxiv.org/html/2608.21832#bib.bib24)\)\.

A natural question follows\. If we strip the agent away and ask a present\-day VLM only to handle the elementary spatial reasoning that GUI grounding requires: to the right ofSave, inside theLayerspanel, the third item in the menu, does it succeed? Existing benchmarks do not isolate this skill\. End\-to\-end benchmarks like OSWorld\([27](https://arxiv.org/html/2608.21832#bib.bib1)\)combine grounding with planning and action; coarse grounding benchmarks like ScreenSpot\([2](https://arxiv.org/html/2608.21832#bib.bib2)\)and ScreenSpot\-Pro\([10](https://arxiv.org/html/2608.21832#bib.bib3)\)measure whether the click lands within the target box but do not separate failures by spatial primitive\. As a result, when an agent clicks the wrong button, we do not yet know*which*elementary skill broke\.

We close this gap with a controlled diagnostic\.GUI\-Primitivesis a benchmark of 994 contrastive\-pair questions over seven spatial relations in GUI screenshots\. Each question is paired with a twin that keeps the screenshot and anchor fixed and changes the relation expression \(left↔\\leftrightarrowright,inside↔\\leftrightarrowoutside, and so on\), so that the correct target moves to the other designated candidate\. In 253 of 497 pairs, the surrounding instruction template also varies lexically \(verb, head noun, anchor quoting\); this variation is not confounded with the relation term \(Appendix[J\.2](https://arxiv.org/html/2608.21832#A10.SS2)\)\. This construction, adapted from the minimal\-pair tradition in linguistic evaluation\([11](https://arxiv.org/html/2608.21832#bib.bib14);[14](https://arxiv.org/html/2608.21832#bib.bib15);[7](https://arxiv.org/html/2608.21832#bib.bib4)\), controls for screenshot\-specific salience and fixed answer preferences: a model that always clicks the most salient or central element scores correctly on one twin and incorrectly on the other\. We evaluate 19 vision\-language models spanning open\-weight and proprietary systems\. Five annotators independently validated a 196\-item subset \(Fleissκ=0\.94\\kappa=0\.94for well\-formedness\) and selected the instruction\-consistent candidate on 96\.9% of retained items; this is annotator target\-selection accuracy on a two\-candidate task, not a model\-matched grounding score\.

The evaluation yields three main findings\. Claude Opus 4\.7, the strongest model, reaches 31% strict point\-in\-box accuracy on the human\-clean subset\. On four of seven primitives \(containment, occlusion, alignment, proximity\), every model we test falls below the two\-candidate reference level of 0\.50\. Candidate\-level analysis \(§[5\.3](https://arxiv.org/html/2608.21832#S5.SS3)\) indicates that this below\-reference accuracy is primarily attributable to predictions outside both candidate regions rather than to the selection of the contrastive element\. On real desktop screenshots from UI\-Vision\([16](https://arxiv.org/html/2608.21832#bib.bib19)\), accuracy is lower still, and candidate\-level analysis attributes this primarily to predictions outside both candidate regions \(96\.2%96\.2\\%of real\-subset predictions\) rather than to relation errors\. Pair consistency is below the value expected under independent responses because carrier\-template variation and off\-candidate prediction also contribute to this dependence; therefore, candidate\-level analysis \(§[5\.3](https://arxiv.org/html/2608.21832#S5.SS3)\) is required to interpret it\.

Yet the diagnosis is not only a problem statement\. Three results follow\. First, the benchmark tracks downstream performance\. Across ten models for which we also run ScreenSpot\-Pro,GUI\-Primitivesaccuracy correlates with grounding accuracy at Spearmanρ=\+0\.74\\rho=\+0\.74\(p=0\.015p=0\.015\)\. The correlation is stable across the real\-screenshot and synthetic slices ofGUI\-Primitives\. Item\-level logistic regression onn=15,810n=15\{,\}810model–item pairs corroborates this association \(pseudo\-R2=0\.40R^\{2\}=0\.40\); we report it conservatively\. Second, we test three training\-free interventions: Set\-of\-Mark \(SoM\) prompting\([30](https://arxiv.org/html/2608.21832#bib.bib9)\), primitive\-aware chain\-of\-thought \(CoT\), and activation steering\([8](https://arxiv.org/html/2608.21832#bib.bib10);[19](https://arxiv.org/html/2608.21832#bib.bib11)\)\. Marking the two designated candidates \(an oracle two\-candidate diagnostic\) raises selection accuracy by 35–57 percentage points, lifting GPT\-5 from 30% to 87% and OS\-Atlas from 10% to 52%; CoT and activation steering yield no significant gain\. Because the intervention supplies the candidate set that the baseline task requires the model to locate, it is an upper bound on candidate discrimination rather than a deployable method \(§[8](https://arxiv.org/html/2608.21832#S8)\)\. Third, three shortcut controls \(blank canvas, shuffled instructions, heavy blur\) reduce accuracy as expected for a grounded benchmark; the blur control produces a significant but sub\-threshold drop, and the negative results for CoT and activation steering are reported alongside the positive result for SoM\.

We make four contributions\. First, we introduceGUI\-Primitives, a 994\-item contrastive\-pair diagnostic benchmark covering seven spatial relations in GUI screenshots, together with a five\-annotator verified 196\-item core \(κ=0\.94\\kappa=0\.94\)\. Second, we evaluate 19 models across five families and localize the dominant failure to candidate localization, predictions fall outside both candidate regions on 60–92% of items, with relation\-word errors confined to containment and occlusion, alongside a 65\-point gap from human performance\. Third, we show that primitive competence correlates with downstream GUI grounding on ScreenSpot\-Pro \(Spearmanρ=\+0\.74\\rho=\+0\.74,p=0\.015p=0\.015\)\. Fourth, we find that SoM transfers across both open and closed models, improving GPT\-5 by 57 points, whereas CoT prompting and activation steering do not; we releaseGUI\-Primitives, all model predictions, and analysis code\.

![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_construction_banner.png)Figure 1:GUI\-Primitivesdataset construction pipeline\.Two source corpora: real desktop screenshots from UI\-Vision and controlled synthetic stimuli from our procedural renderer, feed into per\-primitive minimal\-pair miners, pass through automatic quality gates, are cross\-checked by a five\-annotator human verification pass \(Fleissκ=0\.94\\kappa\{=\}0\.94\), and released as the 994\-item benchmark\. Stage colors follow the primitive palette used throughout the paper\.
## 2Related Work

#### GUI grounding and computer\-use agents\.

A line of recent work targets agents that operate real software\. OSWorld\([27](https://arxiv.org/html/2608.21832#bib.bib1)\)provides a real Linux and Windows VM environment with execution\-based scoring on 369 tasks; a subsequent human\-verified release closes ambiguous tasks\([28](https://arxiv.org/html/2608.21832#bib.bib32)\)\. ScreenSpot\([2](https://arxiv.org/html/2608.21832#bib.bib2)\)and ScreenSpot\-Pro\([10](https://arxiv.org/html/2608.21832#bib.bib3)\)measure pixel\-grounding in isolation, the latter targeting professional software at high resolution; MMBench\-GUI\([22](https://arxiv.org/html/2608.21832#bib.bib26)\)adds a hierarchical multi\-platform evaluation\. Grounding models trained on screenshots include SeeClick, OS\-Atlas\([25](https://arxiv.org/html/2608.21832#bib.bib13)\), UI\-TARS\([18](https://arxiv.org/html/2608.21832#bib.bib12)\), UGround\([4](https://arxiv.org/html/2608.21832#bib.bib20)\), Aria\-UI\([32](https://arxiv.org/html/2608.21832#bib.bib22)\), GUI\-Actor\([24](https://arxiv.org/html/2608.21832#bib.bib21)\), Jedi\([26](https://arxiv.org/html/2608.21832#bib.bib23)\), OpenCUA\([21](https://arxiv.org/html/2608.21832#bib.bib25)\), and the test\-time scaling agent GTA1\([31](https://arxiv.org/html/2608.21832#bib.bib24)\)\. Closest in spirit, GUI\-Cursor\([33](https://arxiv.org/html/2608.21832#bib.bib27)\)probes a cursor\-based agent’s spatial reasoning under reinforcement learning, and Beyond Semantics\([17](https://arxiv.org/html/2608.21832#bib.bib30)\)argues that large vision\-token norms suppress positional encoding, a mechanistic candidate for our failures\. These works ask whether models land the click or how to train them; we ask which elementary spatial skill they lack when they miss\.

#### Spatial reasoning in vision\-language models\.

Several benchmarks reveal that VLMs struggle with spatial relations even on natural images\.What’s\-Up\([7](https://arxiv.org/html/2608.21832#bib.bib4)\)uses controlled minimal pairs to show that contrastive VLMs handle left/right and on/under poorly despite high captioning accuracy\. VSR\([12](https://arxiv.org/html/2608.21832#bib.bib8)\)catalogs 66 spatial relations in natural images; BLINK\([3](https://arxiv.org/html/2608.21832#bib.bib5)\)bundles 14 perceptual primitives and finds even GPT\-4V well below humans; CV\-Bench\([20](https://arxiv.org/html/2608.21832#bib.bib6)\)shows that 2D/3D spatial primitives are bottlenecked by the visual encoder\. SpatialVLM\([1](https://arxiv.org/html/2608.21832#bib.bib7)\)addresses these gaps by co\-training on synthetic 3D\-grounded data\. RocketScience\([5](https://arxiv.org/html/2608.21832#bib.bib33)\)uses contrastive pairs to disentangle object localization from spatial reasoning, and finds that only top reasoning models succeed\.GUI\-Primitivesextends this controlled methodology to GUI screenshots, where the visual prior is starkly different: text labels everywhere, near\-identical icons, dense layout\.

#### Minimal\-pair evaluation\.

Minimal pairs originate in psycholinguistic evaluation\([11](https://arxiv.org/html/2608.21832#bib.bib14);[14](https://arxiv.org/html/2608.21832#bib.bib15)\), where a single lexical or syntactic edit creates a contrast whose disambiguation requires understanding rather than surface statistics\.What’s\-Upbrought the design to vision\-language\. We adopt the same logic for GUI screenshots and add pair\-consistency\([7](https://arxiv.org/html/2608.21832#bib.bib4)\)as an internal diagnostic of relational failure \(§[5\.5](https://arxiv.org/html/2608.21832#S5.SS5)\)\.

#### Inference\-time interventions\.

SoM prompting\([30](https://arxiv.org/html/2608.21832#bib.bib9)\)overlays numbered marks on visual regions so the model names the mark rather than regressing a coordinate\. SoM\-LLaVA\([29](https://arxiv.org/html/2608.21832#bib.bib31)\)shows that open\-source MLLMs need explicit SoM training data to match closed models\. Visual prompting is now a standard tool in GUI\-agent stacks, with recent training\-free zoom and modality\-aware variants: DiMo\-GUI\([23](https://arxiv.org/html/2608.21832#bib.bib28)\)decouples text and icon modalities and applies test\-time scaling; RegionFocus\([13](https://arxiv.org/html/2608.21832#bib.bib29)\)reaches 61\.6% on ScreenSpot\-Pro with a Qwen2\.5\-VL\-72B backbone by iteratively zooming into a focal region\. Activation and attention steering, including the localization\-head method of[8](https://arxiv.org/html/2608.21832#bib.bib10)and the SteerVLM framework\([19](https://arxiv.org/html/2608.21832#bib.bib11)\), modify model internals at inference time without weight updates\. We test one representative from each family\.

## 3TheGUI\-PrimitivesBenchmark

GUI\-Primitivesis designed for one purpose: to ask whether a vision\-language model has each of the seven elementary spatial skills that GUI grounding requires\. Three design choices follow from this purpose: the seven primitives \(§[3\.1](https://arxiv.org/html/2608.21832#S3.SS1)\), the minimal\-pair construction \(§[3\.2](https://arxiv.org/html/2608.21832#S3.SS2)\), and the human verification protocol \(§[3\.4](https://arxiv.org/html/2608.21832#S3.SS4)\)\. The full construction pipeline is summarized in Figure[1](https://arxiv.org/html/2608.21832#S1.F1)\.

### 3\.1Seven elementary primitives

We chose primitives that are unambiguous, common in everyday GUI language, and decomposable\. Table[1](https://arxiv.org/html/2608.21832#S3.T1)lists all seven together with their relation words and chance level\. The primitives split into three groups\.*Cardinal direction*covers horizontal and vertical relative position\.*Topology*covers containment \(inside or outside a panel\)\.*Layout*covers alignment \(sharing a row or a column\), proximity \(which of several elements is nearest\), list ordinal \(the third item of a menu\), and occlusion \(visible or partly covered by a pop\-up\)\.

#### Why these seven?

We require each primitive to be lexically realizable in a short user instruction, separable by a single relation\-expression change into a contrastive pair, and a basic building block of compositional GUI relations\. The seven jointly cover the four broad classes cataloged by prior natural\-image work: cardinal direction, topology, ordinality, and layout\-distance\([12](https://arxiv.org/html/2608.21832#bib.bib8);[7](https://arxiv.org/html/2608.21832#bib.bib4);[5](https://arxiv.org/html/2608.21832#bib.bib33)\)\. We do not claim coverage of every spatial relation, only of the elementary ones from which the rest compose\.

Table 1:The seven elementary spatial primitives inGUI\-Primitives, with their canonical relation words and two\-candidate reference levels\. All seven primitives, including*list\-ordinal*, are realized as binary minimal\-pair contrasts \(71 items per side\), so the reference level is0\.500\.50throughout; §[4](https://arxiv.org/html/2608.21832#S4)explains why this is a forced\-choice reference rather than the chance level of unconstrained clicking\.

### 3\.2Minimal\-pair construction

Every item inGUI\-Primitivesappears as one half of a contrastive pair: the two members share one screenshot and one anchor element, and the relation expression changes, with the correct target as the*contrastive*element \(Figure[2](https://arxiv.org/html/2608.21832#S3.F2), details in Appendix[J\.1](https://arxiv.org/html/2608.21832#A10.SS1)\)\. This construction removes the shortcuts that inflate scores on coarser benchmarks: a model that always selects the most central, salient, or frequently labeled element earns one point and loses one on a pair, so the pair\-level score is zero\. We measure this directly as*pair\-consistency*\(§[5\.5](https://arxiv.org/html/2608.21832#S5.SS5)\)\. Three representative pairs for each of the remaining six primitives are shown in Appendix[P](https://arxiv.org/html/2608.21832#A16)\.

![Refer to caption](https://arxiv.org/html/2608.21832v1/fig1_minimal_pair_teaser.png)Figure 2:A contrastive pair fromGUI\-Primitives\(details in Appendix[P](https://arxiv.org/html/2608.21832#A16)\)\. The screenshot is fixed; one word flips, and the correct target shifts to the other candidate\. A model exploiting an answer prior passes one twin and fails the other\.
### 3\.3Sources: real and synthetic screenshots

GUI\-Primitivesdraws from two sources \(Appendix[J\.2](https://arxiv.org/html/2608.21832#A10.SS2)and[K](https://arxiv.org/html/2608.21832#A11)\), both reported separately throughout\. The*real*arm \(n=290n=290\) uses element\-grounding annotations from UI\-Vision\([16](https://arxiv.org/html/2608.21832#bib.bib19)\)on Web, MacOS, VS Code, Office, and professional applications, covering five primitives whose annotations are recoverable from element labels: horizontal and vertical relative position, alignment, proximity, and a 24\-item list\-ordinal slice mined from real menus and lists\. The*synthetic*arm \(n=704n=704\) is procedurally rendered with known positions, controlled distractors, and a fixed layout language; it is the only way to obtain*containment*and*occlusion*cleanly, and it supplies the remaining*list\-ordinal*items, since real screenshot annotations do not ship the parent–child or overlay–target metadata\. We treat the synthetic arm as a controlled\-stimulus arm in theWhat’s\-Uptradition and disclose every result split by source\.

### 3\.4Human verification

Five annotators independently judged a 196\-item stratified subset \(human\-verified core\), answering two questions per item: well\-formedness and correct target \(Appendix[J\.3](https://arxiv.org/html/2608.21832#A10.SS3)\)\. Agreement is high: FleissκQ​1=0\.942\\kappa\_\{Q1\}=0\.942\(“almost perfect” on the Landis–Koch scale,[9](https://arxiv.org/html/2608.21832#bib.bib16)\) andκQ​2=0\.787\\kappa\_\{Q2\}=0\.787\(“substantial”\)\. Eleven items judged invalid by the majority were dropped, leaving the 185\-itemhuman\-clean core; human accuracy on it is 96\.9%\. Annotator instructions are in Appendix[F](https://arxiv.org/html/2608.21832#A6)\.

## 4Models and Protocol

#### Models\.

We evaluate 19 vision\-language models: seven proprietary models accessed through commercial APIs \(Claude Opus 4\.7, Sonnet 4\.6, Haiku 4\.5, GPT\-5, GPT\-4\.1, GPT\-4o\-mini, Gemini 3\.1 Flash Lite\); three Qwen\-family open \(Qwen2\.5\-VL\-7B, Qwen2\-VL\-7B, OS\-Atlas\-Base\-7B\); InternVL3\-8B; Meta Llama\-3\.2\-11B\-Vision; three sizes of Gemma 3; two PaliGemma 2 variants; and MiniCPM\-V\. Identifiers and endpoints are in Appendix[B](https://arxiv.org/html/2608.21832#A2)\. Pixtral\-12B and Idefics3 8B are excluded because their chat template formats are incompatible with our generic wrapper\.

#### Prompting and decoding\.

All models use greedy decoding \(T=0T=0\) and a fixed seed\. Per\-family coordinate conventions \(pixel, normalized\[0,1\]\[0,1\], normalized\[0,1000\]\[0,1000\]\) are handled by a robust parser whose unit tests are bundled with the artifact\. The three closed providers silently downsample large screenshots server\-side \(Anthropic≤1568\\leq 1568px, OpenAI≈1024\\approx 1024on the long side, Gemini variable\); we pre\-resize to a known long side and rescale predicted coordinates back to the original pixel space\. The rescale was needed on a large fraction of items \(e\.g\., 1,297 of 1,581 Sonnet ScreenSpot\-Pro records\); Appendix[E](https://arxiv.org/html/2608.21832#A5)documents the procedure and the per\-model record counts\.

#### Metrics\.

The primary metric, following ScreenSpot\-Pro, is*point\-in\-box*accuracy: a predictionp^=\(x^,y^\)\\hat\{p\}=\(\\hat\{x\},\\hat\{y\}\)is correct iffx^∈\[x1,x2\]\\hat\{x\}\\in\[x\_\{1\},x\_\{2\}\]andy^∈\[y1,y2\]\\hat\{y\}\\in\[y\_\{1\},y\_\{2\}\]in the original pixel frame\.*Loose*accuracy \(prediction within2×2\\timestarget diagonal of the box center\) separates sub\-pixel imprecision from element misidentification on small icons\. All accuracies carry bootstrap 95% CIs \(2,000 resamples\); paired comparisons use McNemar\([15](https://arxiv.org/html/2608.21832#bib.bib17)\)with Holm–Bonferroni\([6](https://arxiv.org/html/2608.21832#bib.bib18)\)correction across the seven primitives\. All seven primitives are binary minimal\-pair contrasts \(Table[1](https://arxiv.org/html/2608.21832#S3.T1)\), so0\.500\.50serves as a*two\-candidate reference level*throughout\. Because models emit an unconstrained coordinate rather than selecting among candidates,0\.500\.50is a forced\-choice reference rather than the chance level of the prediction task, and a prediction can fall outside both candidate regions\. We flag a cell as*below the reference level*only when the upper bootstrap bound falls below0\.500\.50\. To distinguish selection of the contrastive element from predictions outside both regions, we classify every baseline prediction as target, distractor, neither, or invalid in Appendix[R](https://arxiv.org/html/2608.21832#A18)\(further details in Appendix[L](https://arxiv.org/html/2608.21832#A12)\)\.

## 5Diagnostic Results

We report results in five parts, in this order: \(i\) controls that validate the benchmark, \(ii\) overall accuracy and the human gap, \(iii\) per\-primitive structure of failure, \(iv\) the real\-vs\-synthetic decomposition, and \(v\) pair\-consistency as direct evidence of relational failure\.

### 5\.1Shortcut controls validate the benchmark

A diagnostic is only as good as the shortcuts it forbids \(Appendix[N\.1](https://arxiv.org/html/2608.21832#A14.SS1)\)\. Figure[3](https://arxiv.org/html/2608.21832#S5.F3)reports three controls on Qwen2\.5\-VL\-7B over the full 994\-item benchmark\.

Figure 3:Shortcut controls \(Qwen2\.5\-VL\-7B,n=994n=994\)\. Text\-only and shuffled drops are both highly significant; blur produces a smaller but significant drop, below the pre\-registered ten\-point threshold\.Pairing the instruction with a*blank canvas*drops accuracy from 22\.0% to 6\.5% \(Δ=−15\.5\\Delta=\-15\.5pt,p=2\.8×10−27p=2\.8\\times 10^\{\-27\}\);*shuffling*the screenshot to one from a different item of the same primitive drops it to 14\.8% \(Δ=−7\.2\\Delta=\-7\.2pt,p=1\.3×10−11p=1\.3\\times 10^\{\-11\}\)\. Both behave as a grounded model should: when the visual or the matching instruction is removed, the task collapses\.*Heavy Gaussian blur*\(σ=12\\sigma=12px\) drops accuracy only to 16\.7% \(Δ=−5\.3\\Delta=\-5\.3pt,p=8\.5×10−5p=8\.5\\times 10^\{\-5\}\) significant but below our pre\-registered ten\-point threshold\. VLM grounding appears to rely more on global layout than on fine\-grained OCR of text labels, since removing high\-frequency detail removes only a modest fraction of the signal\. However, blur alone cannot disentangle failures in OCR, spatial encoding, and cross\-modal binding; targeted ablations are discussed in Appendix[S](https://arxiv.org/html/2608.21832#A19)\.

### 5\.2Overall accuracy and the human gap

Table 2:Overall accuracy onGUI\-Primitivesfor 12 representative models, alongside accuracy on the human\-clean 185\-item subset\. The strongest model \(Claude Opus 4\.7\) reaches 32% on the clean core, a 65\-point gap below human accuracy \(96\.9%\)\. All 19 models are in Appendix[A](https://arxiv.org/html/2608.21832#A1)\.Table[2](https://arxiv.org/html/2608.21832#S5.T2)reports the headline numbers, and Figure[4](https://arxiv.org/html/2608.21832#S5.F4)visualizes the gap from humans\.

Figure 4:Human\-clean core \(185 items, 5 annotators,κ=0\.94\\kappa=0\.94\)\. Human accuracy is 96\.9%\. The strongest model is Claude Opus 4\.7 at 32\.4%\.All evaluated models perform substantially below the annotator target\-selection accuracy of 96\.9%, though the two protocols are not matched \(§[3\.4](https://arxiv.org/html/2608.21832#S3.SS4)\): the strongest system \(Claude Opus 4\.7\) reaches 31–32% strict point\-in\-box accuracy and the 65\-point gap holds on both the full benchmark and the cleaned 185\-item core\. Scale alone does not close it; the four strongest proprietary models cluster in a narrow 24–31% range, and the 7B open Qwen2\.5\-VL\-Instruct is statistically tied with closed Claude Haiku 4\.5 on the human\-verified core \(24\.5% vs\. 25\.0%\) despite being roughly two orders of magnitude smaller and runnable on a single GPU\. The rank order is stable across the full 994\-item benchmark, the 196\-item human\-verified core, and an LLM\-judge\-verified 100\-item subset \(Appendix[A](https://arxiv.org/html/2608.21832#A1)\), so the gap is not a noisy\-item artifact\.

### 5\.3Per\-primitive structure of failure

Figure 5:Per\-primitive accuracy across eight representative models, with bootstrap 95% confidence intervals\. Dashed lines mark the two\-candidate reference level \(0\.50 for all primitives; Table[1](https://arxiv.org/html/2608.21832#S3.T1)\)\. Oncontainment,occlusion,alignment, andproximity, every model falls below this reference, including the four strongest proprietary models; candidate\-level analysis \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\) attributes this below\-reference accuracy primarily to predictions outside both candidate regions\.Figure[5](https://arxiv.org/html/2608.21832#S5.F5)unpacks the overall accuracy by primitive \(Appendix[N\.2](https://arxiv.org/html/2608.21832#A14.SS2)\)\. The picture is uneven and informative\. One primitive,list\-ordinal, is well solved by the top five models \(Claude family, GPT\-5, and Qwen2\.5\-VL\), all achieving0\.800\.80–0\.830\.83against the0\.500\.50two\-candidate reference \(Table[1](https://arxiv.org/html/2608.21832#S3.T1)\)\. The shared task here is counting list positions, which the language model can do once it identifies the list\. The other six primitives are uniformly hard\. Onrel\-pos\-horizontal, Claude Opus 4\.7 reaches0\.510\.51, at the reference level; every other model is below\. Onrel\-pos\-vertical, the best model is GPT\-5 at0\.320\.32\. Oncontainment,occlusion,alignment, andproximity,*the upper 95% bootstrap bound for every model we test sits below the reference level*of0\.500\.50, including all four strongest proprietary models\. Because0\.500\.50is a two\-candidate reference level rather than the chance level of unconstrained coordinate prediction, we classify all 18,886 baseline predictions by the candidate region they fall in \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\)\. Predictions fall outside both candidate regions on 60–92% of items, and among predictions inside a candidate region, the target is selected at least as often as the distractor on every primitive\. Forcontainmentandocclusionthe within\-region selection rate is statistically indistinguishable from0\.500\.50\(0\.548 and 0\.554\), indicating no measurable relation\-word signal on these two primitives\. Forrel\-pos\-horizontal,rel\-pos\-vertical, andproximitythe target is selected on 0\.89–0\.90 of within\-region predictions, indicating that the deficit is candidate localization rather than relational interpretation; thealignmentestimate varies with the classification rule \(§[R](https://arxiv.org/html/2608.21832#A18)\) and we do not interpret it\. The associated directional prior is quantified in Appendix[Q](https://arxiv.org/html/2608.21832#A17), and its role is discussed in §[8](https://arxiv.org/html/2608.21832#S8)\. Model\-specific coordinate\-frame anomalies are also quantified in Appendix[Q](https://arxiv.org/html/2608.21832#A17)\.

### 5\.4Real versus synthetic

We next separate two candidate explanations for the per\-primitive results in §[5\.3](https://arxiv.org/html/2608.21832#S5.SS3): small real\-GUI target size and the relative proportion of synthetic items\. The full strict/loose split by source is in Appendix[C](https://arxiv.org/html/2608.21832#A3)and[N\.4](https://arxiv.org/html/2608.21832#A14.SS4)\. On synthetic screenshots,*loose*accuracy is0\.600\.60–0\.760\.76even when strict accuracy is much lower, indicating that synthetic errors are largely sub\-pixel imprecision rather than element misidentification\. On UI\-Vision real screenshots, strict accuracy is essentially zero for every model on every primitive, and*loose*accuracy also stays low \(66–23%23\\%\)\. Candidate\-level analysis \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\) shows that 96% of real\-screenshot predictions fall outside both the target and distractor regions, while among predictions inside a candidate region, the target is selected on 0\.75 pooled \(0\.91 for the two strongest models\)\. Real\-screenshot accuracy is therefore primarily attributable to candidate localization, with target size a contributing factor \(Appendix[Q](https://arxiv.org/html/2608.21832#A17)\), rather than to inverted relational interpretation\.

Figure 6:Pair\-consistency vs\. raw accuracy, per model per primitive\. Independent guessing would place each point neary=x2y=x^\{2\}\(dashed\)\. Almost all points lie*below*this curve, theWhat’s\-Upsignature of relational failure\.
### 5\.5Pair\-consistency: a relational failure signature

Independent draws would put pair\-consistency aty=x2y=x^\{2\}\(accuracy squared\)\. Figure[6](https://arxiv.org/html/2608.21832#S5.F6)shows it consistently*below*that curve, the relational\-failure signature of[7](https://arxiv.org/html/2608.21832#bib.bib4)\. Within a model, the two pair members are negatively correlated: being right on the “left of” twin makes the model more likely than chance to be wrong on the “right of” twin\. The relation word, not the picture, is what confuses the model \(Appendix[N\.5](https://arxiv.org/html/2608.21832#A14.SS5)\)\. Eight representative minimal\-pair failures are analyzed in Appendix[T](https://arxiv.org/html/2608.21832#A20)\.

Figure 7:Per\-modelGUI\-Primitivesaccuracy \(nmodels=10n\_\{\\text\{models\}\}=10\) against ScreenSpot\-Pro grounding accuracy\. Spearmanρ=\+0\.736\\rho=\+0\.736\(p=0\.015p=0\.015\)\.

## 6Primitive Competence Predicts Grounding

A diagnostic is only useful if it predicts a downstream property of interest\. We test this on ScreenSpot\-Pro\([10](https://arxiv.org/html/2608.21832#bib.bib3)\), the standard high\-resolution GUI\-grounding benchmark, which we ran on the ten models for which we have bothGUI\-Primitivesand SS\-Pro predictions\. Figure[7](https://arxiv.org/html/2608.21832#S5.F7)shows the per\-model relationship\. Spearmanρ=\+0\.736\\rho=\+0\.736\(p=0\.015p=0\.015,n=10n=10\); the correlation holds inside the real\-screenshot subset \(ρ=\+0\.705\\rho=\+0\.705,p=0\.023p=0\.023\) and inside the synthetic subset \(ρ=\+0\.760\\rho=\+0\.760,p=0\.011p=0\.011\)\. A leave\-one\-model\-out sensitivity analysis gives a Spearman in\[\+0\.62,\+0\.86\]\[\+0\.62,\+0\.86\]; no single model drives the result, though dropping the strongest or weakest raisesppto≈0\.07\\approx 0\.07\. An item\-level logistic regression with model fixed effects \(details in Appendix[O](https://arxiv.org/html/2608.21832#A15)\) and a log target\-area control reaches pseudo\-R2=0\.404R^\{2\}=0\.404onn=15,810n=15\{,\}810pairs and corroborates the joint association; per\-primitive coefficients are small \(\|coef\|≤0\.07\|\\text\{coef\}\|\\leq 0\.07\) and fragile, so we claim only the model\-level effect \(full forest plot in Appendix[D](https://arxiv.org/html/2608.21832#A4)\)\.

## 7Training\-Free Interventions

If models lack the elementary spatial skills GUI grounding needs, can inference\-time scaffolding put them back? We test three interventions\.SoM\([30](https://arxiv.org/html/2608.21832#bib.bib9)\)overlays numbered marks on candidate elements; the model names the mark, trading pixel regression for symbolic selection\.Primitive\-aware CoTprepends a short decomposition prompt that names the relation explicitly, probing whether the failure is a reasoning sequence rather than perception\.Activation steeringfollows SteerVLM\([19](https://arxiv.org/html/2608.21832#bib.bib11)\): we compute mean\-difference vectors between correctly\- and incorrectly\-grounded items on a 200\-item held\-out contrast set, then addα​𝐯ℓ\\alpha\\mathbf\{v\}\_\{\\ell\}to the residual stream at decoder layersℓ∈\{12,…,19\}\\ell\\in\\\{12,\\dots,19\\\}of Qwen2\.5\-VL\-7B at inference, withα=4\.0\\alpha=4\.0chosen as the largest value from\{1,2,4,6\}\\\{1,2,4,6\\\}preserving fluency\. As a positive control, the same pipeline with a verbosity\-contrast vector shifts output length as expected, so the machinery is functional\.

Figure 8:Intervention deltas vs\. each model’s baseline, with Holm\-adjusted significance \(paired McNemar across primitives\)\. SoM recovers 35–57 points on three of four tested models; the only marginal case is Claude Opus 4\.7, which already has the highest baseline\. CoT and activation steering yield no significant change\.### 7\.1Oracle two\-candidate marking

Figure[8](https://arxiv.org/html/2608.21832#S7.F8)reports intervention deltas with significance\. Oracle two\-candidate marking overlays marks on exactly the target and distractor, so the baseline and marked conditions are not identical\-task comparisons: marking removes the candidate\-search burden that dominates baseline error \(§[5\.3](https://arxiv.org/html/2608.21832#S5.SS3)\)\. It is the only intervention producing large, statistically significant gains:\+57\.1\+57\.1pt onGPT\-5\(30→87%30\\to 87\\%,p<10−50p<10^\{\-50\}the highest absolute accuracy any model reaches under any condition\),\+42\.1\+42\.1pt onOS\-Atlas\-Base\-7B\(10→52%10\\to 52\\%,p<10−50p<10^\{\-50\}\),\+35\.1\+35\.1pt onQwen2\.5\-VL\-7B\(22→57%22\\to 57\\%,p<10−50p<10^\{\-50\}\), and a marginal\+9\.2\+9\.2pt onClaude Opus 4\.7\(31→40%31\\to 40\\%, Holm\-adjustedp=0\.08p=0\.08, not significant\)\. Across the four evaluated models, the observed gain is inversely ordered with baseline accuracy; given four points, we treat this as descriptive rather than a general relationship\.

#### Per\-primitive pattern\.

Figure[9](https://arxiv.org/html/2608.21832#S7.F9)breaks the gain down by primitive on the two closed APIs we ran with SoM\. Six of seven primitives gain at least 40 points on GPT\-5; the only primitive that*regresses*\(by−32\.4\-32\.4points\) islist\-ordinalon Qwen2\.5 VL, which the model had already saturated at 81% baseline\. Converting an already\-solved task into mark selection introduces an indirect cost that can reduce accuracy on primitives the model has already saturated\.

Figure 9:Per\-primitive SoM delta on Claude Opus 4\.7 and GPT\-5\. Stars mark intervals excluding zero\. On GPT\-5, six of seven primitives gain at least 40 points\.

### 7\.2CoT and activation steering: negative results

We report two negative results with the same effort as the positive one \(Appendix[M](https://arxiv.org/html/2608.21832#A13)\)\. Primitive\-aware CoT on Qwen2\.5\-VL\-7B producedΔ=\+0\.5\\Delta=\+0\.5pt \(p=0\.51p=0\.51, n\.s\.\); the model already names the relation correctly in its intermediate reasoning, then still emits the wrong coordinate\. Activation steering producedΔ=\+1\.5\\Delta=\+1\.5pt \(p=0\.066p=0\.066, n\.s\.\), with no configuration in the grid crossing the Holm threshold\. The one intervention that works bypasses the spatial skill it should test; the two that probe internal representation directly do not move the metric\. We read this as evidence that the failure lies closer to perception \(visual encoder, cross\-modal binding\) than to language\-side reasoning or residual\-stream representation\.

## 8Discussion

Accuracy does not increase monotonically with nominal model scale in the evaluated set\. The strongest proprietary models achieve only 24–31% accuracy\. Spatial grounding in GUI screenshots remains a persistent weakness of current vision\-language systems\. Oracle two\-candidate marking provides the largest training\-free improvement we observe, but it supplies the candidate set rather than repairing the skill gap, and is therefore a diagnostic upper bound rather than a solution\. By marking candidate elements, SoM shifts part of the relational grounding problem into symbolic mark selection\. Its effectiveness also depends on the quality of the upstream detector, whose failure modes are discussed in Appendix[I](https://arxiv.org/html/2608.21832#A9)\. Candidate\-level analysis \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\) indicates that below\-reference accuracy is primarily attributable to predictions outside both candidate regions \(60–92% of predictions\) rather than to systematic selection of the contrastive element; within\-region selection follows the relation word on all primitives except containment and occlusion, where it is statistically indistinguishable from0\.500\.50\. The associated layout\-frequency prior is directional: predictions fall within a candidate region 2\.7×\\timesmore often when the target is the left element, and within\-region accuracy is 0\.972 under “left of” versus 0\.717 under “right of” \(Appendix[Q](https://arxiv.org/html/2608.21832#A17)\)\. We report this prior as a hypothesis for the source of the directional asymmetry rather than as a causal claim\. The same decomposition is consistent with the magnitude of the SoM gain: SoM supplies candidate regions directly, addressing the candidate\-localization component that accounts for most baseline errors\. Appendix[U](https://arxiv.org/html/2608.21832#A21)presents six recurring error types with representative examples\. Finally, the failure pattern mirrors relational errors observed inWhat’s\-Upfor natural images\([7](https://arxiv.org/html/2608.21832#bib.bib4)\)\. This suggests that controlled minimal\-pair diagnostics are essential for evaluating spatial competence across visual modalities, including GUI grounding\.

## 9Limitations

This work has several limitations \(Appendix[V](https://arxiv.org/html/2608.21832#A22)\)\.\(i\)Containment and occlusion appear only in the synthetic arm ofGUI\-Primitives\(list\-ordinal includes a 24\-item real slice\), since real GUI datasets lack scalable parent–child or overlay–target metadata for minimal\-pair mining; findings on these primitives therefore have limited external validity, and results are reported separately by source arm\. This scoping bounds the candidate\-level finding of Appendix[R](https://arxiv.org/html/2608.21832#A18): the absence of a measurable relation\-word signal on containment and occlusion is established on synthetic screenshots only\.\(ii\)After model fixed effects absorb between\-model variance, per\-primitive coefficients in our item\-level regression are small \(\|coef\|≤0\.07\|\\text\{coef\}\|\\leq 0\.07\); we claim only the joint model\-level association and report the forest plot \(Appendix[D](https://arxiv.org/html/2608.21832#A4)\) for transparency\.\(iii\)We stop at ScreenSpot\-Pro; end\-to\-end OSWorld task success\([27](https://arxiv.org/html/2608.21832#bib.bib1);[28](https://arxiv.org/html/2608.21832#bib.bib32)\)remains future work\.\(iv\)GUI\-Primitivesv1 covers static pre\-action grounding for English click targets only; planned extensions include drag\-target prediction, bidirectional and CJK layouts, an OSWorld pre\-action slice, and frozen snapshots of shifting closed APIs \(Appendix[B](https://arxiv.org/html/2608.21832#A2)\)\.\(v\)SoM acts as a scaffold by partially converting relational grounding into symbolic mark selection; our setup overlays at most two candidate marks, so gains may not generalize to realistic multi\-candidate scenes, and deployment performance will inherit upstream detector failures\.\(vi\)Point\-in\-box accuracy is binary and coarse: our loose metric \(within2×2\\timestarget diagonal\) partly separates sub\-pixel imprecision from element misidentification, but neither metric captures structured spatial errors \(e\.g\., anchor\-collapse or direction inversion\), which we analyze qualitatively in Appendix[U](https://arxiv.org/html/2608.21832#A21)\. Llama\-3\.2\-11B\-Vision emits a default coordinate on≈42%\\approx 42\\%of items \(1\.2% accuracy\), consistent with ScreenSpot\-Pro reports of<2%<\\\!2\\%for generalist VLMs\.

## Ethical Considerations

Annotators were paid above the local hourly minimum and provided informed consent \(Appendix[F](https://arxiv.org/html/2608.21832#A6)\)\. UI\-Vision screenshots remain under their original license; we redistribute only derived annotations, not the screenshots themselves\.GUI\-Primitivesis intended to support safer GUI\-agent development by exposing failures in elementary spatial grounding before deployment\.

## References

- B\. Chen, Z\. Xu, S\. Kirmani, B\. Ichter, D\. Driess, P\. Florence, D\. Sadigh, L\. Guibas, and F\. XiaSpatialVLM: endowing vision\-language models with spatial reasoning capabilities\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:[Link](https://arxiv.org/abs/2401.12168)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px2.p1.1)\.
- Chenget al\.\(2024\)K\. Cheng, Q\. Sun, Y\. Chu, F\. Xu, Y\. Li, J\. Zhang, and Z\. WuSeeClick: harnessing gui grounding for advanced visual gui agents\.InProceedings of the Annual Meeting of the Association for Computational Linguistics \(ACL\),External Links:[Link](https://arxiv.org/abs/2401.10935)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§1](https://arxiv.org/html/2608.21832#S1.p2.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Fuet al\.\(2024\)X\. Fu, Y\. Hu, B\. Li, Y\. Feng, H\. Wang, X\. Lin, D\. Roth, N\. A\. Smith, W\. Ma, and R\. KrishnaBLINK: multimodal large language models can see but not perceive\.InProceedings of the European Conference on Computer Vision \(ECCV\),External Links:[Link](https://arxiv.org/abs/2404.12390)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px2.p1.1)\.
- Gouet al\.\(2025\)B\. Gou, R\. Wang, B\. Zheng, Y\. Xie, C\. Chang, Y\. Shu, H\. Sun, and Y\. SuNavigating the digital world as humans do: universal visual grounding for GUI agents\.InProceedings of the International Conference on Learning Representations \(ICLR\),Note:OralExternal Links:[Link](https://arxiv.org/abs/2410.05243)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Hoehinget al\.\(2025\)N\. Hoehing, A\. Cosma, T\. Rebedea, and S\. RusetiUnderstanding space is rocket science — only top reasoning models can solve spatial understanding tasks\.arXiv preprint arXiv:2509\.02175\.External Links:[Link](https://arxiv.org/abs/2509.02175)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.21832#S3.SS1.SSS0.Px1.p1.1)\.
- Holm \(1979\)S\. HolmA simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[Appendix L](https://arxiv.org/html/2608.21832#A12.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.21832#S4.SS0.SSS0.Px3.p1.1)\.
- Kamathet al\.\(2023\)A\. Kamath, J\. Hessel, and K\. ChangWhat’s “up” with vision\-language models? investigating their struggle with spatial reasoning\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://arxiv.org/abs/2310.19785)Cited by:[§N\.5](https://arxiv.org/html/2608.21832#A14.SS5.p1.1),[Appendix U](https://arxiv.org/html/2608.21832#A21.SS0.SSS0.Px2.p1.1),[§1](https://arxiv.org/html/2608.21832#S1.p3.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px2.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px3.p1.1),[§3\.1](https://arxiv.org/html/2608.21832#S3.SS1.SSS0.Px1.p1.1),[§5\.5](https://arxiv.org/html/2608.21832#S5.SS5.p1.1),[§8](https://arxiv.org/html/2608.21832#S8.p1.1)\.
- Kanget al\.\(2025\)S\. Kang, J\. Kim, J\. Kim, and S\. J\. KimYour large vision\-language model only needs a few attention heads for visual grounding\.arXiv preprint arXiv:2503\.06287\.External Links:[Link](https://arxiv.org/abs/2503.06287)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p5.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px4.p1.1)\.
- Landis and Koch \(1977\)J\. R\. Landis and G\. G\. KochThe measurement of observer agreement for categorical data\.Biometrics33\(1\),pp\. 159–174\.Cited by:[§J\.3](https://arxiv.org/html/2608.21832#A10.SS3.p1.1),[§3\.4](https://arxiv.org/html/2608.21832#S3.SS4.p1.1)\.
- Liet al\.\(2025\)K\. Li, Z\. Meng, H\. Lin, Z\. Luo, Y\. Tian, J\. Ma, Z\. Huang, and T\. ChuaScreenSpot\-Pro: GUI grounding for professional high\-resolution computer use\.InProceedings of the ACM International Conference on Multimedia \(MM\),External Links:[Link](https://arxiv.org/abs/2504.07981)Cited by:[Appendix O](https://arxiv.org/html/2608.21832#A15.p1.1),[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§1](https://arxiv.org/html/2608.21832#S1.p2.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1),[§6](https://arxiv.org/html/2608.21832#S6.p1.1)\.
- Linzenet al\.\(2016\)T\. Linzen, E\. Dupoux, and Y\. GoldbergAssessing the ability of LSTMs to learn syntax\-sensitive dependencies\.InTransactions of the Association for Computational Linguistics \(TACL\),External Links:[Link](https://aclanthology.org/Q16-1037/)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p3.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px3.p1.1)\.
- Liuet al\.\(2023\)F\. Liu, G\. Emerson, and N\. CollierVisual spatial reasoning\.Transactions of the Association for Computational Linguistics \(TACL\)\.External Links:[Link](https://arxiv.org/abs/2205.00363)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px2.p1.1),[§3\.1](https://arxiv.org/html/2608.21832#S3.SS1.SSS0.Px1.p1.1)\.
- Luoet al\.\(2025\)T\. Luo, L\. Logeswaran, J\. Johnson, and H\. LeeVisual test\-time scaling for GUI agent grounding\.arXiv preprint arXiv:2505\.00684\.External Links:[Link](https://arxiv.org/abs/2505.00684)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px4.p1.1)\.
- Marvin and Linzen \(2018\)R\. Marvin and T\. LinzenTargeted syntactic evaluation of language models\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),External Links:[Link](https://aclanthology.org/D18-1151/)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p3.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px3.p1.1)\.
- McNemar \(1947\)Q\. McNemarNote on the sampling error of the difference between correlated proportions or percentages\.Psychometrika12\(2\),pp\. 153–157\.Cited by:[Appendix L](https://arxiv.org/html/2608.21832#A12.SS0.SSS0.Px4.p1.1),[§4](https://arxiv.org/html/2608.21832#S4.SS0.SSS0.Px3.p1.1)\.
- Nayaket al\.\(2025\)S\. Nayak, X\. Jian, K\. Lin, J\. A\. Rodriguez, M\. Kalsi, R\. Awal, N\. Chapados, M\. T\. Özsu, A\. Agrawal, D\. Vazquez, C\. Pal, P\. Taslakian, S\. Gella, and S\. RajeswarUI\-Vision: a desktop\-centric GUI benchmark for visual perception and interaction\.InProceedings of the International Conference on Machine Learning \(ICML\),PMLR, Vol\.267,pp\. 45817–45851\.External Links:[Link](https://arxiv.org/abs/2503.15661)Cited by:[§J\.2](https://arxiv.org/html/2608.21832#A10.SS2.p2.1),[§1](https://arxiv.org/html/2608.21832#S1.p4.1),[§3\.3](https://arxiv.org/html/2608.21832#S3.SS3.p1.1)\.
- Qiet al\.\(2025\)J\. Qi, J\. Tang, H\. Wu, M\. Sun, and C\. ZhangBeyond semantics: rediscovering spatial awareness in vision\-language models\.arXiv preprint arXiv:2503\.17349\.External Links:[Link](https://arxiv.org/abs/2503.17349)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Qinet al\.\(2025\)Y\. Qin, Y\. Ye, J\. Fang, H\. Wang, S\. Liang, S\. Tian, J\. Zhang, J\. Li, Y\. Li, S\. Huang, W\. Zhong, K\. Li, J\. Yang, Y\. Chen, K\. Lin, Y\. Yang, Y\. Shen, S\. Cheng, M\. Shen, B\. Zhou, and Y\. LiuUI\-TARS: pioneering automated gui interaction with native agents\.arXiv preprint arXiv:2501\.12326\.External Links:[Link](https://arxiv.org/abs/2501.12326)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Sivakumaret al\.\(2025\)A\. Sivakumar, S\. Hu, V\. Suriyakumar, A\. Goel, and S\. JangSteerVLM: robust model control through lightweight activation steering for vision language models\.InFindings of the Association for Computational Linguistics: EMNLP,External Links:[Link](https://arxiv.org/abs/2510.26769)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p5.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2608.21832#S7.p1.1)\.
- Tonget al\.\(2024\)S\. Tong, E\. Brown, P\. Wu, S\. Woo, M\. Middepogu, S\. C\. Akula, J\. Yang, S\. Yang, A\. Iyer, X\. Pan, A\. Wang, R\. Fergus, Y\. LeCun, and S\. XieCambrian\-1: a fully open, vision\-centric exploration of multimodal LLMs\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2406.16860)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2025a\)X\. Wang, T\. Xie, B\. Wang, J\. Wang, J\. Deng, J\. Chen, W\. Hu, Y\. Xu, J\. Yang, and T\. YuOpenCUA: open foundations for computer\-use agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2508.09123)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2025b\)X\. Wang, Z\. Liu, Y\. Li, Y\. Chen, J\. Cheng, Y\. Zheng, H\. Lin, Z\. Cao, and W\. WangMMBench\-GUI: hierarchical multi\-platform evaluation framework for GUI agents\.arXiv preprint arXiv:2507\.19478\.External Links:[Link](https://arxiv.org/abs/2507.19478)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2025a\)H\. Wu, H\. Chen, H\. Cai, Y\. Liu, J\. Ye, L\. Yang, and Y\. WangDiMo\-GUI: advancing test\-time scaling in GUI grounding via modality\-aware visual reasoning\.InProceedings of the Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 26257–26267\.External Links:[Link](https://arxiv.org/abs/2507.00008)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px4.p1.1)\.
- Wuet al\.\(2025b\)Q\. Wu, K\. Cheng, R\. Yang, C\. Zhang, J\. Yang, H\. Jiang, J\. Mu, B\. Peng, B\. Qiao, R\. Tan, S\. Q\. Shi, Y\. Liu, G\. Zheng, B\. Lv, M\. Du, Y\. Ge, Y\. Sui, L\. Liu, W\. Wang, Y\. N\. Zhang, Q\. Lin, S\. Rajmohan, D\. Zhang, and J\. GaoGUI\-Actor: coordinate\-free visual grounding for GUI agents\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2506.03143)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Wuet al\.\(2024\)Z\. Wu, Z\. Wu, F\. Xu, Y\. Wang, Q\. Sun, C\. Jia, K\. Cheng, Z\. Ding, L\. Chen, P\. P\. Liang, and Y\. QiaoOS\-Atlas: a foundation action model for generalist GUI agents\.arXiv preprint arXiv:2410\.23218\.External Links:[Link](https://arxiv.org/abs/2410.23218)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2025\)T\. Xie, J\. Deng, X\. Li, J\. Yang, H\. Wu, J\. Chen, W\. Hu, X\. Wang, Y\. Xu, Z\. Wang, Y\. Su, J\. Yang, C\. Cao, and T\. YuScaling computer\-use grounding via user interface decomposition and synthesis\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:Spotlight, Datasets and BenchmarksExternal Links:[Link](https://arxiv.org/abs/2505.13227)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. YuOSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:[Link](https://arxiv.org/abs/2404.07972)Cited by:[Appendix V](https://arxiv.org/html/2608.21832#A22.SS0.SSS0.Px3.p1.1),[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§1](https://arxiv.org/html/2608.21832#S1.p2.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1),[§9](https://arxiv.org/html/2608.21832#S9.p1.1)\.
- XLANG Lab \(2025\)XLANG LabIntroducing OSWorld\-Verified\.Note:XLANG Lab blog post, 28 July 2025External Links:[Link](https://xlang.ai/blog/osworld-verified)Cited by:[Appendix V](https://arxiv.org/html/2608.21832#A22.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1),[§9](https://arxiv.org/html/2608.21832#S9.p1.1)\.
- Yanet al\.\(2024\)A\. Yan, Z\. Yang, J\. Wu, W\. Zhu, J\. Yang, L\. Li, K\. Lin, J\. Wang, J\. McAuley, J\. Gao, and L\. WangList items one by one: a new data source and learning paradigm for multimodal LLMs\.InConference on Language Modeling \(COLM\),External Links:[Link](https://arxiv.org/abs/2404.16375)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px4.p1.1)\.
- Yanget al\.\(2023\)J\. Yang, H\. Zhang, F\. Li, X\. Zou, C\. Li, and J\. GaoSet\-of\-mark prompting unleashes extraordinary visual grounding in GPT\-4V\.arXiv preprint arXiv:2310\.11441\.External Links:[Link](https://arxiv.org/abs/2310.11441)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p5.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px4.p1.1),[§7](https://arxiv.org/html/2608.21832#S7.p1.1)\.
- Yanget al\.\(2025a\)Y\. Yang, D\. Liu, Y\. Jiang, Y\. Huang, H\. Xu, Y\. Cheng, C\. Zou, M\. Le, and C\. XiongGTA1: GUI test\-time scaling agent\.arXiv preprint arXiv:2507\.05791\.External Links:[Link](https://arxiv.org/abs/2507.05791)Cited by:[§1](https://arxiv.org/html/2608.21832#S1.p1.1),[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Yanget al\.\(2025b\)Y\. Yang, Y\. Lou, Y\. Yuan, X\. Sun, J\. Wei, H\. Liu, Y\. Wang, Y\. Liu, J\. Liang, and J\. LiAria\-UI: visual grounding for GUI instructions\.InFindings of the Association for Computational Linguistics: ACL,External Links:[Link](https://arxiv.org/abs/2412.16256)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.
- Zhaoet al\.\(2025\)W\. Zhao, X\. Chen, J\. Henderson, V\. Carbune, and P\. VergaLearning GUI grounding with spatial reasoning from visual feedback\.arXiv preprint arXiv:2509\.21552\.External Links:[Link](https://arxiv.org/abs/2509.21552)Cited by:[§2](https://arxiv.org/html/2608.21832#S2.SS0.SSS0.Px1.p1.1)\.

## Appendix AFull Per\-Model Results

Table[3](https://arxiv.org/html/2608.21832#A1.T3)reports overall accuracy for all 19 models on the full benchmark, the 196\-item human\-verified core, and the 185\-item human\-clean core\. Per\-primitive accuracies with bootstrap 95% CIs for every cell are released alongside the artifact inruns/diagnostic/analysis\.json\. A model\-by\-primitive accuracy heatmap is shown in Figure[10](https://arxiv.org/html/2608.21832#A1.F10)\.

Table 3:Full benchmark accuracy for all 19 evaluated models on three splits: the fulln=994n\{=\}994benchmark \(Full\), the human\-verifiedn=196n\{=\}196subset \(Core\), and the cleanedn=185n\{=\}185subset after removing items judged invalid by a majority of the five annotators \(Clean\)\. Closed models were not run on the full split because of API cost; the Core column reports every model that was\. Per\-primitive accuracies with bootstrap intervals are released in the artifact\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_heatmap.png)Figure 10:Per\-model per\-primitive accuracy heatmap, all 19 models\. The horizontal band at∼\\sim0\.80 onlist\-ordinalmarks the only primitive any model solves reliably\.
## Appendix BModels, Versions, and Endpoints

Table[4](https://arxiv.org/html/2608.21832#A2.T4)lists the exact model identifiers used in this paper, together with the endpoint each was queried through\. Closed models were accessed via their official APIs in May 2026; open models were run from Hugging Face checkpoints\. We pin identifiers here so later runs can be compared against the snapshot reported in this paper\.

Table 4:Model identifiers used in this paper\. Closed models were queried via official APIs in May 2026; open models were run from Hugging Face checkpoints with greedy decoding \(T=0T=0\) and a fixed seed\.ModelIdentifierFamily / EndpointClaude Opus 4\.7claude\-opus\-4\-7Anthropic APIClaude Sonnet 4\.6claude\-sonnet\-4\-6Anthropic APIClaude Haiku 4\.5claude\-haiku\-4\-5Anthropic APIGPT\-5 \(reasoning\)gpt\-5OpenAI APIGPT\-4\.1gpt\-4\.1OpenAI APIGPT\-4o\-minigpt\-4o\-miniOpenAI APIGemini 3\.1 Flash Litegemini\-3\.1\-flash\-liteGoogle APIQwen2\.5\-VL\-7B\-InstructQwen/Qwen2\.5\-VL\-7B\-InstructHugging FaceQwen2\-VL\-7B\-InstructQwen/Qwen2\-VL\-7B\-InstructHugging FaceOS\-Atlas\-Base\-7BOS\-Copilot/OS\-Atlas\-Base\-7BHugging FaceInternVL3\-8BOpenGVLab/InternVL3\-8BHugging FaceLlama\-3\.2\-11B\-Visionmeta\-llama/Llama\-3\.2\-11B\-Vision\-InstructHugging FaceGemma 3\-27B\-itgoogle/gemma\-3\-27b\-itHugging FaceGemma 3\-12B\-itgoogle/gemma\-3\-12b\-itHugging FaceGemma 3\-4B\-itgoogle/gemma\-3\-4b\-itHugging FacePaliGemma 2\-10B\-mix\-448google/paligemma2\-10b\-mix\-448Hugging FacePaliGemma 2\-3B\-mix\-448google/paligemma2\-3b\-mix\-448Hugging FaceMiniCPM\-V \(ollama\)minicpm\-vOllama \(Q4\)gemma3:27b \(ollama\)gemma3:27bOllama \(Q4\)
## Appendix CReal vs\. Synthetic: Strict and Loose Accuracy

Table[5](https://arxiv.org/html/2608.21832#A3.T5)reports point\-in\-box \(strict\) and within\-2×2\{\\times\}\-diagonal \(loose\) accuracy split by source for seven representative models\. The zero strict accuracy on UI\-Vision real screenshots is not a target\-size artifact: loose accuracy stays low \(66–23%23\\%\), so the model picks the wrong element rather than missing the right one by a few pixels\.

Table 5:Strict \(point\-in\-box\) vs\. loose \(within2×2\\timestarget\-diagonal\) accuracy, split by image source\. In real desktop screenshots, loose accuracy stays low even when the prediction is allowed to miss by a generous margin: failure is element misidentification, not sub\-pixel imprecision\.
## Appendix DItem\-Level Regression Forest Plot

Figure[11](https://arxiv.org/html/2608.21832#A4.F11)shows the item\-level logistic regression of ScreenSpot\-Pro success on per\-primitiveGUI\-Primitivescompetence \(n=15,810n=15\{,\}810, pseudo\-R2=0\.404R^\{2\}=0\.404\), with model fixed effects, a log target\-area control, and item\-clustered standard errors\. After model effects absorb between\-model variance, individual primitive coefficients are small \(\|coef\|≤0\.07\|\\text\{coef\}\|\\leq 0\.07\) and only two cross the Holm\-corrected threshold, so we report this plot for transparency only\.The headline number is the model\-level Spearman in Fig\.[7](https://arxiv.org/html/2608.21832#S5.F7)of the main paper, not the per\-primitive coefficients\.

Figure 11:Item\-level regression forest plot \(n=15,810n=15\{,\}810, pseudo\-R2=0\.404R^\{2\}=0\.404\)\.
## Appendix EClosed\-API Coordinate Rescaling

All three closed\-vision APIs we tested apply server\-side image preprocessing whose output is not visible to the caller\. The predicted click coordinates are returned in the*downsampled*frame rather than the original pixel space, silently misaligning the prediction with the ground\-truth bounding box\.

#### Detection\.

On the first GPT\-5 ScreenSpot\-Pro run, we observed predictions like\(662,541\)\(662,541\)on items whose ground\-truth boxes were at\(1774,1586,2113,1618\)\(1774,1586,2113,1618\)in a3840×21603840\\times 2160image\. The prediction was internally consistent with a10241024\-side downsample; the ground truth was in the original frame\.

#### Mitigation\.

Our wrapper now resizes every image to a known maximum long side \(≤1500\\leq 1500px\) before submission, then rescales the predicted coordinate back to the original space using the recorded resize factor\. Where the provider does further internal tiling \(notably OpenAI\), we additionally validate by re\-querying on the already\-downsized image and confirming the rescale factor is≈1\.0\\approx 1\.0\. Records that required rescaling: 42/196 Claude Haiku diagnostic, 43/196 Claude Opus diagnostic, 219/994 Claude Sonnet diagnostic, 1,297/1,581 Claude Sonnet ScreenSpot\-Pro, and≈1500/1581\\approx 1500/1581GPT\-5 ScreenSpot\-Pro\. The 284 Claude Sonnet ScreenSpot\-Pro items that originally errored with HTTP 400 \(“image\>5\>5MB”\) were retried under the new wrapper, and the final run is clean\.

#### Recommendation\.

We recommend that future closed\-VLM evaluations report whether their wrapper handles provider\-side downsampling\. Otherwise, ScreenSpot\-Pro’s strict accuracy on high\-resolution professional applications is silently and severely under\-reported\.

## Appendix FHuman Annotation Protocol

Five annotators independently labeled the 196\-item stratified subset\. Each annotator answered two questions per item\.

#### Q1: well\-formedness\.

Is the instruction unambiguous on this screenshot?The annotator answeredgoodorbad\. An item was kept only if at least 3 of 5 annotators marked it as well\-formed\. This dropped 11 of 196 items, leaving the 185\-item*human\-clean*core\.

#### Q2: correct target\.

Which of the two candidate boxes does the instruction identify?The annotator answeredt\(the target box\) ord\(the distractor box\)\. Human accuracy is the fraction of items on which the majority answer wast, computed on the human\-clean core only\.

#### Agreement\.

We report Fleissκ\\kappafor both questions on the full 196\-item subset\. FleissκQ​1=0\.942\\kappa\_\{Q1\}=0\.942\(well\-formedness, “almost perfect”\); FleissκQ​2=0\.787\\kappa\_\{Q2\}=0\.787\(target, “substantial”\)\. The pairwiseκ\\kappamatrix is released with the artifact\.

#### Compensation and consent\.

Annotators were paid above the local hourly minimum, were told the purpose of the labeling \(academic publication and public benchmark release\), and could opt out at any point\. No personally\-identifying information was collected\.

## Appendix GSynthetic\-Stimulus Generation

The synthetic arm ofGUI\-Primitivesrenders GUI\-like screenshots with a fixed layout language: a top toolbar of buttons, a left sidebar of list items, a central grid of tiles, and an optional dialog overlay\. Element positions are sampled uniformly within constrained pixel ranges; the seed is fixed to ensure reproducibility of the corpus\. For each primitive, a separate generator constructs minimal pairs by selecting an anchor and the two candidate elements that satisfy the contrastive relation\. Quality gates in the released code verify that both members of every pair use the same screenshot, have non\-empty disjoint targets, and pass per\-primitive coverage checks before items enter the benchmark\.

## Appendix HIntervention Implementation Details

#### SoM\.

We render mark IDs on the candidate boxes \(two per item: target and distractor\) with a deterministic ID shuffle keyed by the item ID so the correct answer is not always “1”\. The model is asked to respond with the mark number to click, and we map the response back to the box center for evaluation\.

#### Primitive\-aware CoT\.

We prepend a one\-paragraph decomposition prompt that names the relation explicitly and asks for a chain of reasoning before the click\. Per\-primitive scaffolds are released in the artifact\.

#### Activation steering\.

We hook the decoder layers of Qwen2\.5\-VL\-7B at layers\{12,…,19\}\\\{12,\\dots,19\\\}and addα⋅𝐯ℓ\\alpha\\cdot\\mathbf\{v\}\_\{\\ell\}to the residual stream at each layer, where𝐯ℓ\\mathbf\{v\}\_\{\\ell\}is the mean\-difference vector between hidden states on correctly\- and incorrectly\-grounded baseline items at that layer\. We searchedα∈\{1,2,4,6\}\\alpha\\in\\\{1,2,4,6\\\};α=4\\alpha=4was the maximum that left sentence\-level fluency intact\.

## Appendix ISoM Failure Modes

SoM replaces the original grounding task with a mark\-selection task\. The substitution is not perfect, and the failure modes are worth listing for downstream users\.

Saturated primitive regression\.On Qwen2\.5\-VL\-7B,list\-ordinalaccuracy drops by 32\.4 points under SoM\. The model had already learned to count list items by their on\-screen order; converting the same task to “select mark X” adds an indirection that the model handles less well than the original\.

Mark placement collisions\.On highly crowded screenshots, two marks placed at nearby element centers can occlude each other, or the model can confuse them\. We did not observe this on the small candidate sets used in our benchmark \(typically 2 marks per item\), but it is a known issue at scale\.

Closed\-API consistency\.On the human\-verified core, Claude Opus 4\.7 gained only\+9\.2\+9\.2points under SoM, in contrast to GPT\-5’s\+57\.1\+57\.1points\. This 47\-point gap on the same items under the same intervention suggests that Opus already grounds the unmarked image with the same logic SoM provides, while GPT\-5 was genuinely held back by an aspect of the task that mark selection removes\.

## Appendix JDetails of the GUI\-Primitives Benchmark

### J\.1Details of minimal\-pair construction

Every item inGUI\-Primitivesappears as one half of a contrastive pair\. A pair shares one screenshot and one anchor element; the relation expression changes between the two members so that the correct target moves to the other designated candidate\. In 253 of 497 pairs, the surrounding instruction template also varies lexically in verb, head noun, and anchor quoting; this variation is not confounded with the relation term \(Appendix[J\.2](https://arxiv.org/html/2608.21832#A10.SS2)\)\. Figure[2](https://arxiv.org/html/2608.21832#S3.F2)shows a real example: in \(A\) the click should land to the right of thebreak debuggericon; in \(B\) the same screenshot is paired withleft of, and the correct target is the icon on the other side\. The construction controls for screenshot\-specific salience and fixed answer preferences: a model that always selects the most central, salient, or frequently labeled element scores correctly on one twin and incorrectly on the other, so its pair\-level score is zero\. We measure this directly with*pair\-consistency*: the fraction of pairs on which both members are correct \(§[5\.5](https://arxiv.org/html/2608.21832#S5.SS5)\)\.

### J\.2Details of sources: real and synthetic screenshots

GUI\-Primitivesdraws screenshots from two sources, both reported separately\.

Real desktop screenshots \(n = 290 items\)\.We use the element\-grounding annotations from UI\-Vision\([16](https://arxiv.org/html/2608.21832#bib.bib19)\)as the source corpus for real GUIs, covering Web, MacOS, VS Code, Office, and various professional applications\. Items synthesized from this source cover the four primitives whose annotations are unambiguously recoverable from element labels: horizontal relative position, vertical relative position, alignment, and proximity\.

Controlled synthetic stimuli \(n = 704 items\)\.The remaining items are rendered from a procedural generator that places elements in known positions, with controlled distractors and a fixed layout language\. This is the only way to obtain*containment*,*list ordinal*, and*occlusion*items in clean form, because those three primitives require a structured parent–child or overlay–target relationships that real screenshot annotations do not ship at scale\.

We disclose the split openly\. The synthetic arm is a controlled\-stimulus arm in theWhat’s\-Uptradition, where the hardest relations are likewise generated under controlled rendering\. All real vs\. synthetic numbers are reported separately throughout the paper\.

### J\.3Details of human verification

Five annotators independently judged a 196\-item stratified subset \(human\-verified core\)\. For each item, the annotator answered two questions: is the instruction well\-formed and unambiguous on the given screenshot, and which of the two candidate boxes is the correct target\. Inter\-annotator agreement is high\. On Q1 \(well\-formedness\), Fleissκ=0\.942\\kappa=0\.942\(“almost perfect” on the Landis–Koch scale,[9](https://arxiv.org/html/2608.21832#bib.bib16)\)\. On Q2 \(correct target\), Fleissκ=0\.787\\kappa=0\.787\(“substantial”\)\. Eleven items were judged invalid by the majority and were dropped, leaving 185 items in thehuman\-clean core\.

Figure 12:Benchmark distribution analysis\.\(A\)Contrastive\-relation balance: every relation is represented by exactly 71 items per side, ruling out relation\-prior exploitation\.\(B\)Target–distractor spatial separation \(normalized by image diagonal\) per primitive; violin width encodes density\.\(C\)Target size distribution \(log\-scale\) by source arm; dashed lines are per\-source medians\.\(D\)Target centre positions \(normalized to\[0,1\]2\[0,1\]^\{2\},y=1y\{=\}1at top\), coloured by primitive; targets span the full screen area\.

## Appendix KBenchmark Distribution and Scenario Diversity

Figure[12](https://arxiv.org/html/2608.21832#A10.F12)summarises the distributional properties ofGUI\-Primitivesacross four dimensions\.

#### Relation balance \(Panel A\)\.

Every contrastive relation is represented by exactly 71 items per side per primitive, a strict 50/50 split enforced by the minimal\-pair miner\. This guarantees that a model exploiting a relation\-frequency prior \(e\.g\., always predictingleft\) cannot achieve better\-than\-reference\-level accuracy at the primitive level, and it rules out skewed\-prior explanations for any observed performance gap\. Table[6](https://arxiv.org/html/2608.21832#A11.T6)reports the per\-primitive composition by source arm: 142 items per primitive; containment and occlusion are synthetic\-only, and list\-ordinal includes a 24\-item real slice\.

Table 6:Per\-primitive composition ofGUI\-Primitivesby source arm\. Every primitive has 142 items, exactly 71 per relation word\.
#### Target–distractor spatial separation \(Panel B\)\.

Target–distractor distances \(normalized by the image diagonal\) span a wide range: median0\.180\.18, IQR\[0\.04,0\.31\]\[0\.04,0\.31\]\.list\-ordinalitems are tightly spaced \(adjacent list rows, median0\.040\.04\), whilecontainmentandocclusionitems have the largest separations \(median0\.320\.32–0\.310\.31\)\. This variation ensures that the benchmark exercises a range of spatial configurations rather than collapsing onto a single difficulty level\.

#### Target size \(Panel C\)\.

Real UI\-Vision targets are considerably smaller than synthetic ones \(median0\.05%0\.05\\%vs\.1\.3%1\.3\\%of image area\), consistent with the fine\-grained element boxes in professional desktop applications\. Both distributions span multiple orders of magnitude \(log10range≈\\approx3\.5\), providing diversity in target granularity within each arm\.

#### Spatial coverage \(Panel D\)\.

Target centers, normalized to the unit square, span the full screen area with no strong clustering, reflecting a mix of toolbar, sidebar, central panel, and dialog overlay targets across both sources\. The slight top\-left bias \(x¯=0\.29\\bar\{x\}=0\.29,y¯=0\.24\\bar\{y\}=0\.24\) reflects the prevalence of toolbar and sidebar elements in real GUI layouts rather than a construction artifact\.

## Appendix LDetails of Models and Protocol

#### Details of models\.

We evaluate 19 vision\-language models: seven proprietary models \(Claude Opus 4\.7, Claude Sonnet 4\.6, Claude Haiku 4\.5, GPT\-5, GPT\-4\.1, GPT\-4o\-mini, Gemini 3\.1 Flash Lite\); three open Qwen\-family models \(Qwen2\.5\-VL\-7B, Qwen2\-VL\-7B, OS\-Atlas\-Base\-7B, which is GUI\-tuned Qwen2\-VL\); InternVL3\-8B; Meta Llama\-3\.2\-11B\-Vision; three sizes of Google Gemma 3 \(4B, 12B, 27B, including an Ollama\-Q4 copy\); two PaliGemma 2 variants \(3B, 10B\); and MiniCPM\-V\. The full list and provenance are in Appendix[B](https://arxiv.org/html/2608.21832#A2)\. We could not include Pixtral\-12B and Idefics3\-8B because their chat\-template formats are incompatible with our generic wrapper; this is disclosed as a known scope limitation rather than silently omitted\.

#### Details of prompting and decoding\.

All models use greedy decoding \(T=0T=0\) and a fixed seed\. The grounding prompt is a single short instruction \(“respond with only the pixel coordinate to click”\)\. Per\-family coordinate conventions \(pixel, normalized\[0,1\]\[0,1\], normalized\[0,1000\]\[0,1000\]\) are handled by a robust parser; the parser is in the released code, and its unit tests are bundled with the artifact\.

#### Details of closed\-API coordinate frames\.

A practical wrinkle complicated the closed\-model runs\. The three closed providers each downsample large screenshots server\-side \(Anthropic to≤1568\\leq 1568px, OpenAI to roughly10241024on the long side, Gemini variable\) and return coordinates in the downsampled frame\. We pre\-resize every image to a known long side and rescale the predicted coordinate back to the original pixel space\. The rescale was needed on a large fraction of items \(e\.g\., 1,297 of 1,581 Sonnet ScreenSpot\-Pro records\)\. Full details in Appendix[E](https://arxiv.org/html/2608.21832#A5); we believe this is a methodological note worth sharing with the community\.

#### Details of metrics\.

The primary metric, following ScreenSpot\-Pro, is*point\-in\-box*accuracy: a predictionp^=\(x^,y^\)\\hat\{p\}=\(\\hat\{x\},\\hat\{y\}\)is correct iffx^∈\[x1,x2\]\\hat\{x\}\\in\[x\_\{1\},x\_\{2\}\]andy^∈\[y1,y2\]\\hat\{y\}\\in\[y\_\{1\},y\_\{2\}\]for ground\-truth boxB=\[x1,y1,x2,y2\]B=\[x\_\{1\},y\_\{1\},x\_\{2\},y\_\{2\}\]in the original pixel frame\.*Loose*accuracy \(prediction within2×2\\timestarget diagonal of the box center\) separates sub\-pixel imprecision from element mis\-identification on small icons\. All accuracies carry bootstrap 95% CIs \(2,000 resamples\)\. Paired comparisons use McNemar\([15](https://arxiv.org/html/2608.21832#bib.bib17)\)tests with Holm–Bonferroni\([6](https://arxiv.org/html/2608.21832#bib.bib18)\)correction across the seven primitives\. All primitives are binary contrasts \(reference level0\.500\.50; Table[1](https://arxiv.org/html/2608.21832#S3.T1)\)\. We flag a cell as below the reference level only when the upper bootstrap bound falls below0\.500\.50; thus, a point estimate dipping below it due to sampling noise does not count\. Because models emit unconstrained coordinates,0\.500\.50is a forced\-choice reference rather than the chance level of the prediction task \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\)\. The below\-reference flags persist under anchor\-level and screenshot\-level cluster resampling; the synthetic generator reuses anchor slots \(containment: 4 distinct anchors; occlusion: 13; alignment: 34; proximity: 43, each over 142 items and 71 screenshots\), and no clustered upper bound reaches0\.500\.50for any of the 19 models on the four sub\-reference primitives\.

## Appendix MDetails of CoT and Activation Steering: Negative Results

We report two negative results, with the same effort as the positive one\.

Primitive\-aware CoT on Qwen2\.5\-VL\-7B producedΔ=\+0\.5\\Delta=\+0\.5points \(Holm\-adjustedp=0\.51p=0\.51, not significant\)\. The decomposition prompt does not seem to address the limiting factor: the model correctly names the relation in its intermediate reasoning, yet still emits the wrong coordinate\.

Activation steering with SteerVLM\-style vectors on Qwen2\.5\-VL\-7B producedΔ=\+1\.5\\Delta=\+1\.5points \(Holm\-adjustedp=0\.066p=0\.066, not significant\)\. The grid search over layers andα\\alphadid not yield a configuration that crossed the Holm threshold\. Either GUI grounding requires a different intervention point than the layers where SteerVLM has helped on linguistic tasks, or the residual\-stream signal is genuinely weak, and a single contrastive vector is not enough to push the model across the decision boundary on minimal pairs\.

#### Reading the intervention table\.

The picture is uneven on purpose\. The one intervention that works, SoM, works by*bypassing*the spatial skill it should test, replacing it with a symbolic naming task\. The two interventions that probe the model’s internal representation directly do not move the metric\. We read this as evidence that the failure is closer to perception \(visual encoder, cross\-modal binding\) than to language\-side reasoning or residual\-stream representation\.

## Appendix NDetails of Diagnostic Results

### N\.1Details of shortcut controls validate the benchmark

A diagnostic is only as good as the shortcuts it forbids\. Figure[3](https://arxiv.org/html/2608.21832#S5.F3)reports three controls on Qwen2\.5\-VL\-7B over the full 994\-item benchmark\.

In the*text\-only*condition, the instruction is paired with a blank canvas of the same size: accuracy falls from 22\.0% to 6\.5% \(Δ=−15\.5\\Delta=\-15\.5pt, McNemarp=2\.8×10−27p=2\.8\\times 10^\{\-27\}\)\. In the*shuffled*condition, the screenshot is replaced with one from a different item of the same primitive: accuracy drops to 14\.8% \(Δ=−7\.2\\Delta=\-7\.2pt,p=1\.3×10−11p=1\.3\\times 10^\{\-11\}\)\. Both controls behave as a truly grounded model should: when the visual or the matching instruction is removed, the model can no longer solve the task\.

The third control,*heavy Gaussian blur*\(σ=12\\sigma=12px\), drops accuracy to 16\.7% \(Δ=−5\.3\\Delta=\-5\.3pt,p=8\.5×10−5p=8\.5\\times 10^\{\-5\}\)\. The drop is statistically significant but below our pre\-registered ten\-point threshold, so under our pre\-registered criterion the blur control does not reach a strict pass\. The result indicates that VLM grounding relies more on global layout than on fine\-grained OCR of text labels, since removing high\-frequency detail removes only a modest fraction of the signal\.

### N\.2Details of overall accuracy and the human gap

Table[2](https://arxiv.org/html/2608.21832#S5.T2)reports the headline numbers\. Figure[4](https://arxiv.org/html/2608.21832#S5.F4)visualizes the gap from human performance\. Three observations follow\.

First, all models perform well below the annotator target\-selection accuracy of 96\.9% on a two\-candidate task, which is not matched to model point\-in\-box accuracy \(§[3\.4](https://arxiv.org/html/2608.21832#S3.SS4)\)\. The strongest system, Claude Opus 4\.7, reaches 31–32%\. The 65\-point gap holds on the full benchmark and on the cleaned 185\-item core alike\.

Second, accuracy does not increase monotonically with nominal scale\. The four strongest proprietary models \(Claude Opus 4\.7, GPT\-5, Claude Sonnet 4\.6, Claude Haiku 4\.5\) occupy the top of the table but cluster within a narrow 24–31% range\. Meanwhile, the 7B open Qwen2\.5\-VL\-Instruct is statistically tied with the closed Claude Haiku 4\.5 on the human\-verified core \(24\.5% vs\. 25\.0%\), a noteworthy parity finding given that Qwen2\.5\-VL is roughly two orders of magnitude smaller than the closed reference and runs on a single GPU\.

Third, the rank order is stable\. The same top group leads on the full 994\-item benchmark, on the 196\-item human\-verified core, and on the LLM\-judge\-verified 100\-item subset \(Appendix[A](https://arxiv.org/html/2608.21832#A1)\)\. This means the human gap is not an artifact of noisy items\.

### N\.3Details of per\-primitive structure of failure

Figure[5](https://arxiv.org/html/2608.21832#S5.F5)unpacks the overall accuracy by primitive\. The picture is uneven and informative\.

One primitive,list\-ordinal, is well solved by the top five models \(Claude family, GPT\-5, and Qwen2\.5\-VL\), all achieving0\.800\.80–0\.830\.83against the0\.500\.50two\-candidate reference \(Table[1](https://arxiv.org/html/2608.21832#S3.T1)\)\. The shared task here is counting list positions, which the language model can do once it identifies the list\.

The other six primitives are uniformly hard\. Onrel\-pos\-horizontalClaude Opus 4\.7 reaches0\.510\.51, at the reference level; every other model is below\. Onrel\-pos\-vertical, the best model is GPT\-5 at0\.320\.32\. Oncontainment,occlusion,alignment, andproximity,*the upper 95% bootstrap bound for every model we test sits below the reference level*of0\.500\.50, including all four strongest proprietary models\. Candidate\-level analysis \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\) indicates that this below\-reference accuracy is primarily attributable to predictions outside both candidate regions \(60–92% of predictions\) rather than to selection of the contrastive element: within\-region selection favors the target at least as often as the distractor on every primitive, with containment and occlusion statistically indistinguishable from0\.500\.50\(0\.548 / 0\.554\) and horizontal, vertical, and proximity at 0\.89–0\.90\. The associated directional prior is discussed in §[8](https://arxiv.org/html/2608.21832#S8)\.

### N\.4Details of real versus synthetic

We separate two candidate explanations for the per\-primitive results in §[5\.3](https://arxiv.org/html/2608.21832#S5.SS3): small real\-GUI target size and the relative proportion of synthetic items\. Table[5](https://arxiv.org/html/2608.21832#A3.T5)reports strict and loose accuracy split by source\.

On synthetic screenshots, models often predict near the correct element: loose accuracy is0\.600\.60–0\.760\.76even when strict box\-edge accuracy is substantially lower, indicating that most synthetic errors are sub\-pixel imprecision rather than element misidentification\. On UI\-Vision real screenshots, strict accuracy is essentially zero for every model on every primitive, and*loose*accuracy also stays low \(6–23%\)\. Candidate\-level analysis \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\) shows that on the real arm, 96% of predictions fall outside both candidate regions, while among predictions inside a candidate region, the target is selected on 0\.75 pooled\. Real\-screenshot accuracy is therefore primarily attributable to candidate localization, with target size a contributing factor, rather than to inverted relational interpretation\.

### N\.5Details of pair\-consistency: a relational failure signature

If a model produced independent draws on the two members of a pair, pair\-consistency would equaly=x2y=x^\{2\}\(accuracy squared\)\. Figure[6](https://arxiv.org/html/2608.21832#S5.F6)shows that pair\-consistency is consistently*lower*than thaty=x2y=x^\{2\}curve\. This pattern, called the relational\-failure signature in[7](https://arxiv.org/html/2608.21832#bib.bib4), means that within a model, the two pair members are negatively correlated: when the model is right on the “left of” twin, it is more likely than chance to be wrong on the “right of” twin\. The information that confuses the model is the relation word, not the picture\.

## Appendix ODetails of Primitive Competence Predicts Grounding

A diagnostic is only useful if it predicts a downstream property of interest\. We test this on ScreenSpot\-Pro\([10](https://arxiv.org/html/2608.21832#bib.bib3)\), the standard high\-resolution GUI\-grounding benchmark, and we run it on the ten models for which we have bothGUI\-Primitivesand SS\-Pro predictions\.

Figure[7](https://arxiv.org/html/2608.21832#S5.F7)shows the per\-model relationship\. Spearmanρ=\+0\.736\\rho=\+0\.736\(p=0\.015p=0\.015,n=10n=10\)\. The correlation holds within the real\-screenshot subset \(ρ=\+0\.705\\rho=\+0\.705,p=0\.023p=0\.023\) and within the synthetic subset \(ρ=\+0\.760\\rho=\+0\.760,p=0\.011p=0\.011\), so it is not an artifact of any single item type\. A leave\-one\-model\-out sensitivity analysis gives a Spearman in\[\+0\.62,\+0\.86\]\[\+0\.62,\+0\.86\], so no single model drives the result; dropping the strongest or weakest does raise thepp\-value to≈0\.07\\approx 0\.07\. We also report a wide non\-parametric bootstrap CI \(\[\+0\.11,\+0\.98\]\[\+0\.11,\+0\.98\],n=10n=10\) for transparency rather than as a precision claim\. This is the load\-bearing result of this section\.

#### Item\-level corroboration\.

We also fit an item\-level logistic regression of ScreenSpot\-Pro success on each item, using the model’s per\-primitive accuracy profile, model fixed effects, a log target\-area control, and item\-clustered standard errors\. The model achieves pseudo\-R2=0\.404R^\{2\}=0\.404onn=15,810n=15\{,\}810model\-item pairs\. The forest plot of per\-primitive coefficients is in Figure[11](https://arxiv.org/html/2608.21832#A4.F11)\.

We urge caution on the per\-primitive coefficients\. After the model fixed effects absorb between\-model variance, the residual between\-primitive variance is small \(all\|coef\|≤0\.07\|\\text\{coef\}\|\\leq 0\.07\), and only two coefficients are individually significant at the Holm\-corrected level\. With four distinct open models contributing to ScreenSpot\-Pro data, the competence vectors are correlated across primitives, making per\-primitive identification fragile\. We therefore claim only the joint, model\-level association, which is strong, and explicitly do not claim that competence at any particular primitive causes downstream success\.

Figure 13:Model sensitivity analysis\.\(A\)Left–right accuracy gap per model, sorted by magnitude\. Red bars = left\-biased \(15/19 models;t=5\.6t=5\.6,p<\.001p<\.001\)\.\(B\)Above–below accuracy gap\. Weaker and not significant overall \(p=\.10p=\.10\); OS\-Atlas is an outlier with a strong above preference\.\(C\)Mean accuracy stratified by target normalized area \(tiny/medium/large\) per primitive, pooled over all 19 models\.\(D\)Parse\-fail \(null output\) and default\-coordinate rates per model; both are infrastructure errors, not spatial\-reasoning failures\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_rel_pos_horizontal_ui_vision.png)Figure 14:rel\-pos\-horizontal\(real screenshots\)\.Three minimal pairs from UI\-Vision\. Within each row, both panels show the same screenshot; only the relation word \(*right of*vs\.*left of*\), and the gold target moves accordingly\. Across rows, the application and anchor change\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_rel_pos_vertical_ui_vision.png)Figure 15:rel\-pos\-vertical\(real screenshots\)\.Three minimal pairs;*above*vs\.*below*the anchor element across three different real GUIs\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_alignment_ui_vision.png)Figure 16:alignment\(real screenshots\)\.Three minimal pairs in which the relation switches between*same row as*and*same column as*the anchor\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_proximity_ui_vision.png)Figure 17:proximity\(real screenshots\)\.Three minimal pairs in which the two correct answers are the elements*nearest to*and*farthest from*the same anchor across three different real applications\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_containment_synthetic.png)Figure 18:containment\(synthetic\)\.Three minimal pairs\. Twin A is the*contained within*a parent; twin B is the parent*containing*that inner element\. Synthetic stimuli are used because clean parent–child annotations are rare in real screenshot corpora\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_list_ordinal_synthetic.png)Figure 19:list\-ordinal\(synthetic\)\.Three minimal pairs of symmetric ordinal queries \(thekk\-th item from the top vs\. from the bottom\)\. The only primitive every top\-tier model reliably solves\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_teaser_occlusion_synthetic.png)Figure 20:occlusion\(synthetic\)\.Three minimal pairs\. Twin A is the element that is*fully visible*, twin B is the element that is*partly hidden*behind an overlay\. A relational\-shortcut model is drawn to the visually unusual \(occluded\) element on both twins, scoring zero at the pair level\.

## Appendix PAdditional Minimal\-Pair Examples

Figure[2](https://arxiv.org/html/2608.21832#S3.F2)shows a singlerel\-pos\-horizontalpair\. To make the minimal\-pair construction intuitive for the other six primitives, Figures[14](https://arxiv.org/html/2608.21832#A15.F14)–[20](https://arxiv.org/html/2608.21832#A15.F20)each present*three*representative pairs for one primitive\. Every row is a distinct pair; within a row, both panels share the same screenshot, only the relation word in the instruction differs \(twin A on the left, twin B on the right\), and the colored box marks only that panel’s gold target\. A relationally\-grounded model must point at the two different boxes for the two instructions; a shortcut\-grounded model points at the same box for both\.

![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_01.png)Figure 21:Case study 1:rel\-pos\-vertical, real screenshot\.A: “Click the element above the replace\.” B: “Click the element below the replace\.” Opus and GPT\-5 collapse to the same coordinate on both twins \(both near\(≈1180,130\)\(\\approx 1180,130\)\), getting B right by luck of which side they defaulted to and missing A\. Qwen2\.5\-VL emits two completely unrelated clicks\. Hollow markers = misses; filled markers = hits\. Anchor\-collapse failure\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_02.png)Figure 22:Case study 2:proximity, real screenshot\.A: “Select the button*nearest*to ‘current file directory’\.” B: “Click the icon*farthest*from the current file directory\.” All four models cluster their predictions in the top toolbar regardless of the instruction\. The anchor \(a sidebar element\) and the two targets \(sidebar items\) are far from the predictions\. This is the population\-level pattern onproximity: 92% of predictions fall outside both candidate regions \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\), while the target is selected on 89% of within\-region predictions, locating the deficit in candidate localization rather than relational interpretation\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_03.png)Figure 23:Case study 3:alignment, synthetic\.A: “Select the control aligned in the same*row*as ‘Paste’\.” B: “Select the control aligned in the same*column*as ‘Paste’\.” Models click*the Paste anchor itself*or a visually similar icon, instead of the aligned neighbor\. Qwen2\.5\-VL emits an*identical coordinate*for both twins, the canonical “relation\-ignored” signature\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_04.png)Figure 24:Case study 4:alignment, real screenshot\.A/B: “aligned in the same row/column as ‘Navigate Backward’\.” The real\-screenshot version of case study 3\. GPT\-5 wanders far from the anchor on both twins\. Qwen2\.5\-VL again emits the same coordinate for both\.alignmenton real GUIs is the hardest cell in the benchmark\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_05.png)Figure 25:Case study 5:occlusion, synthetic\.A: the ‘Reset’ button that is*fully visible*\. B: the ‘Reset’ button that is*partly hidden*behind a dialog\. All four models point to the partly\-hidden button on*both*twins \(paired predictions differ by<<10 px\)\. The models are drawn to the visually unusual element and never report the visible one\. On this pair, the models select the contrastive element on both twins; pooled over all models and items, within\-region selection onocclusionis 0\.554, statistically indistinguishable from0\.500\.50\(Appendix[R](https://arxiv.org/html/2608.21832#A18)\)\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_06.png)Figure 26:Case study 6:containment\.A: target is the element*contained within*the parent\. B: target is the parent*containing*the inner element\. Models repeatedly click the inner element on both twins, missing the parent on twin B\.containmentcarries the weakest relation\-word signal in the benchmark: within\-region selection splits 0\.548/0\.452 between target and distractor, and 92% of predictions fall outside both candidate regions \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\)\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_07.png)Figure 27:Case study 7:rel\-pos\-vertical, synthetic\.The controlled\-stimulus version of case study 1\. Even on the clean, synthetic layout, the same anchor\-collapse signature is visible: predictions cluster near the anchor instead of resolving above/below\.![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_case_08.png)Figure 28:Case study 8:rel\-pos\-horizontal, real screenshot\.A: “element to the right of*X*”\. B: “element to the left of*X*”\. The minimal\-pair construction that opens the paper \(Fig\.[2](https://arxiv.org/html/2608.21832#S3.F2)\) repeats here on a different application\. Same shortcut, same failure mode\.
## Appendix QModel\-Specific Sensitivity Analysis

Figure[13](https://arxiv.org/html/2608.21832#A15.F13)reports four model\-level sensitivity diagnostics computed directly from the existing diagnostic runs\.

#### Left–right asymmetry \(Panel A\)\.

Across 19 models, 15 score higher onleft ofthanright ofitems, with a mean gap of\+0\.169\+0\.169accuracy points \(one\-samplet=5\.6t=5\.6,p<\.001p<\.001, sign testp<\.001p<\.001\)\. The bias is present in every proprietary model \(Opus, Sonnet, Haiku, GPT\-5, Gemini\) and in the strongest open models\. This is consistent with a left\-side layout frequency prior: in Western GUIs, action buttons, primary controls, and frequently clicked elements are disproportionately placed on the left, so models that ground by position salience rather than by relation word will inflate left\-side accuracy\. The candidate\-level analysis \(Appendix[R](https://arxiv.org/html/2608.21832#A18)\) indicates that this asymmetry coexists with relation\-word use rather than replacing it: within\-region selection is 0\.972 under “left of” and 0\.717 under “right of”, and predictions fall within a candidate region 2\.7×\\timesmore often when the target is the left element\. The accuracy asymmetry reported here and the within\-region asymmetry in Appendix[R](https://arxiv.org/html/2608.21832#A18)are two measurements of the same directional effect\. The two exceptions, MiniCPM\-V and PaliGemma2\-3B, are low\-accuracy models that default to near\-origin coordinates on many items \(Panel D\), effectively averaging away the bias\.

#### Above–below asymmetry \(Panel B\)\.

The below\-over\-above preference is weaker and not statistically significant \(meanΔ=\+0\.047\\Delta=\+0\.047,t=1\.7t=1\.7,p=\.10p=\.10, 12/19 models below\-biased\)\. OS\-Atlas is a notable outlier with a strong*above*preference \(Δ=−0\.28\\Delta=\-0\.28\), suggesting a GUI\-specific training prior that differs from the natural\-image convention\.

#### Target\-size sensitivity \(Panel C\)\.

Pooled across all models and primitives, large targets \(\>\>75th percentile of normalized area\) reach a mean accuracy of 0\.092 vs\. 0\.022 for tiny targets \(<<25th percentile\)\. The effect is largest forcontainmentandlist\-ordinal, primitives where the target is a panel or a list row that occupies substantial screen area, and smallest foralignmentandproximity, where the spatial relationship matters more than target visibility\. Caution: the size gap is partially confounded with source \(synthetic targets are larger\), so this should not be interpreted as a pure resolution effect\.

#### Coordinate\-frame and parse\-failure rates \(Panel D\)\.

Three distinct failure modes are visible: Llama\-3\.2\-11B\-Vision emits a near\-origin default on 46% of items \(consistent with a “don’t know” fallback, not a parser bug\); PaliGemma2\-10B and Claude Haiku produce null outputs \(empty or unparseable text\) on 8–11% of items; and InternVL3 and Claude Sonnet show low but non\-zero null rates\. These are infrastructure failures, not spatial\-reasoning failures, and are already annotated as Type C and F errors in Appendix[U](https://arxiv.org/html/2608.21832#A21)\.

## Appendix RCandidate\-Level Classification of Baseline Predictions

The0\.500\.50reference level is a two\-candidate forced\-choice reference rather than the chance level of the prediction task: models emit an unconstrained coordinate, so a prediction can fall outside both candidate regions\. To separate selection of the contrastive element from predictions outside both regions, we classify every baseline prediction from all 19 models \(18,886 records\) geometrically:*target*if the point lies inside the target box,*distractor*if inside the distractor box,*neither*if inside neither, and*invalid*if no coordinate could be parsed\. A point inside both boxes is assigned to the nearer box center; this case does not occur \(0 of 18,886\)\. Table[7](https://arxiv.org/html/2608.21832#A18.T7)reports the four\-way split pooled over models, together with the within\-region selection rate \(target as a fraction of predictions inside either region\), its denominator, and a two\-sided exact binomial test against0\.500\.50\.

Table 7:Candidate\-level classification of all baseline predictions, pooled over 19 models \(strict rule: a prediction is assigned to a candidate only if it lies inside that candidate’s box\)\. Sel\. = within\-region selection rate, target / \(target \+ distractor\), reported with its denominatornn\.ppis a two\-sided exact binomial test against 0\.50\. Selection rates computed on a smallnn\(e\.g\., alignment,n=173n=173\) are correspondingly less precise\.#### Selection between candidates\.

The target is selected at least as often as the distractor on every primitive\. The*neither*column dominates throughout: 60–92% of predictions fall outside both candidate regions\. Forcontainmentandocclusion, within\-region selection is statistically indistinguishable from0\.500\.50, indicating no measurable relation\-word signal on these two primitives; forrel\-pos\-horizontal,rel\-pos\-vertical, andproximity, within\-region selection favors the target \(0\.890\.89–0\.900\.90\)\. At the individual\-model level, the only statistically significant excess of distractor selection is Gemma3\-4B \(0\.3280\.328,n=67n=67\); no model shows such an excess under the permissive rule below\.

#### Sensitivity to the inclusion rule\.

The strict rule yields small denominators, so we repeat the classification under a permissive rule: a prediction is assigned to a candidate if it lies within one box\-diagonal of that candidate’s box center \(normalized per box\), and to*neither*otherwise\. The two rules agree for six of seven primitives: containment 0\.539 \(n=1680n=1680\), occlusion 0\.557 \(n=1256n=1256\), and the four target\-favoring primitives at 0\.69–0\.77 \(n=471n=471–20752075\)\. Thealignmentestimate depends on the rule: 0\.728 under the strict rule \(n=173n=173, 6% of its predictions\) versus 0\.529 under the permissive rule \(n=1021n=1021\)\. We report both values and do not interpret the within\-region selection rate for this primitive\.

#### Real\-screenshot arm\.

On UI\-Vision screenshots, the*neither*column is larger still: pooled over 19 models, 96\.2% of real\-arm predictions fall outside both candidate regions \(87\.8% for GPT\-5 and Claude Opus 4\.7 alone\), while within\-region selection is 0\.747 pooled \(n=150n=150\) and 0\.914 for GPT\-5 and Opus \(n=70n=70\)\. Real\-screenshot accuracy is thus primarily attributable to candidate localization rather than to inverted relational interpretation\.

#### Comparison with the human task\.

The human annotation task is a two\-candidate forced choice, so the within\-region selection rate is the model\-side quantity matched to the human accuracy of 96\.9%\. On the human\-clean core, Claude Opus 4\.7 produces a within\-region prediction on 62 of 185 items and selects the target on 60 of those 62 \(0\.968\);60/185=0\.32460/185=0\.324equals the point\-in\-box accuracy in Table[2](https://arxiv.org/html/2608.21832#S5.T2), since “prediction inside the target box” is the same event under both definitions\. For GPT\-5,54/57=0\.94754/57=0\.947within\-region and54/185=0\.29254/185=0\.292overall\. Within\-region discrimination for the strongest models \(0\.95–0\.97\) is comparable to human accuracy \(0\.969\); the difference in overall accuracy is primarily attributable to the low proportion of predictions \(≈\\approxone third\) that fall within either candidate region\.

#### Directional asymmetry\.

Conditioning within\-region predictions on the relation word givesP⁡\(target∣“left of”\)=0\.972P\(\\text\{target\}\\mid\\text\{\`\`left of''\}\)=0\.972\(n=323n=323\) versus0\.7170\.717\(n=120n=120\) for “right of”; for the vertical primitive, “below”=0\.984=0\.984\(n=253n=253\) versus “above”=0\.816=0\.816\(n=228n=228\)\. Both conditional rates exceed0\.500\.50, indicating that the relational word is used on these primitives\. The left\-side layout prior to Appendix[Q](https://arxiv.org/html/2608.21832#A17)manifests as a difference in the rate at which predictions fall within a candidate region \(2\.7×\\timeshigher when the target is the left element\) together with a lower within\-region accuracy under “right of”\.

#### Cluster\-robustness of the below\-reference flags\.

The synthetic generator reuses anchor slots \(containment: 4 distinct anchors; occlusion: 13; alignment: 34; proximity: 43; 71 screenshots per primitive\), so we recompute all per\-primitive bootstrap intervals with cluster resampling at the anchor level and at the screenshot level\. No upper bound reaches0\.500\.50for any of the 19 models on any of the four sub\-reference primitives under either clustering; the widest interval is Claude Opus 4\.7 on proximity, anchor\-clustered,\[0\.035,0\.398\]\[0\.035,0\.398\]\. The below\-reference intervals therefore persist under clustering, while the candidate\-level classification above indicates that their primary source is off\-candidate prediction\.

## Appendix SOCR, Layout, and Cross\-Modal Binding: Disentangling Failure Sources

The blur\-control experiment \(§[5\.1](https://arxiv.org/html/2608.21832#S5.SS1)\) shows that removing high\-frequency detail, which degrades fine\-grained OCR of text labels, reduces accuracy by only 5\.3 points, substantially less than removing the screenshot entirely \(−\-15\.5 points\) or shuffling it \(−\-7\.2 points\)\. This suggests that models rely more on global layout structure than on reading individual element labels, but it does not isolate*which*component of grounding fails\. Three failure sources are plausible: \(1\)text/OCR dependence: the model cannot read the anchor label and therefore cannot localise the relational neighbour; \(2\)spatial\-encoding failure: the model reads the anchor correctly but misrepresents its position or the spatial relation in the instruction; \(3\)cross\-modal binding failure: the model reads both the anchor and the relation word correctly but fails to compose them into the correct click target\.

#### What the blur result tells us\.

The modest accuracy drop under blur is evidence against strong OCR dependence for the binary outcome: even when text is illegible, models still achieve 16\.7% \(well above the text\-only baseline of 6\.5%\), implying that global spatial structure alone carries residual information\. However, blur does not selectively impair text: it also degrades icon shapes and element boundaries, so we cannot distinguish OCR loss from icon\-recognition loss under this control alone\.

#### What targeted ablations would reveal\.

Three ablations would cleanly disentangle the three failure sources, and we lay them out as concrete future directions:

1. 1\.Text masking / selective blur\.Replace all text regions \(detected via OCR or the accessibility tree\) with uniform grey blocks while preserving spatial layout\. A large drop relative to full\-image accuracy would indicate OCR dependence; a small drop would point to spatial encoding or binding failures\.
2. 2\.Layout preservation with degraded text\.Render a version of each screenshot where element shapes and positions are preserved, but all text is replaced with random strings of the same length\. If accuracy stays near baseline, the model is primarily using spatial structure; if it drops, text semantics matter\.
3. 3\.Icon glyph perturbation\.Swap icon images with visually similar but semantically different glyphs while keeping positions and text unchanged\. This isolates whether failures arise from icon\-identity recognition or from relational reasoning once the elements are identified\.

We do not run these ablations here because they require either a high\-fidelity text\-detection pipeline \(for selective masking\) or controlled re\-rendering of real GUI screenshots \(for glyph swapping\), neither of which is available in the current environment\. We flag them as the highest\-priority next steps for understanding whether the bottleneck is perceptual \(OCR/icon recognition\) or compositional \(cross\-modal binding of the anchor identity to the spatial\-relation word\)\.

## Appendix TQualitative Case Studies

Figures[21](https://arxiv.org/html/2608.21832#A16.F21)–[28](https://arxiv.org/html/2608.21832#A16.F28)present eight concrete minimal\-pair failures drawn from the top\-4 baseline models\. In every figure, the two panels are the two twins of one pair \(instruction A on the left, instruction B on the right\) sharing the same screenshot\. Annotations are programmatic, not hand\-drawn:

- •green box: gold target \(fromitems\.jsonl\)\.
- •dashed terra box: the distractor \(the wrong twin of the minimal pair\)\.
- •coloured markers: each model’s actual click coordinate fromruns/diagnostic/<model\>/diagnostic\.jsonl\.Filledmarkers are predictions that fall inside the gold box \(a hit\);hollowmarkers are misses\.

The case studies span every primitive in both data sources; the chosen pairs are those where at least 5 of 8 model–twin evaluations failed, i\.e\. pairs that the strongest models genuinely struggle with\. Recurring failure modes are summarized at the end\.

#### Cross\-case summary\.

Three patterns recur across all eight case studies and motivate the error analysis in Appendix[U](https://arxiv.org/html/2608.21832#A21):\(i\) Anchor\-collapse: predictions cluster on the anchor noun rather than on its spatial neighbor \(cases 1, 3, 4, 7\)\.\(ii\) Twin\-identical predictions: the two paired clicks land within a few pixels of each other, meaning the relation word was ignored \(cases 3, 4, 5\)\.\(iii\) Off\-candidate prediction: the dominant population\-level failure is that predictions fall outside both members of the pair \(60–92% of predictions; Appendix[R](https://arxiv.org/html/2608.21832#A18)\)\. Case 5 illustrates selection of the contrastive twin at the item level; pooled across all models and items, within\-region selection is near0\.500\.50oncontainmentandocclusionand favors the target on the remaining primitives\.

![Refer to caption](https://arxiv.org/html/2608.21832v1/fig_supp_error_cheatsheet.png)Figure 29:Error\-type cheatsheet: one real example per error type, drawn from the actual prediction JSONLs\.Right\-of\-marker labels sit in 50%\-transparent boxes so the underlying screenshot remains visible\.\(A\) Anchor\-collapse:*click lands on the anchor noun, not the relational neighbor\.*Opus 4\.7’s prediction onui\_vision\-199\-0\(“click the element above the replace”\) falls inside the dashed “anchor” zone \(the “replace” button\), not the green target above it\.\(B\) Twin\-identical prediction:*same click for both twins → relation word ignored\.*Qwen2\.5\-VL emits the same click on the two alignment twins of pair 81; both targets are shown for context\.\(C\) Constant default:*model defaults to a canonical coordinate\.*Llama\-3\.2\-V emits literal\(0,0\)\(0,0\)on∼\\sim46% of diagnostic items; the prediction marker sits in the upper\-left corner regardless of where the target is\.\(D\) Saturated\-primitive regression:*intervention hurts a primitive the model already solved\.*Onlist\-ordinalthe green\-filled marker shows that Qwen2\.5\-VL’s baseline answer was correct, but the SoM adds an indirection and the same model misses \(red×\{\\times\}\)\.\(E\) Contrastive\-option choice:*on this pair, the models select the contrastive target on both twins\.*On theocclusionpair, both Opus \(circles\) and GPT\-5 \(squares\) cluster on the partly\-hidden target on*both*twins, ignoring the visible target\. Pooled over all models and items, within\-region selection onocclusionis statistically indistinguishable from0\.500\.50\(Appendix[R](https://arxiv.org/html/2608.21832#A18)\)\.\(F\) Coordinate\-frame error \(now fixed\):*API returns coords in a downsampled frame\.*GPT\-5’s pre\-fix ScreenSpot\-Pro prediction \(terra hollow circle\) lands in a 1024\-side frame far from the 4K target; after our pre\-resize \+ scale\-back wrapper, the post\-fix prediction \(indigo filled circle\) sits near the gold box\.

## Appendix UError Analysis

We classify every failed prediction \(across the 19×\\times994==18,886 model–item evaluations on the full benchmark\) into one of six error types\. Figure[29](https://arxiv.org/html/2608.21832#A20.F29)shows one real example of each type drawn from the model predictions\. The types are not mutually exclusive; a prediction can belong to more than one, but they are intended to be informative\.

#### Type A: Anchor\-collapse\.

The prediction lands within1×1\{\\times\}the diagonal of the anchor element’s bounding box and not in the target box\. The model correctly locates the anchor \(the noun the instruction refers to\), but emits a click on the anchor itself rather than on the spatial neighbor the relation word specifies\. Most common onrel\-pos\-horizontalandrel\-pos\-vertical\(Example 1 above\)\. Affects 28% of all failed binary\-primitive items on Opus 4\.7, 21% on GPT\-5\.

#### Type B: Twin\-identical prediction\.

The two paired items receive predictions whose Euclidean distance in pixel space is less than 10% of the smaller side of the screenshot\. The model emitted essentially the same click for “left of X” and “right of X”, telling us that the relation word was ignored\. This is the most diagnostic failure mode: it pushes*pair\-consistency*far below*accuracy*\(the “What’s\-Upsignature” of[7](https://arxiv.org/html/2608.21832#bib.bib4), re\-derived in §[5\.5](https://arxiv.org/html/2608.21832#S5.SS5)/ Fig\.[6](https://arxiv.org/html/2608.21832#S5.F6)\)\. Affects 37% of paired failures on Qwen2\.5\-VL, 25% on Opus 4\.7, and 41% on GPT\-5*before*the closed\-API coordinate rescale; after the rescale \(Appendix[E](https://arxiv.org/html/2608.21832#A5)\), the GPT\-5 rate drops to 24%\.

#### Type C: Constant default\.

The model emits a fixed canonical coordinate \(e\.g\.\(0,0\)\(0,\\,0\)or the image center\) on a large fraction of items\. The clearest case is Llama\-3\.2\-11B\-Vision, which emits\(0,0\)\(0,\\,0\)on 46% of diagnostic items and 25% of ScreenSpot\-Pro items\. This is a model\-side “don’t\-know” fallback, not a parser failure:raw\_textis a well\-formedclick\(0, 0\)string\. We document it because the headline 1\.2% accuracy of Llama\-3\.2 onGUI\-Primitives*understates*the model: its actual decision is “refuse to ground” on roughly half the items\.

#### Type D: Saturated\-primitive regression\.

The model already solves the primitive without the intervention, so the intervention*introduces friction*\. The clearest instance is SoM on Qwen2\.5\-VL’slist\-ordinal, which regresses by 32\.4 points \(§[7](https://arxiv.org/html/2608.21832#S7), Appendix[I](https://arxiv.org/html/2608.21832#A9)\)\. The intervention converts a task the model had already learned \(counting list items\) into a mark\-selection task with one indirection\. The lesson is that training\-free interventions should be applied selectively rather than uniformly\.

#### Type E: Contrastive\-option choice \(item level\)\.

On individual pairs, several models select the contrastive element on both twins \(Example 5\)\. At the population level, the candidate\-level classification in Appendix[R](https://arxiv.org/html/2608.21832#A18)shows that below\-reference point\-in\-box accuracy is primarily attributable to predictions outside both candidate regions: pooled over 19 models, the target is selected at least as often as the distractor on every primitive\. Forcontainmentandocclusion, within\-region selection is 0\.548 and 0\.554, statistically indistinguishable from0\.500\.50, indicating no measurable relation\-word signal on these two primitives\. The directional layout prior is quantified in Appendix[Q](https://arxiv.org/html/2608.21832#A17): predictions concentrate on left\-side elements, and within\-region accuracy is lower under “right of” than “left of”\.

#### Type F: Coordinate\-frame error \(now fixed\)\.

Closed\-API providers downsample large screenshots and return coordinates in the downsampled frame\. Before our wrapper handled this, predictions on 4K ScreenSpot\-Pro images landed in a∼\\sim1024\-side coordinate space and missed every target\. This was the dominant error category for Claude Sonnet 4\.6 \(1,297 / 1,581 SS\-Pro records\) and for the first GPT\-5 ScreenSpot\-Pro run \(≈\\approx1,500 / 1,581 records\); both are corrected in the version reported in this paper \(Appendix[E](https://arxiv.org/html/2608.21832#A5)\)\. We flag this category here because future closed\-VLM evaluations that omit the rescale will misattribute this error to genuine model failure\.

#### Cross\-cutting observation\.

Types A, B, and E share a common cause: the model has correctly*detected*the elements but failed to*compose*them with the relation word\. Types C and F are infrastructure/fallback errors\. Type D is an intervention\-design issue\. The fact that training\-free interventions move the needle only on the composition\-side errors, and only when the intervention bypasses the relation \(SoM\) rather than scaffolding it \(CoT, steering\), is consistent with the*discussion*section’s reading that the bottleneck is on the perception / binding side, not on the language side\.

## Appendix VLimitations

#### Two primitives are synthetic\-only\.

Containment and occlusion are present only in the synthetic arm ofGUI\-Primitives; list\-ordinal draws 24 of its 142 items from real UI\-Vision menus and lists, with the remainder synthetic\. Real GUI datasets do not provide scalable parent–child or overlay–target metadata, so containment and occlusion cannot be mined from real screenshots\. Consequently, findings on these two primitives have limited external validity beyond settings where controlled synthetic stimuli are representative of real GUI behavior; the failure rates observed on containment and occlusion may not transfer directly to uncurated real\-world GUIs\. This constraint bounds the candidate\-level result of Appendix[R](https://arxiv.org/html/2608.21832#A18): the absence of a measurable relation\-word signal on containment and occlusion is established on synthetic screenshots only\. We treat the synthetic items as a controlled\-stimulus arm in theWhat’s\-Uptradition and report all results split by source arm\.

#### Per\-primitive regression is conservative\.

After the model fixed effects absorb between\-model variance, individual per\-primitive coefficients in our item\-level regression are small\. We therefore claim only the joint, model\-level association and do not interpret per\-primitive coefficients causally\. The model\-level Spearman is statistically significant; the per\-primitive forest plot is reported for transparency, not as a contribution\.

#### No OSWorld downstream slice\.

We stop at ScreenSpot\-Pro\. End\-to\-end task success on OSWorld\([27](https://arxiv.org/html/2608.21832#bib.bib1)\)and its verified release\([28](https://arxiv.org/html/2608.21832#bib.bib32)\)would provide stronger downstream confirmation, but it is a separate VM\-harness study and is left for future work\.

#### Action and language scope\.

GUI\-Primitivesv1 covers English instructions and click targets\. Future versions could extend the diagnostic to drag\-target prediction, bidirectional and CJK layouts, and an OSWorld pre\-action grounding slice\.

#### SoM is a scaffold, not a fix\.

SoM partially bypasses the underlying perception\-and\-binding problem by converting relational grounding into symbolic mark selection; the observed gains reflect scaffolding rather than repair of the spatial skill gap\. Additionally, our setup overlays at most two candidate marks per item \(target and distractor\), which is far fewer than the crowded mark sets that arise in realistic GUI scenes\. Gains observed under this minimal two\-mark setup may therefore not generalize to settings with many candidates, where disambiguating among marks becomes a bottleneck in its own right\. Finally, in deployment, marks would be proposed by an OmniParser\-style or accessibility\-tree\-driven component, with its failure modes propagating upstream\.

#### Point\-in\-box accuracy is a coarse metric\.

Point\-in\-box accuracy, while standard for GUI grounding, is binary: a click one pixel outside the target box and a click on the entirely wrong element score identically\. The loose metric \(within2×2\\timestarget diagonal of the box center\) partially separates sub\-pixel imprecision from element misidentification, but neither metric captures the structure of spatial errors: whether the model was on the right side of the screen, how far it drifted, or whether it collapsed to the anchor rather than the relational neighbor\. These structured error patterns are documented qualitatively in Appendix[U](https://arxiv.org/html/2608.21832#A21), but a finer\-grained spatial\-error metric remains future work\.

#### Llama\-3\.2\-Vision returns default coordinates\.

Llama\-3\.2\-11B\-Vision emits a constant or near\-origin coordinate on roughly 42% of items\. The 1\.2% accuracy is the model defaulting, not a parsing bug; it is consistent with the original ScreenSpot\-Pro paper’s report of<2<\\\!2% accuracy for generalist VLMs\.

#### Proprietary API behavior may shift\.

The proprietary providers can change downsampling, decoding, or model identity at any time\. We freeze a snapshot of dates and pin model identifiers in Appendix[B](https://arxiv.org/html/2608.21832#A2); future runs may differ\.

## Information About Use Of AI Assistants

AI assistants were used for limited editing support and language refinement\. The final manuscript contents were verified and authored by the research team\.