Lexical Coupling in GUI Element Grounding: Sentence Embeddings Track Labels across Mobile and Web

# Lexical Coupling in GUI Element Grounding:Sentence Embeddings Track Labels across Mobile and Web
Source: [https://arxiv.org/html/2608.21794](https://arxiv.org/html/2608.21794)
Qijia ChenAffiliation:Department of Computer ScienceAffiliation:University of HelsinkiAffiliation:Helsinki, FinlandEmail:[qijia\.chen@helsinki\.fi](mailto:)Giulio JacucciAffiliation:Department of Computer ScienceAffiliation:University of HelsinkiAffiliation:Helsinki, FinlandEmail:[giulio\.jacucci@helsinki\.fi](mailto:)

###### Abstract

GUI grounding evaluations that expose UI elements as text metadata often treat high instruction–element embedding similarity as evidence of semantic grounding\. Across three mobile and web benchmarks, we show that this interpretation is frequently confounded by visible\-label recovery\. Lexical baselines remain competitive at top\-1, label\-poor targets remain weak for text\-only methods, and encoder top\-1 hits are predictable from lexical rank, candidate\-pool size, and label type\. We evaluate each action as a same\-screen ranking task, comparing five off\-the\-shelf single\-vector encoders with lexical baselines\. Encoders recover some lexical misses, but deployable fusion gains are much smaller than target\-aware oracle gains\. These findings show that embedding\-based evaluations can conflate visible\-label recovery with semantic GUI grounding\. Embedding\-based evaluations should therefore report lexical baselines, label\-type stratification, and deployable\-fusion diagnostics\. Our released repository provides analysis scripts and de\-texted per\-step panels:[https://github\.com/qijia123/lexical\-coupling\-release](https://github.com/qijia123/lexical-coupling-release)\.

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.21794v1/Figure_01.png)Figure 1:Overview of the measurement audit\.\(A\)Text\-metadata GUI grounding is treated as same\-screen candidate ranking given an instruction\.\(B\)High instruction–element embedding similarity may reflect visible\-label overlap rather than semantic grounding\.\(C\)We compare off\-the\-shelf single\-vector encoders against lexical controls and diagnose the results with label\-type stratification, target\-text dependence perturbation, LCC, and ESH\.GUI agents must map natural\-language instructions to actionable on\-screen elements\([8](https://arxiv.org/html/2608.21794#bib.bib6);[10](https://arxiv.org/html/2608.21794#bib.bib7);[3](https://arxiv.org/html/2608.21794#bib.bib4);[21](https://arxiv.org/html/2608.21794#bib.bib5)\), and benchmarks supply the element\-level targets that make this mapping measurable\([5](https://arxiv.org/html/2608.21794#bib.bib3);[14](https://arxiv.org/html/2608.21794#bib.bib1)\)\. As agents and benchmarks scale, this mapping is increasingly scored by instruction–element sentence\-embedding similarity over textual metadata\. But high similarity is ambiguous when the instruction repeats the target’s visible label: it may reflect label recovery rather than semantic inference from task or screen context \(Figure[1](https://arxiv.org/html/2608.21794#S1.F1)\)\.

In Messick’s terms\([16](https://arxiv.org/html/2608.21794#bib.bib14)\), embedding distance is a valid grounding proxy only if it carries signal beyond a lexical baseline; this paper is an audit of how GUI grounding is*measured*, not a proposal for a new grounding model\. We focus on off\-the\-shelf single\-vector sentence encoders as plug\-and\-play similarity metrics, with fine\-tuning as a control\. We recast each grounded step as ranking the visible same\-screen UI elements and compare lexical retrievers with off\-the\-shelf encoders across AndroidControl\([14](https://arxiv.org/html/2608.21794#bib.bib1)\), MoTIF\([1](https://arxiv.org/html/2608.21794#bib.bib2)\), and Mind2Web\([5](https://arxiv.org/html/2608.21794#bib.bib3)\)\. We stratify by target label type, summarise lexical coupling and fusion headroom with LCC and ESH, and use STS\-B\([2](https://arxiv.org/html/2608.21794#bib.bib24)\)plus AndroidControl\-Curated\([13](https://arxiv.org/html/2608.21794#bib.bib9)\)to bound generic encoder weakness and annotation noise\.

Three findings follow\. First, across AndroidControl, MoTIF, and Mind2Web, none of the five off\-the\-shelf encoders consistently outperforms BM25 atR​@​1R@1; Figure[2](https://arxiv.org/html/2608.21794#S1.F2)shows the paired cross\-corpus deltas\. Second, performance depends strongly on exposed target labels: all text\-only methods remain weak on label\-poor targets, and masking the target’s exposed text metadata reducesR​@​1R@1by 22–38 percentage points on an equal\-allocation three\-stratum sample\. Encoder hit@1 outcomes are also predictable from lexical rank, label type, and candidate\-pool size, with AUCs of0\.820\.82–0\.880\.88\. Task\-specific fine\-tuning substantially improves top\-1 ranking accuracy but does not remove lexical coupling: the fine\-tuned model retains an LCC of0\.8770\.877\. ItsR​@​1R@1also drops from0\.6940\.694to0\.1510\.151under the target\-text dependence perturbation, showing that the improved ranker continues to rely strongly on text exposed by the target candidate\. Third, embeddings contain complementary signal, but on AndroidControl, standard deployable fusion captures only a small fraction of the target\-aware oracle headroom and can reduce top\-1 accuracy when many correlated retrievers are combined\.

Figure 2:PairedΔ​R​@​1\\Delta R@1of each sentence\-embedding retriever versus BM25 on the three corpora\. Error bars are95%95\\%cluster\-bootstrap CIs by episode orannotation\_id\.Contributions\.\(1\) We show that high off\-the\-shelf instruction–element embedding similarity often reflects visible\-label recovery and therefore does not, by itself, establish semantic GUI grounding\. \(2\) We identify a stable cross\-corpus mechanism behind this effect: instructions often name targets using visible UI labels, while benchmark metadata re\-exposes those labels in candidate text\. \(3\) We develop LCC and ESH and situate them in an audit suite with lexical baselines, label\-type stratification, and a target\-text dependence perturbation, quantifying lexical coupling and separating target\-aware oracle headroom from deployable fusion gains\. \(4\) We turn these findings into a practical reporting protocol for text\-metadata GUI grounding evaluations\.

## 2Related Work

Our audit sits at the intersection of GUI grounding evaluation, construct\-validity and shortcut\-learning audits in NLP, and lexical\-versus\-dense retrieval in IR\.

#### GUI grounding benchmarks and agents\.

GUI grounding benchmarks evaluate whether models map natural\-language instructions to UI elements or screen coordinates\. We use three text\-metadata corpora: AndroidControl\([14](https://arxiv.org/html/2608.21794#bib.bib1)\), MoTIF\([1](https://arxiv.org/html/2608.21794#bib.bib2)\), and Mind2Web\([5](https://arxiv.org/html/2608.21794#bib.bib3)\)\(sizes and splits in §[3\.1](https://arxiv.org/html/2608.21794#S3.SS1)\)\. Pixel\-level visual grounders\([3](https://arxiv.org/html/2608.21794#bib.bib4);[21](https://arxiv.org/html/2608.21794#bib.bib5);[8](https://arxiv.org/html/2608.21794#bib.bib6);[10](https://arxiv.org/html/2608.21794#bib.bib7)\)and end\-to\-end Android agent harnesses\([23](https://arxiv.org/html/2608.21794#bib.bib8)\)lie outside our text\-similarity scope\. This literature asks whether models can ground; we ask what the similarity scores measure\.

#### Construct validity and shortcut learning in NLP\.

We build on construct\-validity audits in language understanding\.[16](https://arxiv.org/html/2608.21794#bib.bib14)establishes that a score\-based metric is informative only insofar as it carries the construct it claims to measure\.[9](https://arxiv.org/html/2608.21794#bib.bib12)and[15](https://arxiv.org/html/2608.21794#bib.bib11)show NLI classifiers can predict labels from annotation artifacts or lexical\-overlap heuristics rather than the entailment relation itself;[7](https://arxiv.org/html/2608.21794#bib.bib13)frame these as shortcut learning\. We adapt the audit logic but not its verdict: in GUI grounding, label overlap is often a valid route to the target, so the construct\-validity question is not whether embeddings exploit a shortcut but whether their scores carry evidence beyond that lexical route\. Our lexical baselines, label\-type stratification, and residual logistic audit are that methodological transfer\.

#### Lexical versus dense retrieval\.

What NLP construct\-validity audits call a shortcut, IR treats as a lexical baseline that should not be skipped: dense and sparse retrievers must be reported side by side before dense similarity is interpreted as semantic evidence\. BM25\([19](https://arxiv.org/html/2608.21794#bib.bib16)\)is the canonical lexical reference; dense retrieval work such as DPR\([12](https://arxiv.org/html/2608.21794#bib.bib17)\)motivates neural semantic matching in open\-domain QA, while BEIR\([20](https://arxiv.org/html/2608.21794#bib.bib15)\)shows that BM25 remains a robust zero\-shot baseline across heterogeneous text\-retrieval tasks\. Our setting asks a different measurement question: when sentence\-embedding similarity is used as a GUI grounding metric, does it carry evidence beyond visible\-label recovery? GUI candidate texts expose the same labels annotators use to describe targets, making lexical matching both a valid grounding route and a confound for interpreting embedding similarity as semantic grounding\. We turn this mechanism into reportable lexical coupling and fusion headroom diagnostics\. Reciprocal\-rank fusion\([4](https://arxiv.org/html/2608.21794#bib.bib18)\)is a standard rank\-fusion baseline; the encoders we audit are standard MTEB\-style sentence encoders\([17](https://arxiv.org/html/2608.21794#bib.bib19)\)\(§[3\.5](https://arxiv.org/html/2608.21794#S3.SS5)\), and STS\-B contrast \(§[4\.4](https://arxiv.org/html/2608.21794#S4.SS4.SSS0.Px2)\) shows the coupling is corpus\-specific rather than evidence of weak encoders\.

#### Audits of GUI grounding quality\.

Two recent audits of GUI grounders sit alongside ours\.[11](https://arxiv.org/html/2608.21794#bib.bib10)use adversarial multi\-instruction generation for a model\-side audit of robustness to query variation\. AndroidControl\-Curated\([13](https://arxiv.org/html/2608.21794#bib.bib9)\)identifies ambiguities and factual errors in AndroidControl and releases a purified split, an annotation\-side cleanup that holds the evaluation metric fixed\. Ours is a metric\-side audit: we hold the model family and benchmark fixed and ask what the similarity score itself measures\. We use AndroidControl\-Curated’s flags in §[4\.4](https://arxiv.org/html/2608.21794#S4.SS4.SSS0.Px2)as a robustness check; their benchmark purification and our metric audit are complementary\.

## 3Method

We operationalise the audit as a lexical\-controlled candidate\-ranking protocol that ranks same\-screen UI elements with lexical and embedding retrievers, stratifies results by target label availability, and summarises lexical coupling and fusion headroom with LCC and ESH\.

### 3\.1Datasets and candidate pools

#### AndroidControl\.

AndroidControl\([14](https://arxiv.org/html/2608.21794#bib.bib1)\)provides 15,283 mobile UI demonstration episodes with per\-step instructions paired to typed actions\. From the accessibility tree at each grounded step we extract every visible UI element and resolve the target at the action coordinate, yielding 58,078 grounded steps and 4\.56M total candidates \(median 62 per step, max 1,657\)\. We use the official train/val/test split; Appendix[B](https://arxiv.org/html/2608.21794#A2)gives extraction and join\-key details\.

#### Label\-type taxonomy\.

For each target element we record which textual affordance is present in the accessibility tree: text\-only \(visibletext\), cd\-only \(onlycontent\_description\), both, resource\-only \(resource ID only\), pseudo \(boilerplate strings such asimage containerorlayout header\), or none\.

#### Main and diagnostic subsets\.

Our main analysis uses clean\-main:clickandlong\_pressactions whose target has visible text or content description \(n=32,646n=32\{,\}646\)\. Label\-poor rows \(resource\-only/pseudo/none\) and the full 58,078\-step set are reported separately as diagnostics rather than averaged into the main metric\.

#### MoTIF\.

For cross\-corpus stress testing we run the same target\-ranking protocol on MoTIF\([1](https://arxiv.org/html/2608.21794#bib.bib2)\), using the episode goal as the query because MoTIF lacks AndroidControl\-style step instructions\. After matching preprocessing this yields 22,427 grounded steps; the clean\-main subset \(text/cd\-labelled click targets\) contains 4,724 rows\. The label\-type taxonomy and ranking protocol are otherwise identical\.

#### Mind2Web \(cross\-domain\)\.

We run the same protocol on Mind2Web\([5](https://arxiv.org/html/2608.21794#bib.bib3)\)’s three test splits \(5,941 usable steps after filtering; mean 531, median 410, max 4,301 candidates per step\) using the releasedpos\+neg\_candidatespool\. The query is the task plus the prefix ofaction\_reprspreceding the target action; we exclude the current step’starget\_action\_reprbecause it is element\-aware and would leak the target identity\. The label\-type taxonomy maps to web counterparts \(text\-only, aria\-only, both, id/class\-only, none\); DOM\-parsing details are in Appendix[B](https://arxiv.org/html/2608.21794#A2)\.

### 3\.2Auxiliary datasets for control checks

#### AndroidControl\-Curated\.

AndroidControl\-Curated\([13](https://arxiv.org/html/2608.21794#bib.bib9)\)provides annotation\-quality flags that we use to test whether the BM25 lead is driven by annotation noise; 2,946 of its rows join to our clean\-main subset\. Per\-flag definitions and counts are in Appendix[B](https://arxiv.org/html/2608.21794#A2)\.

#### STS\-B\.

To contrast corpus\-specific lexical coupling against canonical sentence similarity we evaluate MiniLM, MPNet, BGE, E5, and Qwen3 on the STS\-B dev\+test split\([2](https://arxiv.org/html/2608.21794#bib.bib24)\)\(2,879 sentence pairs with gold similarity in\[0,5\]\[0,5\]\)\.

### 3\.3Candidate\-ranking protocol

For each grounded step we form a ranking queryqqand a candidate setC=\{c1,…,cn\}C=\\\{c\_\{1\},\\dots,c\_\{n\}\\\}consisting of every parsed UI element on the current screen\. The query is the step instruction \(AndroidControl\), the episode goal \(MoTIF\), or the task plus preceding action\-history prefix \(Mind2Web; see §[3\.1](https://arxiv.org/html/2608.21794#S3.SS1)\)\. The positive candidate is the element at the recorded target index\. A retriever scores each candidate; the target’s rank is its position when candidates are sorted by descending score, with deterministic tie\-break \(Appendix[D](https://arxiv.org/html/2608.21794#A4)\)\. We report standard ranking metrics:R​@​1R@1,R​@​5R@5,R​@​10R@10, and mean reciprocal rank \(MRR\)\. All metrics treat each grounded step as one ranking event and aggregate uniformly across steps within the chosen subset\.

### 3\.4Element Textualisation

Each UI element is serialised to a single retriever input string from visibletextand accessibility descriptions, with resource\-id or class\-name fallbacks when no readable label is present\. Lexical and embedding retrievers consume the same textualised candidate strings, so any lexical advantage is not due to BM25 receiving richer UI text\. Corpus\-specific parsing, tokenisation, and stoplists are in Appendix[B](https://arxiv.org/html/2608.21794#A2); alternative variants used only in construct correlation are in Appendix[L](https://arxiv.org/html/2608.21794#A12)\.

### 3\.5Retrievers

#### Random\.

A reference baseline that ranks candidates uniformly at random per step\. ExpectedR​@​1R@1is the per\-step mean of1/nx1/n\_\{x\}; we report it to make candidate\-pool difficulty explicit\.

#### Jaccard\.

For each candidate we compute the Jaccard coefficient between query and candidate token sets\. Empty sets receive score zero\.

#### BM25\.

We score candidates with BM25 using fixed, standard hyperparameters \(k1=1\.5k\_\{1\}=1\.5,b=0\.75b=0\.75\) with no corpus\-specific tuning\. Each screen is treated as its own collection so that document frequencies reflect the local distractor pool rather than the global vocabulary, matching how a deployed grounding system would rank elements on a single screen\. Appendix[D](https://arxiv.org/html/2608.21794#A4)confirms the per\-screen scheme is not what drives BM25’s advantage\.

#### Sentence embeddings\.

We rank candidates with five off\-the\-shelf single\-vector encoders spanning 22M to 0\.6B parameters: MiniLM\-L6 \(all\-MiniLM\-L6\-v2, 22M\) and MPNet \(all\-mpnet\-base\-v2, 110M\) sentence\-transformer models\([18](https://arxiv.org/html/2608.21794#bib.bib20)\), BGE \(BAAI/bge\-large\-en\-v1\.5, 335M\)\([22](https://arxiv.org/html/2608.21794#bib.bib21)\), E5 \(intfloat/e5\-large\-v2, 335M\), and Qwen3\-Embedding\-0\.6B \(Qwen/Qwen3\-Embedding\-0\.6B, 0\.6B\)\([24](https://arxiv.org/html/2608.21794#bib.bib22)\)\. Embeddings areℓ2\\ell\_\{2\}\-normalised and ranking uses cosine similarity\. Model\-specific instructions are respected: queries to BGE are prefixed with the documented instruction string and E5 inputs are prefixed withquery:andpassage:for the two sides; Qwen3 queries use the model\-card instruction format\.

### 3\.6Fusion

We test whether the embeddings carry ranking signal beyond BM25 under fusion, in two ways\.

#### Oracle min\-rank ceiling\.

For each step and retriever setMM, letrjr\_\{j\}be the target rank under retrieverjj\. The oracle target rank isrMor=minj∈M⁡rjr^\{\\mathrm\{or\}\}\_\{M\}=\\min\_\{j\\in M\}r\_\{j\}\. This is a target\-aware per\-step selector: it is not deployable, because it uses the true target to choose the retriever that ranked that target best\. We use it as a headroom diagnostic before testing deployable fusion\.

#### True Reciprocal Rank Fusion\.

For each candidateccand methodjjwith rankrj​\(c\)r\_\{j\}\(c\)we form

srrf​\(c\)=∑j∈M1k\+rj​\(c\),k=60,s\_\{\\mathrm\{rrf\}\}\(c\)=\\sum\_\{j\\in M\}\\frac\{1\}\{k\+r\_\{j\}\(c\)\},\\quad k=60,\(1\)following[4](https://arxiv.org/html/2608.21794#bib.bib18)\. The target’s combined rank is1\+\|\{c≠t:srrf​\(c\)\>srrf​\(t\)\}\|1\+\|\\\{c\\neq t:s\_\{\\mathrm\{rrf\}\}\(c\)\>s\_\{\\mathrm\{rrf\}\}\(t\)\\\}\|, withsrrfs\_\{\\mathrm\{rrf\}\}ties resolved by the deterministic tie\-break of §[3\.9](https://arxiv.org/html/2608.21794#S3.SS9)\. Unlike the oracle, this is a deployable combiner: it uses per\-candidate ranks under each method, not the target’s privileged rank\. We report both the oracle ceiling and the true\-RRF realisation so that the gap between them quantifies how much of the oracle headroom is actually captured by a standard fusion rule\.

We report lexical fusion, single\-embedding BM25\+encoder fusion \(BGE, E5, Qwen3\), and aggregate fusion over all five main encoders; the full list is given in Table[8](https://arxiv.org/html/2608.21794#A9.T8)\.

### 3\.7LCC and ESH

We formalise two scalar diagnostics that summarise an encoder’s behaviour at the \(corpus, encoder\) cell level\. LCC operates independently of the fusion machinery in §[3\.6](https://arxiv.org/html/2608.21794#S3.SS6); ESH compresses its oracle and RRF combiners into a single headroom pair\.

#### Definition 1 \(Lexical Coupling Coefficient, LCC\)\.

For a benchmark𝒟\\mathcal\{D\}and an encoderMM, letyM\(x\)=𝟙\[rM\(x\)=1\]y\_\{M\}\(x\)=\\mathbb\{1\}\[r\_\{M\}\(x\)=1\]be the binary indicator thatMMranks the true target at position11on stepxx\. Define the control vector

ϕ⁡\(x\)=\(CLOSE\\displaystyle\\phi\(x\)=\\bigl\(log⁡\(1\+rbm25​\(x\)\),log⁡\(1\+rjaccard​\(x\)\),\\displaystyle\\log\(1\{\+\}r\_\{\\textsc\{bm25\}\}\(x\)\),\\,\\log\(1\{\+\}r\_\{\\textsc\{jaccard\}\}\(x\)\),\(2\)OPENlog⁡n⁡\(x\),1ℓ⁡\(x\)\),\\displaystyle\\log n\(x\),\\,\\mathbf\{1\}\_\{\\ell\(x\)\}\\bigr\),where𝟏ℓ⁡\(x\)\\mathbf\{1\}\_\{\\ell\(x\)\}is a one\-hot encoding of the target element’s label\-type partition\. The two rank terms are lexical; we add candidate\-pool sizelog⁡n⁡\(x\)\\log n\(x\)and label type as structural controls so that success a model could obtain from pool shape or label availability alone is not credited as semantic signal\. The Lexical Coupling Coefficient is the AUC of a logistic regression ofyMy\_\{M\}onϕ\\phi, fit on𝒟\\mathcal\{D\}withℓ2\\ell\_\{2\}regularisation \(C=1C=1\) on standardised features:

LCC​\(M,𝒟\)=AUC⁡\(Pr^​\[yM\|ϕ\],yM\)\.\\textrm\{LCC\}\(M;\\,\\mathcal\{D\}\)\\;=\\;\\mathrm\{AUC\}\\\!\\left\(\\widehat\{\\Pr\}\\\!\\bigl\[\\,y\_\{M\}\\,\\big\|\\,\\phi\\,\\bigr\],\\;y\_\{M\}\\right\)\.\(3\)LCC takes values in\[0\.5,1\.0\]\[0\.5,\\,1\.0\]and measures how reproducible an encoder’s R@1 successes are from these lexical and structural controls without reference to the encoder itself\. AtLCC=0\.85\\textrm\{LCC\}\{=\}0\.85the controls assign a higher predicted hit probability to a random encoder hit than to a random miss85%85\\%of the time; we useLCC\>0\.85\\textrm\{LCC\}\>0\.85as a descriptive high\-coupling regime, not a hypothesis\-test cutoff, and report the underlying AUCs throughout\.

#### Definition 2 \(Effective Semantic Headroom, ESH\)\.

For an encoder set𝒮=\{M1,…,Mk\}\\mathcal\{S\}=\\\{M\_\{1\},\\dots,M\_\{k\}\\\}and the oracle min\-rank combiner of §[3\.6](https://arxiv.org/html/2608.21794#S3.SS6),

ESHoracle​\(𝒮,𝒟\)\\displaystyle\\textrm\{ESH\}^\{\\textrm\{oracle\}\}\(\\mathcal\{S\};\\,\\mathcal\{D\}\)=Pr\[r𝒮∪\{bm25\}min\(x\)=1\]\\displaystyle=\\Pr\\\!\\bigl\[r\_\{\\mathcal\{S\}\\cup\\\{\\textsc\{bm25\}\\\}\}^\{\\textrm\{min\}\}\(x\)=1\\bigr\]−Pr\[rbm25\(x\)=1\]\.\\displaystyle\\quad\-\\Pr\\\!\\bigl\[r\_\{\\textsc\{bm25\}\}\(x\)=1\\bigr\]\.\(4\)It measures the R@1 gain of the target\-aware per\-step selector when𝒮\\mathcal\{S\}is added to BM25, and is therefore a headroom diagnostic, not a deployable fusion result\. Its deployable counterpart,ESHreal​\(𝒮,𝒟\)\\textrm\{ESH\}^\{\\textrm\{real\}\}\(\\mathcal\{S\};\\,\\mathcal\{D\}\), is defined identically with the RRF combiner of §[3\.6](https://arxiv.org/html/2608.21794#S3.SS6)\(k=60k=60\([4](https://arxiv.org/html/2608.21794#bib.bib18)\)\) replacing the oracle, and is the R@1 gain a standard hybrid retriever actually delivers\. The pair\(ESHoracle,ESHreal\)\(\\textrm\{ESH\}^\{\\textrm\{oracle\}\},\\textrm\{ESH\}^\{\\textrm\{real\}\}\)summarises both the in\-principle complementarity of𝒮\\mathcal\{S\}and what a conventional fusion rule realises\.

Matching thresholds for unrealisable headroom \(ESHoracle−ESHreal\\textrm\{ESH\}^\{\\textrm\{oracle\}\}\-\\textrm\{ESH\}^\{\\textrm\{real\}\}\) and destructive fusion \(ESHreal≤0\\textrm\{ESH\}^\{\\textrm\{real\}\}\\leq 0\), together with per\-retriever rank caching requirements, are in Appendix[C](https://arxiv.org/html/2608.21794#A3)\.

### 3\.8Fine\-tuning control

We include a task\-supervised positive control to test whether the observed limitations are specific to plug\-and\-play similarity metrics rather than dense\-retriever capacity under task supervision\. MiniLM\-L6\-v2 \(22M\) is fine\-tuned for one contrastive epoch on the AndroidControl train split and evaluated on the held\-out clean\-main test subset under the same ranking protocol\. We then rerun the LCC audit and target\-text dependence perturbation on the fine\-tuned ranks to measure lexical coupling and reliance on target\-exposed text after supervision\. Results are in §[4\.4](https://arxiv.org/html/2608.21794#S4.SS4.SSS0.Px1); full training details are in Appendix[K](https://arxiv.org/html/2608.21794#A11)\.

### 3\.9Statistical inference and reproducibility

All paired comparisons use a paired cluster bootstrap withB=2000B=2000resamples by corpus\-specific unit:episode\_idfor AndroidControl and MoTIF, andannotation\_idfor Mind2Web\. 95% CIs are percentile intervals\. Ranking ties are broken with a hash\-based deterministic jitter keyed on \(episode id, step index, retriever, candidate index\) so rank assignment is reproducible across subsets\. Appendix[D](https://arxiv.org/html/2608.21794#A4)gives full inference and implementation details\. Textualisation code, ranking scripts, and de\-texted per\-step panels are released alongside the paper \(Appendix[A](https://arxiv.org/html/2608.21794#A1)\)\.

## 4Results

Single\-cell numbers are verified against per\-step CSV panels released with the paper; all paired comparisons use aB=2000B\{=\}2000paired cluster bootstrap on the corpus\-specific unit defined in §[3\.9](https://arxiv.org/html/2608.21794#S3.SS9)\.

### 4\.1Target ranking and label availability on AndroidControl

On the clean\-main subset \(n=32,646n\{=\}32\{,\}646steps over12,39312\{,\}393episodes; §[3\.1](https://arxiv.org/html/2608.21794#S3.SS1)\), with random ranking givingR​@​1=0\.022R@1\{=\}0\.022as the difficulty floor, BM25 obtainsR​@​1=0\.557R@1\{=\}0\.557andMRR=0\.632\\text\{MRR\}\{=\}0\.632, exceeding every individual sentence\-embedding baseline on top\-1 accuracy and MRR \(Table[1](https://arxiv.org/html/2608.21794#S4.T1)\)\. The three larger encoders in the main comparison, BGE\-large, E5\-large\-v2, and Qwen3\-Embedding\-0\.6B, underperform BM25 by3\.73\.7,5\.05\.0, and7\.67\.6pp onR​@​1R@1, respectively; a more recent 0\.6B single\-vector embedding control does not alter the top\-1 pattern\. Embeddings recover advantage only atR​@​5R@5and beyond: BGE\-large attains the bestR​@​5=0\.724R@5\{=\}0\.724andR​@​10=0\.785R@10\{=\}0\.785, a pattern consistent with embeddings finding the target in a broader top\-kkshortlist but ranking it less reliably at position 1\. Paired cluster\-bootstrap deltas confirm the BM25 lead over all five encoders on bothR​@​1R@1and MRR; the smallest is BGE\-large at\+3\.7\+3\.7ppR​@​1R@1and\+1\.6\+1\.6pp MRR, with all CIs excluding zero \(Appendix[E](https://arxiv.org/html/2608.21794#A5)\)\.

Table 1:Target\-ranking metrics on the clean\-main subset \(n=32,646n\{=\}32\{,\}646\)\.#### Mechanism: label\-type stratification\.

Stratifying by target label type \(Table[2](https://arxiv.org/html/2608.21794#S4.T2);n=58,078n\{=\}58\{,\}078all DB rows\) separates three regimes\. On text\-rich targets, lexical and embedding methods both work but BM25 wins top\-1; on content\-description\-only targets all methods sit within a few points of each other; on label\-poor targets all methods remain weak\. The 25,430 resource\-only/pseudo/none rows show the boundary: pseudo and none rows sit at random scale, while resource\-only rows top out atR​@​1=0\.124R@1\{=\}0\.124under Qwen3, far below text\-rich performance\. Thus embedding alignment does not reliably recover targets that lack a human\-readable label; the same low\-label collapse holds across corpora \(Appendix[F](https://arxiv.org/html/2608.21794#A6)\)\. Masking the target’s exposed text metadata dropsR​@​1R@1by2222–3838pp on an equal\-allocation stratified sample of text\-only, cd\-only, and both\-labelled targets \(Appendix[G](https://arxiv.org/html/2608.21794#A7)\), providing a controlled measure of target\-text dependence that complements the cross\-corpus label\-type collapse\. Theinput\_textdiagnostic subset, excluded from clean\-main because the DB target resolves to the system keyboard rather than the intended input field, is documented in Appendix[B](https://arxiv.org/html/2608.21794#A2)\.

Table 2:R​@​1R@1by target label\-type on all DB rows\. Best non\-random per row in bold\.To complement the quantitative label\-type and masking analyses, Appendix[O](https://arxiv.org/html/2608.21794#A15)provides a qualitative error analysis with de\-identified examples for the three recurring regimes: lexical label recovery, residual embedding recovery, and label\-poor failure\. These examples are illustrative; the population\-level evidence is Table[2](https://arxiv.org/html/2608.21794#S4.T2)and the target\-text dependence perturbation in Appendix[G](https://arxiv.org/html/2608.21794#A7)\.

### 4\.2Cross\-corpus generalisation on MoTIF and Mind2Web

Because the corpora differ simultaneously in query granularity, domain, candidate\-pool size, and annotation protocol, this section tests whether the within\-corpus BM25\-vs\-embedding gap*generalises*across these conditions\. We therefore interpret cross\-corpus absolute scores descriptively and focus on the pairedΔ\\Deltawithin each corpus, where BM25 and the encoder see identical queries and candidate pools\.

We apply the same candidate\-ranking audit to MoTIF \(mobile, episode goal as query\) and Mind2Web \(web, task\-with\-history; §[3\.1](https://arxiv.org/html/2608.21794#S3.SS1)\): on neither corpus does an off\-the\-shelf embedding significantly beat BM25 at top\-1, and the low\-label weak regime recurs on both, across independent corpora collected under different annotation protocols \(Table[3](https://arxiv.org/html/2608.21794#S4.T3)\)\. On 4,724 MoTIF clean\-main click targets, BM25 wins top\-1 withR​@​1=0\.224R@1\{=\}0\.224versus E5\-large’sR​@​1=0\.212R@1\{=\}0\.212, BGE\-large’sR​@​1=0\.201R@1\{=\}0\.201, and Qwen3\-Embedding\-0\.6B’sR​@​1=0\.167R@1\{=\}0\.167; absolute scores are lower than on AndroidControl because the query is a goal rather than a step instruction\. Embeddings recover advantage further down the list: MiniLM attains the best MRR \(0\.3420\.342\), BGE\-large the bestR​@​5R@5\(0\.4790\.479\), and MPNet the bestR​@​10R@10\(0\.6330\.633\)\. On the union of the three Mind2Web test splits \(n=5,941n\{=\}5\{,\}941usable steps; pool median410410\), no encoder significantly exceeds BM25 atR​@​1R@1\(BGE\-large and E5\-large tie within paired CIs, while Qwen3 is below BM25\); BM25 retains significant MRR leads over MPNet, BGE\-large, E5\-large, and Qwen3\. The low\-label weak regime recurs on both: on MoTIF’s 17,703 label\-poor rowsR​@​1R@1ranges only0\.0390\.039–0\.0730\.073against random0\.0420\.042, and on Mind2Web’s 474id/class\-only or unlabelled rows every retriever scoresR​@​1≤0\.005R@1\\leq 0\.005\.

Table 3:Cross\-corpus target\-ranking metrics on MoTIF and Mind2Web\. Bold marks the best non\-random value in each metric column\.
### 4\.3Corpus\-invariant lexical coupling and bounded headroom

Lexical coupling is stable across corpora, while oracle semantic headroom remains bounded; the LCC and oracle ESH diagnostics \(§[3\.7](https://arxiv.org/html/2608.21794#S3.SS7)\) quantify both at the encoder level \(Table[4](https://arxiv.org/html/2608.21794#S4.T4)\)\. BGE\-large, E5\-large, and Qwen3\-Embedding\-0\.6B remain strongly coupled across corpora \(BGE LCC0\.8450\.845–0\.8800\.880; E5 LCC0\.8490\.849–0\.8820\.882; Qwen3 LCC0\.8350\.835–0\.8600\.860\): an encoder’s R@1 successes are predictable from\(log⁡rBM25,log⁡rJaccard,log⁡ncands,label\-type\)\(\\log r\_\{\\text\{BM25\}\},\\log r\_\{\\text\{Jaccard\}\},\\log n\_\{\\text\{cands\}\},\\text\{label\-type\}\)on every corpus we test\. Oracle ESH\(BM25\+\+BGE\) ranges from\+0\.027\+0\.027to\+0\.071\+0\.071across corpora; Mind2Web has the smallest oracle headroom because its larger and more diverse candidate pool diffuses min\-rank gains\.

Table 4:Cross\-corpus LCC and oracle ESH\. BM25 isR​@​1R@1; ESHorisESHoracle\\textrm\{ESH\}^\{\\textrm\{oracle\}\}\(BM25\+\+BGE\)\.
### 4\.4Robustness and fine\-tuning control

#### Fine\-tuning control\.

As a task\-supervised positive control, we fine\-tune MiniLM\-L6\-v2 for one contrastive epoch on AndroidControl and evaluate it on the held\-out2,9492\{,\}949\-row clean\-main test subset under the same ranking protocol\. The fine\-tuned encoder reachesR​@​1=0\.694R@1\{=\}0\.694overall \(\+13\.6\+13\.6pp over BM25,\+17\.6\+17\.6pp over zero\-shot MiniLM\) and recovers37\.4%37\.4\\%of BM25\-miss rows at rank 1\. However, its hits remain predictable from lexical controls \(LCC0\.8770\.877\), so task supervision improves the ranker without removing the lexical\-coupling pattern\. ItsR​@​1R@1also drops from0\.6940\.694to0\.1510\.151under the target\-text dependence perturbation \(Appendix[K](https://arxiv.org/html/2608.21794#A11), Tables[11](https://arxiv.org/html/2608.21794#A11.T11)–[12](https://arxiv.org/html/2608.21794#A11.T12)\), showing that the supervised gains remain strongly dependent on target\-exposed text\.

#### Ruling out annotation noise and generic encoder weakness\.

Two controls bound this interpretation: BM25 still leads on reviewed AndroidControl\-Curated rows, while all five encoders strongly outperform Jaccard on STS\-B\. Thus the pattern is not explained by annotation noise or generic encoder weakness; full counts and STS\-B breakdowns are in Appendices[B](https://arxiv.org/html/2608.21794#A2)and[M](https://arxiv.org/html/2608.21794#A13)\.

### 4\.5Residual signal and deployable fusion

#### Conditional recovery\.

Embeddings recover1414–16%16\\%of clean\-main BM25\-miss rows at rank 1, concentrated on label\-rich targets \(BGE recovers23\.3%23\.3\\%of both\-labelled misses\); they agree with BM25 on80\.9%80\.9\\%of BM25\-hit rows: redundant on easy cases, disagreeing only on a non\-trivial residual\.

#### Oracle vs\. deployable fusion\.

The min\-rank oracle over BM25, Jaccard, and all five main encoders attainsR​@​1=0\.706R@1\{=\}0\.706on clean\-main \(\+0\.151\+0\.151over BM25\)\. The lift is concentrated where labels expose multiple affordances:\+0\.227\+0\.227on both,\+0\.210\+0\.210on cd\-only, and\+0\.119\+0\.119on text\-only\. True RRF over the same candidate\-level ranks leaves most oracle headroom unrealised \(Appendix[I](https://arxiv.org/html/2608.21794#A9), Figure[4](https://arxiv.org/html/2608.21794#A9.F4)\): single\-embedding pairs BM25\+\+BGE, BM25\+\+E5, and BM25\+\+Qwen3 gain only\+0\.012\+0\.012,\+0\.011\+0\.011, and\+0\.005\+0\.005R​@​1R@1, respectively, and the lexical\-plus\-five\-embedding combiner whose oracle lifts by\+0\.151\+0\.151instead*decreases*R​@​1R@1by0\.0090\.009under deployable RRF\. The largest deployable top\-1 fusion gain remains the lexical pair BM25\+\+Jaccard \(R​@​1=0\.578R@1\{=\}0\.578,\+0\.023\+0\.023\[\+0\.021,\+0\.025\+0\.021,\+0\.025\] over BM25\); the full oracle\-vs\-RRF table is in Appendix[I](https://arxiv.org/html/2608.21794#A9)\. Oracle headroom identifies complementary signal but substantially overstates deployable fusion gains; validation\-tuned score fusion reaches the same conclusion \(Appendix[J](https://arxiv.org/html/2608.21794#A10)\)\.

## 5Discussion

#### Two theses\.

The results support two claims\. Off\-the\-shelf sentence\-embedding similarity is not a sufficient grounding proxy without lexical controls\. Its top\-1 behaviour is largely predictable from lexical rank and candidate structure, and deployable RRF gains remain far smaller than target\-aware oracle headroom\. Fine\-tuning sharpens this conclusion\. It substantially improves ranking, ruling out a failure of dense retrieval as a class, but the resulting model remains lexically coupled and label\-sensitive\.

#### Similarity alone is not grounding evidence\.

Sentence embeddings contribute real signal to GUI grounding, but on their own their similarity does not establish semantic grounding without lexical controls\. Across the corpora, instruction–label overlap explains much of top\-1 target recovery because users name elements by visible words that UI metadata re\-exposes as text or accessibility labels\.LCC​\(BGE\-large\)\\textrm\{LCC\}\(\\text\{BGE\-large\}\)remains in\[0\.85,0\.88\]\[0\.85,0\.88\]across mobile and web \(Table[4](https://arxiv.org/html/2608.21794#S4.T4)\), so this is a property of GUI candidate spaces and not of any one corpus\. In that setting, a high embedding score can mean that a model recovered the visible label, not that it inferred the target from task semantics, screen context, or affordance\. Removing only the target candidate’s visible label, while leaving non\-target labels intact, dropsR​@​1R@1by2222–3838pp across BM25 and the embeddings on the same equal\-allocation stratified subset \(Appendix[G](https://arxiv.org/html/2608.21794#A7)\)\. This controlled drop shows that exposed target text supports a substantial share of top\-1 recovery\. Together with the lexical baselines, label\-type stratification, and LCC results, it establishes visible\-label recovery as a major component of performance in this evaluation setting\. This matters because many GUI\-agent evaluations use alignment scores as evidence that an agent understands an instruction\. Our results show that such a construct\-validity claim requires evidence beyond embedding distance alone\.

#### Embeddings still carry residual signal\.

Embeddings recover∼15%\\sim 15\\%of BM25\-miss rows and improve deeper\-recall coverage, but standard RRF captures only a small part of this residual and aggregate combiners can lose top\-1 accuracy, a pattern we name destructive fusion \(§[3\.7](https://arxiv.org/html/2608.21794#S3.SS7)\)\. Highly correlated, lexically coupled retrievers may aggregate without adding enough target information, while their disagreements on non\-target candidates can accumulate\. The gap between oracle and deployable headroom is itself useful: it separates the existence of residual non\-lexical signal from the claim that a practical system can exploit it\.

#### Two deployment paths\.

For deployed GUI agents, off\-the\-shelf embeddings should not be the sole top\-1 ranker\. Two paths follow: lexical retrieval followed by semantic or multimodal reranking, and task\-specific fine\-tuning, where even a small encoder escapes the off\-the\-shelf ceiling on AndroidControl \(§[4\.4](https://arxiv.org/html/2608.21794#S4.SS4.SSS0.Px1)\)\. The same lexical\-control argument extends to multimodal GUI evaluations: improvements on text\-label\-heavy benchmarks should be compared against text\-only lexical retrieval before they count as visual grounding\.

#### Reporting recommendations\.

Among the controls we examine, label\-type stratification is the most revealing: aggregate scores merge two qualitatively different regimes\. When targets expose readable labels, lexical retrieval is already strong; when they do not, both lexical and text\-only embedding methods perform poorly\. We therefore recommend that text\-metadata GUI grounding evaluations report \(1\) lexical baselines using the same candidate pool and element textualization; \(2\) performance stratified by target label type; \(3\) results by candidate\-pool\-size bucket; \(4\) target\-text dependence perturbations where feasible; and \(5\) deployable fusion gains separately from target\-aware oracle headroom\. Without these controls, aggregate scores can be dominated by label\-explicit cases and obscure failures where the exposed candidate text is insufficient\.

#### Benchmark\-design recommendations\.

Lexical coupling should also inform future corpus construction\. Demonstration\-style annotation can encourage label\-explicit instructions: for example, an annotator may write “tap Settings” while viewing a button labeled “Settings\.” Future benchmarks should therefore \(1\) annotate how directly each instruction mentions or paraphrases the target’s exposed label; \(2\) balance label\-explicit examples with cases in which identifying the target requires broader task or screen context, including visual evidence where relevant; and \(3\) where feasible, collect paired instructions for the same screen and target, one with and one without a direct label mention\. Such native pairs complement post\-hoc masking by separating direct label recovery from grounding that relies on broader contextual evidence\.

## Limitations

Our protocol targets text\-metadata candidate\-ranking evaluations with enumerable UI\-element candidate pools \(AndroidControl, MoTIF, Mind2Web\), where elements are represented by accessibility\-tree or DOM text; it does not cover screenshot\-, OCR\-, or layout\-only visual grounding without additional perception controls\. Image\-and\-bounding\-box benchmarks such as ScreenSpot\([3](https://arxiv.org/html/2608.21794#bib.bib4)\)and its OS\-Atlas refinement\([21](https://arxiv.org/html/2608.21794#bib.bib5)\)require OCR or UI parsing, which introduces a separate perception variable\.

Our claim is bounded to grounding\-metric validity rather than downstream behavioural\-difficulty prediction: step\-level embedding distances do not predict local repair actions on AndroidControl, where repair signals are sparse by design \(Appendix[N](https://arxiv.org/html/2608.21794#A14)\)\.

Our conclusions concern off\-the\-shelf single\-vector sentence encoders under candidate\-ranking evaluation\. Reciprocal\-rank fusion and a validation\-tuned linear score\-fusion stress test leave most oracle headroom unrealised \(Appendix[J](https://arxiv.org/html/2608.21794#A10)\); richer learned combiners remain future work\.

One epoch of contrastive fine\-tuning substantially improves a 22M encoder on the same protocol \(§[4\.4](https://arxiv.org/html/2608.21794#S4.SS4.SSS0.Px1)\); stronger visual representations or end\-to\-end agent training may change absolute performance\. The measurement recommendation remains: read such gains relative to lexical controls and a label\-type breakdown\.

## References

- A\. Burns, D\. Arsan, S\. Agrawal, R\. Kumar, K\. Saenko, and B\. A\. PlummerA dataset for interactive vision\-language navigation with unknown command feasibility\.InEuropean Conference on Computer Vision,pp\. 312–328\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p2.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.21794#S3.SS1.SSS0.Px4.p1.1)\.
- Ceret al\.\(2017\)D\. Cer, M\. Diab, E\. Agirre, I\. Lopez\-Gazpio, and L\. SpeciaSemEval\-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation\.InProceedings of the 11th international workshop on semantic evaluation \(SemEval\-2017\),pp\. 1–14\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p2.1),[§3\.2](https://arxiv.org/html/2608.21794#S3.SS2.SSS0.Px2.p1.1)\.
- Chenget al\.\(2024\)K\. Cheng, Q\. Sun, Y\. Chu, F\. Xu, L\. YanTao, J\. Zhang, and Z\. WuSeeclick: harnessing gui grounding for advanced visual gui agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9313–9332\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p1.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2608.21794#Sx1.p1.1)\.
- Cormacket al\.\(2009\)G\. V\. Cormack, C\. L\. Clarke, and S\. BuettcherReciprocal rank fusion outperforms condorcet and individual rank learning methods\.InProceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval,pp\. 758–759\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px3.p1.1),[§3\.6](https://arxiv.org/html/2608.21794#S3.SS6.SSS0.Px2.p1.2),[§3\.7](https://arxiv.org/html/2608.21794#S3.SS7.SSS0.Px2.p1.2)\.
- Denget al\.\(2023\)X\. Deng, Y\. Gu, B\. Zheng, S\. Chen, S\. Stevens, B\. Wang, H\. Sun, and Y\. SuMind2Web: towards a generalist agent for the web\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 28091–28114\.External Links:[Document](https://dx.doi.org/10.52202/075280-1220)Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p1.1),[§1](https://arxiv.org/html/2608.21794#S1.p2.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.21794#S3.SS1.SSS0.Px5.p1.1)\.
- Formalet al\.\(2021\)T\. Formal, B\. Piwowarski, and S\. ClinchantSPLADE: sparse lexical and expansion model for first stage ranking\.InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval,External Links:[Document](https://dx.doi.org/10.1145/3404835.3463098)Cited by:[Appendix P](https://arxiv.org/html/2608.21794#A16.p1.1)\.
- Geirhoset al\.\(2020\)R\. Geirhos, J\. Jacobsen, C\. Michaelis, R\. Zemel, W\. Brendel, M\. Bethge, and F\. A\. WichmannShortcut learning in deep neural networks\.Nature Machine Intelligence2\(11\),pp\. 665–673\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px2.p1.1)\.
- Gouet al\.\(2025\)B\. Gou, D\. R\. Wang, B\. Zheng, Y\. Xie, C\. Chang, Y\. Shu, H\. Sun, and Y\. SuNavigating the digital world as humans do: universal visual grounding for gui agents\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 30851–30883\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p1.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1)\.
- Gururanganet al\.\(2018\)S\. Gururangan, S\. Swayamdipta, O\. Levy, R\. Schwartz, S\. Bowman, and N\. A\. SmithAnnotation artifacts in natural language inference data\.InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 \(Short Papers\),pp\. 107–112\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px2.p1.1)\.
- Honget al\.\(2024\)W\. Hong, W\. Wang, Q\. Lv, J\. Xu, W\. Yu, J\. Ji, Y\. Wang, Z\. Wang, Y\. Dong, M\. Ding,et al\.Cogagent: a visual language model for gui agents\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 14281–14290\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p1.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1)\.
- Jandialet al\.\(2026\)S\. Jandial, Y\. Li, J\. Wagle, and K\. KoishidaDo gui grounders truly understand ui elements?\.InFindings of the Association for Computational Linguistics: EACL 2026,pp\. 2772–2785\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px4.p1.1)\.
- Karpukhinet al\.\(2020\)V\. Karpukhin, B\. Oguz, S\. Min, P\. Lewis, L\. Wu, S\. Edunov, D\. Chen, and W\. YihDense passage retrieval for open\-domain question answering\.InProceedings of the 2020 conference on empirical methods in natural language processing \(EMNLP\),pp\. 6769–6781\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px3.p1.1)\.
- Leunget al\.\(2025\)H\. F\. Leung, X\. Xi, and F\. ZuoAndroidControl\-curated: revealing the true potential of gui agents through benchmark purification\.arXiv preprint arXiv:2510\.18488\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p2.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px4.p1.1),[§3\.2](https://arxiv.org/html/2608.21794#S3.SS2.SSS0.Px1.p1.1)\.
- Liet al\.\(2024\)W\. Li, W\. Bishop, A\. Li, C\. Rawles, F\. Campbell\-Ajala, D\. Tyamagundlu, and O\. RivaOn the effects of data scale on UI control agents\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 92130–92154\.External Links:[Document](https://dx.doi.org/10.52202/079017-2925)Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p1.1),[§1](https://arxiv.org/html/2608.21794#S1.p2.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1),[§3\.1](https://arxiv.org/html/2608.21794#S3.SS1.SSS0.Px1.p1.1)\.
- McCoyet al\.\(2019\)R\. T\. McCoy, E\. Pavlick, and T\. LinzenRight for the wrong reasons: diagnosing syntactic heuristics in natural language inference\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 3428–3448\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px2.p1.1)\.
- Messick \(1995\)S\. MessickValidity of psychological assessment: validation of inferences from persons’ responses and performances as scientific inquiry into score meaning\.\.American psychologist50\(9\),pp\. 741\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p2.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px2.p1.1)\.
- Muennighoffet al\.\(2023\)N\. Muennighoff, N\. Tazi, L\. Magne, and N\. ReimersMteb: massive text embedding benchmark\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 2014–2037\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px3.p1.1)\.
- Reimers and Gurevych \(2019\)N\. Reimers and I\. GurevychSentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing \(EMNLP\-IJCNLP\),pp\. 3982–3992\.Cited by:[§3\.5](https://arxiv.org/html/2608.21794#S3.SS5.SSS0.Px4.p1.1)\.
- Robertson and Walker \(1994\)S\. E\. Robertson and S\. WalkerSome simple effective approximations to the 2\-poisson model for probabilistic weighted retrieval\.InSIGIR’94: Proceedings of the Seventeenth Annual International ACM\-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University,pp\. 232–241\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px3.p1.1)\.
- Thakuret al\.\(2021\)N\. Thakur, N\. Reimers, A\. Rücklé, A\. Srivastava, and I\. GurevychBEIR: a heterogeneous benchmark for zero\-shot evaluation of information retrieval models\.InProceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks,Vol\.1\.Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px3.p1.1)\.
- Wuet al\.\(2025\)Z\. Wu, Z\. Wu, F\. Xu, Y\. Wang, Q\. Sun, C\. Jia, K\. Cheng, Z\. Ding, L\. Chen, P\. P\. Liang,et al\.OS\-atlas: foundation action model for generalist gui agents\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 5090–5108\.Cited by:[§1](https://arxiv.org/html/2608.21794#S1.p1.1),[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1),[Limitations](https://arxiv.org/html/2608.21794#Sx1.p1.1)\.
- Xiaoet al\.\(2024\)S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. NieC\-pack: packed resources for general chinese embeddings\.Cited by:[§3\.5](https://arxiv.org/html/2608.21794#S3.SS5.SSS0.Px4.p1.1)\.
- Xuet al\.\(2025\)Y\. Xu, X\. Liu, X\. Sun, S\. Cheng, H\. Yu, H\. Lai, S\. Zhang, D\. Zhang, J\. Tang, and Y\. DongAndroidLab: training and systematic benchmarking of android autonomous agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 2144–2166\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.107)Cited by:[§2](https://arxiv.org/html/2608.21794#S2.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, M\. Li, D\. Long, X\. Zhang, H\. Lin, B\. Yang, P\. Xie, A\. Yang, D\. Liu, J\. Lin, F\. Huang, and J\. ZhouQwen3 embedding: advancing text embedding and reranking through foundation models\.External Links:2506\.05176Cited by:[§3\.5](https://arxiv.org/html/2608.21794#S3.SS5.SSS0.Px4.p1.1)\.

## Appendix overview

The appendix is organised as follows\. Appendices[A](https://arxiv.org/html/2608.21794#A1)–[D](https://arxiv.org/html/2608.21794#A4)cover reproducibility: artifact licensing and the data statement, data preprocessing per corpus, LCC/ESH thresholds and rank\-caching, and full statistical\-inference and implementation details\. Appendices[E](https://arxiv.org/html/2608.21794#A5)–[K](https://arxiv.org/html/2608.21794#A11)and[M](https://arxiv.org/html/2608.21794#A13)provide supplementary tables and analyses underlying the main results, including paired BM25\-vs\-encoder deltas, label\-type heatmaps, the target\-text dependence perturbation, LCC cross\-validated AUC, oracle min\-rank vs\. RRF, calibrated fusion, fine\-tuning breakdowns, and STS\-B per\-encoder Spearman\. Appendices[L](https://arxiv.org/html/2608.21794#A12)and[N](https://arxiv.org/html/2608.21794#A14)document the construct\-validity boundary: theM3M\_\{3\}\-to\-Jaccard convergence and the behavioural validity limit our claims respect\. Appendix[O](https://arxiv.org/html/2608.21794#A15)provides a qualitative error analysis with de\-identified examples for the three ranking regimes discussed in §[4](https://arxiv.org/html/2608.21794#S4); Appendix[P](https://arxiv.org/html/2608.21794#A16)adds Qwen3 paired intervals and the sparse\-retrieval stress test\.

## Appendix AArtifact use, license, and data statement

We use five public research datasets \(AndroidControl, MoTIF, Mind2Web, AndroidControl\-Curated, and STS\-B\), five public sentence\-embedding checkpoints \(MiniLM, MPNet, BGE, E5, and Qwen3\-Embedding\-0\.6B\), and one public sparse\-retrieval checkpoint \(SPLADE\)\. We cite their creators in Sections[2](https://arxiv.org/html/2608.21794#S2)and[3](https://arxiv.org/html/2608.21794#S3)and Appendix[P](https://arxiv.org/html/2608.21794#A16), and use each only for the evaluation purpose it was released for\.

We do not redistribute the original datasets or checkpoints\. We release only derived per\-step evaluation panels \(join keys and numeric/categorical fields\), analysis scripts, and summary reports, under the MIT License and subject to the terms of the original artifacts; to recover the raw inputs, users fetch the corpora and checkpoints from their official sources\. The code and panels are available in the project repository linked in the abstract\.

Some source screens carry third\-party content; AndroidControl Gmail screens, for instance, can expose user email addresses\. We therefore strip all free\-text columns \(instructions, element descriptions, DOM text, STS sentence pairs\) from the released panels, keeping only join keys and numeric/categorical fields, and every numeric result in the paper is reproducible from these de\-texted panels\. We do not otherwise screen the corpora for offensive content beyond their providers’ curation\.

## Appendix BElement textualisation and per\-corpus preprocessing

#### AndroidControl\.

The corpus provides 83,848 actions across 8 typed action classes \(click,long\_press,input\_text,scroll,wait,navigate\_back,navigate\_home,open\_app\)\. We read all 20 TFRecord shards under the official 13,603/137/1,543\-episode train/val/test split\. For each tap/long\-press/text step we parse the accessibility tree of the on\-screen frame and enumerate every visible UI element with non\-empty bounds\. The target element is the smallest\-area visible element whose bounding box contains the recorded action coordinate\. When multiple boxes coincide, we tie\-break deterministically on tree\-traversal index\. The join key\(episode\_id,step\_index\)\(\\texttt\{episode\\\_id\},\\texttt\{step\\\_index\}\)is exact and verified between the action trace and the accessibility\-tree screen extracts\.

#### Diagnostic subsets\.

Theinput\_textdiagnostic subset \(n=6,033n\{=\}6\{,\}033; excluded from clean\-main\) is included as an illustration of target\-resolution failure: DB target resolution for text\-entry actions lands on the system keyboard rather than the input field, so every retriever scores it as if it were a different target\. All retrievers fall*below*random on this subset: randomR​@​1=0\.034R@1\{=\}0\.034, BM25R​@​1=0\.021R@1\{=\}0\.021, BGE\-largeR​@​1=0\.004R@1\{=\}0\.004\. The clean\-no\-dup variant \(n=26,832n=26\{,\}832\) removes within\-episode duplicate step instructions, which arise when annotators recycle the previous step’s instruction for awaitaction and would inflate spurious M3–friction correlations because the recycled instruction no longer describes the visible target\.

#### Mind2Web DOM parsing\.

For each candidate identified bybackend\_node\_idin the releasedcleaned\_html, we aggregate descendant<text\>content,aria\-label,alt,placeholder, andtitleattributes into a textual representation\. The label\-type taxonomy maps as text\-only = visible inner text with no aria\-label; aria\-only = aria\-label / title / placeholder only; both = both; id/class\-only = only DOMidorclass; none = tag only\. After filtering steps with zero or multiple positive candidates the usable set contains 5,941 rows \(test\_domain3,780,test\_task1,225,test\_website936\); the label\-rich subset \(text\- or aria\-labelled targets, used as “M2W rich” in Table[4](https://arxiv.org/html/2608.21794#S4.T4)\) contains 5,467 rows\.

#### AndroidControl\-Curated flags\.

We use the released AndroidControl\-Curated quality flags to test whether the BM25 lead is driven by annotation noise\. Of 8,377 Curated rows, 2,946 fall in our clean\-main subset; the relevant flags cover task rewrites \(527 full; 144 clean\-main\), candidate\-disagreement review \(1,272 full; 274 clean\-main\), and action\-text typo fixes \(86 full; 0 clean\-main\)\. PairedR​@​1R@1deltas of BM25 over BGE\-large on these rows \(cluster bootstrap by episode,B=2000B\{=\}2000\) are: overall\+4\.4\+4\.4pp \[\+2\.9,\+6\.0\+2\.9,\+6\.0\], candidate\-disagreement review\+8\.8\+8\.8pp \[\+3\.3,\+14\.3\+3\.3,\+14\.3\], and task rewrites\+6\.3\+6\.3pp \[\+0\.0,\+12\.9\+0\.0,\+12\.9\] \(lower bound at zero under the smaller subset\)\.

#### Element textualisation stoplist\.

Queries and candidates use lowercased alphanumeric tokens with a short UI stoplist, so lexical baselines are not boosted by generic UI verbs or class names that appear in both query and candidate by construction\.

## Appendix CLCC/ESH thresholds and rank\-caching requirements

We adopt three reading thresholds, calibrated against the cross\-corpus patterns in §[4](https://arxiv.org/html/2608.21794#S4)\. We use these thresholds as descriptive reading aids, not as hypothesis tests:

- •LCC​\(M\)\>0\.85\\textrm\{LCC\}\(M\)\>0\.85: hit@1 outcomes are discriminated from misses with AUC above0\.850\.85by the lexical and structural controls, without reference to the encoder\.
- •ESHoracle​\(𝒮\)−ESHreal​\(𝒮\)≥0\.05\\textrm\{ESH\}^\{\\textrm\{oracle\}\}\(\\mathcal\{S\}\)\-\\textrm\{ESH\}^\{\\textrm\{real\}\}\(\\mathcal\{S\}\)\\geq 0\.05at R@1: in\-principle complementary signal is largely unrealisable by standard RRF: the embeddings carry information BM25 lacks but RRF cannot cheaply extract it\.
- •ESHreal​\(𝒮\)≤0\\textrm\{ESH\}^\{\\textrm\{real\}\}\(\\mathcal\{S\}\)\\leq 0at R@1: destructive fusion; adding𝒮\\mathcal\{S\}to BM25 under RRF hurts top\-1 accuracy\.

LCC needs only the per\-step target rank under each retriever\.ESHreal\\textrm\{ESH\}^\{\\textrm\{real\}\}additionally requires per\-candidate per\-method ranks at compute time \(needed to compute each candidate’s RRF score\), though only the target’s combined rank is retained afterwards\. Both diagnostics extend to any GUI grounding benchmark with element\-level ground truth, a candidate pool, and at least one lexical baseline\.

## Appendix DStatistical inference and implementation details

#### Models, hyperparameters, and compute\.

The encoders are off\-the\-shelf checkpoints used without modification: MiniLM \(all\-MiniLM\-L6\-v2, 22M\), MPNet \(all\-mpnet\-base\-v2, 110M\), BGE \(BAAI/bge\-large\-en\-v1\.5, 335M\), E5 \(intfloat/e5\-large\-v2, 335M\), and Qwen3\-Embedding\-0\.6B \(Qwen/Qwen3\-Embedding\-0\.6B, 0\.6B\)\. BM25 uses standard, untuned parametersk1=1\.5k\_\{1\}\{=\}1\.5,b=0\.75b\{=\}0\.75, and reciprocal\-rank fusion uses the standard constantk=60k\{=\}60\(not tuned\)\. We perform no hyperparameter search for the zero\-shot retrievers\. For Qwen3, queries use the model\-card templateInstruct: Given a GUI instruction, retrieve the target UI element\.followed byQuery: \{instruction\}; candidates use the same UI\-element textualisation as all other retrievers and no candidate\-side instruction\. We use the checkpoint’s default tokenizer truncation andSentenceTransformer\.encode\(\.\.\., normalize\_embeddings=True\), then score candidates by dot product over the normalised vectors\. The single training run is a one\-epoch contrastive fine\-tune of MiniLM\-L6\-v2 withMultipleNegativesRankingLoss, batch size6464, learning rate2​e−52\\mathrm\{e\}\{\-\}5,10%10\\%warmup, and up to three same\-screen hard negatives per example\. All experiments run on a single consumer\-grade GPU with FP16 inference; the workload is dominated by embedding extraction and per\-screen candidate ranking \(inference only\), with the one\-epoch fine\-tune as the sole training cost, for a total on the order of a few GPU\-hours\. Encoders are loaded and fine\-tuned with thesentence\-transformerslibrary; Mind2Web DOM parsing usesBeautifulSoup; bootstrap inference and correlations useNumPy/SciPy; BM25 is our own implementation with the parameters above\. A pinned dependency list ships with the released code\.

All paired comparisons \(single\-retriever deltas, fusion deltas versus BM25, and partial\-correlation deltas\) use a paired cluster bootstrap withB=2000B=2000resamples byepisode\_idon AndroidControl and MoTIF, and byannotation\_idon Mind2Web \(matching the corpus’s unit of within\-cluster correlation\)\. Reporting by episode/annotation respects the non\-independence between consecutive steps of the same task\. 95% confidence intervals are reported as percentile intervals; we say an effect “actively excludes zero” when both endpoints lie on the same side of zero\. Per\-step Spearman and Pearson correlations use the standard formulae without further adjustment; we never report app\-value without the corresponding effect\-size CI\.

All ranking ties are broken with a hash\-based deterministic jitter keyed on the episode id, step index, retriever, and candidate index so that rank assignment is independent of traversal order and reproducible across subsets\. Random seeds, the cluster\-bootstrap RNG seed, and the BM25 hyperparameters are fixed; the full pipeline is released alongside the per\-step CSV panels \(license and artifact terms in Appendix[A](https://arxiv.org/html/2608.21794#A1)\)\.

#### BM25 IDF on small per\-screen collections\.

Each screen is treated as its own BM25 collection, so IDF is estimated from a median of 62 candidates and is necessarily coarse\. This does not drive the lexical advantage\. In the limit of uninformative IDF, BM25 reduces to length\-normalised query–candidate token overlap, the same surface\-label signal Jaccard captures\. Jaccard tracks BM25 to within a few points ofR​@​1R@1on every corpus \(Tables[1](https://arxiv.org/html/2608.21794#S4.T1),[3](https://arxiv.org/html/2608.21794#S4.T3)\)\. The lexical lead therefore reflects instruction–label overlap rather than precise IDF calibration\.

## Appendix EPaired BM25\-vs\-encoder deltas on AndroidControl

Table[5](https://arxiv.org/html/2608.21794#A5.T5)reports the paired cluster\-bootstrap deltas of BM25 over each encoder on clean\-main, summarised in §[4\.1](https://arxiv.org/html/2608.21794#S4.SS1)\.

Table 5:Paired cluster\-bootstrap deltas of BM25 over each encoder on clean\-main \(n=32,646n\{=\}32\{,\}646;B=2000B\{=\}2000byepisode\_id\)\. Positive favours BM25; all CIs exclude zero\.
## Appendix FLabel\-type collapse heatmap

![Refer to caption](https://arxiv.org/html/2608.21794v1/figure_3_label_type_collapse.png)Figure 3:Per\-cellR​@​1R@1on AndroidControl \(all DB rows,n=58,078n\{=\}58\{,\}078; six\-way label\-type partition\), MoTIF \(all steps,n=22,427n\{=\}22\{,\}427; four\-way partition: text\-only, resource\-only, pseudo, none\), and Mind2Web \(all test splits,n=5,941n\{=\}5\{,\}941; five\-way label\-type partition\)\.
## Appendix GTarget\-text dependence perturbation

To quantify how strongly target ranking depends on text exposed by the gold candidate, we run an asymmetric target\-label ablation on AndroidControl clean\-main\. We use equal allocation across three label strata, sampling1,0001\{,\}000text\_only,1,0001\{,\}000cd\_only, and1,0001\{,\}000bothrows\. For each row, we blank only the target candidate’stextandcontent\_description; the query, candidate pool, non\-target candidates, targetresource\_id, and target class name remain fixed\. The target textualisation changes for2,998/3,0002\{,\}998/3\{,\}000rows\. The resultingR​@​1R@1drops are large for BM25 and all five single\-vector encoders\. The delta measures dependence on target\-text availability\. Because a UI label carries both lexical and semantic information, the perturbation is most informative as part of the full audit suite, alongside the lexical baselines, label\-type stratification, and LCC\. After masking, candidate textualisation falls back to the target’sresource\_idor class name\. Resource IDs can retain label\-related tokens, so the perturbation conservatively removes the human\-facing fields rather than every potentially informative target string\. The aggregate gives equal weight to the three sampled label strata and is a diagnostic average rather than a corpus\-prevalence\-weighted estimate\.

Table 6:Target\-text dependence on an equal\-allocation stratified AndroidControl clean\-main sample \(n=3,000n\{=\}3\{,\}000;1,0001\{,\}000per label stratum\)\. Negative deltas measure the loss in top\-1 recovery when the gold candidate’s exposed label is removed while the query and non\-target candidates stay fixed\.
## Appendix HLCC cross\-validated AUC

We report five\-fold grouped cross\-validated AUC of the LCC logistic regression alongside the in\-sample AUC used in Table[4](https://arxiv.org/html/2608.21794#S4.T4)\. Folds are split byepisode\_idon AndroidControl and MoTIF, and byannotation\_idon Mind2Web \(matching the cluster unit used elsewhere for these corpora\), so that no episode / annotation appears in both train and test of any fold\.

Table 7:LCC full\-data AUC vs\. 5\-fold episode\-grouped CV AUC\. The two diverge by at most0\.0130\.013\(Qwen3\-Embedding\-0\.6B, M2W all\), confirming that the lexical coupling signal is not an in\-sample artefact\.
## Appendix IOracle min\-rank vs\. true RRF: full table

Table[8](https://arxiv.org/html/2608.21794#A9.T8)gives the full candidate\-level oracle\-vs\-RRF comparison summarised in §[4\.5](https://arxiv.org/html/2608.21794#S4.SS5)and plotted in Figure[4](https://arxiv.org/html/2608.21794#A9.F4)\. Oracle min\-rank uses the privileged target rank per row to upper\-bound any combiner; true RRF combines per\-candidate ranks and is therefore deployable\. The two diverge sharply: oracle liftsR​@​1R@1by up to\+0\.151\+0\.151, while the same combiner under true RRF*loses*R​@​1R@1\. Single\-embedding RRF pairs with BGE, E5, and Qwen3 give only small deployable gains, and the largest deployable top\-1 lift comes from the lexical pair BM25\+\+Jaccard\.

Figure 4:Oracle min\-rank \(ESHoracle\\textrm\{ESH\}^\{\\textrm\{oracle\}\}, grey\) vs\. deployable reciprocal\-rank fusion \(ESHreal\\textrm\{ESH\}^\{\\textrm\{real\}\}, red\)Δ​R​@​1\\Delta R@1over BM25 on AndroidControl clean\-main\. Error bars showB=2000B\{=\}2000cluster\-bootstrap95%95\\%CIs for deployable RRF deltas; oracle bars are target\-aware upper bounds\.Table 8:Oracle min\-rank vs\. true RRF on AndroidControl clean\-main \(n=32,646n\{=\}32\{,\}646\)\. True\-RRFΔ\\Deltavs BM25 95% CIs are paired cluster bootstrap byepisode\_idwithB=2000B\{=\}2000\. “5 emb\.” denotes MiniLM, MPNet, BGE\-large, E5\-large, and Qwen3\-Embedding\-0\.6B; “lex” adds Jaccard to BM25\.
## Appendix JValidation\-tuned calibrated fusion stress test

#### Calibrated fusion stress test\.

Beyond the fixed RRF rule used in the main analysis, we test whether a minimal validation\-tuned linear score fusion recovers more of the oracle headroom on AndroidControl clean\-main\. For each screen, raw BM25 scores and embedding cosine scores are z\-normalized over the candidate pool, then combined assα​\(c\)=α​zBM25​\(c\)\+\(1−α\)​zemb​\(c\)s\_\{\\alpha\}\(c\)=\\alpha z\_\{\\mathrm\{BM25\}\}\(c\)\+\(1\-\\alpha\)z\_\{\\mathrm\{emb\}\}\(c\)\. A single interpolation weightα∈\{0,0\.05,…,1\}\\alpha\\in\\\{0,0\.05,\\ldots,1\\\}is selected on the official validation split \(n=261n\{=\}261clean\-main rows\) to maximizeR​@​1R@1, with MRR as a tie\-breaker, and then evaluated on the held\-out test split \(n=2,949n\{=\}2\{,\}949\)\. This is not a learned reranker, but a simple calibration check\.

Table 9:Validation\-tuned calibrated score fusion on AndroidControl clean\-main test \(n=2,949n\{=\}2\{,\}949\)\. Scores are z\-normalized within each candidate pool;α\\alphais selected on validation byR​@​1R@1with MRR as a tie\-breaker\. Oracle rows are target\-aware upper bounds, not deployable systems\. CIs use paired cluster bootstrap by episode withB=2000B\{=\}2000\.Calibrated fusion recovers limited deployable signal across the five main encoders\. Validation\-tuned fusion yields statistically stable gains for MiniLM \(\+2\.27\+2\.27pp, CI\[\+1\.35,\+3\.23\]\[\+1\.35,\+3\.23\]\), MPNet \(\+2\.81\+2\.81pp, CI\[\+1\.89,\+3\.81\]\[\+1\.89,\+3\.81\]\), E5 \(\+1\.46\+1\.46pp, CI\[\+0\.52,\+2\.42\]\[\+0\.52,\+2\.42\]\), and Qwen3 \(\+1\.80\+1\.80pp, CI\[\+0\.87,\+2\.79\]\[\+0\.87,\+2\.79\]\)\. For BGE, validation selectsα=1\.00\\alpha\{=\}1\.00, so the calibrated system reduces exactly to BM25 and yields no held\-out change\. Then=261n\{=\}261validation subset limits the reliability ofα\\alphaselection; this outcome should be read as “no consistent gain detected on validation”, not necessarily “no signal exists”\. All five calibrated results remain far below their corresponding oracle headroom \(\+5\.97\+5\.97to\+6\.88\+6\.88pp\)\. Fixed RRF yields only small held\-out changes \(−0\.54\-0\.54to\+1\.19\+1\.19pp\), with a significant gain only for BM25\+MPNet\. Thus, calibrated fusion supports the same measurement recommendation: oracle headroom and deployable fusion gains should be reported separately\.

## Appendix KFine\-tuning: full breakdown and coupling audit

We fine\-tune MiniLM\-L6\-v2 with a one\-epoch contrastive objective on the AndroidControl training split\. We sample29,42629\{,\}426training examples from29,43629\{,\}436clean\-main train steps; each example carries a query, the target candidate, and up to three same\-screen hard negatives, in addition to the in\-batch negatives provided by MultipleNegativesRankingLoss\. The evaluation protocol mirrors the zero\-shot evaluation: per\-screen candidate ranking on the held\-out2,9492\{,\}949\-row clean\-main test subset\.

Table 10:FullR​@​1R@1breakdown of fine\-tuning evaluation on the AndroidControl test clean\-main subset\.M0M\_\{0\}= zero\-shot MiniLM\-L6\-v2;MftM\_\{\\mathrm\{ft\}\}= same model after one epoch; BGE0= zero\-shot BGE\-large\.We rerun two complementary diagnostics on the same held\-out test rows \(Tables[11](https://arxiv.org/html/2608.21794#A11.T11)–[12](https://arxiv.org/html/2608.21794#A11.T12)\): LCC measures whether supervision changes lexical coupling, while the target\-text perturbation measures whether the improved ranker remains dependent on text exposed by the gold candidate\. Fine\-tuning substantially improves target recovery, including on BM25 misses, but the hit pattern remains highly predictable from lexical and structural controls, and removing the gold candidate’s exposed text sharply reduces top\-1 accuracy\.

Table 11:Fine\-tuned MiniLM coupling audit on the same held\-out AndroidControl clean\-main test rows \(n=2,949n\{=\}2\{,\}949\)\. LCC is 5\-fold episode\-grouped CV AUC using BM25 rank, Jaccard rank, candidate\-pool size, and label type\. BM25\-miss recovery is the fraction of rows with BM25 rank\>1\>1that the method ranks first\.Table 12:Held\-out target\-text dependence perturbation for the fine\-tuning control \(R​@​1R@1\)\. The perturbation blanks the gold candidate’s visible text and content description while retaining the query, non\-target candidates, resource IDs, class names, and candidate pools\. Negative deltas measure sensitivity to target\-exposed text\.
## Appendix LConstruct correlation: M3 converges to Jaccard

Throughout this appendix and Appendix[N](https://arxiv.org/html/2608.21794#A14)we writeM3​\(q,t\)M\_\{3\}\(q,t\)for the per\-step instruction–target cosine distance1−cos⁡\(ϕ⁡\(q\),ϕ⁡\(t\)\)1\-\\mathrm\{cos\}\(\\phi\(q\),\\phi\(t\)\)under a sentence encoderϕ\\phi, andM1M\_\{1\}for its episode\-level analogue \(defined in Appendix[N](https://arxiv.org/html/2608.21794#A14)\)\.M3M\_\{3\}is the step\-level alignment quantity that the Limitations section refers to when bounding our claim to grounding\-metric validity rather than behavioural prediction\.

Beyond ranking, we measure how strongly each embedding family’s step\-level distance to the textualised target correlates with a matching Jaccard baseline\. Three textualisation variants are compared: current \(the form used for ranking\), label\-only \(text \+ content description, no class or action prefix\), and action\-aware \(action verb prepended\)\. For every variant and embedding family we compute the per\-step cosine distance and the per\-step Jaccard distance between the same query and the same textualised target, then report Spearman correlation across the clean\-strict subset of the step\-local panel \(n=32,636n=32\{,\}636steps with a valid M3, target text or content description, and at least one subsequent action\)\.

On this subset, M3 is highly correlated with the matching token\-level Jaccard distance across all three textualisation variants and all five encoder families \(Table[13](https://arxiv.org/html/2608.21794#A12.T13)\); Spearman clusters in\[0\.68,0\.82\]\[0\.68,\\,0\.82\], with action\-aware textualisation producing the tightest coupling for MiniLM, MPNet, and BGE\-large \(E5\-large is the exception, where the*current*textualisation is marginally tighter than action\-aware\)\. Model\-to\-model agreement \(same variant\) is even higher, with pairwise Spearman in\[0\.80,0\.91\]\[0\.80,\\,0\.91\]\.

Table 13:Step\-level M3–Jaccard Spearman \(n=32,636n\{=\}32\{,\}636\)\. All cellsp<10−300p<10^\{\-300\}\.
## Appendix MSTS\-B per\-encoder Spearman

Table 14:Spearman correlations on STS\-B dev\+test \(2,8792\{,\}879pairs\)\. All five encoders beat Jaccard against gold similarity by1616–2424points; embedding–Jaccard agreement remains in\[0\.608,0\.749\]\[0\.608,0\.749\], lower than the corresponding\[0\.68,0\.82\]\[0\.68,0\.82\]range on AndroidControl clean\-strict \(Appendix[L](https://arxiv.org/html/2608.21794#A12)\)\.

## Appendix NBehavioural validity boundary

We test whether instruction\-target alignment metrics predict behavioural friction proxies at both the step and the episode level\.

#### Step\-level friction labels\.

For each step in the no\-duplicate subset, we mark whether either of the next two actions is a back, wait, or scroll action\. We also use two robustness variants: a non\-terminal back\-out label that excludes repeated terminal exits, and a broader friction label that also counts home/open\-app transitions\. For each alignment metric and friction proxy we report marginal Spearman, decile contrasts, and within\-episode pairwise deltas comparing the highest\- and lowest\-M3 steps in the same episode\.

Step\-level M3 variants do not predict local repair signals\. On the no\-duplicate subset \(n=26,832n\{=\}26\{,\}832\), current M3 is near zero against back actions \(−0\.008\-0\.008\[−0\.020,\+0\.004\-0\.020,\+0\.004\]\), non\-terminal back\-outs \(−0\.007\-0\.007\[−0\.018,\+0\.005\-0\.018,\+0\.005\]\), and the combined back/wait/scroll friction proxy \(−0\.009\-0\.009\[−0\.020,\+0\.003\-0\.020,\+0\.003\]\)\. The action\-aware variant is similar in magnitude and direction\. A lexical Jaccard baseline on the same labels is slightly larger but same\-direction \(combined\-friction Spearman−0\.020\-0\.020\[−0\.031,−0\.008\-0\.031,\-0\.008\]\), indicating that the step\-local null is not an embedding\-specific artefact\.

#### Episode\-level mismatch\.

Letggbe an episode’s goal string andL=\(l1,…,ln\)L=\(l\_\{1\},\\dots,l\_\{n\}\)its prospective step instructions\. The full\-TFRecord episode\-level mismatch is

M1​\(g,L\)=1−cos⁡\(ϕ⁡\(g\),ϕ⁡\(l1​‖⋯‖​ln\)\),M\_\{1\}\(g,L\)=1\-\\mathrm\{cos\}\\bigl\(\\phi\(g\),\\,\\phi\(l\_\{1\}\\,\\\|\\,\\cdots\\,\\\|\\,l\_\{n\}\)\\bigr\),\(5\)whereϕ\\phiis the MiniLM\-L6 encoder and∥\\\|denotes whitespace\-joined concatenation\. We computeM1M\_\{1\}over all 15,283 episodes and correlate it against episode\-level back, wait, and scroll frequencies and incidence indicators using marginal Spearman and partial Spearman controlling for episode length, instruction diversity, and goal–step lexical distance\.

FullM1M\_\{1\}has small but bounded positive associations with aggregate friction: marginal Spearman is\+0\.092\+0\.092\[\+0\.077,\+0\.107\+0\.077,\+0\.107\] for back frequency and\+0\.124\+0\.124\[\+0\.108,\+0\.140\+0\.108,\+0\.140\] for scroll frequency\. After the controls, these shrink to\+0\.102\+0\.102\[\+0\.088,\+0\.117\+0\.088,\+0\.117\] and\+0\.038\+0\.038\[\+0\.023,\+0\.053\+0\.023,\+0\.053\], below 0\.15 throughout\. We read this as weak task\-structure signal rather than direct semantic difficulty: fullM1M\_\{1\}correlates with goal–step Jaccard distance at Spearman\+0\.715\+0\.715, and removing the lexical and length components shrinks the residual to small positive values\.

#### Typing\-correction sparsity\.

Typing\-correction events \(consecutive same\-field re\-entries\) occur in only 82 instances across 63 of the 15,283 episodes \(0\.4%0\.4\\%episode incidence\), too sparse to serve as a friction proxy\. This confirms the broader pattern: AndroidControl annotators were instructed to avoid unnecessary or unrelated actions, so behavioural friction surfaces are systematically sparse\.

## Appendix OQualitative error analysis

Table[15](https://arxiv.org/html/2608.21794#A15.T15)reports de\-identified problematic examples from all three datasets\. Rows are selected to illustrate three recurring regimes in the observed ranks: lexical label recovery, residual embedding recovery, and label\-poor failure\. Text fields are paraphrased to avoid exposing user or third\-party dataset content, while ranks and label types are unchanged\. For compactness, rank summaries report BM25 and four representative encoders; all five encoders are covered in the quantitative analyses\.

Table 15:Qualitative error analysis for recurring ranking regimes\. AC abbreviates AndroidControl and M2W abbreviates Mind2Web\. Text is paraphrased to avoid exposing user or third\-party dataset content; the tabulated ranks and label types are unchanged from the source rows\. Lower rank is better; rank 1 indicates the top\-ranked target\.

## Appendix PAdditional embedding and retrieval stress tests

Table[16](https://arxiv.org/html/2608.21794#A16.T16)reports the pairedR​@​1R@1intervals and coupling diagnostics for Qwen3\-Embedding\-0\.6B, which is included in the main single\-vector encoder comparison, and for one appendix\-only broader retrieval stress test\. SPLADE \([6](https://arxiv.org/html/2608.21794#bib.bib23);naver/splade\-cocondenser\-ensembledistil\) is included separately as a learned sparse retriever, not as a main single\-vector encoder baseline\. These rows show that the measurement diagnosis persists for both a more recent single\-vector embedding control and a broader sparse retrieval architecture: Qwen3 remains below BM25 atR​@​1R@1on AndroidControl clean\-main, MoTIF clean\-main, and Mind2Web all test splits, while SPLADE is below BM25 on AndroidControl clean\-main and MoTIF clean\-main and statistically tied with BM25 on Mind2Web all test splits; both remain highly predictable from lexical and candidate controls\.

Table 16:Qwen3 paired intervals and sparse\-retrieval stress test\. Deltas are method minus BM25 atR​@​1R@1; negative deltas mean the method is below BM25\. For SPLADE, Coupling AUC is the same hit@1 predictability diagnostic, not the encoder\-specific LCC definition\.