ECHOES：通过递归历史压缩与特权在线蒸馏洞察当前图像之外

# ECHOES: Seeing Beyond the Current Image via Recursive History Compression and Privileged On-Policy Distillation
Source: [https://arxiv.org/html/2610.22408](https://arxiv.org/html/2610.22408)
Tengda Guo Jiawang Hu Yifei Tao Yin Wang Haotian Hu Fanyi Wang Liujuan TangAffiliation:Tsinghua University StepFun \(Shanghai Jieyue Xingchen Intelligent Technology Co\., Ltd\.\) Affiliation:Nanyang Technological University Zhejiang Leapmotor Technology Co\., Ltd\. Honor

###### Abstract

Long\-horizon GUI agents rely on visual history to maintain task states, but retaining historical screenshots increases visual\-token usage and inference costs, while textual summaries can discard critical state information, undermining long\-horizon reliability\. We propose ECHOES, which leverages training\-time privileged information to guide textual history compression and strengthen long\-horizon reasoning\. Specifically, ECHOES adopts a Compression–Think–Action \(CTA\) format to recursively update a textual history state and reason\. The same model, given privileged multi\-image inputs, serves as a teacher that provides dense token\-level on\-policy distillation on student\-generated CTA rollouts\. Our Perception–Memory Fidelity \(PMF\) Gap\-guided OPD–RL framework dynamically balances privileged distillation and task\-reward optimization, with periodic teacher updates supporting continued improvement\. We further introduce ECHOESBMK, a fine\-grained diagnostic benchmark derived from real GUI agent failure trajectories to jointly evaluate visual perception and cross\-modal state retention\. Experiments on four GUI benchmarks and ECHOESBMK demonstrate substantial gains in task performance and memory fidelity\. Without retaining historical screenshots at deployment, ECHOES achieves competitive performance with a favorable performance–inference\-cost trade\-off and exhibits positive zero\-shot transfer to unseen visual planning tasks\.

## 1Introduction

Recent advances in multimodal large language models \(MLLMs\) have enabled agent systems capable of perceiving, reasoning, and acting in complex visual environments, with increasing success in long\-horizon visual reasoning tasks\([Xu et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib3);[Liu et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib4)\)\. As a representative application, graphical user interface \(GUI\) agents have made substantial progress in single\-step visual decision\-making and grounding\([Cheng et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib1);[Lin et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib2)\), yet remain brittle in long\-horizon control tasks\([Tian et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib6);[Zhou et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib12)\)\. A key cause is state\-retention failure, which causes decisions and execution to drift out of alignment with the actual environment state\. As interaction trajectories grow, long\-horizon GUI agents must therefore not only understand the current interface but also preserve past states that remain relevant to future decisions\. Existing approaches support long\-horizon reasoning by retaining visual history\([Zhou et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib12)\), generating textual summaries\([Liu et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib4)\), or maintaining structured states\([Tian et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib6)\)\. However, these gains exhibit diminishing returns as trajectories lengthen, while accumulating visual tokens, context length, and KV\-cache overhead hinder efficient deployment\([Huang et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib8);[Zhou et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib12)\)\. Long\-horizon visual agents thus face a fundamental tension: additional historical screenshots increase inference costs, whereas directly converting visual history into text can omit critical task states through cross\-modal information loss\. We term this loss the Perception–Memory Fidelity Gap \(PMF Gap\)\. This motivates a different question: Can multi\-image visual history be used only during training to transfer its decision\-making capabilities to a model that relies solely on the current image and low\-cost reasoning over compressed history at deployment?

![Refer to caption](https://arxiv.org/html/2610.22408v1/intro.png)Figure 1:Motivation and method comparison\.\(a\) History\-independent versus history\-dependent decisions\. \(b\) ECHOES’s performance–cost advantage on GUI\-Odyssey\. \(c\) Existing methods versus ECHOES with recursive CTA and training\-time multi\-image supervision\.To address this question, we propose ECHOES, a training method for long\-horizon visual agents, as illustrated in Figure[1](https://arxiv.org/html/2610.22408#S1.F1)\. We model task\-relevant compressed history as a learnable approximate sufficient statistic of multi\-image visual history for subsequent decisions\. We first adopt the Compression–Think–Action \(CTA\) reasoning mode, which continuously maintains long\-horizon task states through recursive Compression\. We then introduce training\-time privileged on\-policy distillation \(OPD\)\([Agarwal et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib9)\), using the stronger multi\-image*privileged view*as the teacher to provide dense token\-level supervision on CTA trajectories generated by the single\-image \+ Compression*deployable view*itself\. This encourages the textual state to progressively preserve information in the full visual history that is relevant to subsequent decisions\. Building on this design, we construct a PMF Gap\-guided OPD–RL training framework that dynamically coordinates the optimization weights of OPD and GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib10)\)across rounds according to the information gap between the privileged and deployable observation modes\. When the gap is large, the framework emphasizes privileged\-policy supervision; as it narrows, task\-reward optimization receives greater weight, gradually shifting learning from absorbing the privileged policy’s information advantage to directly optimizing the actual task objective\. To evaluate GUI agents’ long\-horizon performance at a finer granularity, we additionally introduce ECHOESBMK, a benchmark constructed from real failure trajectories that spans seven task categories across three levels and jointly evaluates perception and cross\-modal information retention\.

Experiments show that ECHOES substantially improves performance on long\-horizon GUI benchmarks and ECHOESBMK with lower inference overhead, while demonstrating strong zero\-shot generalization to non\-GUI visual planning environments\. Further experiments reveal pronounced improvements on longer\-horizon and history\-dependent tasks, and show that these gains stem from the CTA reasoning mode and the training design\. In summary, our main contributions are as follows:

1. 1\.We propose ECHOES, a training\-time privileged OPD method that adopts CTA reasoning and uses additional historical visual information to provide dense supervision, transferring the decision\-making advantage of multi\-image inputs to a low\-cost deployment mode that relies only on the current observation and a recursive textual state\.
2. 2\.We propose a PMF Gap\-guided OPD–RL training framework that dynamically balances privileged distillation and task\-reward optimization according to the privileged–deployable information gap\. A round\-wise self\-evolving teacher continuously strengthens both observation modes, substantially improving long\-horizon visual decision\-making\.
3. 3\.We construct ECHOESBMK, a fine\-grained benchmark derived from real GUI agent failure trajectories, to jointly evaluate visual perception and memory fidelity in the conversion of visual information into textual states\. We further demonstrate the advantages of ECHOES across multiple benchmarks and analyze its performance along multiple dimensions\.

## 2Related Work

Long\-horizon VLM agents require both effective history representations and policies trained for sequential decision\-making\. Beyond visual grounding and action modeling\([Cheng et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib1);[Qin et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib11)\), recent approaches use textual summaries, history\-aware policy optimization, and latent memory to preserve decision\-relevant context\([Liu et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib4);[Zhou et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib12);[Zhang et al\., 2026a](https://arxiv.org/html/2610.22408#bib.bib7)\)\. These approaches address the trade\-off between historical information and inference cost, including learning to align compressed and richer context representations\. In parallel, reinforcement learning improves GUI policies through task rewards\([Lu et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib14);[Luo et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib15)\), while on\-policy distillation supplies teacher feedback on student\-generated sequences\([Agarwal et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib9)\)\. Recent studies further explore target\-region cues and future observations as privileged contexts\([Zhang et al\., 2026c](https://arxiv.org/html/2610.22408#bib.bib19);[Li et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib21)\), as well as combining distillation with reinforcement learning\([Wu et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib20)\)\. Building on these directions, our work targets the capability gap between multi\-image historical observations and single\-image deployment with recursive textual compression\. We use recent visual history as training\-time privileged evidence to supervise complete Compression–Think–Action sequences, jointly guiding state updates, reasoning, and action generation\. With a round\-wise refreshed teacher, we adapt the distillation–RL balance to the measured privileged–deployable performance gap\. This design transfers history\-informed decision\-making into the deployable policy without retaining historical screenshots at inference\. Appendix[E](https://arxiv.org/html/2610.22408#A5)provides an extended discussion\.

## 3Method

### 3\.1Compression–Think–Action Reasoning and Task\-State Modeling

Long\-horizon visual decision\-making requires historical context beyond the current observation, yet accumulating screenshots increases visual\-token usage and inference cost with trajectory length\. Since relevant history varies across decision stages, the model should autonomously select, compress, and reuse historical information rather than rely on a fixed external summarizer\. We therefore adopt Compression–Think–Action \(CTA\), which integrates recursive textual\-state compression, explicit reasoning, and action generation into a single autoregressive sequence, as shown in Figure[2](https://arxiv.org/html/2610.22408#S3.F2)\.

![Refer to caption](https://arxiv.org/html/2610.22408v1/framework_v5.png)Figure 2:Overview of ECHOES\.Given the task, current screenshot, and previousCC, the policy generates a CTA sequence\. Training combines OPD from the same model’s privileged multi\-image view with GRPO under PMF\-guided weighting, refreshing the teacher at round boundaries\.At steptt, the deployment state comprises the task instructionqq, current visual observationoto\_\{t\}, and previous compressionCt−1C\_\{t\-1\}\. Conditioned on this state, the model generates the complete CTA sequence in a single autoregressive rollout:

st=\(q,ot,Ct−1\),\(Ct,Tt,At\)∼πθ\(⋅∣st\)\.s\_\{t\}=\(q,o\_\{t\},C\_\{t\-1\}\),\\qquad\(C\_\{t\},T\_\{t\},A\_\{t\}\)\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\.Here,CtC\_\{t\}is the updated textual task state,TtT\_\{t\}is explicit reasoning, andAtA\_\{t\}belongs to a unified nine\-category GUI action space\. After executingAtA\_\{t\}, the environment returnsot\+1o\_\{t\+1\}, while onlyCtC\_\{t\}is retained across steps\. CTA thus maintains historical context byrecursively rewriting its textual stateinstead of accumulating screenshots\.

Compression aims to preserve decision\-relevant information, not reconstruct the past\. LetHtVH\_\{t\}^\{V\}denote the complete visual interaction history available at steptt\. Ideally, the compressed deployment state should support a decision distribution approaching that conditioned on the full history:

DKL\(p\(Tt,At∣q,HtV\)∥p\(Tt,At∣st\)\)⟶0,\|Ct\|≤B,D\_\{\\mathrm\{KL\}\}\\\!\\left\(p\(T\_\{t\},A\_\{t\}\\mid q,H\_\{t\}^\{V\}\)\\,\\big\\\|\\,p\(T\_\{t\},A\_\{t\}\\mid s\_\{t\}\)\\right\)\\longrightarrow 0,\\qquad\|C\_\{t\}\|\\leq B,whereBBdenotes the textual capacity budget and the second distribution marginalizes over the generatedCtC\_\{t\}\. This design aims to learn how to encode and recursively update historical context inCtC\_\{t\}, such that the resulting joint distribution over reasoning and actions approximates its full\-history counterpart\. Under this budget,CtC\_\{t\}should prioritize task progress, critical state changes, completed operations, and observed execution outcomes, while discarding details irrelevant to future decisions\.

### 3\.2Multi\-Image–Single\-Image Training\-Time Privileged Self\-Distillation

Algorithm 1PMF Gap\-Guided Privileged OPD–RL1:Initial student

θ\\theta, fixed validation set

𝒟val\\mathcal\{D\}\_\{\\mathrm\{val\}\}, rounds

RR, weight bounds

λmin,λmax\\lambda\_\{\\min\},\\lambda\_\{\\max\},

ϵ\>0\\epsilon\>0
2:Trained deployable CTA policy

πθD\\pi\_\{\\theta\}^\{D\}
3:

G0←max⁡\{0,SP​\(θ\)−SD​\(θ\)\}G\_\{0\}\\leftarrow\\max\\\{0,S^\{P\}\(\\theta\)\-S^\{D\}\(\\theta\)\\\}
4:Define

f⁡\(g\)=clip⁡\(λmin\+\(λmax−λmin\)​gG0\+ϵ,λmin,λmax\)f\(g\)=\\operatorname\{clip\}\\\!\\left\(\\lambda\_\{\\min\}\+\(\\lambda\_\{\\max\}\-\\lambda\_\{\\min\}\)\\frac\{g\}\{G\_\{0\}\+\\epsilon\},\\lambda\_\{\\min\},\\lambda\_\{\\max\}\\right\);

λ1←f⁡\(G0\)\\lambda\_\{1\}\\leftarrow f\(G\_\{0\}\)
5:for

r=1,…,Rr=1,\\ldots,Rdo

6:

θ¯←copy⁡\(θ\)\\bar\{\\theta\}\\leftarrow\\operatorname\{copy\}\(\\theta\)⊳\\trianglerightFreeze teacher within round

7:foreach student update in round

rrdo

8:Sample grouped CTA rollouts

Yt=\(Ct,Tt,At\)∼πθD\(⋅∣q,ot,Ct−1\)Y\_\{t\}=\(C\_\{t\},T\_\{t\},A\_\{t\}\)\\sim\\pi\_\{\\theta\}^\{D\}\(\\cdot\\mid q,o\_\{t\},C\_\{t\-1\}\)
9:

pt,iP←πθ¯\(⋅∣q,ot,𝒱t,Yt,<i\),pt,iD←πθ\(⋅∣q,ot,Ct−1,Yt,<i\)p\_\{t,i\}^\{P\}\\leftarrow\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid q,o\_\{t\},\\mathcal\{V\}\_\{t\},Y\_\{t,<i\}\),\\quad p\_\{t,i\}^\{D\}\\leftarrow\\pi\_\{\\theta\}\(\\cdot\\mid q,o\_\{t\},C\_\{t\-1\},Y\_\{t,<i\}\)
10:

ℒOPDCTA←meanvalid CTA tokensDKL\(sg\[pt,iP\]∥pt,iD\)\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\\mathrm\{CTA\}\}\\leftarrow\\operatorname\{mean\}\_\{\\text\{valid CTA tokens\}\}D\_\{\\mathrm\{KL\}\}\\\!\\left\(\\operatorname\{sg\}\[p\_\{t,i\}^\{P\}\]\\,\\\|\\,p\_\{t,i\}^\{D\}\\right\)
11:

θ←Update⁡\(θ,λr​ℒOPDCTA\+\(1−λr\)​ℒGRPOCTA\)\\theta\\leftarrow\\operatorname\{Update\}\\\!\\left\(\\theta,\\,\\lambda\_\{r\}\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\\mathrm\{CTA\}\}\+\(1\-\\lambda\_\{r\}\)\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{CTA\}\}\\right\)
12:endfor

13:

Gr←max⁡\{0,SP​\(θ\)−SD​\(θ\)\},λr\+1←f⁡\(Gr\)G\_\{r\}\\leftarrow\\max\\\{0,S^\{P\}\(\\theta\)\-S^\{D\}\(\\theta\)\\\},\\quad\\lambda\_\{r\+1\}\\leftarrow f\(G\_\{r\}\)
14:endfor

CTA requires compression to preserve historical information needed for decisions\. However, supervision confined to the compressed view does not expose decision\-relevant visual evidence omitted from textual state\. We therefore introduce training\-time privileged self\-distillation based on on\-policy distillation \(OPD\)\([Agarwal et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib9)\), using the same model’s multi\-image view as teacher to guide the deployable single\-image CTA policy\. This observation asymmetry makes visual history a training\-time reference for learning recursive compression and history\-informed decisions, rather than a deployment input\. At steptt, the student and teacher receive:

xtD=\(q,ot,Ct−1\),xtP=\(q,ot,𝒱t\),x\_\{t\}^\{D\}=\(q,o\_\{t\},C\_\{t\-1\}\),\\qquad x\_\{t\}^\{P\}=\(q,o\_\{t\},\\mathcal\{V\}\_\{t\}\),where D and P denote deployable and privileged views, andVtV\_\{t\}contains recent historical screenshots preceding steptt\. The teacher uses the same architecture as the student, exploiting additional visual evidence rather than greater model capacity\. The student samplesYt=\(Ct,Tt,At\)∼πθD\(⋅∣xtD\)Y\_\{t\}=\(C\_\{t\},T\_\{t\},A\_\{t\}\)\\sim\\pi\_\{\\theta\}^\{D\}\(\\cdot\\mid x\_\{t\}^\{D\}\)under deployment inputs\. Both policies are then evaluated on the same student\-generated prefixes via teacher\-forced scoring:

pt,iP=πθ¯\(⋅∣xtP,Yt,<i\),pt,iD=πθ\(⋅∣xtD,Yt,<i\),p\_\{t,i\}^\{P\}=\\pi\_\{\\bar\{\\theta\}\}\\left\(\\cdot\\mid x\_\{t\}^\{P\},Y\_\{t,<i\}\\right\),\\qquad p\_\{t,i\}^\{D\}=\\pi\_\{\\theta\}\\left\(\\cdot\\mid x\_\{t\}^\{D\},Y\_\{t,<i\}\\right\),whereθ¯\\bar\{\\theta\}denotes the teacher parameters andYt,<iY\_\{t,<i\}the student’s token prefix\. We distill all Compression, Think, and Action tokens using

ℒOPDCTA=1N∑i∈ℐC∪ℐT∪ℐADKL\(sg\[pt,iP\]∥pt,iD\),\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\\mathrm\{CTA\}\}=\\frac\{1\}\{N\}\\sum\_\{i\\in\\mathcal\{I\}\_\{C\}\\cup\\mathcal\{I\}\_\{T\}\\cup\\mathcal\{I\}\_\{A\}\}D\_\{\\mathrm\{KL\}\}\\left\(\\operatorname\{sg\}\[p\_\{t,i\}^\{P\}\]\\,\\\|\\,p\_\{t,i\}^\{D\}\\right\),whereℐC\\mathcal\{I\}\_\{C\},ℐT\\mathcal\{I\}\_\{T\}, andℐA\\mathcal\{I\}\_\{A\}denote the corresponding token positions,NNcounts valid distilled tokens, andsg⁡\[⋅\]\\operatorname\{sg\}\[\\cdot\]stops teacher gradients\. SupervisingCtC\_\{t\}guides the retention and recursive updating of historical information, while supervisingTtT\_\{t\}andAtA\_\{t\}aligns reasoning and action generation with the teacher’s richer visual context\. Together, these signals train the deployable CTA policy to approximate its privileged counterpart:

πθD\(⋅∣q,ot,Ct−1\)→OPDπθ¯P\(⋅∣q,ot,𝒱t\)\.\\pi\_\{\\theta\}^\{D\}\(\\cdot\\mid q,o\_\{t\},C\_\{t\-1\}\)\\;\\xrightarrow\{\\;\\mathrm\{OPD\}\\;\}\\;\\pi\_\{\\bar\{\\theta\}\}^\{P\}\(\\cdot\\mid q,o\_\{t\},\\mathcal\{V\}\_\{t\}\)\.The privileged visual history is used only during training; inference retains the single\-image \+ Compression input without additional visual or teacher overhead\.

### 3\.3PMF Gap\-Guided OPD–RL Training Framework

OPD provides dense supervision but targets privileged\-policy behavior, whereas GRPO directly optimizes task rewards from relatively sparse feedback\. We introduce PMF Gap\-guided OPD–RL to balance knowledge transfer and reward\-driven improvement according to the remaining advantage of privileged visual history\. Rather than using a fixed mixture, this framework adapts the balance to the model’s evolving performance under the two observation modes\. Each round comprises deployable CTA rollouts and student updates using OPD \(Sec\. 3\.2\) and GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib10)\):

ℒ\(r\)=λr​ℒOPDCTA\+\(1−λr\)​ℒGRPOCTA\.\\mathcal\{L\}^\{\(r\)\}=\\lambda\_\{r\}\\mathcal\{L\}\_\{\\mathrm\{OPD\}\}^\{\\mathrm\{CTA\}\}\+\(1\-\\lambda\_\{r\}\)\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}^\{\\mathrm\{CTA\}\}\.Both losses are averaged over their respective valid tokens or rollout samples before combination\. GRPO rewards action correctness, valid and parseable CTA formatting, and penalizes excessive Compression length\. We adopt a round\-wise self\-evolving teacher, with student and teacher parametersθr\\theta\_\{r\}andθ¯r\\bar\{\\theta\}\_\{r\}\. The teacher andλr\\lambda\_\{r\}remain fixed within each round; the updated student refreshes the teacher at the round boundary:

θ¯r\+1←θr\+1,∇θ¯rℒ\(r\)=0\.\\bar\{\\theta\}\_\{r\+1\}\\leftarrow\\theta\_\{r\+1\},\\qquad\\nabla\_\{\\bar\{\\theta\}\_\{r\}\}\\mathcal\{L\}^\{\(r\)\}=0\.
At each round’s end, the latest student is evaluated under both views on a fixed small\-scale validation set, yielding scoresSrPS\_\{r\}^\{P\}andSrDS\_\{r\}^\{D\}\. Their nonnegative difference defines the training\-time PMF Gap:

Gr=max⁡\(0,SrP−SrD\)\.G\_\{r\}=\\max\(0,S\_\{r\}^\{P\}\-S\_\{r\}^\{D\}\)\.Because both scores use the same model parameters, this gap reflects the relative benefit of privileged visual history rather than differences between model checkpoints\. LetG0G\_\{0\}denote the initial gap before training\. We map the current gap relative toG0G\_\{0\}to the next\-round OPD weight:

λr\+1=clip⁡\(λmin\+\(λmax−λmin\)​GrG0\+ϵ,λmin,λmax\)\.\\lambda\_\{r\+1\}=\\operatorname\{clip\}\\left\(\\lambda\_\{\\min\}\+\(\\lambda\_\{\\max\}\-\\lambda\_\{\\min\}\)\\frac\{G\_\{r\}\}\{G\_\{0\}\+\\epsilon\},\\lambda\_\{\\min\},\\lambda\_\{\\max\}\\right\)\.Here,ϵ\>0\\epsilon\>0ensures numerical stability, and\[λmin,λmax\]\[\\lambda\_\{\\min\},\\lambda\_\{\\max\}\]bounds the weight\. Larger gaps favor OPD, whereas shrinking gaps shift weight toward GRPO\. The weight can increase again if the gap widens, enabling gap\-responsive adaptation rather than a prescribed decay\. This balances privileged\-policy transfer with reward\-driven improvement beyond imitation\.

## 4Benchmark

### 4\.1Failure\-Driven GUI Perception and Memory\-Fidelity Evaluation

Existing GUI benchmarks cover grounding, screen understanding, and task execution\([Cheng et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib1);[Hsiao et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib24);[Rawles et al\., 2023](https://arxiv.org/html/2610.22408#bib.bib25);[Xie et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib26);[Lu et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib27)\), but overall success rates conflate perception, memory, planning, and action errors, obscuring the bottlenecks of long\-horizon agents\. Meanwhile, fine\-grained evaluations primarily assess information acquisition from visual inputs\([Cheng et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib1);[Hsiao et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib24)\), with less attention to whether this information remains usable after compression into textual memory\. Our failure analysis shows that click and filtering\-state errors are among the most common failure modes, suggesting that screen perception and state retention may underlie many execution errors\. We therefore construct a failure\-driven diagnostic benchmark that jointly examines visual state understanding and its preservation in textual memory for fine\-grained analysis of long\-horizon execution\. Specifically, we evaluate Perception together with the Perception–Memory Fidelity Gap \(PMF Gap\), defined asGPMF=max⁡\(0,SV−SM\)G\_\{\\mathrm\{PMF\}\}=\\max\(0,S^\{V\}\-S^\{M\}\), whereSVS^\{V\}andSMS^\{M\}denote normalized scores using direct visual input and only model\-generated textual memory, respectively\. A smaller gap indicates less performance degradation after compression; interpreted alongside direct\-vision performance, it helps distinguish failures to understand visual states from failures to retain the acquired state information\.

LevelTaskDiagnostic target\# InstancesL1L2L3BBox\-to\-FunctionElement semanticsSemantic\-to\-BBoxVisual groundingActive\-TabNavigation stateForeground\-LayerForeground stateBlocking\-StateInteraction blockingScreen\-SelectionSelection stateScreen\-State DiffCross\-frame changesFigure 3:Task taxonomy and statistics of ECHOESBMK\.Seven tasks span element\-level perception \(L1\), whole\-screen state understanding \(L2\), and cross\-frame state changes \(L3\)\. Bars indicate evaluation\-instance counts\.Following the observed failure patterns, we derive seven diagnostic tasks from real GUI agent rollouts and failure records by unifying screenshots, component trees, and trajectories into standardized screen states\. These tasks span L1 element\-level perception, L2 whole\-screen state understanding, and L3 cross\-frame state changes\. Single\-screen tasks use interface elements and state annotations, while cross\-frame tasks sample pre\- and post\-action observations to preserve state transitions from real execution\. Tasks retain their natural output formats, including coordinate localization, categorical judgments, and natural\-language descriptions, rather than being uniformly converted into action prediction\. This reduces confounding from policy behavior and action\-space design when measuring underlying perception capabilities\.

### 4\.2Benchmark Statistics and Quality Control

Figure[3](https://arxiv.org/html/2610.22408#S4.F3)summarizes the task taxonomy, diagnostic targets, and instance distributions\. The benchmark combines dense local queries with whole\-screen and cross\-frame evaluations\. Automatic construction and LLM\-assisted annotation are followed by manual verification to remove ambiguous or incorrect samples, and evaluation uses metrics matched to each task’s output\. Construction details, counting conventions, annotation checks, and evaluation protocols are provided in Appendix[F](https://arxiv.org/html/2610.22408#A6)\.

## 5Experiments

### 5\.1Experimental Setup

Training\.We first perform CTA SFT on Qwen3\-VL\-8B\([Bai et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib29)\)using 10K self\-constructed samples, followed by privileged OPD–RL on 9\.5K samples with PMF Gap\-guided weighting\.

Benchmarks and baselines\.We evaluate on GUI\-Odyssey\([Lu et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib27)\), AndroidControl\-v2\-High\([Li et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib34)\), AITW\([Rawles et al\., 2023](https://arxiv.org/html/2610.22408#bib.bib25)\), and AMEX\-L3\([Chai et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib35)\), with MageBench\-Sokoban\([Zhang et al\., 2026b](https://arxiv.org/html/2610.22408#bib.bib36)\)for zero\-shot transfer and ECHOESBMK \(Sec\. 4\) for perception and memory\-fidelity diagnostics\. Baselines include general VLMs from the Qwen, GLM, and Step3 families\([Bai et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib29);[Bai et al\., 2025b](https://arxiv.org/html/2610.22408#bib.bib42);[Qwen Team, 2026](https://arxiv.org/html/2610.22408#bib.bib43);[GLM\-V Team, 2025](https://arxiv.org/html/2610.22408#bib.bib44);[Huang et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib30)\)\(GLM\-4\.1V\-9B denotes the Thinking variant\); GUI agents MAI\-UI\-8B\([Zhou et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib31)\), AgentCPM\-GUI\-8B\([Zhang et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib16)\), GUI\-Owl\-1\.5\-8B/32B\([Xu et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib32)\), UI\-TARS\-1\.5\-7B\([Qin et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib11)\), UI\-R1\-3B\([Lu et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib14)\), and InfiGUI\-R1\-3B\([Liu et al\., 2025b](https://arxiv.org/html/2610.22408#bib.bib33)\); and proprietary models GPT\-5\.5\([OpenAI, 2026a](https://arxiv.org/html/2610.22408#bib.bib40)\)and Gemini 3\.1 Pro\([Google DeepMind, 2026](https://arxiv.org/html/2610.22408#bib.bib38)\)\.

Evaluation protocol\.Baselines use favorable model\-specific protocols, following official/native inference procedures where available and retaining native or GT history where applicable\. Unless otherwise specified, ECHOES receives only the task instruction, current screenshot, and self\-generated recursiveCC, without historical screenshots or GT history\. Efficiency is measured by per\-step input token counts and wall\-clock latency\. Evaluation and training details appear in Appendices[A](https://arxiv.org/html/2610.22408#A1)and[D](https://arxiv.org/html/2610.22408#A4), respectively\.

### 5\.2Main Results, Efficiency, and Transfer

Table 1:Main results on four GUI benchmarks\. All scores are reported in percentage \(%\)\. Best and second\-best results are marked inboldandunderline, respectively\.As shown in Table[1](https://arxiv.org/html/2610.22408#S5.T1), ECHOES consistently improves over Qwen3\-VL\-8B across all four GUI benchmarks, with gains in action prediction, action\-type recognition, and visual grounding\. It remains competitive with larger general VLMs and GUI\-specialized models while relying only on the current screenshot and recursiveCC, whereas some baselines access ground\-truth interaction history\. These results support transferring the decision\-making advantage of training\-time visual history into a compact deployment representation\. Additional results on InternVL3\.5\-8B support transferability across backbones \(Appendix[B\.1](https://arxiv.org/html/2610.22408#A2.SS1)\)\. Beyond GUI tasks, ECHOES also outperforms its base model in normalized task score and fully solved tasks on MageBench\-Sokoban under zero\-shot evaluation, as shown in Table[4](https://arxiv.org/html/2610.22408#S5.F4)\. Both models use the same native online interaction pipeline, while ECHOES retains its recursiveCCstate without task\-specific adaptation\. This suggests that its gains transfer beyond GUI tasks rather than arising from a GUI\-specific action or compression format\.

Efficiency comparisons further show that ECHOES maintains high performance at relatively low inference cost, as shown in Figure[4](https://arxiv.org/html/2610.22408#S5.F4)\. Compared with GUI\-Owl and UI\-TARS under longer\-history settings, it uses substantially fewer input tokens and achieves lower per\-step latency\. In controlled comparisons with the Qwen3\-VL\-8B backbone fixed, increasing the number of historical screenshots expands the visual context and raises latency, while performance gains gradually saturate\. RecursiveCCachieves a better performance–cost trade\-off without retaining historical screenshots\. Although action\-only history is cheaper, it remains less accurate, suggesting that the benefit comes from modeling task\-relevant historical states rather than simply adding textual context\. The additional training\-time overhead is 16\.7%, as detailed in Appendix[B\.5](https://arxiv.org/html/2610.22408#A2.SS5)\.

Figure 4:Efficiency on GUI\-Odyssey\.EM versus input tokens and latency; dashed lines connect Qwen3\-VL\-8B history settings\.Table 2:Zero\-shot transfer on MageBench\-Sokoban without additional training\.

### 5\.3Ablation Studies

Table[3](https://arxiv.org/html/2610.22408#S5.T3)\(a\) shows that both RL and privileged OPD improve GUI\-Odyssey performance from the same SFT initialization\. RL optimizes task rewards, while OPD provides dense multi\-image supervision on student\-generated CTA prefixes\. Combining them yields further gains; PMF Gap\-guided scheduling outperforms predefined linear OPD–RL scheduling by 3\.46 percentage points in EM \(68\.58 vs\. 65\.12\), supporting adaptive coordination beyond a prescribed transition between the two objectives\. Trajectory\-level paired bootstrap results are provided in Appendix[B\.4](https://arxiv.org/html/2610.22408#A2.SS4)\.

Inference\-time interventions in Table[3](https://arxiv.org/html/2610.22408#S5.T3)\(b\) remove memory \(NoCC\), substitute another trajectory’s state \(ShuffledCC\), or reuse an outdated state \(StaleCC\)\. All three reduce performance, supporting reliance on trajectory\-consistent, updated history rather than additional text alone\. Generic recursive summarization \(Naive Summary\) partially recovers performance but remains below learned recursive updates \(RecursiveCC\)\. These results support retaining decision\-relevant states rather than merely summarizing the past, consistent with our objective of learning a task\-sufficient textual state\.

Table 3:Ablation studies on GUI\-Odyssey\.Left: contributions of RL, privileged OPD, and PMF\-guided joint optimization\. Right: interventions on the recurrent textual stateCC\.\(a\) Training objectives

\(b\) Compression intervention

### 5\.4Long\-Horizon, Training Dynamics, and Hyperparameter Analysis

Table 4:Memory dependency\.GUI\-Odyssey results on history\-dependent \(Dep\.\) and independent \(Indep\.\) steps\.
Figure 5:Round\-wise self\-evolution\.Privileged/deployable scores andλOPD\\lambda\_\{\\mathrm\{OPD\}\}\.Table 5:Trajectory\-length analysis on GUI\-Odyssey\.Performance on trajectories with≤12\\leq 12or≥13\\geq 13steps, comparing history settings \(left\) and models \(right\)\.

Table 6:Hyperparameter sensitivity on GUI\-Odyssey\.We vary the teacher\-refresh interval and the OPD–RL transition range while keeping the remaining training configuration unchanged\.\(a\) Teacher refresh interval

\(b\) OPD–RL transition range

Table[5](https://arxiv.org/html/2610.22408#S5.T5)shows that ECHOES leads on memory\-dependent steps, with larger gains on dependent than independent steps across all listed baselines \(e\.g\., 6\.6 vs\. 1\.3 percentage points over GUI\-Owl\-1\.5\-32B\)\. It also leads both trajectory\-length groups in Table[5](https://arxiv.org/html/2610.22408#S5.T5), achieving 70\.49 EM for≤12\\leq 12steps and 67\.84 for longer trajectories\. Together with theCCinterventions, these results support the role of updated, trajectory\-consistent memory in long\-horizon action decision without retaining historical screenshots\. Appendix[C](https://arxiv.org/html/2610.22408#A3)illustrates history\-conditioned decisions on similar screens\.

Figure[5](https://arxiv.org/html/2610.22408#S5.T5)shows an overall narrowing of the privileged–deployable test\-set gap despite round\-wise fluctuations\. The deployable score rises from 0\.5624 to 0\.6858, slightly exceeding the final privileged score of 0\.6760\. Concurrently, the validation\-driven OPD weight decreases overall from 0\.8 to 0\.2, shifting emphasis toward task\-reward optimization\. Round\-wise test scores and validation\-driven weights are reported in Appendix[B\.3](https://arxiv.org/html/2610.22408#A2.SS3)\.

Under a fixed training budget, Table[6](https://arxiv.org/html/2610.22408#S5.T6)compares teacher\-refresh intervals of 3, 6, and 12 steps and three OPD–RL transition ranges\. The default 6\-step interval andλOPD∈\[0\.2,0\.8\]\\lambda\_\{\\mathrm\{OPD\}\}\\in\[0\.2,0\.8\]achieve the highest EM \(68\.58\) in their respective comparisons; more frequent refreshes or a wider range offer no improvement\.

### 5\.5Diagnostic Benchmark and PMF Analysis

Table[7](https://arxiv.org/html/2610.22408#S5.T7)evaluates element semantics, grounding, whole\-screen understanding, and cross\-frame state changes\. ECHOES leads all four L2 state\-understanding tasks and remains competitive with larger VLMs on Semantic\-to\-BBox, suggesting improved screen understanding beyond action prediction\. Model rankings vary across perception dimensions, showing that a single grounding or end\-to\-end metric provides an incomplete assessment\. Stronger direct perception also does not necessarily imply higher memory fidelity\. Using the PMF Gap defined in Sec\. 4\.1, ECHOES achieves the lowest average gap across the four L2 tasks \(0\.021 versus 0\.054 for Qwen3\-VL\-8B\), alongside higher direct\-vision scores, supporting improved preservation of visual states in textual memory\. Task\-wise PMF results in Appendix[B\.2](https://arxiv.org/html/2610.22408#A2.SS2)show that Active\-Tab generally incurs larger gaps than Foreground and Blocking, including for ECHOES\. Together, these findings highlight improvements in both state understanding and memory fidelity, while identifying fine\-grained state retention as a remaining bottleneck not captured by perception accuracy alone\.

Table 7:Perception and memory fidelity on ECHOESBMK\.We report direct\-vision scores \(↑\\uparrow\) for seven tasks and Avg\. PMF \(↓\\downarrow\), the mean nonnegative performance drop from vision to text\-only memory replay across four L2 tasks\. Bold indicates the best results\.

## 6Conclusion

We present ECHOES, which treats recent visual history as training\-time privileged information rather than a persistent deployment input\. It combines CTA recursive state modeling, multi\-image–single\-image privileged OPD, and PMF Gap\-guided OPD–RL with a round\-wise refreshed teacher, transferring history\-informed decision\-making to a policy conditioned on the task instruction, current screenshot, and self\-generated textual stateCC\. Experiments show improvements over Qwen3\-VL\-8B across four GUI benchmarks, favorable performance–cost trade\-offs, and zero\-shot transfer beyond GUI\-specific action spaces\. Memory\-dependency analyses, ablations, and Compression interventions support the role of task\-relevant historical\-state modeling rather than additional text alone\. ECHOESBMK further distinguishes visual\-state understanding from memory fidelity, showing improved direct perception and a smaller average PMF Gap, while fine\-grained state retention and long\-horizon degradation remain challenges\. Together, these findings support using privileged visual history during training to learn compact textual states that preserve decision\-relevant information without retaining historical screenshots at deployment\.

## AI use statement

We used generative AI tools to assist with manuscript language refinement, supervised fine\-tuning \(SFT\) data construction, and the construction and verification of ECHOESBMK\. All AI\-assisted content and data were reviewed by the authors, who take full responsibility for the final manuscript, experimental results, and associated artifacts\.

## References

- R\. Agarwal, N\. Vieillard, Y\. Zhou, P\. Stanczyk, S\. R\. Garea, M\. Geist, and O\. BachemOn\-policy distillation of language models: learning from self\-generated mistakes\.InInternational Conference on Learning Representations,Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p2.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1),[§3\.2](https://arxiv.org/html/2610.22408#S3.SS2.p1.1)\.
- Anthropic \(2026\)AnthropicIntroducing Claude Opus 5\.Note:[https://www\.anthropic\.com/news/claude\-opus\-5](https://www.anthropic.com/news/claude-opus-5)Accessed: 2026\-09\-20Cited by:[§F\.2](https://arxiv.org/html/2610.22408#A6.SS2.p1.1)\.
- Baiet al\.\(2025a\)S\. Bai, Y\. Cai, R\. Chen,et al\.Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Baiet al\.\(2025b\)S\. Bai, K\. Chen, X\. Liu,et al\.Qwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Chaiet al\.\(2025\)Y\. Chai, S\. Huang, Y\. Niu, H\. Xiao, L\. Liu, G\. Wang, D\. Zhang, S\. Ren, and H\. LiAMEX: android multi\-annotation expo dataset for mobile gui agents\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 2138–2156\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.110)Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Chenget al\.\(2024\)K\. Cheng, Q\. Sun, Y\. Chu, F\. Xu, L\. Yantao, J\. Zhang, and Z\. WuSeeClick: harnessing GUI grounding for advanced visual GUI agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 9313–9332\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.505)Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1),[§4\.1](https://arxiv.org/html/2610.22408#S4.SS1.p1.1)\.
- Fanet al\.\(2026\)Z\. Fan, H\. Yu, Y\. Shen, H\. Yan, G\. Peng, T\. Peng, Y\. Zhang, X\. Zhang, K\. Tan, Z\. Ge, X\. Zhang, and D\. JiangSkill\-guided continuation distillation for GUI agents\.arXiv preprint arXiv:2606\.18890\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1)\.
- GLM\-V Team \(2025\)GLM\-V TeamGLM\-4\.1v\-thinking and glm\-4\.5v: towards versatile multimodal reasoning with scalable reinforcement learning\.arXiv preprint arXiv:2507\.01006\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Google DeepMind \(2026\)Google DeepMindGemini 3\.1 pro model card\.Note:Official model cardCited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Hsiaoet al\.\(2025\)Y\. Hsiao, F\. Zubach, G\. Baechler, S\. Sunkara, V\. Carbune, J\. Lin, M\. Wang, Y\. Zhu, and J\. ChenScreenQA: large\-scale question\-answer pairs over mobile app screenshots\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 9427–9452\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.477)Cited by:[§4\.1](https://arxiv.org/html/2610.22408#S4.SS1.p1.1)\.
- Huanget al\.\(2026\)A\. Huang, C\. Yao, C\. Han,et al\.STEP3\-vl\-10b technical report\.arXiv preprint arXiv:2601\.09668\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Huanget al\.\(2025\)K\. Huang, H\. Qiu, Y\. Dai, C\. Xiong, and C\. WuGUI\-kv: efficient GUI agents via KV cache with spatio\-temporal awareness\.arXiv preprint arXiv:2510\.00536\.Cited by:[§1](https://arxiv.org/html/2610.22408#S1.p1.1)\.
- Liet al\.\(2024\)W\. Li, W\. Bishop, A\. Li, C\. Rawles, F\. Campbell\-Ajala, D\. Tyamagundlu, and O\. RivaOn the effects of data scale on ui control agents\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Liet al\.\(2026\)W\. Li, J\. Liu, T\. Chu, H\. Yu, and W\. LiThe next screenshot knows: gated hindsight distillation for mobile GUI agents\.arXiv preprint arXiv:2608\.06065\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.
- Linet al\.\(2025\)K\. Q\. Lin, L\. Li, D\. Gao, Z\. Yang, S\. Wu, Z\. Bai, W\. Lei, L\. Wang, and M\. Z\. ShouShowUI: one vision\-language\-action model for GUI visual agent\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19498–19508\.Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p1.1)\.
- Linet al\.\(2026\)Z\. Lin, F\. Liu, Y\. Yang, J\. Lyu, Y\. Gao, Y\. Liu, Z\. Lu, Y\. Yu, M\. Yang, J\. Li, D\. Ye, and J\. JiangUI\-Voyager: a self\-evolving GUI agent learning via failed experience\.arXiv preprint arXiv:2603\.24533\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1)\.
- Liuet al\.\(2025a\)T\. Liu, C\. Wang, R\. Li, Y\. Yu, X\. He, and S\. BaiGUI\-rise: structured reasoning and history summarization for gui navigation\.InAdvances in Neural Information Processing Systems,Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.
- Liuet al\.\(2025b\)Y\. Liu, P\. Li, C\. Xie, X\. Hu, X\. Han, S\. Zhang, H\. Yang, and F\. WuInfiGUI\-r1: advancing multimodal gui agents from reactive actors to deliberative reasoners\.arXiv preprint arXiv:2504\.14239\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Liuet al\.\(2025c\)Z\. Liu, J\. Li, W\. X\. Zhao, D\. Gao, Y\. Li, and J\. WenPAL\-ui: planning with active look\-back for vision\-based GUI agents\.arXiv preprint arXiv:2510\.00413\.Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1)\.
- Luet al\.\(2025\)Q\. Lu, W\. Shao, Z\. Liu, L\. Du, F\. Meng, B\. Li, B\. Chen, S\. Huang, K\. Zhang, and P\. LuoGUIOdyssey: a comprehensive dataset for cross\-app gui navigation on mobile devices\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 22404–22414\.Cited by:[§4\.1](https://arxiv.org/html/2610.22408#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Luet al\.\(2026\)Z\. Lu, Y\. Chai, Y\. Guo, X\. Yin, L\. Liu, H\. Wang, H\. Xiao, S\. Ren, P\. Zhao, G\. Liu, G\. Xiong, and H\. LiUI\-R1: enhancing efficient action prediction of GUI agents by reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 17608–17616\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Luoet al\.\(2025\)R\. Luo, L\. Wang, W\. He, and X\. XiaGUI\-R1: a generalist r1\-style vision\-language action model for GUI agents\.arXiv preprint arXiv:2504\.10458\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.
- OpenAI \(2026a\)OpenAIGPT\-5\.5\.Note:[https://openai\.com/index/introducing\-gpt\-5\-5/](https://openai.com/index/introducing-gpt-5-5/)Accessed: 2026\-09\-20Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- OpenAI \(2026b\)OpenAIGPT\-5\.6: frontier intelligence that scales with your ambition\.Note:Official model releaseCited by:[§F\.2](https://arxiv.org/html/2610.22408#A6.SS2.p1.1)\.
- Qinet al\.\(2025\)Y\. Qin, Y\. Ye, J\. Fang, H\. Wang, S\. Liang, S\. Tian, J\. Zhang, J\. Li, Y\. Li, S\. Huang,et al\.UI\-TARS: pioneering automated GUI interaction with native agents\.arXiv preprint arXiv:2501\.12326\.Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Qwen Team \(2026\)Qwen TeamQwen3\.5: towards native multimodal agents\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Rawleset al\.\(2023\)C\. Rawles, A\. Li, D\. Rodriguez, O\. Riva, and T\. LillicrapAndroid in the wild: a large\-scale dataset for android device control\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§4\.1](https://arxiv.org/html/2610.22408#S4.SS1.p1.1),[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Rosset al\.\(2011\)S\. Ross, G\. Gordon, and D\. BagnellA reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,Proceedings of Machine Learning Research, Vol\.15,pp\. 627–635\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. GuoDeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p2.1),[§3\.3](https://arxiv.org/html/2610.22408#S3.SS3.p1.1)\.
- Tianet al\.\(2025\)S\. Tian, H\. Wen, Y\. Chen, J\. Liu, S\. Zhao, G\. Liu, J\. Ren, Y\. Liu, and Y\. LiAgentProg: empowering long\-horizon GUI agents with program\-guided context management\.arXiv preprint arXiv:2512\.10371\.Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p1.1)\.
- Wanget al\.\(2025a\)H\. Wang, H\. Zou, H\. Song, J\. Feng, J\. Fang, J\. Lu, L\. Liu, Q\. Luo, S\. Liang, S\. Huang,et al\.UI\-TARS\-2 technical report: advancing GUI agent with multi\-turn reinforcement learning\.arXiv preprint arXiv:2509\.02544\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1)\.
- Wanget al\.\(2025b\)W\. Wang, Z\. Gao, L\. Gu, H\. Pu, L\. Cui, X\. Wei, Z\. Liu, L\. Jing, S\. Ye, J\. Shao,et al\.InternVL3\.5: advancing open\-source multimodal models in versatility, reasoning, and efficiency\.arXiv preprint arXiv:2508\.18265\.Cited by:[§B\.1](https://arxiv.org/html/2610.22408#A2.SS1.SSS0.Px1.p1.1)\.
- Wuet al\.\(2026\)Y\. Wu, Z\. Cai, L\. Ning, H\. Wang, Z\. Chen, Y\. Tang, and H\. ChenLiteGUI: distilling compact GUI agents with reinforcement learning\.arXiv preprint arXiv:2605\.07505\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.
- Xieet al\.\(2024\)T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei, Y\. Liu, Y\. Xu, S\. Zhou, S\. Savarese, C\. Xiong, V\. Zhong, and T\. YuOSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.InAdvances in Neural Information Processing Systems,Vol\.37\.Cited by:[§4\.1](https://arxiv.org/html/2610.22408#S4.SS1.p1.1)\.
- Xuet al\.\(2026\)H\. Xu, X\. Zhang, H\. Liu, J\. Wang, Z\. Zhu, S\. Zhou, X\. Hu, F\. Gao, J\. Cao, Z\. Wang, Z\. Chen, J\. Liao, Q\. Zheng, J\. Zeng, Z\. Xu, S\. Bai, J\. Lin, J\. Zhou, and M\. YanMobile\-agent\-v3\.5: multi\-platform fundamental gui agents\.arXiv preprint arXiv:2602\.16855\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Xuet al\.\(2025\)Y\. Xu, Z\. Wang, J\. Wang, D\. Lu, T\. Xie, A\. Saha, D\. Sahoo, T\. Yu, and C\. XiongAguvis: unified pure vision agents for autonomous GUI interaction\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 69772–69805\.Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p1.1)\.
- Zhanget al\.\(2026a\)G\. Zhang, Y\. Ling, F\. Meng, K\. Wang, and S\. YanMem\-w: latent memory\-native GUI agents\.arXiv preprint arXiv:2605\.09317\.Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.
- Zhanget al\.\(2026b\)M\. Zhang, Q\. Dai, Y\. Yang, J\. Bao, D\. Chen, K\. Qiu, C\. Luo, X\. Geng, and B\. GuoMageBench: bridging large multimodal models to agents\.InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision,pp\. 1415–1427\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Zhanget al\.\(2026c\)Y\. Zhang, D\. Wu, H\. Shen, Y\. Zhou, and C\. MaLearn where to click from yourself: on\-policy self\-distillation for GUI grounding\.arXiv preprint arXiv:2605\.00642\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.
- Zhanget al\.\(2025\)Z\. Zhang, Y\. Lu, Y\. Fu, Y\. Huo, S\. Yang, Y\. Wu, H\. Si, X\. Cong, H\. Chen, Y\. Lin, J\. Xie, W\. Zhou, W\. Xu, Y\. Zhang, Z\. Su, Z\. Zhai, X\. Liu, Y\. Mei, J\. Xu, H\. Tian, C\. Wang, C\. Chen, Y\. Yao, Z\. Liu, and M\. SunAgentCPM\-GUI: building mobile\-use agents with reinforcement fine\-tuning\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp\. 155–180\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Zhaoet al\.\(2026\)S\. Zhao, Z\. Xie, M\. Liu, J\. Huang, G\. Pang, F\. Chen, and A\. GroverSelf\-distilled reasoner: on\-policy self\-distillation for large language models\.arXiv preprint arXiv:2601\.18734\.Cited by:[§E\.2](https://arxiv.org/html/2610.22408#A5.SS2.p1.1)\.
- Zhenget al\.\(2023\)L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. StoicaJudging LLM\-as\-a\-judge with MT\-Bench and chatbot arena\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§F\.3](https://arxiv.org/html/2610.22408#A6.SS3.p1.1)\.
- Zhouet al\.\(2025\)H\. Zhou, X\. Zhang, P\. Tong, J\. Zhang, L\. Chen, Q\. Kong, C\. Cai, C\. Liu, Y\. Wang, J\. Zhou, and S\. HoiMAI\-ui technical report: real\-world centric foundation gui agents\.arXiv preprint arXiv:2512\.22047\.Cited by:[§5\.1](https://arxiv.org/html/2610.22408#S5.SS1.p2.1)\.
- Zhouet al\.\(2026\)X\. Zhou, G\. Chen, Y\. Xie, Z\. Li, K\. Zhou, S\. Wang, S\. Yang, Z\. Tian, and R\. ShaoHiconAgent: history context\-aware policy optimization for GUI agents\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§E\.1](https://arxiv.org/html/2610.22408#A5.SS1.p1.1),[§1](https://arxiv.org/html/2610.22408#S1.p1.1),[§2](https://arxiv.org/html/2610.22408#S2.p1.1)\.

## Appendix AAdditional Experimental Details

### A\.1Benchmarks and Metrics

GUI benchmarks\.We report TM and EM on GUI\-Odyssey\. AndroidControl\-v2\-High reports overall accuracy, action\-type accuracy, and grounding accuracy, while AITW and AMEX\-L3 report step\-level and action\-type accuracy\. These metrics characterize complementary aspects of GUI decision\-making, including action selection and visual grounding\.

Zero\-shot visual planning\.We evaluate MageBench\-Sokoban without additional training and report the normalized task score and the number of fully solved tasks\. In the comparison between ECHOES and its Qwen3\-VL\-8B base model, both models follow the same MageBench native online interaction pipeline\. This evaluation examines whether the training gains transfer beyond GUI tasks\.

Diagnostic evaluation\.ECHOESBMK comprises seven tasks covering element\-level perception, whole\-screen state understanding, and cross\-frame state changes\. In addition to task\-specific perception scores, we report the PMF Gap to measure the nonnegative performance drop from direct visual access to reliance on model\-generated textual memory\. The reported average PMF Gap is macro\-averaged over the four L2 whole\-screen state tasks\. Benchmark construction and task\-specific evaluation protocols are provided in the diagnostic benchmark appendix\.

### A\.2Baseline Inference Context

We retain favorable model\-specific inference settings rather than imposing a uniform current\-only input on all baselines\. Official or native inference procedures are used whenever possible, including native or ground\-truth interaction history where applicable\. We distinguish four context settings:*current\-only*,*native history*,*GT history*, and*Current\+CC*\. Native history follows the baseline’s own history\-handling mechanism, whereas GT history denotes ground\-truth interaction information supplied during evaluation\. In contrast, under Current\+CC, ECHOES receives the task instruction and current screenshot together with its recursively generated Compression, without accessing historical screenshots or ground\-truth historical information\. Thus, the cross\-model comparison preserves each baseline’s supported inference setup rather than enforcing an identical historical\-information budget\.

### A\.3Efficiency Evaluation

We jointly report input\-token count and wall\-clock latency per step to characterize inference cost\. Cross\-model comparisons follow the model\-specific protocols described above\. Complementarily, controlled comparisons use the same Qwen3\-VL\-8B backbone with current\-only inputs, textual action history, and varying numbers of historical screenshots, alongside the Current\+CCdeployment setting\. These comparisons assess the trade\-off between task performance and inference cost under different history representations and context budgets\.

## Appendix BAdditional Results

### B\.1Evaluation on an Additional VLM Backbone

#### Experimental design\.

To examine whether ECHOES extends beyond Qwen3\-VL, we repeat the training pipeline with InternVL3\.5\-8B\([Wang et al\., 2025b](https://arxiv.org/html/2610.22408#bib.bib39)\)\. We compare the released checkpoint \(Base\), CTA supervised fine\-tuning \(CTA\-SFT\), and the full ECHOES model\. The experiments use the same approximately 10K SFT samples and 9\.5K OPD–RL samples, data splits, action space, and evaluation metrics as the main experiments\. CTA\-SFT and ECHOES share the deployment input\(q,ot,Ct−1\)\(q,o\_\{t\},C\_\{t\-1\}\), allowing us to assess the additional benefit of privileged OPD–RL beyond CTA initialization\. Both the student and the privileged teacher use InternVL3\.5\-8B\. The teacher receives\(q,ot,Vt\)\(q,o\_\{t\},V\_\{t\}\)withoutCt−1C\_\{t\-1\}and is refreshed from the student at round boundaries\. The OPD weight follows the same PMF\-guided rule but is computed from InternVL’s own validation\-set gaps rather than copied from the Qwen3\-VL run\.

#### Results\.

Table[8](https://arxiv.org/html/2610.22408#A2.T8)compares the two backbones across four GUI benchmarks\. For InternVL3\.5\-8B, ECHOES improves over Base by 17\.54, 6\.45, 9\.38, and 10\.26 percentage points on GUI\-Odyssey EM, AC\-v2\-High accuracy, AITW step accuracy, and AMEX\-L3 step accuracy, respectively\. The corresponding gains over CTA\-SFT are 8\.91, 3\.86, 5\.24, and 5\.44 points\. The gains vary across benchmarks rather than uniformly matching those of Qwen3\-VL\. Our primary comparison is between ECHOES and CTA\-SFT within the same backbone, assessing additional optimization beyond supervised initialization rather than requiring one backbone to consistently outperform the other\.

Table 8:Additional\-backbone comparison on four GUI benchmarks\.All scores are percentages, andΔ\\Deltadenotes differences in percentage points\. Qwen3\-VL\-8B results are reproduced from the main\-results table\. Base denotes the released checkpoint before our training\.

### B\.2Detailed PMF results on ECHOESBMK

Table[9](https://arxiv.org/html/2610.22408#A2.T9)breaks down the PMF Gap across the four L2 state categories\. ECHOES achieves the lowest gap on Active\-Tab, Foreground\-Layer, and Screen\-Selection, yielding the lowest macro\-average of 0\.021, compared with 0\.054 for Qwen3\-VL\-8B\. Active\-Tab incurs the largest gap for every evaluated model; ECHOES reduces this gap from 0\.138 to 0\.065 relative to its base model, although navigation\-state retention remains its largest source of degradation\. Foreground\-Layer and Screen\-Selection are preserved more reliably, with gaps of only 0\.000 and 0\.008\. On Blocking\-State, ECHOES ranks second, behind its base model \(0\.012 versus 0\.004\)\. Interpreted alongside the direct\-vision scores in Table[7](https://arxiv.org/html/2610.22408#S5.T7), these results support improved overall visual\-to\-text memory fidelity while highlighting fine\-grained navigation states as a remaining challenge\.

Table 9:Detailed PMF results on the diagnostic GUI benchmark\.We report the visual\-to\-text performance gap for each of the four L2 state categories\. Lower PMF indicates less information loss during visual\-to\-text transformation\.
### B\.3Round\-wise Training Dynamics

Table[10](https://arxiv.org/html/2610.22408#A2.T10)reports GUI\-Odyssey test\-set EM for the privileged and deployable views of the same model checkpoint, together with the logged OPD weight\. Importantly,SrPS\_\{r\}^\{P\}andSrDS\_\{r\}^\{D\}are measured on the test set, whereasλr\\lambda\_\{r\}is determined from the privileged–deployable gap on a separate validation set at round boundaries; the displayed test\-set gap is not used to compute the weight\. The deployable score increases from 0\.5624 to 0\.6858, while the signed test\-set gap narrows from \+0\.0849 to \-0\.0098, with the final deployable view slightly outperforming the privileged view\. Meanwhile,λr\\lambda\_\{r\}decreases from 0\.80 to 0\.20, showing that the narrowing test\-set gap accompanies the validation\-driven shift toward greater emphasis on task\-reward optimization\.

Table 10:Round\-wise privileged–deployable gap and OPD weight\.
### B\.4Statistical Significance of Ablation Results

Table[11](https://arxiv.org/html/2610.22408#A2.T11)reports paired EM improvements of the full ECHOES model over its training ablations on GUI\-Odyssey, with 95% confidence intervals obtained using trajectory\-level paired bootstrap with 10,000 resamples\. ECHOES improves over SFT, SFT\+RL, and SFT\+OPD by 12\.34, 4\.93, and 4\.57 percentage points, respectively\. All reported confidence intervals exclude zero, and all reportedpp\-values are at most 0\.002\. In particular, the improvement over linear\-weight OPD–RL is 3\.46 percentage points, with a 95% confidence interval of\[1\.77,5\.12\]\[1\.77,5\.12\]andp=0\.002p=0\.002, supporting the additional contribution of PMF Gap\-guided scheduling beyond combining the two training objectives with a linear weight\.

Table 11:Statistical significance of the ablation comparisons on GUI\-Odyssey\.We report paired EM improvements of the full model over each training variant, with 95% confidence intervals obtained using trajectory\-level paired bootstrap with 10,000 resamples\.
### B\.5Training\-Time Overhead

We characterize the additional training\-time cost of ECHOES relative to standard OPD–RL using normalized wall\-clock time, with the baseline runtime set to 100\. The reported 16\.7% overhead corresponds to a normalized ECHOES runtime of 116\.70\. Table[12](https://arxiv.org/html/2610.22408#A2.T12)provides a training\-time overhead result across round\-boundary validation and teacher\-refresh operations, based on an unchanged baseline training workload and non\-overlapping additional operations\.

Table 12:Training\-time overhead\.Runtime is normalized to the OPD–RL baseline\.ComponentOPD–RLECHOESAdditional timeBaseline training workload100\.00100\.000\.00Additional deployable\-view validation0\.004\.834\.83Additional privileged\-view validation0\.009\.129\.12Additional teacher refresh/synchronization0\.002\.752\.75Total normalized runtime100\.00116\.7016\.70Relative overhead0\.0%16\.7%—The relative overhead is computed as

Δtime=τECHOES−τOPD​\-​RLτOPD​\-​RL×100%=16\.7%\.\\Delta\_\{\\mathrm\{time\}\}=\\frac\{\\tau\_\{\\mathrm\{ECHOES\}\}\-\\tau\_\{\\mathrm\{OPD\\text\{\-\}RL\}\}\}\{\\tau\_\{\\mathrm\{OPD\\text\{\-\}RL\}\}\}\\times 100\\%=16\.7\\%\.Under the illustrative allocation, dual\-view validation contributes 13\.95 normalized time units, while teacher refresh and synchronization contribute 2\.75\. Validation estimates the privileged–deployable gap for updatingλOPD\\lambda\_\{\\mathrm\{OPD\}\}, whereas teacher refresh transfers the latest student parameters to the frozen teacher used in the subsequent round\. Any corresponding operations already performed by the baseline belong to the baseline workload rather than the additional\-cost terms\.

These operations are confined to training\. Deployment retains the task instruction, current screenshot, and recursively generated stateCC, requiring neither dual\-view validation nor teacher refreshes\. The additional training\-time cost therefore does not introduce corresponding operations into the inference pipeline\.

## Appendix CCase Study

Figure[6](https://arxiv.org/html/2610.22408#A3.F6)presents an 11\-step cross\-app example \(steps 00–10\) involving a search for Paris Fashion Week dates in Opera and reminder creation in TickTick\. At each step, ECHOES conditions on the task instructionqq, current screenshototo\_\{t\}, and previous compressionCt−1C\_\{t\-1\}to generate\(Ct,Tt,At\)\(C\_\{t\},T\_\{t\},A\_\{t\}\)\. The example highlights two roles of recursive compression: preserving facts across application switches and tracking task progress to distinguish decisions on visually similar screens\.

#### Cross\-app fact retention\.

At step 05,C5C\_\{5\}records the date range June 18–23, 2024, from the displayed search result and notes that the reminder has not yet been created\. These dates remain inC6,…,C9C\_\{6\},\\ldots,C\_\{9\}as the current observation changes from Opera to the Android home screen and then TickTick\. Meanwhile, the compression updates the immediate task state, including whether the reminder interface is open and whether the title field is focused\. The later reference screenshots show a June 18 reminder entry and its saved state, which are reflected inC9C\_\{9\}andC10C\_\{10\}\. Thus, the textual state retains task\-relevant facts while incorporating newly observed progress, without requiring the earlier browser screenshot as an input\.

#### History\-conditioned action selection\.

Steps 00 and 07 show similar empty TickTick Inbox screens, but correspond to different task stages\. At step 00, the event dates have not been retrieved, so the model selectsPRESS\_HOMEto begin navigating to Opera\. At step 07, the search is complete and the dates are retained in the compression, so the model instead selects the add button to begin creating the reminder\. This contrast illustrates why current\-screen appearance alone does not specify the appropriate next action: the decision also depends on what has already been accomplished\. Recursive compression represents this distinction through both retained facts and updated task progress\.

![Refer to caption](https://arxiv.org/html/2610.22408v1/ECHOES_case_study.png)Figure 6:Cross\-app fact retention through recursive compression\.An 11\-step example involving information retrieval in Opera and reminder creation in TickTick\. Each panel shows the current screenshot and the corresponding Compression, Think, and Action outputs\. Dates recorded at step 05 remain available after leaving Opera, while similar Inbox screens at steps 00 and 07 correspond to different task stages and actions\. Each decision uses only the task instruction, current screenshot, and previous compression, without historical screenshot inputs\.

## Appendix DTraining and Implementation Details

Training data and supervision\.We use Qwen3\-VL\-8B as the base model and construct all post\-training data ourselves\. The training set contains approximately 10K SFT samples and 9\.5K OPD–RL samples\. Each SFT sample follows the Compression–Think–Action \(CTA\) format and includes explicit supervision for the recurrent textual stateCC\. The targetCCannotations are generated by GPT\-5\.6 using the task instruction and the corresponding interaction trajectory as context, providing an initialization for task\-state writing before on\-policy optimization\. The training and evaluation data are constructed from disjoint splits: no evaluation sample or trajectory is included in either SFT or OPD–RL training\.

Deployable and privileged observations\.At interaction steptt, the deployable policy receives only the task instructionqq, the current screenshototo\_\{t\}, and the recursively generated stateCt−1C\_\{t\-1\}:

xtD=\(q,ot,Ct−1\)\.x\_\{t\}^\{D\}=\(q,o\_\{t\},C\_\{t\-1\}\)\.The privileged teacher receives up to four preceding screenshots\. Accordingly, its visual input contains at most five temporally ordered observations, consisting of the current screenshot and the four most recent historical screenshots:

Vt=\(omax⁡\(1,t−4\),…,ot−1\),xtP=\(q,ot,𝒱t\)\.V\_\{t\}=\(o\_\{\\max\(1,t\-4\)\},\\ldots,o\_\{t\-1\}\),\\qquad x\_\{t\}^\{P\}=\(q,o\_\{t\},\\mathcal\{V\}\_\{t\}\)\.When fewer than four historical observations are available, the privileged view uses the available history\. The teacher and student share the same model architecture, tokenizer, and CTA output format, but receive different observation contexts:\(q,ot,Vt\)\(q,o\_\{t\},V\_\{t\}\)and\(q,ot,Ct−1\)\(q,o\_\{t\},C\_\{t\-1\}\), respectively\. The teacher is frozen within each training round and refreshed from the latest student at the round boundary\. Compression is constrained to a maximum length ofBC=128B\_\{C\}=128tokens\.

OPD–RL optimization\.During OPD–RL training, the deployable policy first generates CTA rollouts underxtDx\_\{t\}^\{D\}\. The privileged teacher does not generate an independent trajectory; instead, it performs teacher\-forced scoring on the same student\-generated CTA sequence\. OPD is applied to valid tokens in Compression, Think, and Action, while GRPO performs group\-relative optimization over the student rollouts\.

The GRPO reward combines action correctness, output\-format validity, and a length penalty on Compression:

R=Ract\+Rfmt−Rlen\.R=R\_\{\\mathrm\{act\}\}\+R\_\{\\mathrm\{fmt\}\}\-R\_\{\\mathrm\{len\}\}\.Here,RactR\_\{\\mathrm\{act\}\}evaluates whether the predicted action type and its associated grounding or parameters match the target;RfmtR\_\{\\mathrm\{fmt\}\}encourages valid and parseable CTA outputs; andRlenR\_\{\\mathrm\{len\}\}discourages unnecessarily verboseCC\. Unless otherwise specified, the teacher is refreshed every 6 optimization steps\. The default PMF\-guided OPD–RL transition range isλOPD∈\[0\.2,0\.8\]\\lambda\_\{\\mathrm\{OPD\}\}\\in\[0\.2,0\.8\], with the weight updated only at round boundaries according to the privileged–deployable performance gap\. For GRPO, we use a rollout group size ofK=8K=8and sample trajectories with temperature1\.01\.0\.

Unified action space\.We normalize the GUI post\-training trajectories into the following nine basic action categories\.

Table 13:Unified GUI action space used for post\-training\.
## Appendix ERelated Work

### E\.1Long\-Horizon VLM Agent Reasoning

With the development of multimodal large language models, visual Agents have gradually expanded from single\-step interface understanding and grounding to long\-horizon tasks involving planning, reasoning, and sequential decision\-making\. Early works such as SeeClick, Aguvis, ShowUI, and UI\-TARS mainly improve GUI operation capabilities by enhancing visual grounding, action modeling, and explicit reasoning\([Cheng et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib1);[Xu et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib3);[Lin et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib2);[Qin et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib11)\); however, as interaction trajectories grow, current decisions increasingly depend on previous interface states, action outcomes, and task progress, making historical information an important component of long\-horizon reasoning\. Recent works further focus on history modeling; for example, GUI\-Rise, HiconAgent, PAL\-UI, AgentProg, and Mem\-W retain long\-term interaction information through textual summarization, history compression, visual retrieval, structured states, or latent\-space memory, respectively\([Liu et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib4);[Zhou et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib12);[Liu et al\., 2025c](https://arxiv.org/html/2610.22408#bib.bib5);[Tian et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib6);[Zhang et al\., 2026a](https://arxiv.org/html/2610.22408#bib.bib7)\)\. These methods demonstrate the importance of historical context for long\-horizon visual decision\-making, but generally still rely on the continuous storage, compression, or access of historical information during inference, and therefore require a trade\-off between information completeness and visual tokens, context length, and computational overhead\. Different from this inference\-time history management paradigm, this work explores how to use the recent visual history only as training\-time privileged information and transfer its decision\-making advantage to a deployment mode that relies only on the current observation and a compact textual state\.

### E\.2Agentic Post\-Training and Self\-Evolution

Beyond context modeling, post\-training paradigms have also continued to advance the long\-horizon capabilities of VLM Agents\. Traditional supervised fine\-tuning relies on fixed expert trajectories and has difficulty covering the state distribution induced by the model’s own execution\([Ross et al\., 2011](https://arxiv.org/html/2610.22408#bib.bib13)\); reinforcement learning represented by GRPO\([Shao et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib10)\)further optimizes reasoning and action capabilities through model\-generated rollouts and task rewards, and has been applied to GUI Agents by UI\-R1, GUI\-R1, AgentCPM\-GUI, GUI\-Rise, UI\-TARS\-2, and other works\([Lu et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib14);[Luo et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib15);[Zhang et al\., 2025](https://arxiv.org/html/2610.22408#bib.bib16);[Liu et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib4);[Wang et al\., 2025a](https://arxiv.org/html/2610.22408#bib.bib17)\)\. However, outcome rewards in long\-horizon environments are usually relatively sparse, making it difficult for intermediate decisions to receive fine\-grained supervision\([Lin et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib22);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22408#bib.bib19)\)\. Complementarily, GKD/On\-Policy Distillation \(OPD\) uses teacher distributions to provide dense token\-level signals on states visited by the student itself\([Agarwal et al\., 2024](https://arxiv.org/html/2610.22408#bib.bib9)\); recent works including Self\-Distilled Reasoner, GUI\-SD, LiteGUI, and GHD have further explored same\-model self\-distillation with privileged contexts and the combination of OPD with RL\([Zhao et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib18);[Zhang et al\., 2026c](https://arxiv.org/html/2610.22408#bib.bib19);[Wu et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib20);[Li et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib21)\)\. Meanwhile, UI\-Voyager and SGCD construct new supervision by continuously utilizing trajectories generated by the model itself, reflecting the trend of Agent training from static data toward self\-evolution\([Lin et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib22);[Fan et al\., 2026](https://arxiv.org/html/2610.22408#bib.bib23)\)\. Existing works mainly use answers, target regions, future states, or external feedback to improve the current policy, whereas this work focuses on the cross\-observation\-mode capability gap caused by visual\-history deployment constraints, and further dynamically coordinates privileged\-policy distillation and task\-reward optimization according to this gap, allowing the model to gradually transition from imitating a stronger observation mode to directly optimizing the actual task objective\.

## Appendix FDiagnostic Benchmark Details

### F\.1Task Construction and Statistics

Data construction\.ECHOESBMK is grounded in manually designed and reviewed tasks involving real applications\. We execute these tasks with multiple GUI agents and derive diagnostic queries from their observations and failure records\. Screenshots, component annotations, and interaction trajectories are standardized into records containing image identifiers, dimensions, element annotations, and task\-specific reference answers\. Cross\-frame records additionally link temporally ordered observations to the corresponding interaction context\. The benchmark preserves task\-native outputs rather than converting all queries into action prediction\.

Tasks and statistics\.The benchmark contains 9,061 instances across seven tasks\. At L1,*BBox\-to\-Function*\(4,102\) identifies the function of a localized element, while*Semantic\-to\-BBox*\(604\) localizes a control from a functional request\. At L2,*Active\-Tab*\(123\) identifies the active navigation item;*Foreground\-Layer*\(577\) detects and describes foreground interface layers;*Blocking\-State*\(246\) determines whether the requested interaction is blocked; and*Screen\-Selection*\(118\) identifies selected items and values\. At L3,*Screen\-State Diff*\(3,291\) describes changes between pre\- and post\-action screenshots, including navigation, overlay, and selection\-state transitions\. Multiple queries may share a screenshot, while cross\-frame instances contain two observations; instance counts therefore differ from unique\-image counts\. Alternative MCQ, judge, and memory\-replay views are not counted as additional instances\.

### F\.2Annotation and Quality Control

Review procedure\.Following multi\-model execution and candidate construction, GPT\-5\.6 Sol\([OpenAI, 2026b](https://arxiv.org/html/2610.22408#bib.bib37)\)and Claude Opus 5\([Anthropic, 2026](https://arxiv.org/html/2610.22408#bib.bib41)\)perform preliminary checking and filtering, followed by sample\-by\-sample human verification\. Reviewers check screenshot–question consistency, target\-element and state annotations, and whether the reference answer is unambiguously supported by the supplied observations\. Element\-level checks verify bounding boxes and functional descriptions; screen\-level checks verify navigation, foreground, blocking, and selection states; cross\-frame checks verify the observed transition and its direction\. Ambiguous or incorrectly annotated instances are removed\.

Verification coverage\.All 9,061 retained instances receive human review, covering 4,706 L1, 1,064 L2, and 3,291 L3 instances\. This constitutes 100% review coverage, not an independently measured annotation\-accuracy rate\. Final annotations are determined through human inspection of visual evidence rather than model agreement alone\. Review distinguishes foreground presence from interaction blocking, element visibility from selection, and intended actions from observed outcomes\. Benchmark reference annotations are verified separately from the model\-generated summaries evaluated for memory fidelity\.

### F\.3Evaluation Protocols

Task\-specific scoring\.Semantic\-to\-BBox uses region hit rate: predicted coordinates are mapped to the original image and checked against the reference bounding box, with parsing failures recorded separately\. Active\-Tab, Blocking\-State, and foreground\-presence judgments use accuracy, while Screen\-Selection uses rule\-based matching of selected items and values\. BBox\-to\-Function and Screen\-State Diff support MCQ and LLM\-Judge\([Zheng et al\., 2023](https://arxiv.org/html/2610.22408#bib.bib28)\)evaluation; foreground descriptions can additionally be assessed semantically\. MCQ distractors are preferentially drawn from same\-screen elements or semantically similar alternatives\. Task\-specific answer extraction preserves correctness\-critical distinctions, including numerical values, state polarity, element identity, and transition direction\.

Direct vision and memory replay\.The 1,064 L2 instances support paired evaluation\. Each model first generates its own textual screen summary, covering foreground information, the active navigation tab, and selected items or values\. Summaries are cached for reuse across applicable queries\. The same model then answers each query using either the screenshot or its generated summary alone, with identical questions and reference answers across conditions\. Memory\-only inputs contain no screenshot and do not substitute human reference captions or summaries from another model\. This screen\-level protocol measures visual\-to\-text state retention, complementing the long\-horizon experiments on recursive CTA execution\.

PMF Gap\.For L2 taskkk, letSkVS\_\{k\}^\{V\}andSkMS\_\{k\}^\{M\}denote normalized task scores on matched instances under direct\-vision and memory\-only inputs, respectively\. We compute

GPMF,k=max⁡\(0,SkV−SkM\),G¯PMF=14​∑k∈𝒦L​2GPMF,k,G\_\{\\mathrm\{PMF\},k\}=\\max\(0,S\_\{k\}^\{V\}\-S\_\{k\}^\{M\}\),\\qquad\\overline\{G\}\_\{\\mathrm\{PMF\}\}=\\frac\{1\}\{4\}\\sum\_\{k\\in\\mathcal\{K\}\_\{L2\}\}G\_\{\\mathrm\{PMF\},k\},where𝒦L​2\\mathcal\{K\}\_\{L2\}contains the four whole\-screen tasks\. The nonnegative operation is applied to each task\-level difference before unweighted macro\-averaging; L1 and L3 are excluded\. Smaller gaps indicate less performance degradation after replacing visual evidence with text, but must be interpreted alongside direct\-vision scores rather than as proof of lossless memory\. These diagnostic test scores are separate from the validation\-set privileged–deployable gap used to schedule OPD–RL training\.

### F\.4Task Examples in ECHOESBMK

Figure[7](https://arxiv.org/html/2610.22408#A6.F7)presents one example for each of the seven diagnostic tasks in ECHOESBMK\. The tasks span three levels: element\-level perception \(L1\), whole\-screen state understanding \(L2\), and cross\-frame state changes \(L3\)\. Each panel pairs a task prompt with the corresponding visual observation and example answer, with enlarged regions highlighting the relevant interface details\.

#### L1: Element\-level perception\.

Panels \(a\) and \(b\) illustrate two complementary directions of element understanding\.BBox\-to\-Functionasks the model to infer the function of a highlighted interface element, such as recognizing that the boxed download button saves a book for offline reading\. Conversely,Semantic\-to\-BBoxprovides a functional description and asks the model to localize the corresponding element, such as identifying the share button for a rain\-sound session\. Together, these tasks examine both the interpretation of localized visual elements and the grounding of semantic requests in the interface\.

#### L2: Whole\-screen state understanding\.

Panels \(c\)–\(f\) examine four aspects of the current screen state\.Active\-Tabidentifies the selected navigation tab, which is “All” in the example\.Foreground\-Layerdetects and describes a foreground interface layer, here a confirmation modal for leaving dictation\.Blocking\-Statedetermines whether the requested interaction is blocked and identifies the blocking condition; the example shows a VIP\-membership paywall preventing playback\. Unlike foreground\-layer detection, this task concerns whether the requested operation can proceed, rather than merely whether an overlay is present\. Finally,Screen\-Selectionidentifies currently selected options, illustrated by a 30% down payment and a loan term of 36 installments\. These tasks assess navigation state, foreground presentation, interaction constraints, and selected values beyond locating individual interface elements\.

#### L3: Cross\-frame state changes\.

Panel \(g\) illustratesScreen\-State Diff, which compares a before–after screenshot pair rather than a single observation\. The example requires identifying that the 22:15 alarm changes from off to on and that the next\-alarm countdown also updates\. The target is the change between observations, rather than an independent description of each screen\.

The examples retain task\-specific output formats, including functional descriptions, coordinate localization, categorical judgments, selected\-value sets, and change descriptions, rather than uniformly converting all tasks into action prediction\. They illustrate the direct\-vision component of ECHOESBMK; memory\-fidelity analysis additionally compares direct\-vision and text\-only memory\-replay scores on the four L2 tasks\.

![Refer to caption](https://arxiv.org/html/2610.22408v1/ECHOESBMK_category_examples.png)Figure 7:Representative examples of the seven ECHOESBMK tasks\.Panels \(a\)–\(b\) illustrate L1 element\-level perception: BBox\-to\-Function and Semantic\-to\-BBox\. Panels \(c\)–\(f\) illustrate L2 whole\-screen state understanding: Active\-Tab, Foreground\-Layer, Blocking\-State, and Screen\-Selection\. Panel \(g\) illustrates L3 cross\-frame state changes through Screen\-State Diff\. Each example includes a task prompt, visual observation\(s\), and a corresponding answer\. Full screenshots are accompanied by enlarged details for readability\.