Paper page - Focused Omni-Modal GUI Agents

Source: https://huggingface.co/papers/2610.26203

Abstract

FocusGUI is a lightweight adapter that dynamically selects relevant multimodal evidence for GUI agents to prevent compounding errors from irrelevant context, improving action accuracy without fine-tuning the backbone.

Graphical User Interface (GUI) agents increasingly act over streamingomni-modal observationsspanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound intoSequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement asFocused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this inFocusGUI, a lightweightadapterthat dynamically prioritizesmultimodal channelsagainst a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compactlatentand subtly injects it into thefrozen decoder’sembeddings, guiding the model’s attention without altering its native parameter space.FocusGUItrains fromaction supervisionalone, requiring neither modality-relevance labels norbackbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.

View arXiv pageView PDFProject pageGitHub4Add to collection

Get this paper in your agent:

hf papers read 2610\.26203

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.26203 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.26203 in a dataset README.md to link it from this page.

Spaces citing this paper1

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.