Paper page - MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

Source: https://huggingface.co/papers/2605.18652

Abstract

MementoGUI presents a memory framework for GUI agents that uses learned controllers for selective memory management and retrieval, improving long-horizon task performance through compressed visual and textual representations.

RecentGUI agentshave made substantial progress invisual groundingandaction prediction, yet they remain brittle in long-horizon tasks that require maintaining task state across many interface transitions. Existing agents typically rely on raw history replay or text-only memory, which either overwhelms the model with redundant screenshots or discards localized visual evidence needed for future decisions. To address these limitations, we introduce MementoGUI, a plug-in agentic memory framework that equipsMLLM-basedGUI agentswithMementoCore, a learned controller for onlinememory selection, compression, and retrieval. Rather than treating interaction history as a fixed context, MementoGUI formulates long-horizon GUI control as an onlinememory-control problem:working memoryselectively preserves task-relevant interface events with textual summaries and ROI-level visual evidence, whileepisodic memoryretrieves reusable past trajectories through learned relevance selection.MementoCoremodularizes memory control into specialized operators for step processing,memory compression, episodic writing, and episodic selection, enabling plug-in memory augmentation without finetuning the GUI agent backbone. We further develop a scalable data curation pipeline that converts computer-use trajectories into memory-controller training data, introduce MementoGUI-Bench for evaluating long-horizon decision-making inGUI agents, and designMLLM-basedmetrics forsemantic action matching,task progress, andmemory consistency. Experiments on GUI-Odyssey, MM-Mind2Web, and MementoGUI-Bench show that MementoGUI consistently improvesGUI agentsover no-history, history-replay, and text-only memory baselines, with largerMementoCorebackbones further strengthening memory-augmented GUI control.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2605\.18652

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2605.18652 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2605.18652 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2605.18652 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.