Paper page - The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents

Source: https://huggingface.co/papers/2608.06065

Abstract

Gated Hindsight Distillation improves GUI agent training by using future screenshots as privileged evidence to recover correct reasoning when standard imitation fails.

GUI agents are commonly trained offline from successful interaction trajectories. Standard training decomposes each trajectory into prefix-action pairs: the agent predicts an action from the current screen and interaction history, while the subsequent observation is discarded. This removes the rationale of why an action is correct: the evidence often appears only on the subsequent screen. For example, to enable Soft Wrap, the agent should click Edit or View, but nothing reveals this until the menu opens. Without such evidence, standard imitation gives the model little chance of ever sampling and thus learning the correct reasoning. To address this issue, we proposeGated Hindsight Distillation(GHD), which uses the next screenshot asprivileged informationduring training. A student predicts from the observable trajectory prefix, while aparameter-sharing teacheradditionally observes the next screenshot and re-scores the student’son-policy responses. We apply distillation only when the student fails and the hindsight-conditioned teacher recovers the demonstrated action. GHD improves task success overGRPOon AndroidWorld and AndroidLab across twovision-language models. The code and checkpoints will be made available.

View arXiv pageView PDFAdd to collection

Get this paper in your agent:

hf papers read 2608\.06065

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.06065 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.06065 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.06065 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.