@_akhaliq: StateAct Program State, before Pixels, for Long-Horizon Computer-Use Agents paper: https://huggingface.co/papers/2607.2…
Summary
StateAct proposes a code-first, multi-agent system for long-horizon computer-use agents that operates directly on program state instead of screenshots, achieving higher success rates and lower costs on OSWorld2.0.
View Cached Full Text
Cached at: 07/28/26, 06:31 AM
StateAct
Program State, before Pixels, for Long-Horizon Computer-Use Agents
paper: https://t.co/zoRp9WfYsq https://t.co/BBSSxFCX1e
Paper page - StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Source: https://huggingface.co/papers/2607.22798
Abstract
Computer-useagentsareusuallyimprovedbystrengtheningperception:bettermodelsforreadingascreenshotandchoosingwheretoclick.Yetascreenshotisonlyalossyrenderingoftheunderlyingprogramstate,e.g.,thefiles,applicationbackends,andDOMthatholdthetaskdata.Differentstatescanproducethesamepixels,whilecodecaninspectandmodifythatstatedirectly.StateActisacode-first,multi-agentharnessbuiltaroundthisdistinction.Itsmainagentworksdirectlywithprogramstatebyusingcode,whileadedicatedGUIsubagenthandlesscreenshot-and-clickinteractiononthefewsubgoalsthatneedit,just28of108tasksand1.1%ofmain-agentsteps.Thesamedirectaccesstoprogramstatealsosupportsverification:anindependentfinishgatedouble-checksthesavedresultforstructuralfailures,e.g.,outputthatismissing,unsaved,orwrittentothewrongpath.Tostayontrackoverhundredsofsteps,themainagenthandssubgoalstofreshsubagents,keepingitsowncontextfocused.OnOSWorld2.0,StateActliftsClaudeOpus4.8from20.6%to26.9%onbinarysuccess,andfrom54.8%to61.6%onpartialsuccess,at~9xlowercostpertaskthanthesamemodeldrivenbyscreenshotsalone;acode-onlyvariantwithnoGUIsubagentreachesonly45.9%partial,belowthatscreenshot-basedbaseline’s54.8%.Ingeneral,groundingaction,verification,andmemoryinstate,whatwecallstate-grounding,shiftsthemainbottleneckfromperceptiontowardreasoning:failuresdependmoreonwhattheagentthinksthanonwhatitsees.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2607\.22798
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2607.22798 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2607.22798 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2607.22798 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
@dair_ai: Outstanding paper on computer-using agents. (bookmark it) Computer-using agents drive real software through the screen,…
PreAct compiles successful agent runs into small state-machine programs, enabling 8.5-13x faster replay on repeated tasks without per-step language model calls, with runtime screen checks to ensure correctness.
PreAct: Computer-Using Agents that Get Faster on Repeated Tasks
PreAct compiles successful task runs of computer-using agents into small state-machine programs, allowing fast replay (8.5–13× faster) on repeated tasks by skipping per-step language model calls, while verifying screen states at each step and falling back to the agent when mismatches occur.
What Should Agents Say? Action-state Communication for Efficient Multi-Agent Systems
This paper introduces PACT, a method for structuring agent-to-agent communication in multi-agent LLM systems that uses compact action-state records to reduce token consumption while maintaining or improving task performance, with demonstrated gains on SWE-agent and OpenHands.
StateComp: Learning When to Compress History in Long Horizon Agents
StateComp is a framework that compresses historical interactions in long-horizon agents based on the current state, reducing token usage by 52.27% and achieving a 12.67× speedup in representation extraction while maintaining task performance.
@yoheinakajima: more ppl are now trying out this approach of agents communicating with a shared state (vs talking to each other)
Azalia Mirhoseini highlights DeLM, a decentralized language model approach where agents communicate via shared state, achieving ~10% improvement on SWE-bench Verified with Gemini-3 Flash at less than half the cost.