Unified Agent: Managing Interactions across Devices

# Unified Agent: Managing Interactions across Devices
Source: [https://arxiv.org/html/2608.05729](https://arxiv.org/html/2608.05729)
Xinshuang Liu Runfa Blark Li Shaoxiu Wei Xin Lin Truong Nguyen University of California, San Diego San Diego, CA, USA \{xil235, rul002, shwei, xil321, tqn001\}@ucsd\.edu

###### Abstract

As capabilities rapidly increase, AI agents can move from running inside one app to acting across a user’s devices over time\. Yet existing agent systems still fall short in this scenario\. This is because observations are scattered across devices and moments, but mainstream systems are not designed around this fact: a single agent that treats devices as tools lacks effective state management for all devices across time, and multi\-agent systems coordinate across agents but do not maintain the compact carried state a cross\-device, cross\-time request needs\. We argue that the agent should maintain an effectively designed state that organizes engagement evidence, stated facts, and the standing request in a compact, action\-ready form for deciding its action given the current observation\. To compare state designs, we construct a benchmark of user\-agent interaction across devices and time\. We instantiate this principle in Unified Agent, a stateful agent that carries interaction evidence across devices and moments and uses it with the current observation to act\. In the default setting, it significantly outperforms our adaptations of four published designs\. Across changes in multimodal large language model \(MLLM\) family, capability, and reasoning effort, it remains ahead of all compared systems, demonstrating that the state\-design advantage is robust across MLLM settings\. Our code and data will be publicly available on GitHub\.

Unified Agent: Managing Interactions across Devices

Xinshuang Liu Runfa Blark Li Shaoxiu Wei Xin Lin Truong NguyenUniversity of California, San DiegoSan Diego, CA, USA\{xil235, rul002, shwei, xil321, tqn001\}@ucsd\.edu

## 1Introduction

![Refer to caption](https://arxiv.org/html/2608.05729v1/x1.png)Figure 1:One agent serving one user across devices and over time\.After comparing restaurants on several devices, the user later decides to make a reservation but has forgotten the restaurant’s name\. The user therefore asks to book “the restaurant I’ve been looking at” without specifying a device\. A stateless agent may need to ask, “Which device was it on?” In contrast, Unified Agent carries a compact, action\-ready state and resolves the reference to the restaurant on the phone, where the user’s engagement was strongest\. Better state design makes the request answerable\.As capabilities rapidly increase, AI agents can move from acting inside a single app to managing tasks across a user’s devices over time\. Figure[1](https://arxiv.org/html/2608.05729#S1.F1)shows an everyday scenario: a user compares restaurants on several devices, and the agent must later interpret a device\-unspecified request\. However, today’s mainstream agent systems are not designed for this setting\. A single\-agent system can treat devices as tools, but without an effective cross\-time state, it treats each request as fresh; once the deciding cue has passed, the relevant earlier evidence is no longer available\. Multi\-agent systems can distribute work and coordinate across agents, but without a unified cross\-device state, they can still fail when a user’s request depends on information spread across devices and moments\.

We argue that an agent needs a well\-organized state integrating information across devices and moments\. In Figure[1](https://arxiv.org/html/2608.05729#S1.F1), a stateless agent can only ask, “Which device was it on?”; an agent whose state has absorbed core interaction evidence across time and devices instead infers that the user engaged most with the phone and books the restaurant there\.

We therefore build Unified Agent, which maintains a state across time and devices\. Unified Agent updates the state with each observation and presents the evidence relevant to the current decision in a compact summary\. Each action is selected using both the state and the latest observation\. The state records*engagement evidence*,*stated facts*, and the*standing request*\. It does not need to store a transcript of everything seen; in the example, retaining that the phone received the strongest engagement is what lets Unified Agent resolve “the restaurant I’ve been looking at” to the restaurant on the phone\. The action\-facing summary should hold what a later action may need once the original cue is gone\.

To compare state designs, we construct a benchmark of user\-agent interaction across time and devices\. Our benchmark presents one user interacting with one agent across multiple devices in a 3D scene\. Interactions share a designed cross\-device sequence across varied scenes, layouts, device placements, topics, and actions, so the correct answers are known by construction and the rendered visual semantics provide ground truth for construction checks\. Real\-photo results ground the findings in practice\. On this benchmark, Unified Agent significantly outperforms our adaptations of four published designs in the default setting\. Across changes in MLLM family, capability, and reasoning effort, it remains ahead of all compared systems, showing that the state\-design advantage is robust across MLLM settings\. Since Unified Agent adds no extra mechanism beyond the carried state, the gain traces to how the state is represented and used, not to added system complexity\.

Our contributions are summarized as follows:

- •We formulate the problem of user\-agent interaction across time and devices\.
- •We introduce a unified agent that manages a compact state of engagement evidence, stated facts, and the standing request, and acts based on that state and the latest observation\.
- •We construct a benchmark to study user\-agent interaction across devices and time\.
- •We find that Unified Agent significantly outperforms our adaptations of four published designs in the default setting and leads all compared systems across MLLM families, capabilities, and reasoning efforts\.

![Refer to caption](https://arxiv.org/html/2608.05729v1/x2.png)Figure 2:Six tasks organized as observe–think–act\.The agent perceives the current interaction and identifies the engaged device, reasons with information carried across moments, and determines how to act across devices\.
## 2Related Work

#### Reading the present\.

An agent serving one user across multiple devices often receives only a local observation from a single device camera and the current utterance, while the prior interaction history is not re\-supplied\. The immediate challenge is therefore attribution: which device is the user engaging with, and what evidence remains once the cue disappears? Prior work infers addressees and engagement from cues such as utterances, gaze, and pointingInoue et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib15)\); Yu et al\. \([2025b](https://arxiv.org/html/2608.05729#bib.bib56)\); Li et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib25)\); Lee et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib24),[2025](https://arxiv.org/html/2608.05729#bib.bib23)\); cross\-view grounding binds references to perspectiveTang et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib43)\)\. Once the actor or target is identified, device\-local and embodied agents can act through screen policies, tool use, or robot affordancesQin et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib35)\); Agashe et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib4)\); Han et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib11)\); Acikgoz et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib3)\); Ichter et al\. \([2022](https://arxiv.org/html/2608.05729#bib.bib14)\); Huang et al\. \([2022](https://arxiv.org/html/2608.05729#bib.bib13)\); Liang et al\. \([2023](https://arxiv.org/html/2608.05729#bib.bib26)\); Zitkovich et al\. \([2023](https://arxiv.org/html/2608.05729#bib.bib62)\); Yao et al\. \([2023](https://arxiv.org/html/2608.05729#bib.bib53)\)\. Situated agents further track intent as a user moves, clarify referents, negotiate roles, or infer household intent over timeMin et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib31)\); Yu et al\. \([2025a](https://arxiv.org/html/2608.05729#bib.bib55)\); Abrini and Chetouani \([2026](https://arxiv.org/html/2608.05729#bib.bib2)\); Ma et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib28)\); Arakawa et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib5)\)\. These works primarily address present\-cue interpretation or action once the actor or target has been identified\. We study the complementary setting, where engagement evidence must be bound to a physical device and preserved for a later device\-unspecified request\. Unified Agent addresses this setting by carrying device\-bound engagement evidence beyond the moment in which it is visible\.

#### Carrying state across moments\.

A second line of work examines what should persist across turns\. Situated dialogue systems track belief or common ground over shared scenesVanderHoeven et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib47)\); Kottur et al\. \([2021](https://arxiv.org/html/2608.05729#bib.bib19)\); Abdessaied et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib1)\); Udagawa and Aizawa \([2019](https://arxiv.org/html/2608.05729#bib.bib46)\); Teng et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib45)\), with text\-only slot–value tracking providing an earlier formBudzianowski et al\. \([2018](https://arxiv.org/html/2608.05729#bib.bib6)\)\. Agent memory systems store and retrieve extracted facts, evolving notes, temporal graphs, or user profilesChhikara et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib8)\); Xu et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib51)\); Packer et al\. \([2023](https://arxiv.org/html/2608.05729#bib.bib33)\); Rasmussen et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib36)\); Shu et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib41)\); Shaikh et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib40)\); Yao et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib54)\); Rezazadeh et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib37)\)\. Recent work also frames shared state as a missing layer over present snapshotsWang et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib49)\)and argues for explicit world models as common ground for robotsKwok et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib21)\)\. Memory benchmarks evaluate recall and update in long conversations, first\-person histories, or changing living worldsMaharana et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib29)\); Wu et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib50)\); Yadav et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib52)\); Meng et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib30)\)\. These system\-oriented methods provide durable stores, typically within a channel, scene, profile, or task, while the benchmarks test whether agents can retain and use information\. Unified Agent instead addresses the cross\-device case, where the current observation is insufficient and later decisions require earlier evidence: it preserves per\-device engagement evidence, stated facts, and the standing request in one carried state\.

#### Acting across devices\.

A third line of work examines how an intent is executed across devices or agents\. Cross\-device handoff maintains continuity by moving an artifact, such as a file, clipboard, or assistant embodiment, in response to a sensed cueJoshi et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib17)\); Tejwani et al\. \([2021](https://arxiv.org/html/2608.05729#bib.bib44)\)\. Orchestrators decompose a given intent and route it through enumerated capabilities, endpoints, or active actors, from capability\-routed task graphs to robot teams and smart homesZhang et al\. \([2025a](https://arxiv.org/html/2608.05729#bib.bib59),[b](https://arxiv.org/html/2608.05729#bib.bib61)\); Tan et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib42)\); Hasan et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib12)\); Rivkin et al\. \([2023](https://arxiv.org/html/2608.05729#bib.bib38)\); Zhan et al\. \([2026](https://arxiv.org/html/2608.05729#bib.bib58)\)\. Embodied multi\-agent systems coordinate preassigned agents executors on a shared taskChang et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib7)\); Zhang et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib60)\); Liu et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib27)\)\. These systems provide multi\-device control, but typically resolve the target from the present request, a transferred payload, or a static set of registered endpoints rather than from engagement evidence accumulated over time\. Unified Agent instead first infers the relevant device from carried engagement evidence, then recalls the information associated with that device and routes the request\. This yields a compact cross\-device state that is action\-ready for the user’s requests, without learned memory, retrieval, or fine\-tuning\. These properties make the design compatible with different agent systems while supporting efficient integration\.

## 3Problem Formulation

We study one agent acting for one user across heterogeneous devices\. The agent coordinates these devices through a shared decision interface\. It uses language to specify which devices should respond and what they should do, while each selected device carries out the instruction using native controls that its manufacturer is best positioned to design and maintain\. This separation allows one agent to serve heterogeneous devices while each device retains responsibility for its own execution\. The interaction unfolds as a sequence of moments\. At each moment, the agent receives one local observation through one device camera, with no global room view and no replay of earlier moments, paired with the user’s words\. The hard case is a device\-unspecified request whose deciding cue has already passed: the engaged screen may be dark, the pointing gesture may be gone, and the request may say only “the one I have been using\.” The agent must still decide from the current observation and whatever it has carried forward\.

We organize each moment around the six observe–think–act tasks shown in Figure[2](https://arxiv.org/html/2608.05729#S1.F2):*Detect user and devices*;*Identify engaged device*\(Eng\);*Infer user’s intent*\(Int\);*Recall relevant information*\(Inf\);*Pick responding device*\(Rsp\); and*Decide next action*\(Nxt\)\. The first is the perception task; the remaining five are downstream decisions made from the current perception and whatever information the agent carries forward\. Rsp is represented as a device set that may be empty, so the agent is not forced to name a device when none applies\. Together, the six tasks define the cross\-device interaction problem independently of any particular implementation\.

The decisions draw on the current view and earlier evidence: which device the user engaged, which facts were stated for that device, and what standing request the user made\. A useful cross\-device agent therefore needs carried state that is leaner than raw history and more actionable than a list of previous predictions\. Unified Agent carries this evidence forward, so later requests remain answerable when the original cue is no longer visible\.

![Refer to caption](https://arxiv.org/html/2608.05729v1/x3.png)Figure 3:Overview of Unified Agent\.At momenttt,St−1S\_\{t\-1\}carries evidence accumulated across devices over moments1,…,t−11,\\ldots,t\-1\. The agent folds the current observationOtO\_\{t\}intoSt−1S\_\{t\-1\}to produceStS\_\{t\}, which stores engagement evidence, stated facts, and the standing request\. It then acts by readingStS\_\{t\}together withOtO\_\{t\}, and carriesStS\_\{t\}forward to the next moment\.
## 4Method

An agent acting across a user’s devices must decide each action from evidence spread across devices and moments\. This evidence may come from an earlier moment or another device and may no longer be available when the agent must act\. The agent must therefore carry information forward\. Retaining the full history is costly: it grows with the interaction and competes for the model’s context window, while a later action typically depends on only a small part of it\. The agent needs a compact carried state that preserves the evidence later actions need\.

#### The carried state\.

Unified Agent meets this need with one carried stateStS\_\{t\}\. It folds each new observation into the state and reads the updated state together with the current observation to act \(Figure[3](https://arxiv.org/html/2608.05729#S3.F3)\)\. The state keeps*evidence*in three streams, each serving a distinct need of later action\.*Engagement evidence*accumulates device\-associated interaction cues over time, supporting identification of the relevant device\.*Stated facts*preserve user\-provided information by topic and, when identifiable, by device, so same\-topic facts remain distinguishable across devices\. The*standing request*retains the latest requested action\. Together, the three streams carry evidence rather than decoded downstream answers\. These explicit records are updated with each observation and remain directly available to later decisions\.

#### Updating the state\.

At each momenttt, the agent folds the current observationOtO\_\{t\}into the state:

St=U​\(St−1,Ot\)\.S\_\{t\}=U\(S\_\{t\-1\},\\,O\_\{t\}\)\.\(1\)The update incorporates new evidence while carrying earlier evidence forward\. It accumulates engagement evidence, records stated facts with any identifiable device association, and updates the standing request when a new actionable request is recognized\. Because folding precedes action, evidence inOtO\_\{t\}can affect the current decision\.

#### Acting from the state\.

The agent acts by reading the updated state together with the current observation:

at=D​\(St,Ot\)\.a\_\{t\}=D\(S\_\{t\},\\,O\_\{t\}\)\.\(2\)Here,OtO\_\{t\}provides evidence from the current moment, whileStS\_\{t\}carries evidence accumulated through that moment\. Reading both lets evidence from an earlier interaction cue inform action after the cue is no longer present inOtO\_\{t\}, while new evidence can revise the decision\. The agent uses engagement evidence to identify the relevant device and stated facts to recover information when needed\. Device capabilities are supplied separately rather than carried inStS\_\{t\}and are omitted from the notation\. The decoder obtains the action from the standing request and combines it with engagement evidence and device capabilities to decide which device or devices should respond\. Unified Agent communicates the resulting instruction in language through the shared interface, while each selected device carries it out using its native controls\. This separation is consistent with prior work on tool use and robot skillsSchick et al\. \([2023](https://arxiv.org/html/2608.05729#bib.bib39)\); Ichter et al\. \([2022](https://arxiv.org/html/2608.05729#bib.bib14)\)\.

## 5Benchmark

OurUA\-Benchis a reproducible benchmark of interaction across devices and time\.UA\-Benchtargets devices in a 3D space\. The agent must combine local device\-camera views of user–device interaction with the user’s words before deciding\. Controlled rendering preserves the need for visual and spatial reasoning about situated device interaction\. As in visual and long\-context benchmarksJohnson et al\. \([2017](https://arxiv.org/html/2608.05729#bib.bib16)\); Kuratov et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib20)\), it also provides exact semantic ground truth for construction checks and supports systematic variation and reproducibility\.

An episode is a structured cross\-device interaction in a rendered 3D homePuig et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib34)\); Khanna et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib18)\), observed through device cameras and paired at each frame with the user’s words \(Figure[4](https://arxiv.org/html/2608.05729#S5.F4)\)\. The cast includes a rendered user and screen devices such as a computer, a laptop, and a phone; episodes may additionally field a fetch robot, which extends the responding\-device choices beyond screens\. Each episode follows a designed temporal progression: an initial observation establishes the scene; an earlier interaction establishes user–device engagement and associated context through visual and linguistic cues; and a later device\-unspecified request and recall questions test whether that information remains available as the immediate visual evidence changes\.UA\-Benchinstantiates the six tasks of Section[3](https://arxiv.org/html/2608.05729#S3)\. Device\-camera observations provide the visual input for*Detect user and devices*; the five downstream targets are*Identify engaged device*\(Eng\),*Infer user’s intent*\(Int\),*Recall relevant information*\(Inf\),*Pick responding device*\(Rsp\), and*Decide next action*\(Nxt\)\. Evaluation targets are determined directly from the controlled episode specification rather than from model\-generated annotations, so every target is exact by construction\.

![Refer to caption](https://arxiv.org/html/2608.05729v1/figures/bench_example_rgb.png)

\(a\) Device\-camera view

![Refer to caption](https://arxiv.org/html/2608.05729v1/figures/bench_example_sem.png)

\(b\) Semantic map

Figure 4:RenderedUA\-Benchframe\.\(a\) An agent\-visible device\-camera view shows user–device interaction in a scene containing screen devices and a fetch robot\. \(b\) The paired semantic ground truth supports construction\-time validation of object identity and visibility and is not available to evaluated systems\.Benchmark construction information is strictly separated from evaluated\-system inputs\. Semantic ground truth and render metadata are used only during benchmark construction to establish exact ground truth for entity visibility and validate object identity and screen state\. System inputs are derived solely from rendered device\-camera views, the user’s words, and a public description of the task and devices, including device identities, appearances, and capabilities; the controlled episode specification and evaluation targets remain unavailable to evaluated systems\. Together, these system inputs and evaluation targets provide a common basis for comparing different cross\-device agent designs\. Within each MLLM setting, all MLLM\-based methods are evaluated using a shared per\-frame perception output\. When the decoder is also shared, the comparison isolates differences in state design while preserving Unified Agent’s perceive–fold–act pipeline\.

UA\-Benchcontains 100 matched pairs \(200 samples\) spanning varied indoor scenes, layouts, device placements, topics, and actions\. Each pair varies the preceding interaction under a common task structure and evaluation protocol, enabling direct comparison of how systems use information carried from earlier moments\. Every admitted episode passes automatic checks of construction consistency, user and candidate\-device visibility, and matched\-pair integrity\. Rendered batches also undergo human review for image legibility and rendering quality\. Appendix[B](https://arxiv.org/html/2608.05729#A2)documents the construction and admission protocol\. Section[6\.5](https://arxiv.org/html/2608.05729#S6.SS5)complements the rendered benchmark with photographs\.

Table 1:State\-design comparison\.Per\-system scores for the five downstream decisions on GPT\-5\.6\-LunaOpenAI \([2026](https://arxiv.org/html/2608.05729#bib.bib32)\)\(low reasoning effort\); bold marks the best overall performance\. With its compact, action\-ready state, Unified Agent achieves the highest overall performance among the compared systems without retaining raw history\.Decision score \(↑\\uparrow\)SystemEngIntInfRspNxtOverallGap \[95% CI\]Unified Agent \(Ours\)0\.7770\.3070\.7450\.7340\.7770\.668\[\.64,\.70\]State controlsFull context0\.7650\.3380\.7350\.7570\.4720\.613 \[\.58,\.64\]\+0\.055 \[\+0\.035, \+0\.075\]Self\-notes0\.7600\.3310\.2070\.7360\.3670\.480 \[\.46,\.50\]\+0\.188 \[\+0\.160, \+0\.214\]Answer cache0\.1720\.3210\.0000\.6680\.0570\.244 \[\.23,\.26\]\+0\.424 \[\+0\.389, \+0\.459\]Text only0\.0350\.3690\.0000\.6470\.0000\.210 \[\.20,\.22\]\+0\.458 \[\+0\.426, \+0\.489\]Observation only0\.0320\.3540\.0000\.6050\.0000\.198 \[\.19,\.21\]\+0\.470 \[\+0\.437, \+0\.502\]Published baselinesMixture\-of\-Agents0\.6800\.2990\.4800\.6970\.2150\.474 \[\.45,\.50\]\+0\.194 \[\+0\.161, \+0\.226\]Debate\-or\-vote0\.7030\.3010\.4350\.6890\.2420\.474 \[\.45,\.50\]\+0\.194 \[\+0\.161, \+0\.225\]Mem00\.0820\.3040\.3950\.6240\.0350\.288 \[\.27,\.31\]\+0\.380 \[\+0\.338, \+0\.420\]MM\-DST0\.1950\.2710\.0250\.6490\.1620\.260 \[\.24,\.28\]\+0\.408 \[\+0\.363, \+0\.451\]
Appendix[D](https://arxiv.org/html/2608.05729#A4)details the setup of each compared system\.Overallis the macro\-average of the five downstream decisions, giving each required decision equal weight regardless of its number of scored instances; its brackets show the one\-sample 95% bootstrap interval clustered by matched pair \(Appendix[E](https://arxiv.org/html/2608.05729#A5)\)\.Gapis the difference between Unified Agent’s Overall score and that of each comparison system\. Brackets report the corresponding paired 95% interval; positive values favor Unified Agent\. All gaps remain significant after Holm adjustment within their respective comparison families \(p<10−3p<10^\{\-3\}\)\.

## 6Experiments

We first compare Unified Agent with state controls and with memory and multi\-agent baselines\. We then test how key design elements support later decisions, whether the advantage generalizes across MLLM settings, and whether the same principle holds on real photographs\.

### 6\.1Experimental setup

![Refer to caption](https://arxiv.org/html/2608.05729v1/x4.png)Figure 5:Carried\-state size\.Characters of retained values after each frame, averaged over the 200 episodes; fixed template text is excluded identically for every system\. Unified Agent stays bounded; Full context grows linearly to 11 times the size of Unified Agent’s state\.We compare methods under a common evaluation harness\. Within each MLLM setting, all MLLM\-based comparison systems use fixed zero\-shot prompt templates and share the same perception record, model, final output schema, and scorer, while each retains its own mechanism for representing and carrying information\. Appendices[C](https://arxiv.org/html/2608.05729#A3)and[D](https://arxiv.org/html/2608.05729#A4)detail the evaluation protocol, model and inference settings, output schema, and baseline implementations used in the reported comparisons\. We evaluate four MLLM settings that vary in reasoning effort, model family, and model capability\. Unless otherwise noted, GPT\-5\.6\-LunaOpenAI \([2026](https://arxiv.org/html/2608.05729#bib.bib32)\)at low reasoning effort is the default setting\. Relative to the default, GPT\-5\.6\-Luna at medium reasoning effort varies the reasoning setting; Gemini 3\.1 Flash\-LiteGoogle DeepMind \([2026](https://arxiv.org/html/2608.05729#bib.bib10)\)at low thinking level varies the model family; and GPT\-5\.6\-SolOpenAI \([2026](https://arxiv.org/html/2608.05729#bib.bib32)\)at low reasoning effort varies model capability\. Within each MLLM setting, every MLLM\-based method receives a common per\-frame perception output\. The methods differ in what information they carry forward and how they use it\. This protocol holds the perception input constant across methods, enabling a direct comparison\. Table[8](https://arxiv.org/html/2608.05729#A3.T8)in Appendix[C\.3](https://arxiv.org/html/2608.05729#A3.SS3)reports the quality of shared perception across settings\. Unified Agent continues to perceive, fold each observation into its state, and act from the updated state\.

We report visual\-perception diagnostics supporting*Detect user and devices*separately from the five downstream decisions\. Holding perception fixed within each MLLM setting, the comparison scores Eng, Int, Inf, Rsp, and Nxt\. Engagement, recall, routing, and next action are graded deterministically against by\-construction ground truth; intent is scored by a fixed same\-meaning equivalence MLLM judge \(Appendix[C\.4](https://arxiv.org/html/2608.05729#A3.SS4)\)\. Uncertainty is quantified by a bootstrap clustered on matched pairs \(Appendix[E](https://arxiv.org/html/2608.05729#A5)\)\.

### 6\.2State\-design comparison

Finding 1:A well\-organized carried state provides a performance advantage for cross\-device agents: Unified Agent achieves the highest overall score among all compared systems\.

This subsection examines a central question in state design: how accumulated information should be represented and used for later action\. We compare Unified Agent with the five state controls of Table[1](https://arxiv.org/html/2608.05729#S5.T1)and with four published\-design baselines \(Appendix[D\.2](https://arxiv.org/html/2608.05729#A4.SS2)\):

- •Memory and dialogue\-state methods:Mem0Chhikara et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib8)\)and MM\-DST \(multimodal dialogue state tracking\)Le et al\. \([2022](https://arxiv.org/html/2608.05729#bib.bib22)\); Abdessaied et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib1)\)\.
- •Multi\-agent methods:Mixture\-of\-AgentsWang et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib48)\)and Debate\-or\-voteChoi et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib9)\)\.

Table[1](https://arxiv.org/html/2608.05729#S5.T1)shows that Unified Agent leads all compared systems in overall performance, including the strongest control, Full context\. A paired bootstrap clustered on matched pairs places a 95% confidence interval on every reported gap, and each gap remains significant after Holm correction within its comparison family \(all adjustedp<10−3p<10^\{\-3\}; Appendix[E](https://arxiv.org/html/2608.05729#A5)\)\. Full context, which retains all prior perception records and utterances from which Unified Agent derives its state, scores slightly higher on intent and routing, making it a strong control\. Figure[5](https://arxiv.org/html/2608.05729#S6.F5)reports the size of the carried state across the interaction for both designs\. Full context grows linearly because it retains every prior frame; Unified Agent’s state scales with the observed devices and stated facts rather than with the interaction length, and therefore remains bounded\. Unified Agent achieves the highest overall performance, with its largest advantage on next action\.

Across individual decisions, Unified Agent leads every published design on engagement, recall, routing, and next action\. Mem0 and MM\-DST manage cross\-frame information through retrieved fact memory and updated dialogue state, while Mixture\-of\-Agents and Debate\-or\-vote coordinate device agents through proposal synthesis and peer\-feedback voting\. Observation only decodes from the current shared perception record; its contrast with Unified Agent demonstrates the value of carrying engagement evidence across frames\.

Table 2:Ablations of key design elements\.Each row reports the signed difference between a variant’s per\-decision score and the corresponding Unified Agent score in Table[1](https://arxiv.org/html/2608.05729#S5.T1); negative values favor Unified Agent, and positive values favor the variant\. Shading indicates the absolute magnitude\. All variants use the same fixed per\-frame record and differ from Unified Agent only in the specified carried\-state, visual\-evidence, or routing component\. Appendix[D\.3](https://arxiv.org/html/2608.05729#A4.SS3)defines each variant\.Table 3:Performance across MLLM settings\.Overall performance across the five downstream decisions for Unified Agent and the five state controls; Gap values are computed from unrounded scores; within each column the MLLM\-based systems share the same MLLM and inference setting\. Bold marks the best value in each column, and Gap is Unified Agent’s overall performance minus that of the strongest control\. The state advantage persists across MLLM family, model capability, and reasoning effort\.
### 6\.3Ablations of key design elements

Finding 2:Different tasks need different information: ablations of Unified Agent show that engagement evidence supports identifying the engaged device, prior state supports recall, and device abilities support routing\.

Table[2](https://arxiv.org/html/2608.05729#S6.T2)evaluates four controlled variants\. Count\-free evidence removes multiplicity from activity and pointing evidence; No pointing channel removes pointing from carried and current\-frame inputs; Reduced\-ability routing removes only the per\-device ability clauses; and Frame\-local state resets prior state before folding the unchanged current frame\. The targeted contrasts show losses on Eng without counts or pointing, on Rsp without per\-device abilities, and on Inf without prior state\. All four targeted gaps are significant after Holm correction \(all adjustedp<10−3p<10^\{\-3\}; Appendix[E](https://arxiv.org/html/2608.05729#A5)\)\.

Within each MLLM setting, the compared state designs use the same perception record, whose quality is evaluated separately in Table[8](https://arxiv.org/html/2608.05729#A3.T8)\. Because the controlled variants share this record, their contrasts isolate the contribution of state and evidence pathways after perception\. The results distinguish complementary roles for carried information: carried engagement cues identify the engaged device, retained facts support recall, and device constraints guide routing\. Together, these mechanisms connect earlier interaction evidence to later decisions\.

![Refer to caption](https://arxiv.org/html/2608.05729v1/x5.png)Figure 6:Real\-photo matched pair\.The user engages with the laptop in episode A and the tablet in episode B; the shared request frame is identical across the pair while the earlier engagement is swapped\.
### 6\.4Generalization across MLLM settings

Finding 3:The benefit of a well\-organized state generalizes across MLLM settings: Unified Agent outperforms all compared systems across MLLM families, capabilities, and reasoning efforts\.

This subsection tests whether the state\-design advantage generalizes across MLLM settings\. Table[3](https://arxiv.org/html/2608.05729#S6.T3)evaluates four settings spanning MLLM families, model capabilities, and reasoning efforts and reports overall performance for Unified Agent and the five state controls\.

In every evaluated setting, Unified Agent achieves the highest overall performance among the compared systems and leads Full context, the strongest control, with the ordering preserved across MLLM families, model capabilities, and reasoning efforts\.

### 6\.5Carried state on real photographs

Finding 4:Carried state supports cross\-device decisions from real photographs: Unified Agent identifies the engaged device and recalls the relevant information\.

Table 4:Real\-photograph case study\.Per\-decision outcomes under GPT\-5\.6\-Luna with low reasoning effort\.✓\\checkmarkdenotes a correct result in both episodes;∘\\circdenotes either a correct result in one episode or a partially correct decision; and×\\timesdenotes an incorrect result\. Appendix[F](https://arxiv.org/html/2608.05729#A6)provides episode\-level details\.We test the same state principle with real photographs\. In a home\-desk scenario, a person uses a laptop, tablet, and phone to book a family birthday dinner \(Figure[6](https://arxiv.org/html/2608.05729#S6.F6)\)\. They compare restaurants with different stated times, work mainly on one device, then later pick up their phone and ask, with both work screens dark and without naming a device, to text everyone the time for “the one I was just working on\.”

In the case study, Unified Agent runs its full perceive–fold–act pipeline directly on the photographs, with hand\-on\-device contact as the engagement cue\. Throughout the real\-photo evaluation, it identifies the device with the strongest accumulated engagement from its carried state and recalls the time stated for that device rather than the decoy time \(Table[4](https://arxiv.org/html/2608.05729#S6.T4); Appendix[F](https://arxiv.org/html/2608.05729#A6)\)\.

## 7Conclusion

Cross\-device agents can fail when a request depends on evidence scattered across devices and moments but no longer visible at the time of action\. We argued that such agents need a compact carried state, and we instantiated this design in Unified Agent, whose state stays bounded as the interaction grows\. In the experiments, Unified Agent achieves the highest overall performance among the compared systems, ahead of Full context and the memory and multi\-agent baselines\. Against the four published baselines, Overall gains span 0\.194–0\.408, with all paired\-bootstrap 95% intervals above zero\. The controlled ablations further show that evidence counts, pointing, prior state, and device abilities each support their targeted downstream decisions\. The advantage persists across MLLM settings\. The findings highlight effective state design as an important complement to model strength for cross\-device agents\.

## 8Limitations

Privacy in any stateful design turns on how much user context is carried across devices and time; designs that keep the raw interaction record retain the most\. That record also grows with the interaction, crowding the model’s context, whereas Unified Agent uses its carried state to present a compact, action\-facing summary of engagement evidence, stated facts, and the standing request for later decisions\. Because this state is explicit rather than latent, retained information remains directly inspectable and selectively revisable, allowing the record to be minimized without losing the context needed to answer the user’s requests\.

UA\-Benchprovides exact, traceable evaluation targets for cross\-device interaction\. Its structured targets derive directly from each episode’s design and device metadata, without model annotation\. Render metadata and semantic maps provide construction\-time validation and remain unavailable to evaluated systems\. The resulting structured ground truth is exact rather than estimated and is verified by deterministic recovery checks, while batch\-level human review screens the rendered images for legibility and rendering quality\. A matched\-pair real\-photo case study complements the rendered benchmark\.

## 9Ethical Considerations

Agents that manage interactions across a user’s devices can lower the everyday effort of coordination, keeping a request answerable when the screens are dark and no device is named, and sparing the user the work of reconstructing which device held what\. Keeping the selected device, recalled information, and proposed action aligned with the user’s intent is central to dependable cross\-device coordination\. Unified Agent keeps these elements explicit, so they can be reviewed together before execution\. To preserve privacy across devices, Unified Agent retains a compact, explicit record for later action instead of the complete interaction history: data minimization by design\. Engagement signals relevance, not identity or authorization, particularly on shared devices\. Uncertain or stale evidence should therefore prompt confirmation before information or instructions are routed\. Our code and data will be released publicly, supporting reproducibility and further development of carried\-state designs\.

## References

- Abdessaied et al\. \(2024\)Adnen Abdessaied, Manuel Hochmeister, and Andreas Bulling\. 2024\.Olvit: Multi\-modal state tracking via attention\-based embeddings for video\-grounded dialog\.In*Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation \(LREC\-COLING 2024\)*, pages 12348–12358\.
- Abrini and Chetouani \(2026\)Mouad Abrini and Mohamed Chetouani\. 2026\.Clue: Crossmodal disambiguation via language\-vision understanding with attention\.*arXiv preprint arXiv:2602\.08999*\.
- Acikgoz et al\. \(2025\)Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani\-Tur, and Gokhan Tur\. 2025\.Can a single model master both multi\-turn conversations and tool use? coalm: A unified conversational agentic language model\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 12370–12390\.
- Agashe et al\. \(2025\)Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang\. 2025\.Agent s2: A compositional generalist\-specialist framework for computer use agents\.*arXiv preprint arXiv:2504\.00906*\.
- Arakawa et al\. \(2025\)Riku Arakawa, Prasoon Patidar, Will Page, Jill Lehman, and Mayank Goel\. 2025\.Scaling context\-aware task assistants that learn from demonstration and adapt through mixed\-initiative dialogue\.In*Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology*, pages 1–19\.
- Budzianowski et al\. \(2018\)Paweł Budzianowski, Tsung\-Hsien Wen, Bo\-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic\. 2018\.Multiwoz\-a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.In*Proceedings of the 2018 conference on empirical methods in natural language processing*, pages 5016–5026\.
- Chang et al\. \(2025\)Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, et al\. 2025\.Partnr: A benchmark for planning and reasoning in embodied multi\-agent tasks\.In*International Conference on Learning Representations*, volume 2025, pages 65205–65268\.
- Chhikara et al\. \(2025\)Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav\. 2025\.Mem0: Building production\-ready ai agents with scalable long\-term memory\.*arXiv preprint arXiv:2504\.19413*\.
- Choi et al\. \(2025\)Hyeong Kyu Choi, Xiaojin Zhu, and Sharon Li\. 2025\.Debate or vote: Which yields better decisions in multi\-agent large language models?In*Advances in Neural Information Processing Systems*\.
- Google DeepMind \(2026\)Google DeepMind\. 2026\.[Gemini 3\.1 Flash\-Lite](https://deepmind.google/models/model-cards/gemini-3-1-flash-lite/)\.Technical report, Google DeepMind\.
- Han et al\. \(2026\)Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, et al\. 2026\.Vlaa\-gui: Knowing when to stop, recover, and search, a modular framework for gui automation\.*arXiv preprint arXiv:2604\.21375*\.
- Hasan et al\. \(2026\)Shaid Hasan, Breenice Lee, Sujan Sarker, and Tariq Iqbal\. 2026\.M2hri: An llm\-driven multimodal multi\-agent framework for personalized human\-robot interaction\.*arXiv preprint arXiv:2604\.11975*\.
- Huang et al\. \(2022\)Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al\. 2022\.Inner monologue: Embodied reasoning through planning with language models\.*arXiv preprint arXiv:2207\.05608*\.
- Ichter et al\. \(2022\)Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, and 26 others\. 2022\.Do as I can, not as I say: Grounding language in robotic affordances\.In*Conference on Robot Learning, CoRL 2022, 14\-18 December 2022, Auckland, New Zealand*, volume 205 of*Proceedings of Machine Learning Research*, pages 287–318\. PMLR\.
- Inoue et al\. \(2025\)Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara\. 2025\.An llm benchmark for addressee recognition in multi\-modal multi\-party dialogue\.In*Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology*, pages 330–334\.
- Johnson et al\. \(2017\)Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei\-Fei, C\. Lawrence Zitnick, and Ross B\. Girshick\. 2017\.CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning\.In*2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21\-26, 2017*, pages 1988–1997\. IEEE Computer Society\.
- Joshi et al\. \(2024\)Nikhita Joshi, Richard Li, Jiannan Li, Leonardo Pavanatto, Michel Pahud, Jatin Sharma, Bongshin Lee, Hugo Romat, William Buxton, Nicolai Marquardt, et al\. 2024\.Opportunistic nudges for task migration between personal devices\.In*Extended Abstracts of the CHI Conference on Human Factors in Computing Systems*, pages 1–8\.
- Khanna et al\. \(2024\)Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X\. Chang, and Manolis Savva\. 2024\.Habitat synthetic scenes dataset \(HSSD\-200\): an analysis of 3d scene scale and realism tradeoffs for objectgoal navigation\.In*CVPR*, pages 16384–16393\. IEEE\.
- Kottur et al\. \(2021\)Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi\. 2021\.Simmc 2\.0: A task\-oriented dialog dataset for immersive multimodal conversations\.In*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 4903–4912\.
- Kuratov et al\. \(2024\)Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Y\. Sorokin, and Mikhail Burtsev\. 2024\.Babilong: Testing the limits of llms with long context reasoning\-in\-a\-haystack\.In*Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 \- 15, 2024*\.
- Kwok et al\. \(2026\)Kenneth Kwok, Basura Fernando, Qianli Xu, Vigneshwaran Subbaraju, Dongkyu Choi, and Boon Kiat Quek\. 2026\.Explicit world models for reliable human\-robot collaboration\.*arXiv preprint arXiv:2601\.01705*\.
- Le et al\. \(2022\)Hung Le, Nancy Chen, and Steven Hoi\. 2022\.Multimodal dialogue state tracking\.In*Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 3394–3415\.
- Lee et al\. \(2025\)Geonsun Lee, Min Xia, Nels Numan, Xun Qian, David Li, Yanhe Chen, Achin Kulshrestha, Ishan Chatterjee, Yinda Zhang, Dinesh Manocha, et al\. 2025\.Sensible agent: A framework for unobtrusive interaction with proactive ar agents\.In*Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology*, pages 1–22\.
- Lee et al\. \(2024\)Jaewook Lee, Jun Wang, Elizabeth Brown, Liam Chu, Sebastian S Rodriguez, and Jon E Froehlich\. 2024\.Gazepointar: A context\-aware multimodal voice assistant for pronoun disambiguation in wearable augmented reality\.In*Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems*, pages 1–20\.
- Li et al\. \(2026\)Xinpeng Li, Shijian Deng, Bolin Lai, Weiguo Pian, James M Rehg, and Yapeng Tian\. 2026\.Towards online multi\-modal social interaction understanding\.*Transactions on Machine Learning Research \(TMLR\)*\.
- Liang et al\. \(2023\)Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng\. 2023\.Code as policies: Language model programs for embodied control\.In*2023 IEEE International conference on robotics and automation \(ICRA\)*, pages 9493–9500\. IEEE\.
- Liu et al\. \(2025\)Kehui Liu, Zixin Tang, Dong Wang, Zhigang Wang, Xuelong Li, and Bin Zhao\. 2025\.Coherent: Collaboration of heterogeneous multi\-robot system with large language models\.In*2025 IEEE International Conference on Robotics and Automation \(ICRA\)*, pages 10208–10214\. IEEE\.
- Ma et al\. \(2025\)Chenyang Ma, Kai Lu, Ruta Desai, Xavier Puig, Andrew Markham, and Niki Trigoni\. 2025\.Coopera: Continual open\-ended human\-robot assistance\.In*Proceedings of the Conference on Neural Information Processing Systems \(NeurIPS\)*\.
- Maharana et al\. \(2024\)Adyasha Maharana, Dong\-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang\. 2024\.Evaluating very long\-term conversational memory of llm agents\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 13851–13870\.
- Meng et al\. \(2026\)Fanqing Meng, Lingxiao Du, Zijian Wu, Guanzheng Chen, Xiangyan Liu, Jiaqi Liao, Chonghe Jiang, Zhenglin Wan, Jiawei Gu, Pengfei Zhou, et al\. 2026\.Clawmark: A living\-world benchmark for multi\-turn, multi\-day, multimodal coworker agents\.*arXiv preprint arXiv:2604\.23781*\.
- Min et al\. \(2024\)So Yeon Min, Xavi Puig, Devendra Singh Chaplot, Tsung\-Yen Yang, Akshara Rai, Priyam Parashar, Ruslan Salakhutdinov, Yonatan Bisk, and Roozbeh Mottaghi\. 2024\.Situated instruction following\.In*European Conference on Computer Vision*, pages 202–228\. Springer\.
- OpenAI \(2026\)OpenAI\. 2026\.[GPT\-5\.6 system card](https://deploymentsafety.openai.com/gpt-5-6)\.
- Packer et al\. \(2023\)Charles Packer, Vivian Fang, Shishir G\. Patil, Kevin Lin, Sarah Wooders, and Joseph E\. Gonzalez\. 2023\.Memgpt: Towards llms as operating systems\.*CoRR*, abs/2310\.08560\.
- Puig et al\. \(2024\)Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung\-Yen Yang, Ruslan Partsey, Ruta Desai, Alexander Clegg, Michal Hlavac, So Yeon Min, et al\. 2024\.Habitat 3\.0: A co\-habitat for humans, avatars, and robots\.In*International Conference on Learning Representations*, volume 2024, pages 15306–15336\.
- Qin et al\. \(2025\)Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al\. 2025\.Ui\-tars: Pioneering automated gui interaction with native agents\.*arXiv preprint arXiv:2501\.12326*\.
- Rasmussen et al\. \(2025\)Preston Rasmussen, Pavlo Paliychuk, Travis Beauvais, Jack Ryan, and Daniel Chalef\. 2025\.Zep: a temporal knowledge graph architecture for agent memory\.*arXiv preprint arXiv:2501\.13956*\.
- Rezazadeh et al\. \(2025\)Alireza Rezazadeh, Zichao Li, Ange Lou, Yuying Zhao, Wei Wei, and Yujia Bao\. 2025\.Collaborative memory: Multi\-user memory sharing in llm agents with dynamic access control\.*arXiv preprint arXiv:2505\.18279*\.
- Rivkin et al\. \(2023\)Dmitriy Rivkin, Francois Hogan, Amal Feriani, Abhisek Konar, Adam Sigal, Steve Liu, and Greg Dudek\. 2023\.Sage: smart home agent with grounded execution\.*arXiv preprint arXiv:2311\.00772*\.
- Schick et al\. \(2023\)Timo Schick, Jane Dwivedi\-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom\. 2023\.Toolformer: Language models can teach themselves to use tools\.In*Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 \- 16, 2023*\.
- Shaikh et al\. \(2025\)Omar Shaikh, Shardul Sapkota, Shan Rizvi, Eric Horvitz, Joon Sung Park, Diyi Yang, and Michael S Bernstein\. 2025\.Creating general user models from computer use\.In*Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology*, pages 1–23\.
- Shu et al\. \(2026\)Yiheng Shu, Saisri Padmaja Jonnalagedda, Xiang Gao, Bernal Jiménez Gutiérrez, Weijian Qi, Kamalika Das, Huan Sun, and Yu Su\. 2026\.Remem: Reasoning with episodic memory in language agent\.*arXiv preprint arXiv:2602\.13530*\.
- Tan et al\. \(2025\)Huajie Tan, Cheng Chi, Xiansheng Chen, Yuheng Ji, Zhongxia Zhao, Xiaoshuai Hao, Yaoxu Lyu, Mingyu Cao, Junkai Zhao, Huaihai Lyu, et al\. 2025\.Roboos\-next: A unified memory\-based framework for lifelong, scalable, and robust multi\-robot collaboration\.*arXiv preprint arXiv:2510\.26536*\.
- Tang et al\. \(2024\)Zineng Tang, Lingjun Mao, and Alane Suhr\. 2024\.Grounding language in multi\-perspective referential communication\.In*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 19727–19741\.
- Tejwani et al\. \(2021\)Ravi Tejwani, Boris Katz, and Cynthia Breazeal\. 2021\.Migratable ai: Investigating users’ affect on identity and information migration of a conversational ai agent\.In*International Conference on Social Robotics*, pages 257–267\. Springer\.
- Teng et al\. \(2026\)Zhuyu Teng, Pei Chen, Yichen Cai, Ruoqing Lu, Zhaoqu Jiang, Jiayang Li, Weitao You, and Lingyun Sun\. 2026\.Seeing eye to eye: Enabling cognitive alignment through shared first\-person perspective in human\-ai collaboration: Seeing eye to eye\.In*Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, CHI 2026, Barcelona, Spain, April 13\-17, 2026*, pages 19:1–19:19\. ACM\.
- Udagawa and Aizawa \(2019\)Takuma Udagawa and Akiko Aizawa\. 2019\.A natural language corpus of common grounding under continuous and partially\-observable context\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 33, pages 7120–7127\.
- VanderHoeven et al\. \(2025\)Hannah VanderHoeven, Brady Bhalla, Ibrahim Khebour, Austin C Youngren, Videep Venkatesha, Mariah Bradford, Jack Fitzgerald, Carlos Mabrey, Jingxuan Tu, Yifan Zhu, et al\. 2025\.Trace: Real\-time multimodal common ground tracking in situated collaborative dialogues\.In*Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(System Demonstrations\)*, pages 40–50\.
- Wang et al\. \(2025\)Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou\. 2025\.Mixture\-of\-agents enhances large language model capabilities\.In*The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025*\. OpenReview\.net\.
- Wang et al\. \(2026\)Zhiyuan Wang, Erzhen Hu, Mark Rucker, and Laura E Barnes\. 2026\.Psi: Shared state as the missing layer for coherent ai\-generated instruments in personal ai agents\.*arXiv preprint arXiv:2604\.08529*\.
- Wu et al\. \(2025\)Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai\-Wei Chang, and Dong Yu\. 2025\.Longmemeval: Benchmarking chat assistants on long\-term interactive memory\.In*The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24\-28, 2025*\. OpenReview\.net\.
- Xu et al\. \(2025\)Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang\. 2025\.A\-mem: Agentic memory for llm agents\.*arXiv preprint arXiv:2502\.12110*\.
- Yadav et al\. \(2025\)Karmesh Yadav, Yusuf Ali, Gunshi Gupta, Yarin Gal, and Zsolt Kira\. 2025\.Findingdory: A benchmark to evaluate memory in embodied agents\.*arXiv preprint arXiv:2506\.15635*\.
- Yao et al\. \(2023\)Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R\. Narasimhan, and Yuan Cao\. 2023\.React: Synergizing reasoning and acting in language models\.In*The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1\-5, 2023*\. OpenReview\.net\.
- Yao et al\. \(2025\)Yiqun Yao, Naitong Yu, Xiang Li, Xin Jiang, Xuezhi Fang, Wenjia Ma, Xuying Meng, Jing Li, Aixin Sun, and Yequan Wang\. 2025\.Egomem: Lifelong memory agent for full\-duplex omnimodal models\.*arXiv preprint arXiv:2509\.11914*\.
- Yu et al\. \(2025a\)Albert Yu, Chengshu Li, Luca Macesanu, Arnav Balaji, Ruchira Ray, Raymond Mooney, and Roberto Martín\-Martín\. 2025a\.Mixed\-initiative dialog for human\-robot collaborative manipulation\.*arXiv preprint arXiv:2508\.05535*\.
- Yu et al\. \(2025b\)Yangchen Yu, Yin Chen, Jia Li, Peng Jia, Yu Zhang, Li Dai, Zhenzhen Hu, Meng Wang, and Richang Hong\. 2025b\.Generalizable engagement estimation in conversation via domain prompting and parallel attention\.In*Proceedings of the 33rd ACM International Conference on Multimedia*, pages 14170–14177\.
- Zakka et al\. \(2022\)Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors\. 2022\.[MuJoCo Menagerie: A collection of high\-quality simulation models for MuJoCo](http://github.com/google-deepmind/mujoco_menagerie)\.
- Zhan et al\. \(2026\)Zhonghao Zhan, Krinos Li, Yefan Zhang, and Hamed Haddadi\. 2026\.Hearthnet: Edge multi\-agent orchestration for smart homes\.In*Proceedings of the ACM Conference on AI and Agentic Systems, CAIS 2026, San Jose, CA, USA, May 26\-29, 2026*, pages 1123–1127\. ACM\.
- Zhang et al\. \(2025a\)Chaoyun Zhang, Liqun Li, He Huang, Chiming Ni, Bo Qiao, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al\. 2025a\.UFO3: Weaving the digital agent galaxy\.*arXiv preprint arXiv:2511\.11332*\.
- Zhang et al\. \(2024\)Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan\. 2024\.Building cooperative embodied agents modularly with large language models\.In*International Conference on Learning Representations*, volume 2024, pages 19373–19401\.
- Zhang et al\. \(2025b\)Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An\. 2025b\.Agentorchestra: Orchestrating multi\-agent intelligence with the tool\-environment\-agent \(tea\) protocol\.*arXiv preprint arXiv:2506\.12508*\.
- Zitkovich et al\. \(2023\)Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al\. 2023\.Rt\-2: Vision\-language\-action models transfer web knowledge to robotic control\.In*Conference on Robot Learning*, pages 2165–2183\. PMLR\.

## Appendix Contents

## Appendix ACarried State Specification

### A\.1State representation

For an episode with device set𝒟\\mathcal\{D\}, let𝒟eng⊆𝒟\\mathcal\{D\}\_\{\\mathrm\{eng\}\}\\subseteq\\mathcal\{D\}be the screen devices eligible to be labeled as the engaged device\. After framett, Unified Agent carries

St=\(Ct,Pt,Kt,rt\),S\_\{t\}=\(C\_\{t\},P\_\{t\},K\_\{t\},r\_\{t\}\),\(3\)whereCtC\_\{t\}andPtP\_\{t\}encode engagement evidence,KtK\_\{t\}stores stated facts, andrtr\_\{t\}stores the standing request\. InCtC\_\{t\},mtm\_\{t\}counts nonempty engagement observations, activity in the rendered benchmark and attended\-device contact in the real\-photo case, andlitt​\(d\)\\mathrm\{lit\}\_\{t\}\(d\)counts those containingdd, for eachd∈𝒟engd\\in\\mathcal\{D\}\_\{\\mathrm\{eng\}\}\.PtP\_\{t\}retains attended\-device cues: pointing in the rendered benchmark and hand\-on\-device contact in the real\-photo case\. The decision interface summarizes these cues as per\-device tallies\.KtK\_\{t\}stores the first stated fact for each device–topic pair when the device is known and for each topic otherwise\.rtr\_\{t\}stores the latest requested action and, in the real\-photo case, the queried topic\.

### A\.2State update and decision interface

The state evolves by folding in each observation,

St=U​\(St−1,Ot\)\.S\_\{t\}=U\(S\_\{t\-1\},\\,O\_\{t\}\)\.\(4\)For eachOtO\_\{t\}, activity and attended\-device cues update engagement evidence; a stated fact at a nonempty activity frame is added only when no fact is already recorded for that device and topic; and an eligible command replaces the standing request\. Other components persist\. The agent foldsOtO\_\{t\}intoSt−1S\_\{t\-1\}before predicting fromStS\_\{t\}and the current observation\.

Eng uses accumulated activity and attended\-device cues; Int interprets the current utterance; Inf retrieves the fact for the queried topic, resolving same\-topic facts by device; Rsp combines the requested action, engagement evidence, and card\-declared device abilities; and Nxt combines the standing request with engagement evidence\. The decoder receives the per\-device activity and cue tallies, one record for each stored fact, the standing request, and the current observation\.

## Appendix BBenchmark Design and Construction

UA\-Benchconsists of controlled matched pairs in which later decisions depend on evidence from earlier interaction frames\. This section presents the benchmark scope, matched\-pair design, and episode construction and admission\.

### B\.1Benchmark scope

Each episode contains a temporally ordered interaction among one user, candidate screen devices, and, in episodes that include it, a fetch robot\. Table[5](https://arxiv.org/html/2608.05729#A2.T5)summarizes the benchmark scale, frame structure, device roles, and recall vocabulary\.

Table 5:UA\-Benchstructure and coverage\.Dataset scale, episode timeline, device roles, and recall vocabulary\.
### B\.2Matched\-pair design

Each matched pair contains two episodes with identical opening, request, and recall frames \(Table[5](https://arxiv.org/html/2608.05729#A2.T5)\)\. The intervening engagement frames exchange the engaged and decoy roles and their associated values, yielding different Eng, Inf, Rsp, and Nxt targets while preserving Int across the role swap\. Figure[7](https://arxiv.org/html/2608.05729#A2.F7)summarizes this construction\.

![Refer to caption](https://arxiv.org/html/2608.05729v1/x6.png)Figure 7:Matched\-pair design\.The paired episodes exchange earlier engagement roles and associated values while sharing the later request and recall frames\. The paired construction makes earlier interaction evidence the distinguishing context for the later decisions\.
### B\.3Episode construction and admission

Each episode is scripted in a rendered 3D household scene containing one user, screen devices, additional screen distractors, and, in episodes that field it, a MuJoCo Menagerie humanoid robot \(Unitree G1\)Zakka et al\. \([2022](https://arxiv.org/html/2608.05729#bib.bib57)\)in the fetch\-robot role\.

#### Agent inputs\.

The agent receives temporally ordered device\-camera images and utterances, a device roster, and a device card\. The roster specifies device identity, category, appearance, and any screen\-glow color; the card specifies the device set, each device’s fixed abilities, the observation rule, and the closed answer vocabularies\.

#### Construction records\.

Each episode’s construction record comprises the scene structure, engagement schedules, role assignments, render metadata, semantic maps, pair identities, and target labels, supporting benchmark generation and validation\. Agent\-facing episodes contain only the inputs defined above\.

#### Ground truth and admission\.

Eng, Inf, Rsp, and Nxt are deterministic functions of the construction script, device metadata, and controlled role assignment; Int uses an authored reference validated before admission\. Admission requires recoverable targets, designated\-frame constraints on vocabulary and actions, leakage and device\-naming checks, image\-level checks of pointing\-gesture and candidate\-device visibility, and the matched\-pair invariants of the preceding subsection\. Rendered batches additionally undergo manual screening for legibility and rendering quality\.

The construction enforces the following invariants for every admitted pair\.

1. 1\.Shared\-frame identity\.The shared frames are rendered once and reused, making them identical across the pair\.
2. 2\.Role swap\.The engaged and decoy roles are exchanged over the engagement frames, so the engaged\-device label differs across the pair\.
3. 3\.Value collision\.The same topic receives distinct values on two devices\. Statement order is reversed within each pair and balanced across pairs\.
4. 4\.Anonymous wording\.Utterances use device\-unspecified wording, and recall questions omit the queried value\.
5. 5\.Ability\-constrained routing\.Responding\-device targets follow the card\-declared abilities: a device supports a request exactly when its card covers the requested action, so targets range over device sets, including a single device and the empty set, while engagement targets are defined over the candidate screen devices\.

Table 6:Downstream output specification\.Output space and scoring rule for each evaluated decision\.
Table 7:Evaluated MLLM settings\.GPT\-5\.6\-Luna at low reasoning effort is the primary setting\.

## Appendix CEvaluation Protocol and Scoring

Within each MLLM setting, the evaluation fixes the shared perception record, output specification, and scorer\. This section specifies the output specification, MLLM settings, shared records, and scoring rules; Appendix[D](https://arxiv.org/html/2608.05729#A4)describes the compared systems\.

### C\.1Output specification and MLLM settings

Table[7](https://arxiv.org/html/2608.05729#A2.T7)defines the output space and scoring rule for the five downstream decisions, and Table[7](https://arxiv.org/html/2608.05729#A2.T7)summarizes the evaluated MLLM settings\. The default configuration uses GPT\-5\.6\-Luna at low reasoning effort\. All prompts are fixed zero\-shot templates\.

### C\.2Shared\-perception protocol

The evaluation holds a shared perception pipeline fixed across systems\. For each setting, the MLLM processes each episode in temporal order and produces a per\-frame perception record\. Every MLLM\-based comparison system then receives this shared record and constructs its own cross\-frame state and downstream predictions\. Controlled variants apply the common decoder to Unified Agent’s fixed records and utterances with only the specified factor changed\. All systems use the same output schema and scorer\.

### C\.3Per\-frame evidence and diagnostics

For each frame, the shared perception record contains the episode position; the set of active devices; the attended device, if any; a binary indicator of whether the user is attending to a device; the action, stated value, and queried topic expressed in the utterance, when applicable; and the requested action, together with a binary indicator of whether a request is present\. Each system additionally receives the public device card, device roster, and current utterance\.

For rendered episodes, prior activity observations determine when the perception pipeline extracts pointing cues, while the shared record provides the current\-frame cue rather than accumulated engagement\. Stated values come from the current utterance, and recall is produced during downstream prediction\. Shared frames therefore yield identical inputs across the pair\. Table[8](https://arxiv.org/html/2608.05729#A3.T8)reports current\-frame perception diagnostics for these shared records across the four MLLM settings\.

Table 8:Shared\-perception diagnostics\.Current\-frame active\-screen set\-F1F\_\{1\}and pointed\-device accuracy on the rendered benchmark for the four evaluated MLLM settings \(Table[7](https://arxiv.org/html/2608.05729#A2.T7)\)\.Active\-screen set\-F1F\_\{1\}uses all 2,400 frame instances; pointed\-device accuracy uses the 1,600 pointing\-frame instances\.

### C\.4Scoring

#### Deterministic decisions\.

The construction fixes Eng, Inf, Rsp, and Nxt without model annotation; render metadata is used only for validation\. Eng and Nxt use exact match, Inf uses case\-insensitive exact match, and set\-valued Rsp uses set\-F1F\_\{1\}, with the empty target matched exactly by the empty prediction\. An Inf prediction of “none” receives credit only when “none” is the target\.

#### Intent and overall performance\.

Int is evaluated under one rubric by a fixed same\-meaning judge, a text\-only GPT\-5\.6\-Sol at medium reasoning effort; the same judge scores every system, every MLLM setting, and the real\-photo case study\. The rubric credits paraphrases that preserve the action, target, device category, and material constraints\. Empty or malformed predictions score zero\. Overall is the macro\-average of the downstream decisions\.

## Appendix DComparison Systems and Variants

Alongside the compared systems in Table[1](https://arxiv.org/html/2608.05729#S5.T1), we evaluate controlled variants\.

### D\.1Comparison protocol

All systems follow the protocol in Appendix[C\.2](https://arxiv.org/html/2608.05729#A3.SS2)\. Compared systems begin from the shared perception record, restricted to the input fields their designs specify, and retain their method\-specific state\-management mechanisms\. Controlled variants share Unified Agent’s records, utterances, and decoder and change only the specified factor in carried state and, when applicable, current\-frame input\.

### D\.2Controls and baseline adaptations

#### State controls\.

*Full context*retains the complete record and utterance history\.*Self\-notes*updates a free\-text memo at each frame, limited to 600 characters\.*Observation only*uses the current record\.*Answer cache*carries forward only the most recent answers, without counts\.*Text only*reads the utterances with no visual input; its near\-floor engagement and recall confirm that the pair\-separating evidence is visual rather than textual\.

#### Baseline adaptations\.

We adapt the memory and dialogue\-state methods Mem0Chhikara et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib8)\)and MM\-DSTLe et al\. \([2022](https://arxiv.org/html/2608.05729#bib.bib22)\); Abdessaied et al\. \([2024](https://arxiv.org/html/2608.05729#bib.bib1)\), together with the multi\-agent methods Debate\-or\-voteChoi et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib9)\)and Mixture\-of\-AgentsWang et al\. \([2025](https://arxiv.org/html/2608.05729#bib.bib48)\)\. Mem0 maintains a fact memory built by extraction and consolidation and retrieved by lexical matching with a recency fallback; MM\-DST maintains a slot–value state that keeps the most recent value and is read out each turn\. Debate\-or\-vote coordinates per\-device agents through peer revision and field\-wise voting; Mixture\-of\-Agents synthesizes per\-device proposals from per\-proposer running summaries\.

### D\.3Controlled variants

Controlled variants change only the specified state, evidence, or routing factor\.

#### Variant definitions\.

*Frame\-local state*resets the carried state before each frame\.*Count\-free evidence*retains activity and pointing presence but removes their multiplicities\.*No pointing channel*removes pointing from carried and current\-frame inputs while retaining activity evidence\.*Reduced\-ability routing*removes only per\-device ability clauses while retaining Unified Agent’s carried state and general card context\.

Across the targeted contrasts, count and pointing evidence support Eng, prior state supports Inf, and per\-device abilities support Rsp\.

## Appendix EStatistical Analysis

We evaluate overall\-performance differences using a paired analysis clustered by matched pair\. Table[1](https://arxiv.org/html/2608.05729#S5.T1)carries the resulting gap intervals, and every adjusted bootstrap tail\-areapp\-value falls below10−310^\{\-3\}\.

#### Endpoint and clustering\.

For each sample, scored instances are first averaged within each downstream decision; Overall is then the equal\-weight mean of Eng, Int, Inf, Rsp, and Nxt\. Each contrast is Unified Agent minus the comparison, aligned by matched\-pair identity and episode role\. Resampling is clustered by matched pair, with the paired episodes retained jointly because the role exchange makes their outcomes dependent\.

#### Designated frames\.

For every system, Eng averages the shared request and recall frames,f10f\_\{10\}–f12f\_\{12\}, and Rsp averages the construction\-designated routing frames,f9f\_\{9\}andf12f\_\{12\}\. Each resulting score enters Overall with the same weight as the other downstream decisions\.

#### Bootstrap procedure and decision criterion\.

Each of theB=10,000B=10\{,\}000replicates resamples matched pairs with replacement, retains the paired episodes jointly, and recomputes the Overall difference\. Ifkkof theBBreplicates yield a difference at or below zero, the one\-sided tail\-areapp\-value is the add\-one estimate\(1\+k\)/\(B\+1\)\(1\+k\)/\(B\+1\)\. Conclusions are unchanged when resampling instead clusters the 46 rendered scenes that host the 100 pairs\. The Gap brackets in Table[1](https://arxiv.org/html/2608.05729#S5.T1)report pointwise percentile intervals from this distribution; the Overall brackets instead use a one\-sample clustered bootstrap for each system\. The paired\-gap and one\-sample interval procedures use the same fixed random seed\. Holm correction is applied to the tail\-areapp\-values\.

The four adapted published\-method comparisons under GPT\-5\.6\-Luna at low reasoning effort are adjusted together with Holm’s procedure\. Full context, Self\-notes, Answer cache, Text only, and Observation only form five additional comparisons that are adjusted separately\. The decision criterion for the published\-method comparisons is a Holm\-adjustedp<0\.05p<0\.05together with an estimated difference of at least0\.100\.10\.

#### Ablation contrasts\.

The endpoint\-aligned ablation contrasts are Holm\-adjusted as a separate group; every adjustedppfalls below10−310^\{\-3\}\.

## Appendix FReal\-Photo Case Study

#### Protocol and image preparation\.

We construct one controlled matched pair from prepared photographs of a home\-desk interaction involving a laptop, tablet, and phone; the user’s face is not visible\. All evaluated systems receive the same utterances, device card, roster, and closed answer vocabulary, and every system with visual input receives the same prepared photographs, while the episode script supplies the evaluation targets\. Fixed display masks standardize screen state while preserving foreground hand and finger regions\. Each shared frame is prepared once and reused across the pair\.

#### Matched\-pair construction\.

The user compares restaurants on the laptop and tablet and states a reservation time for each\. Subsequent hand\-on\-device contact favors the laptop in episodeAAand the tablet in episodeBB\. In the shared final frame, both work screens are dark, and the user asks from the phone, without naming a device, to text everyone the time for “the one I was just working on” \(Figure[6](https://arxiv.org/html/2608.05729#S6.F6)\)\. The shared setup, fact\-stating, and request frames hold context fixed, while the intervening frames vary engagement and produce different engaged\-device and recalled\-time targets\.

#### Results\.

Unified Agent folds hand\-on\-device evidence from the prepared photographs into its carried state and predicts from that state and the current observation\. Across the pair, it identifies the device with the largest accumulated engagement tally and recalls the associated time\. Only Unified Agent also recalls the stated time on both episodes; the other designs, including Full context, do not \(Table[4](https://arxiv.org/html/2608.05729#S6.T4)\)\.