Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions

# Personalizing Embodied Multimodal Large Language Model Agents over Long-term User Interactions
Source: [https://arxiv.org/html/2605.26256](https://arxiv.org/html/2605.26256)
Jeongeun Lee1Chanyoung Park2Dongha Lee1 1Yonsei University2KAIST \{ljeadec31, donalee\}@yonsei\.ac\.kr cy\.park@kaist\.ac\.kr

###### Abstract

Multimodal large language model \(MLLM\)\-based embodied agents have shown strong potential for solving complex tasks in physical environments\. However, personalized assistance requires more than following generic instructions or recognizing object at the category level\. In real\-world scenarios, the intended target is often specified only implicitly through prior interactions, requiring agents to ground user\-intended instances from personalized context accumulated over time\. In this work, we proposePolar, a multimodal memory\-augmented framework for personalized embodied agents over long\-term user interactions\.Polarorganizes prior interactions into a multimodal knowledge graph with semantic memory for personalized context, and episodic memory for past embodied experiences such as agent trajectories\. ThereforePolarretrieves candidate object memories to ground the user\-intended target instance and guide subsequent planning\. We evaluatePolaracross multiple MLLM backbones and diverse evaluation scenarios to study how prior interactions should be represented and used for long\-term personalization\. Results show that current inputs alone are insufficient for personalized instance grounding, while raw prior interactions are difficult to use directly due to their unstructured form\.Polarimproves performance by converting prior interactions into task\-relevant memory\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/fig/scenario_v2.png)Figure 1:Personalization over long\-term user interactions\.In daily life, users often refer to objects through personal context accumulated over prior interactions rather than explicit target references\. When multiple similar objects are present, conventional embodied agents may fail to determine which specific instance the user intends, as they focus on finding the category “shoes” rather than “which” shoes\. This motivates personalized instance grounding, where embodied agents use long\-term interaction history to resolve user intent\.
## 1Introduction

With great advancements of multimodal large language models \(MLLMs\)Teamet al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib36)\); Hurstet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib37)\); Baiet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib35)\), MLLM\-based embodied agents have been increasingly studied for solving complex tasks while interacting with physical environmentsMuet al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib57)\); Driesset al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib30)\); Zitkovichet al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib58)\); Szotet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib13)\); Zhanget al\.\([2025a](https://arxiv.org/html/2605.26256#bib.bib31)\)\. By integrating visual perception and multimodal reasoning for vision\-driven decision making, these agents are expected to assist human in daily life, including locatingAyubet al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib49)\); Ramrakhyaet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib32)\), navigating toZhaoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib59)\); Daiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib47)\); Qiaoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib12)\), and delivering objectsTanet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib61)\); Korekataet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib60)\)in response to user requests\.

While prior works have largely focused on solving generic instructions, however, category\-level instance recognition is not enough for personalized assistance\. As illustrated in FigureLABEL:fig:intro, when a user asks for shoe under specific circumstances and multiple pairs of shoes exist in the house, the agent struggles to determine which pair to bring without personalized context\. Since such natural language\-only instructions often fail to fully cover user requestLeeet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib41)\), some embodied agents learn to perform tasks on specific target instances with instance\-level groundingLeiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib71)\); Barsellottiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib43)\); Taioliet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib86)\)\. Still, these methods generally assume that the target reference is explicitly and directly given\. In real\-world, however, such personalized context is more often conveyed implicitly through previous interactions, requiring the agent to interpret them to perform personalized instructions accordingly\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/x1.png)Figure 2:Preliminary experiments on the PinNED datasetBarsellottiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib43)\)\.Success Rate measures correct target\-instance navigation, while Category Match counts cases that reach the correct category object but the wrong instance\. 0\-turn and 10\-turn denote the number of intervening interactions between the target reference and the final target instruction\. Since the final instruction contains no reference, the agent must identify the target from prior interactions\. We implement the agent in NavBenchQiaoet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib26)\)\.Although leveraging long\-term context is important for MLLM\-based embodied agents, prior works have shown that MLLMs often degrade in long\-context settingsKimet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib46)\); Xueet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib54)\); Beiet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib55)\)\. They often fail to reason effectively over accumulated multimodal context and instead rely on superficial cuesXueet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib54)\); Beiet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib55)\); Xuet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib89)\)\. They also struggle to track how information evolves over time, making it difficult to capture updates or resolve conflicts across interactionsFuet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib72)\)\. To further examine whether this challenge also arises in embodied multi\-turn interactions, we provide a preliminary analysis in Figure[2](https://arxiv.org/html/2605.26256#S1.F2)\. The results show that multiple MLLM\-based embodied agents often fail to ground the intended target instance from prior interactions, highlighting their difficulty in using accumulated user\-specific information for personalized instance grounding\.

For personalized embodied assistance, the agent should \(1\) manage information over long\-term interactions to build personalized knowledge and \(2\) use that knowledge to interpret and adapt to individual user requests\. This requires more than simply remembering prior interactions, requiring the agent to organize them into task\-relevant memory structures that can be selectively retrieved and applied to current instruction following and planning\. Therefore, in this work, we proposePolar, a multimodal memory\-augmented framework forpersonalized MLLM\-based embodiedagent for long\-term userinteractions\. To this end,Polarleverages multimodal memory to accumulate personalized knowledge from prior interactions and retrieve relevant information for task execution\. While many existing memory methods primarily focus on compressing past interactions, the key idea ofPolaris to convert them into reusable personalized knowledge that can directly support future embodied tasks\. In particular, the memory organizes visual concepts together with personalized context \(i\.e\., semantic memory\) and agent trajectories \(i\.e\., episodic memory\)Tulving and others \([1972](https://arxiv.org/html/2605.26256#bib.bib63)\)\. We organize these memories into a multimodal knowledge graph that connects semantic and episodic memories across objects and interactions\.

We study howPolarstructures prior interactions for long\-term personalized embodied agents across multiple MLLM backbones and evaluation scenarios\. The results show that current observations alone are insufficient for personalized instance grounding\. Raw prior interactions can contain useful user\-specific information, but their unstructured form makes them unstable to use directly for the current task\.Polarimproves performance by converting prior interactions into task\-relevant memory\. Further analyses show that semantic memory provides finer\-grained retrieval of user\-specific information, while episodic memory converts past trajectories into planning\-relevant experience\. These findings suggest that long\-term personalization requires memory representations that support both personalized instance grounding and subsequent embodied planning\.

## 2Related Works

Vision\-driven Embodied Agents\.Early works on embodied agents have primarily leveraged large language models \(LLMs\) with its high reasoning and planning capabilitiesShridharet al\.\([2020](https://arxiv.org/html/2605.26256#bib.bib8)\); Huanget al\.\([2022a](https://arxiv.org/html/2605.26256#bib.bib14),[b](https://arxiv.org/html/2605.26256#bib.bib15)\); Qianet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib29)\)\. They show that LLMs can decompose complex instructionsAhnet al\.\([2022](https://arxiv.org/html/2605.26256#bib.bib21)\); Cao and Lee \([2023](https://arxiv.org/html/2605.26256#bib.bib20)\); Zhouet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib11)\), reason over environment feedbackSinghet al\.\([2022](https://arxiv.org/html/2605.26256#bib.bib23)\); Bhatet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib22)\); Kwonet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib24)\), and compose robot policiesKwonet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib24)\); Lianget al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib18)\); Chenget al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib25)\)\. However, these approaches typically relied on textual scene descriptionsZhouet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib11)\); Zhanget al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib27),[2025b](https://arxiv.org/html/2605.26256#bib.bib19)\), external perception modulesHuanget al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib16)\), or predefined skill librariesAhnet al\.\([2022](https://arxiv.org/html/2605.26256#bib.bib21)\); Chuet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib28)\), limiting their ability to interpret subtle visual cue in raw observations\. To directly perceive fine\-grained visual information, recent research has increasingly explored MLLM\-based embodied agents, which integrate visual perception and language reasoning for vision\-driven decision makingSzotet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib13)\); Qiaoet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib26)\); Yanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib7)\)\. These agents have demonstrated promising performance across a wide range of embodied tasks, including robot manipulationDriesset al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib30)\); Zhanget al\.\([2025a](https://arxiv.org/html/2605.26256#bib.bib31)\), household tasksSzotet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib13)\); Ramrakhyaet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib32)\); Xiaoet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib33)\), and embodied navigationQiaoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib12),[2026](https://arxiv.org/html/2605.26256#bib.bib26)\); Xuet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib34)\)in simulated and real\-world environmentsYanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib7)\); Zhanget al\.\([2025d](https://arxiv.org/html/2605.26256#bib.bib81)\)\. Nevertheless, as using MLLMs as step\-by\-step controllers over raw observations can be computationally inefficient for embodied control, particularly in long\-horizon tasksYueet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib78)\)\. Recent studies therefore leverage MLLMs as high\-level planners, using them to predict abstract action plans, such as future waypointsZhaoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib59)\)or sequential action plansYanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib7)\), rather than invoking them for per\-step embodied decision making\.

Personalized Embodied Agents\.Personalizing MLLMs aims to recognize user\-specific visual entities beyond generic onesKimet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib46)\); Nguyenet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib44)\); Alalufet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib45)\)\. For example, while an off\-the\-shelf MLLMTeamet al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib36)\); Baiet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib35)\)may only recognizea dog, a personalized MLLMNguyenet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib44)\); Ohet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib52)\)is expected to identifythe user’s dog\. Beyond recognition, recent works have also explored retrieval\-augmented personalization, where user\-specific information is stored in an external database and dynamically retrieved to provide personalized contextHaoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib51)\); Daset al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib83)\)\. The need for multimodal personalization has recently been extended to embodied agents, where agents are required to interpret user instructions by grounding user\-specific objects in physical environmentsDaiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib47)\); Leeet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib41)\); Barsellottiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib43)\); Taioliet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib86)\); Ziliottoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib39)\); Wanget al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib38)\)\. This shifts the problem from generic object recognition to identifying which specific object instance is intended for the user\. However, existing approaches typically assume that such personalized references are explicitly available, whereas in realistic settings they are often implicit and must be inferred from prior interactions\.

Memory\-augmented Agents\.Memory modules support long\-term interaction in \(M\)LLM agents beyond the limited context windowZhanget al\.\([2025c](https://arxiv.org/html/2605.26256#bib.bib62)\)\. Early studies mainly developed memory systems for LLM agentsPackeret al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib69)\); Chhikaraet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib68)\); Kanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib70)\); Yuet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib84)\), where the agent actively manages memory operations such as storing and retrieving past information\. However, as these approaches are text\-centric, recent work has therefore extended memory to incorporate multimodal observationsLonget al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib42)\); Liuet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib67)\)\. Nevertheless, maintaining coherent long\-term multimodal memory remains challenging, as agents still struggle to revise outdated memories, track evolving information, and resolve conflicts across interactionsXueet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib54)\); Fuet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib72)\); Liuet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib67)\)\.

## 3Personalized Embodied Task over Long\-term Interactions

### 3\.1Task Formulation

We formulate our task as a partially observable Markov decision process \(POMDP\), defined by

𝒫=\(𝒮,𝒜,Ω,𝒯,𝒪,ℐ,ℛ\),\\mathcal\{P\}=\(\\mathcal\{S\},\\mathcal\{A\},\\Omega,\\mathcal\{T\},\\mathcal\{O\},\\mathcal\{I\},\\mathcal\{R\}\),where𝒮\\mathcal\{S\}denotes the state space,𝒜\\mathcal\{A\}denotes the action space,Ω\\Omegadenotes the observation space,𝒯\\mathcal\{T\}denotes the transition function,𝒪\\mathcal\{O\}denotes the observation function,ℐ\\mathcal\{I\}denotes the task instruction, andℛ\\mathcal\{R\}denotes the reward function\.

For each episodekk, the agent is given a task instructionℐk\\mathcal\{I\}\_\{k\}and previous episodesℰ<k=\{e1,…,ek−1\}\\mathcal\{E\}\_\{<k\}=\\\{e\_\{1\},\\dots,e\_\{k\-1\}\\\}for long\-term memory retrieval, where each episodeeie\_\{i\}consists of a user instruction and the corresponding interaction trajectory\. At each time steptt, the agent receives a partial visual observationvt∈Ωv\_\{t\}\\in\\Omega\. Given the interaction historyτt=\(v1,a1,…,vt−1,at−1,vt\)\\tau\_\{t\}=\(v\_\{1\},a\_\{1\},\\dots,v\_\{t\-1\},a\_\{t\-1\},v\_\{t\}\), the MLLM policyπ\\pipredicts the next action conditioned on the current instruction and the retrieved memory:

at=π​\(τt,ℐk,ℰ<k\)\.a\_\{t\}=\\pi\(\\tau\_\{t\},\\mathcal\{I\}\_\{k\},\\mathcal\{E\}\_\{<k\}\)\.
In this work, we focus on navigation as the target embodied task\. This task requires the agent to ground the intended target instance using prior interactions and move toward it in a physical environment\. It also avoids additional confounding factors from manipulation or tool\-use skills, allowing to study personalized target interpretation and spatial decision making in a controlled setting\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/x2.png)Figure 3:Overview ofPolar\. In thememorizationstage \(left\),Polarbuilds an object\-centric memory graph with semantic memory for personalized context and episodic memory for past trajectories\. In theutilizationstage \(right\),Polarretrieves relevant memories for candidate objects to ground the target and guide subsequent planning\. Timestamps are omitted for brevity\.
### 3\.2Polar: Memory\-augmented framework for Embodied Agents

Figure[3](https://arxiv.org/html/2605.26256#S3.F3)illustrates thePolarframework, which consists of two stages: memorization and utilization\. Given prior episodesE<kE\_\{<k\}, the memorization stage converts previous interactions into the memory module, asE<k→ℳE\_\{<k\}\\rightarrow\\mathcal\{M\}\. Rather than storing interactions as raw logs or simple summaries,Polarorganizes them into personalized task memory associated with objects\. During utilization,Polarretrieves memories conditioned on the current instructionℐk\\mathcal\{I\}\_\{k\}, asℛk=ρ​\(Ik,ℳ\)\\mathcal\{R\}\_\{k\}=\\rho\(I\_\{k\},\\mathcal\{M\}\)\. The retrieved memory setℛk\\mathcal\{R\}\_\{k\}is then used by the embodied policy to predict the next action,at=π​\(τt,Ik,ℛk\)a\_\{t\}=\\pi\(\\tau\_\{t\},I\_\{k\},\\mathcal\{R\}\_\{k\}\)\. In this way,Polarconnects prior interactions to current task execution through task\-relevant memory\.

#### 3\.2\.1Memorization

We represent memory as a multimodal knowledge graph,ℳ=\(𝒱,ℰ\),\\mathcal\{M\}=\(\\mathcal\{V\},\\mathcal\{E\}\),where𝒱\\mathcal\{V\}denotes the set of memory nodes andℰ\\mathcal\{E\}denotes the set of edges between them\. The node set𝒱\\mathcal\{V\}contains three types of nodes: object nodes, semantic memory nodes, and episodic memory nodes\. Each object node corresponds to a unique object identity, represented by an object ID\. For each episode,Polargenerates two types of memory: semantic memory and episodic memory\.

Semantic memorycaptures personalized knowledge associated with an object\.Polarreformulates it into a set of concise semantic statements, where each statement preserves a single user\-specific attribute or association about the target object\. This fine\-grained formulation avoids entangling multiple pieces of personalized information in a single memory entry, which can weaken retrieval accuracyChenet al\.\([2024b](https://arxiv.org/html/2605.26256#bib.bib73)\)\. It also helps organize memory across objects and supports selective updates over time\. For objectoio\_\{i\},Polarconstructs\{si,1,si,2,…,si,Ki\},\\\{s\_\{i,1\},s\_\{i,2\},\\dots,s\_\{i,K\_\{i\}\}\\\},where eachsi,ks\_\{i,k\}is a short textual statement describing user\-specific knowledge aboutoio\_\{i\}\.

Episodic memorystores the agent trajectory together with the raw user instruction and, when available, the reference image\. As raw trajectories are often long and noisy,Polarconverts the full trajectory into a compact textual memorypip\_\{i\}\. Rather than preserving the entire episode in detail,pip\_\{i\}captures planning\-relevant experience, including whether the trajectory succeeded or failed, which rooms were unpromising, and which transitions or observations were informative\. This allows episodic memory to support future planning by reusing prior search experienceLiuet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib80)\)\.

After generating new memories,Polarlinks them to an existing object node if the object ID matches or the reference image is visually similar to an existing one; otherwise, it creates a new object node\. Edges are timestamped to support temporal updates and recency\-aware retrieval\. To avoid redundancy, if a newly generated semantic memory is sufficiently similar to an existing semantic memory node,Polarlinks the corresponding object node to the existing node instead of creating a duplicate\. As a result,Polarincrementally builds an object\-centric structured memory that preserves personalized context over long\-term interactions and supports effective retrieval for future tasks\.

#### 3\.2\.2Utilization

Given an instructionℐ\\mathcal\{I\},Polarretrieves the top\-kksemantic memory nodes fromℳ\\mathcal\{M\}using embedding similarity,sim​\(ℐ,s\)=ϕ​\(ℐ\)⋅ϕ​\(s\),\\mathrm\{sim\}\(\\mathcal\{I\},s\)=\\phi\(\\mathcal\{I\}\)\\cdot\\phi\(s\),whereϕ​\(⋅\)\\phi\(\\cdot\)is the BGE\-M3 encoderChenet al\.\([2024a](https://arxiv.org/html/2605.26256#bib.bib74)\)\. We setk=5k=5to balance memory recall and retrieval noise, as larger values may introduce irrelevant context that confuses object grounding\. For each retrieved semantic node,Polarfollows its graph connection to the corresponding object nodezjz\_\{j\}, and gathers the associated instruction and episodic memorypjp\_\{j\}\.Polarthen reasons over the retrieved candidate objects with the current instruction, grounds the intended target, and leverages the corresponding episodic memory to guide planning\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/x3.png)Figure 4:Overview of the baseline MLLM embodied agent\.The agent follows a hierarchical planning framework with high\-level planning \(top\), which predicts a coarse graph\-based path toward a likely destination, and low\-level planning \(bottom\), which selects executable actions from egocentric observations\. The controller grounds these planned actions in the physical environment\.

### 3\.3MLLM Embodied Agent

Action Space\.Motivated byYanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib7)\), we model the action space hierarchically with two levels: high\-level and low\-level\. The low\-level actions are directly executable actions, defined as

𝒜low=\{MOVE FORWARD,TURN LEFT,TURN RIGHT,STOP\}\.\\mathcal\{A\}\_\{\\mathrm\{low\}\}=\\\{\\texttt\{MOVE FORWARD\},\\texttt\{TURN LEFT\},\\texttt\{TURN RIGHT\},\\texttt\{STOP\}\\\}\.A high\-level action specifies movement toward a selected waypoint or destination\. This abstract navigation decision is further translated into a sequence of low\-level actions,\[at,at\+1,…,at\+n\],\[a\_\{t\},a\_\{t\+1\},\\dots,a\_\{t\+n\}\],which are executable in the environment\.

High\-level Planning\.Since searching for a target in a spacious environment cannot be efficiently solved from local observations aloneQiaoet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib26)\); Yanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib7)\), the high\-level planning module provides coarse guidance for long\-horizon exploration\. Before task execution, the agent first traverses the environment to build a coarse spatial layout by identifying distinct regions and assigning region labels \(e\.g\., living room\)Zhanget al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib27)\); Ziliottoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib39)\); Yinet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib9)\)\. It then converts this layout into a scene graph𝒢\\mathcal\{G\}, where nodes correspond to rooms and edges represent adjacency between neighboring rooms\. Each node contains a waypoint that specifies a navigable target position within the corresponding room\. During exploration, given the scene graph and task context, the agent reasons over the global spatial structure to generate a promising movement directive toward the room most likely to contain the target\. The resulting waypoint sequence is then passed to the low\-level module for execution\.

Low\-level Planning\.The low\-level planning module directly interacts with raw visual observations to make short\-horizon navigation decisions\. It takes multimodal inputs, including a natural language instruction, egocentric visual observations, and optionally a reference image of the target object\. At each step, the agent observes front, left, and right views from its current position, allowing it to compare candidate movement directions using local spatial evidence\. Based on these observations, it selects the next movement direction\. When the target object is detected in any of the current views, the agent selectsSTOP\.

Controller\.The controller executes primitive actions in the Habitat simulatorPuiget al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib75)\)\. Specifically,MOVE FORWARDmoves the agent by 1 meter, whileTURN LEFTandTURN RIGHTrotate the agent by 30 degrees, grounding planned actions into physical movement\.

## 4Experiments

### 4\.1Experimental Settings

#### 4\.1\.1Dataset and Task Setup

Dataset Construction\.We build our evaluation setup on top of PinNEDBarsellottiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib43)\), a personalized instance grounding dataset constructed in Habitat\-Matterport3D \(HM3D\) scenesRamakrishnanet al\.\([2021](https://arxiv.org/html/2605.26256#bib.bib56)\)with photorealistic 3D objects\. We use 4 scenes with 1,817 episodes in total\. FollowingKwonet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib40)\), we generate personalized context using GPT\-5\.111We perform a two\-step filtering process: manual screening and then MTurk\-based validation of quality criteria, including naturalness, personalization clarity, ambiguity, and target consistency\.For each target object, we generate an explicit instruction containing the personalized context and an evaluation instruction that requires the agent to infer the target from prior interactions\. In the acquisition stage, the agent executes the explicit instruction to construct prior user–agent interactions\. Although both instructions refer to the same target object, we vary object locations and agent initial positions across stages to prevent direct trajectory reuse\.

Evaluation Scenarios\.We design three evaluation scenarios to assess different challenges of personalized interpretation under long\-term interactions\.

- •Compositional Scenario\.We consider two variants under the compositional scenario:singleandjoint\. In the single setting, the target can be identified from a single prior episode\. In the joint setting, the agent must combine information from multiple \(2–3\) prior interactions to determine which object is intended\. This evaluates whether the agent can compose distributed personalized evidence across episodes to identify the correct target\.
- •Distractor Scenario\.The environment contains multiple candidate objects from the same category as the target\. This makes coarse category\-level grounding insufficient, and requires the agent to distinguish the intended target from semantically or visually similar distractors\.
- •Temporal Scenario\.The agent must capture for how personalized context changes over time\. We consider two forms of change\. First, the context associated with a specific object may evolve as the user’s routines or preferences change\. Second, the object referred to by the same context may itself change over time\. This scenario tests whether the agent can use the most recent valid interactions rather than rely on outdated associations\.

#### 4\.1\.2Evaluation Protocol

MLLM Backbones\.We evaluate across multiple backbone MLLMs, including the open\-source models Qwen3\-VL\-8B\-Instruct and Qwen2\.5\-VL\-8B\-InstructBaiet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib35)\), as well as the proprietary models GPT\-5, GPT\-4o\-miniHurstet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib37)\), and Gemini\-2\.5\-FlashTeamet al\.\([2023](https://arxiv.org/html/2605.26256#bib.bib36)\)\.

Evaluation Setup\.Before evaluation, we first run the acquisition stage, where the embodied agent executes the explicit personalized instructions\. The resulting episodes, consisting of user instructions and agent trajectories, are then used as prior user–agent interactions across evaluation scenarios within each scene\. We compare the result from acquisition and evaluation stage in Figure[5](https://arxiv.org/html/2605.26256#S4.F5)\.

Baselines\.Based on these prior interactions, we comparePolarwith two baselines: \(1\) ano\-priorbaseline, which uses only the current instruction and observations, and \(2\) araw\-interactionbaseline, where the agent is additionally given prior raw interactions\. Because it is infeasible to include all prior interactions due to context limitations, we randomly sample 15 prior interactions, including the gold interaction, for the raw\-interaction baseline\.

Evaluation Metrics\.We use two widely\-adopted metrics for embodied navigationZhaoet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib59)\); Qiaoet al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib26)\); Yanget al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib7)\): success rate \(SR\) and success weighted by path length \(SPL\)Andersonet al\.\([2018](https://arxiv.org/html/2605.26256#bib.bib85)\)\. SR measures whether the agent successfully reaches the target object\. An episode is considered successful if the distance between the agent’s final position and the goal object is less than two meters\. SPL further accounts for navigation efficiency and is defined asSPL=1N​∑i=1NSi​limax⁡\(pi,li\),\\mathrm\{SPL\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}S\_\{i\}\\frac\{l\_\{i\}\}\{\\max\(p\_\{i\},l\_\{i\}\)\},whereSiS\_\{i\}is a binary success indicator for episodeii,pip\_\{i\}is the agent’s path length, andlil\_\{i\}is the shortest\-path distance to the goal object\. We set the maximum number of steps to 700 for evaluation\.

Table 1:Performance comparison across multiple backbone MLLMs and evaluation scenarios\.### 4\.2Comparison with Baselines for Long\-Term Personalization

We organize our evaluation around the following research questions:

- •RQ1:Do the current instruction and observations provide enough information for personalized instance grounding?
- •RQ2:Are raw prior interactions sufficient for long\-term personalized task execution?
- •RQ3:How does the memory design ofPolaraffect target grounding and planning?

Main Results\.Table[4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2)presents the main evaluation results across multiple backbone MLLMs and evaluation scenarios\. Theno\-priorbaseline often performs worse than bothPolarand theraw\-interactionbaseline across all scenarios\. This suggests that the current observations alone often lack the information needed to interpret the target instruction, highlighting the importance of prior interactions for personalized task execution\. Theraw\-interactionbaseline sometimes improves overno\-priorby leveraging user\-specific information, but the gains are inconsistent\. This suggests that unstructured prior interactions can introduce distracting context, making them difficult to use reliably for the current task\. The advantage ofPolaris especially clear in the joint and temporal scenarios, showing that the object\-centric graph helps connect information distributed across multiple prior interactions\. Furthermore, we analyze whetherPolarimproves instance\-level grounding beyond category\-level navigation\. FollowingBarsellottiet al\.\([2024](https://arxiv.org/html/2605.26256#bib.bib43)\), we report Category Match \(CM\), which measures within\-category false positives where the agent reaches the correct category but the wrong target instance\. Lower CM indicates fewer same\-category target errors, andPolarconsistently yields lower CM than theraw\-interactionbaseline, showing that it better resolves the user\-intended instance among same\-category candidates\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/x4.png)Figure 5:Performance gap between the acquisition and evaluation stages\. In the acquisition stage, explicit personalized instructions with references are provided\.Polaryields the smallest gap when resolving underspecified instructions from prior interactions\.Effect of Retrieved Memory Utilization\.To further examine the effectiveness of the utilization process, Figure[7](https://arxiv.org/html/2605.26256#S4.F7)compares the success rate \(SR\) ofPolarand the raw\-interaction baseline under a controlled retrieval setting, using cases wherePolarretrieves the target instance and the raw\-interaction baseline includes the gold interaction\.Polarachieves higher SR across all scenarios, showing that successful task execution requires more than retrieving the correct memory\. The agent must also reason over the retrieved candidates to identify the intended target and apply the selected information to the current task\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/x5.png)
Figure 6:Within\-category false positive \(i\.e\., CM\)\.![Refer to caption](https://arxiv.org/html/2605.26256v1/x6.png)
Figure 7:Task success under a controlled retrieval setting\.![Refer to caption](https://arxiv.org/html/2605.26256v1/x7.png)
Figure 8:Retrieval recall for task\-relevant information\.
### 4\.3Analysis of Memory Design inPolar

Effect of Semantic Memory on Retrieval\.Figure[8](https://arxiv.org/html/2605.26256#S4.F8)analyzes how semantic memory affects retrieval of task\-relevant personalized information from prior interactions\. We comparePolarwith sparse retrieval \(BM25Robertson and Zaragoza \([2009](https://arxiv.org/html/2605.26256#bib.bib90)\)\) and dense retrieval \(BGEChenet al\.\([2024a](https://arxiv.org/html/2605.26256#bib.bib74)\)\), both of which retrieve directly from raw prior interactions by matching them to the target instruction\. YetPolarretrieves semantic memory nodes, where prior interactions are reformulated into concise, object\-specific statements\. The higher recall ofPolarshows that raw interaction retrieval is too coarse\-grained to isolate request\-relevant information, while semantic memory provides a finer, representation\.

Effect of Episodic Memory on Planning\.To examine the effect of episodic memory representation, we compare several memory representations under the same retrieval pipeline used inPolar\. Recent memory\-augmented agents often either preserve raw trajectoriesFenget al\.\([2026](https://arxiv.org/html/2605.26256#bib.bib82)\)or store

Table 2:Ablation on trajectory representations\.SettingSRInstruction only31\.1\+ Raw trajectory31\.9\+ Summary34\.5\\rowcolorgray\!15 \+ Episodic memory \(Ours\)36\.2

compact summaries of past interactionsLiuet al\.\([2025](https://arxiv.org/html/2605.26256#bib.bib67)\)\. Based on this design space, Table[2](https://arxiv.org/html/2605.26256#S4.T2)evaluates four variants: retrieved instructions only, raw trajectories, trajectory summaries, and the episodic memory used inPolar\. The comparison shows that retaining more history alone does not necessarily improve performance\. As raw trajectories are long and include many task\-irrelevant details, they can be difficult to exploit directly, whereas episodic memory extracts planning\-relevant experience from past interactions, enabling the agent to retrieve more actionable cues for the current task\.

![Refer to caption](https://arxiv.org/html/2605.26256v1/x8.png)Figure 9:Case study of personalized embodied assistance\.Given the same instruction, “trip to\-go,” different users intend different target objects, such as a backpack, headphones, or a book, depending on their prior interactions\. Without prior interactions, the agent relies on commonsense scene priors and gives the same generic answer, such as searching the living room\. In contrast,Polarretrieves user\-specific memories to infer the intended target and to guide the planning\.### 4\.4Case Study

Figure[9](https://arxiv.org/html/2605.26256#S4.F9)provides a qualitative example of how the same instruction can lead to different behaviors depending on personalized memory\. Without prior interactions, the agent can only rely on generic commonsense and scene priors, which leads to similar interpretations across users\. On the other hand,Polaruses retrieved semantic and episodic memories to condition both target grounding and subsequent planning on each user’s prior interactions\. This demonstrates that personalization affects not only target grounding, but also embodied search strategy\.

## 5Conclusion

In this work, we study long\-term personalization for embodied agents, since real\-world personalized assistance often requires agents to interpret user\-specific context implicitly accumulated through prior interactions, rather than relying on explicit target references\. We proposePolar, a memory\-augmented framework that organizes personalized context and embodied experiences into multimodal memory\. Experiments across multiple MLLM backbones and scenarios show thatPolaroutperforms no\-prior and raw\-interaction baselines\. These results highlight that effective long\-term personalization requires not merely longer context, but structured memory that supports target interpretation and embodied planning\.

## References

- \[1\]M\. Ahn, A\. Brohan, N\. Brown, Y\. Chebotar, O\. Cortes, B\. David, C\. Finn, C\. Fu, K\. Gopalakrishnan, K\. Hausman,et al\.\(2022\)Do as i can, not as i say: grounding language in robotic affordances\.arXiv preprint arXiv:2204\.01691\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[2\]Y\. Alaluf, E\. Richardson, S\. Tulyakov, K\. Aberman, and D\. Cohen\-Or\(2024\)Myvlm: personalizing vlms for user\-specific queries\.InEuropean Conference on Computer Vision,pp\. 73–91\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[3\]P\. Anderson, A\. Chang, D\. S\. Chaplot, A\. Dosovitskiy, S\. Gupta, V\. Koltun, J\. Kosecka, J\. Malik, R\. Mottaghi, M\. Savva,et al\.\(2018\)On evaluation of embodied navigation agents\.arXiv preprint arXiv:1807\.06757\.Cited by:[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p4.5)\.
- \[4\]A\. Ayub, C\. L\. Nehaniv, and K\. Dautenhahn\(2023\)A personalized household assistive robot that learns and creates new breakfast options through human\-robot interaction\.In2023 32nd IEEE International Conference on Robot and Human Interactive Communication \(RO\-MAN\),pp\. 2387–2393\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1)\.
- \[5\]S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge,et al\.\(2025\)Qwen3\-vl technical report\.arXiv preprint arXiv:2511\.21631\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p1.1)\.
- \[6\]L\. Barsellotti, R\. Bigazzi, M\. Cornia, L\. Baraldi, and R\. Cucchiara\(2024\)Personalized instance\-based navigation toward user\-specific objects in realistic environments\.Advances in Neural Information Processing Systems37,pp\. 11228–11250\.Cited by:[Figure 2](https://arxiv.org/html/2605.26256#S1.F2.2.1),[Figure 2](https://arxiv.org/html/2605.26256#S1.F2.4.2),[§1](https://arxiv.org/html/2605.26256#S1.p2.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1),[§4\.1\.1](https://arxiv.org/html/2605.26256#S4.SS1.SSS1.p1.1),[§4\.2](https://arxiv.org/html/2605.26256#S4.SS2.p2.1)\.
- \[7\]Y\. Bei, T\. Wei, X\. Ning, Y\. Zhao, Z\. Liu, X\. Lin, Y\. Zhu, H\. Hamann, J\. He, and H\. Tong\(2026\)Mem\-gallery: benchmarking multimodal long\-term conversational memory for mllm agents\.arXiv preprint arXiv:2601\.03515\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p3.1)\.
- \[8\]V\. Bhat, A\. U\. Kaypak, P\. Krishnamurthy, R\. Karri, and F\. Khorrami\(2024\)Grounding llms for robot task planning using closed\-loop state feedback\.arXiv preprint arXiv:2402\.08546\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[9\]Y\. Cao and C\. Lee\(2023\)Robot behavior\-tree\-based task generation with large language models\.arXiv preprint arXiv:2302\.12927\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[10\]J\. Chen, S\. Xiao, P\. Zhang, K\. Luo, D\. Lian, and Z\. Liu\(2024\)M3\-embedding: multi\-linguality, multi\-functionality, multi\-granularity text embeddings through self\-knowledge distillation\.InFindings of the association for computational linguistics: ACL 2024,pp\. 2318–2335\.Cited by:[§3\.2\.2](https://arxiv.org/html/2605.26256#S3.SS2.SSS2.p1.8),[§4\.3](https://arxiv.org/html/2605.26256#S4.SS3.p1.1)\.
- \[11\]T\. Chen, H\. Wang, S\. Chen, W\. Yu, K\. Ma, X\. Zhao, H\. Zhang, and D\. Yu\(2024\)Dense x retrieval: what retrieval granularity should we use?\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,pp\. 15159–15177\.Cited by:[§3\.2\.1](https://arxiv.org/html/2605.26256#S3.SS2.SSS1.p2.4)\.
- \[12\]G\. Cheng, C\. Zhang, W\. Cai, L\. Zhao, C\. Sun, and J\. Bian\(2024\)Empowering large language models on robotic manipulation with affordance prompting\.arXiv preprint arXiv:2404\.11027\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[13\]P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav\(2025\)Mem0: building production\-ready ai agents with scalable long\-term memory\.arXiv preprint arXiv:2504\.19413\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[14\]K\. Chu, X\. Zhao, C\. Weber, M\. Li, W\. Lu, and S\. Wermter\(2024\)Large language models for orchestrating bimanual robots\.In2024 IEEE\-RAS 23rd International Conference on Humanoid Robots \(Humanoids\),pp\. 328–334\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[15\]Y\. Dai, R\. Peng, S\. Li, and J\. Chai\(2024\)Think, act, and ask: open\-world interactive personalized robot navigation\.In2024 IEEE international conference on robotics and automation \(ICRA\),pp\. 3296–3303\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[16\]D\. Das, D\. Talon, Y\. Wang, M\. Mancini, and E\. Ricci\(2025\)Training\-free personalization via retrieval and reasoning on fingerprints\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 9683–9692\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[17\]D\. Driess, F\. Xia, M\. S\. Sajjadi, C\. Lynch, A\. Chowdhery, B\. Ichter, A\. Wahid, J\. Tompson, Q\. Vuong, T\. Yu,et al\.\(2023\)Palm\-e: an embodied multimodal language model\.arXiv preprint arXiv:2303\.03378\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[18\]J\. Feng, B\. Xu, J\. Chen, M\. Dai, C\. Wu, H\. Li, B\. Zeng, Y\. Xie, H\. Liang, M\. Lu,et al\.\(2026\)M2A: multimodal memory agent with dual\-layer hybrid memory for long\-term personalized interactions\.arXiv preprint arXiv:2602\.07624\.Cited by:[§4\.3](https://arxiv.org/html/2605.26256#S4.SS3.p2.1)\.
- \[19\]B\. Fu, Y\. Du, C\. Chang, B\. Jin, W\. Deng, M\. Xu, H\. Yan, W\. Song, and Y\. Wan\(2026\)MMKU\-bench: a multimodal update benchmark for diverse visual knowledge\.arXiv preprint arXiv:2603\.15117\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p3.1),[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[20\]H\. Hao, J\. Han, C\. Li, Y\. Li, and X\. Yue\(2025\)Rap: retrieval\-augmented personalization for multimodal large language models\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 14538–14548\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[21\]W\. Huang, P\. Abbeel, D\. Pathak, and I\. Mordatch\(2022\)Language models as zero\-shot planners: extracting actionable knowledge for embodied agents\.InInternational conference on machine learning,pp\. 9118–9147\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[22\]W\. Huang, C\. Wang, R\. Zhang, Y\. Li, J\. Wu, and L\. Fei\-Fei\(2023\)Voxposer: composable 3d value maps for robotic manipulation with language models\.arXiv preprint arXiv:2307\.05973\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[23\]W\. Huang, F\. Xia, T\. Xiao, H\. Chan, J\. Liang, P\. Florence, A\. Zeng, J\. Tompson, I\. Mordatch, Y\. Chebotar,et al\.\(2022\)Inner monologue: embodied reasoning through planning with language models\.arXiv preprint arXiv:2207\.05608\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[24\]A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p1.1)\.
- \[25\]J\. Kang, M\. Ji, Z\. Zhao, and T\. Bai\(2025\)Memory os of ai agent\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 25972–25981\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[26\]J\. Kim, W\. Kim, W\. Park, and J\. Do\(2026\)MMPB: it’s time for multi\-modal personalization\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track,External Links:[Link](https://openreview.net/forum?id=ElX4WxEa0W)Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p3.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[27\]R\. Korekata, Q\. Xie, Y\. Bisk, and K\. Sugiura\(2026\)Affordance rag: hierarchical multimodal retrieval with affordance\-aware embodied memory for mobile manipulation\.IEEE Robotics and Automation Letters11\(3\),pp\. 2706–2713\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1)\.
- \[28\]T\. Kwon, D\. Choi, H\. Kim, S\. Kim, S\. Moon, B\. Kwak, K\. Huang, and J\. Yeo\(2026\)Embodied agents meet personalization: investigating challenges and solutions through the lens of memory utilization\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=E5L43l5EIu)Cited by:[§4\.1\.1](https://arxiv.org/html/2605.26256#S4.SS1.SSS1.p1.1)\.
- \[29\]T\. Kwon, N\. Di Palo, and E\. Johns\(2024\)Language models as zero\-shot trajectory generators\.IEEE Robotics and Automation Letters9\(7\),pp\. 6728–6735\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[30\]S\. Lee, S\. Mo, and W\. Han\(2025\)Bring my cup\! personalizing vision\-language\-action models with visual attentive prompting\.arXiv preprint arXiv:2512\.20014\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p2.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[31\]X\. Lei, M\. Wang, W\. Zhou, L\. Li, and H\. Li\(2024\)Instance\-aware exploration\-verification\-exploitation for instance imagegoal navigation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16329–16339\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p2.1)\.
- \[32\]J\. Liang, W\. Huang, F\. Xia, P\. Xu, K\. Hausman, B\. Ichter, P\. Florence, and A\. Zeng\(2023\)Code as policies: language model programs for embodied control\.In2023 IEEE International conference on robotics and automation \(ICRA\),pp\. 9493–9500\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[33\]J\. Liu, Y\. Sun, W\. Cheng, H\. Lei, Y\. Chen, L\. Wen, X\. Yang, D\. Fu, P\. Cai, N\. Deng,et al\.\(2025\)Memverse: multimodal memory for lifelong learning agents\.arXiv preprint arXiv:2512\.03627\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1),[§4\.3](https://arxiv.org/html/2605.26256#S4.SS3.p3.1)\.
- \[34\]Z\. Liu, J\. Kim, X\. Luo, D\. Li, and Y\. Yang\(2026\)Exploratory memory\-augmented LLM agent via hybrid on\- and off\-policy optimization\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=UOzxviKVFO)Cited by:[§3\.2\.1](https://arxiv.org/html/2605.26256#S3.SS2.SSS1.p3.2)\.
- \[35\]L\. Long, Y\. He, W\. Ye, Y\. Pan, Y\. Lin, H\. Li, J\. Zhao, and W\. Li\(2026\)Seeing, listening, remembering, and reasoning: a multimodal agent with long\-term memory\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PMz29A7Muq)Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[36\]Y\. Mu, Q\. Zhang, M\. Hu, W\. Wang, M\. Ding, J\. Jin, B\. Wang, J\. Dai, Y\. Qiao, and P\. Luo\(2023\)Embodiedgpt: vision\-language pre\-training via embodied chain of thought\.Advances in Neural Information Processing Systems36,pp\. 25081–25094\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1)\.
- \[37\]T\. Nguyen, H\. Liu, Y\. Li, M\. Cai, U\. Ojha, and Y\. J\. Lee\(2024\)Yo’llava: your personalized language and vision assistant\.Advances in Neural Information Processing Systems37,pp\. 40913–40951\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[38\]Y\. Oh, D\. Chung, J\. Shin, S\. Park, J\. Barthelemy, J\. Mok, and S\. Yoon\(2026\)RePIC: reinforced post\-training for personalizing multi\-modal language models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=DG0F1cdjN7)Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[39\]C\. Packer, V\. Fang, S\. Patil, K\. Lin, S\. Wooders, and J\. Gonzalez\(2023\)MemGPT: towards llms as operating systems\.\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[40\]X\. Puig, E\. Undersander, A\. Szot, M\. D\. Cote, R\. Partsey, J\. Yang, R\. Desai, A\. W\. Clegg, M\. Hlavac, T\. Min, T\. Gervet, V\. Vondruš, V\. Berges, J\. Turner, O\. Maksymets, Z\. Kira, M\. Kalakrishnan, J\. Malik, D\. S\. Chaplot, U\. Jain, D\. Batra, A\. Rai, and R\. Mottaghi\(2023\)Habitat 3\.0: a co\-habitat for humans, avatars and robots\.Cited by:[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p4.1)\.
- \[41\]C\. Qian, P\. Han, Q\. Luo, B\. He, X\. Chen, Y\. Zhang, H\. Du, J\. Yao, X\. Yang, D\. Zhang,et al\.\(2024\)Escapebench: pushing language models to think outside the box\.arXiv e\-prints,pp\. arXiv–2412\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[42\]Y\. Qiao, H\. Hong, W\. Lyu, D\. An, S\. Zhang, Y\. Xie, X\. Wang, and Q\. Wu\(2026\)NavBench: probing multimodal large language models for embodied navigation\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=nf8PKQKtl2)Cited by:[Figure 2](https://arxiv.org/html/2605.26256#S1.F2),[Figure 2](https://arxiv.org/html/2605.26256#S1.F2.4.2.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1),[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p4.5)\.
- \[43\]Y\. Qiao, W\. Lyu, H\. Wang, Z\. Wang, Z\. Li, Y\. Zhang, M\. Tan, and Q\. Wu\(2025\)Open\-nav: exploring zero\-shot vision\-and\-language navigation in continuous environment with open\-source llms\.In2025 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 6710–6717\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[44\]S\. K\. Ramakrishnan, A\. Gokaslan, E\. Wijmans, O\. Maksymets, A\. Clegg, J\. Turner, E\. Undersander, W\. Galuba, A\. Westbury, A\. X\. Chang,et al\.\(2021\)Habitat\-matterport 3d dataset \(hm3d\): 1000 large\-scale 3d environments for embodied ai\.arXiv preprint arXiv:2109\.08238\.Cited by:[§4\.1\.1](https://arxiv.org/html/2605.26256#S4.SS1.SSS1.p1.1)\.
- \[45\]R\. Ramrakhya, M\. Chang, X\. Puig, R\. Desai, Z\. Kira, and R\. Mottaghi\(2025\)Grounding multimodal llms to embodied agents that ask for help with reinforcement learning\.arXiv preprint arXiv:2504\.00907\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[46\]S\. Robertson and H\. Zaragoza\(2009\)The probabilistic relevance framework: bm25 and beyond\.Vol\.4,Now Publishers Inc\.Cited by:[§4\.3](https://arxiv.org/html/2605.26256#S4.SS3.p1.1)\.
- \[47\]M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht\(2020\)Alfworld: aligning text and embodied environments for interactive learning\.arXiv preprint arXiv:2010\.03768\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[48\]I\. Singh, V\. Blukis, A\. Mousavian, A\. Goyal, D\. Xu, J\. Tremblay, D\. Fox, J\. Thomason, and A\. Garg\(2022\)Progprompt: generating situated robot task plans using large language models\.arXiv preprint arXiv:2209\.11302\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[49\]A\. Szot, B\. Mazoure, O\. Attia, A\. Timofeev, H\. Agrawal, D\. Hjelm, Z\. Gan, Z\. Kira, and A\. Toshev\(2025\)From multimodal llms to generalist embodied agents: methods and lessons\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 10644–10655\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[50\]F\. Taioli, E\. Zorzi, G\. Franchi, A\. Castellini, A\. Farinelli, M\. Cristani, and Y\. Wang\(2025\)Collaborative instance object navigation: leveraging uncertainty\-awareness to minimize human\-agent dialogues\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 18781–18792\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p2.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[51\]S\. Tan, D\. Zhou, X\. Shao, J\. Wang, and G\. Sun\(2025\)Language\-conditioned open\-vocabulary mobile manipulation with pretrained models\.arXiv preprint arXiv:2507\.17379\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1)\.
- \[52\]G\. Team, R\. Anil, S\. Borgeaud, J\. Alayrac, J\. Yu, R\. Soricut, J\. Schalkwyk, A\. M\. Dai, A\. Hauth, K\. Millican,et al\.\(2023\)Gemini: a family of highly capable multimodal models\.arXiv preprint arXiv:2312\.11805\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p1.1)\.
- \[53\]E\. Tulvinget al\.\(1972\)Episodic and semantic memory\.Organization of memory1\(381\-403\),pp\. 1\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p4.1)\.
- \[54\]H\. Wang, J\. Zhu, and H\. Dong\(2026\)User\-centric object navigation: a benchmark with integrated user habits for personalized embodied object search\.arXiv preprint arXiv:2602\.06459\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1)\.
- \[55\]A\. Xiao, N\. Janaka, T\. Hu, A\. Gupta, K\. Li, C\. Yu, and D\. Hsu\(2024\)Robi butler: multimodal remote interaction with a household robot assistant\.arXiv preprint arXiv:2409\.20548\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[56\]D\. Xu, Z\. Yang, J\. Chen, Y\. Yuan, M\. Hu, L\. Sun, L\. Van Gool, D\. P\. Paudel, and C\. Feng\(2026\)MultiHaystack: benchmarking multimodal retrieval and reasoning over 40k images, videos, and documents\.arXiv preprint arXiv:2603\.05697\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p3.1)\.
- \[57\]Y\. Xu, Y\. Pan, Z\. Liu, and H\. Wang\(2025\)Flame: learning to navigate with multimodal llm in urban environments\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 9005–9013\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[58\]H\. Xue, F\. Tang, M\. Hu, Y\. Liu, Q\. Huang, Y\. Li, C\. Liu, Z\. Xu, C\. Zhang, C\. Feng,et al\.\(2025\)Mmrc: a large\-scale benchmark for understanding multimodal large language model in real\-world conversation\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 22477–22503\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p3.1),[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[59\]R\. Yang, H\. Chen, J\. Zhang, M\. Zhao, C\. Qian, K\. Wang, Q\. Wang, T\. V\. Koripella, M\. Movahedi, M\. Li, H\. Ji, H\. Zhang, and T\. Zhang\(2025\)EmbodiedBench: comprehensive benchmarking multi\-modal large language models for vision\-driven embodied agents\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=DgGF2LEBPS)Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1),[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p1.2),[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p2.1),[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p4.5)\.
- \[60\]H\. Yin, X\. Xu, Z\. Wu, J\. Zhou, and J\. Lu\(2024\)Sg\-nav: online 3d scene graph prompting for llm\-based zero\-shot object navigation\.Advances in neural information processing systems37,pp\. 5285–5307\.Cited by:[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p2.1)\.
- \[61\]Y\. Yu, L\. Yao, Y\. Xie, Q\. Tan, J\. Feng, Y\. Li, and L\. Wu\(2026\)Agentic memory: learning unified long\-term and short\-term memory management for large language model agents\.arXiv preprint arXiv:2601\.01885\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[62\]Y\. Yue, Y\. Wang, B\. Kang, Y\. Han, S\. Wang, S\. Song, J\. Feng, and G\. Huang\(2024\)Deer\-vla: dynamic inference of multimodal large language models for efficient robot execution\.Advances in Neural Information Processing Systems37,pp\. 56619–56643\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[63\]S\. Zhang, Z\. Xu, P\. Liu, X\. Yu, Y\. Li, Q\. Gao, Z\. Fei, Z\. Yin, Z\. Wu, Y\. Jiang,et al\.\(2025\)Vlabench: a large\-scale benchmark for language\-conditioned robotics manipulation with long\-horizon reasoning tasks\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 11142–11152\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[64\]Y\. Zhang, Z\. He, J\. Li, J\. Lin, Q\. Guan, and W\. Yu\(2024\)MapGPT: an autonomous framework for mapping by integrating large language model and cartographic tools\.Cartography and Geographic Information Science51\(6\),pp\. 717–743\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1),[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p2.1)\.
- \[65\]Y\. Zhang, T\. Ma, Z\. Wang, Y\. Qiao, and P\. Kordjamshidi\(2025\)Vision\-and\-language navigation with analogical textual descriptions in llms\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,pp\. 15028–15036\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[66\]Z\. Zhang, Q\. Dai, X\. Bo, C\. Ma, R\. Li, X\. Chen, J\. Zhu, Z\. Dong, and J\. Wen\(2025\)A survey on the memory mechanism of large language model\-based agents\.ACM Transactions on Information Systems43\(6\),pp\. 1–47\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p3.1)\.
- \[67\]Z\. Zhang, Y\. Wang, H\. Xie, S\. Zhao, M\. Liu, Y\. Lu, X\. He, Z\. Cheng, and Y\. Peng\(2025\)Evaluating multimodal large language models with daily composite tasks in home environments\.arXiv preprint arXiv:2509\.17425\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[68\]X\. Zhao, W\. Cai, L\. Tang, and T\. Wang\(2025\)ImagineNav: prompting vision\-language models as embodied navigator through scene imagination\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vQFw9ryKyK)Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1),[§2](https://arxiv.org/html/2605.26256#S2.p1.1),[§4\.1\.2](https://arxiv.org/html/2605.26256#S4.SS1.SSS2.p4.5)\.
- \[69\]G\. Zhou, Y\. Hong, and Q\. Wu\(2024\)Navgpt: explicit reasoning in vision\-and\-language navigation with large language models\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.38,pp\. 7641–7649\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p1.1)\.
- \[70\]F\. Ziliotto, J\. R\. Akkara, A\. Daniele, L\. Ballan, L\. Serafini, and T\. Campari\(2025\)Personal: towards a comprehensive benchmark for personalized embodied agents\.arXiv preprint arXiv:2509\.19843\.Cited by:[§2](https://arxiv.org/html/2605.26256#S2.p2.1),[§3\.3](https://arxiv.org/html/2605.26256#S3.SS3.p2.1)\.
- \[71\]B\. Zitkovich, T\. Yu, S\. Xu, P\. Xu, T\. Xiao, F\. Xia, J\. Wu, P\. Wohlhart, S\. Welker, A\. Wahid,et al\.\(2023\)Rt\-2: vision\-language\-action models transfer web knowledge to robotic control\.InConference on Robot Learning,pp\. 2165–2183\.Cited by:[§1](https://arxiv.org/html/2605.26256#S1.p1.1)\.