PRISM: Perception Reasoning Interleaved for Sequential Decision Making

# PRISM: Perception Reasoning Interleaved for Sequential Decision Making
Source: [https://arxiv.org/html/2605.05407](https://arxiv.org/html/2605.05407)
Clemence GrislainClement RomacLaure SoulierMohamed ChetouaniOlivier SigaudNicolas Thome

###### Abstract

Scaling LLM\-based embodied agents from text\-only environments to complex multimodal settings remains a major challenge\. Recent work identifies a perception–reasoning–decision gap in standalone Vision–Language Models \(VLMs\), which often overlook task\-critical information\. In this paper, we introduceprism, a framework that tightly couples perception \(VLM\) and decision \(LLM\) through a dynamic question–answer \(DQA\) pipeline\. Instead of passively accepting the VLM’s description, the LLM critiques it, probes the VLM with goal\-oriented questions, and synthesizes a compact image description\. This closed\-loop interaction yields a sharp, task\-driven understanding of the scene\. We evaluateprismon the ALFWorld and Room\-to\-Room \(R2R\) benchmarks\. We show that: \(1\)prismsignificantly outperforms state\-of\-the\-art image\-based models, \(2\) our Interactive goal\-oriented perception pipeline yields systematic and substantial gains, and \(3\)prismis fully automatic, eliminating the need for handcrafted questions or answers\.

Machine Learning, ICML

## 1Introduction

![Refer to caption](https://arxiv.org/html/2605.05407v1/sec/Assets/intro/firstfig.png)Figure 1:Illustration ofprism’s interleaved perception \(VLM\) and reasoning \(LLM\)\.Given an image and goal,prism’s dynamic question\-answering \(DQA\) enables the LLM to query the VLM for task\-critical information\. This yields fine\-grained, task\-driven descriptions \(bottom\), ensuring accurate action prediction\. Conversely, methods decoupling the VLM and LLM yield task\-agnostic, incomplete descriptions, causing the agent to fail\.The field of Embodied Artificial Intelligence, which aims to develop agents capable of perceiving and autonomously interacting with the physical world, has attracted significant attention since the emergence of foundation models\. In particular, Large Language Models \(LLMs\) and Vision\-Language Models \(VLMs\) have demonstrated remarkable emergent capabilities in logical reasoning, semantic understanding, and common\-sense knowledge\(Ichteret al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib49); Zhaiet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib32); Liet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib8)\)\. This naturally led to growing interest in studying how to integrate them as components in embodied agents that interact with the world\. Our work studies how to incorporate them as components of agents interacting with visual environments in order to solve natural language instructions\.

Prior research has shown that LLMs can be leveraged for sequential decision\-making when provided with a textual description of the environment, either zero\-shot\(Yaoet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib2); Shinnet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib26)\), or through fine\-tuning\(Cartaet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib45); Wanget al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib57)\)\. These approaches typically assume perfect perception, where prompts contain all necessary information\. However, transitioning from such idealized text to raw visual input remains a major challenge\.

The impressive ability of VLMs to process images and generate text naturally positions them as candidates for this challenge\(Alayracet al\.,[2022](https://arxiv.org/html/2605.05407#bib.bib27); Liuet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib28)\)\. Recent works have explored two primary integration strategies: \(1\) directly replacing the LLM with a VLM\(Zhaiet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib32); Yanget al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib31)\), or \(2\) employing the VLM as an intermediate perception module to generate textual descriptions for a decision\-making LLM\(Panet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib33); Zhouet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib34); Aissiet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib48)\)\. However, performance still lags behind that achieved in textual environments\. For the latter, this gap highlights the limited ability of VLMs to capture task\-critical, fine\-grained visual information when perception \(VLM\) and decision\-making \(LLM\) are loosely coupled\.

In contrast, perception and decision\-making are deeply intertwined in humans, where sensory inputs prioritize information essential to the current goal\(Gibson,[1986](https://arxiv.org/html/2605.05407#bib.bib51)\)\. To mirror this interactivity, recent works have proposed introducing feedback mechanisms from decision\-making to perception, using human clarification or question\-answering\(Gaoet al\.,[2022](https://arxiv.org/html/2605.05407#bib.bib38); Shenet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib37); Longet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib36)\)\. However, the integration of perception and reasoning for decision\-making is achieved through naive solutions,e\.g\.,mere concatenation of textual interactions\. More importantly, these approaches rely on predefined questions or answers, restricting their applicability to realistic settings\.

In this paper, we introduceprism111Code can be found[here](https://github.com/sal1717lim/PRISM-Perception-Reasoning-Interleaved-for-Sequential-Decision-Making.):Perception andReasoningInterleaved forSequential Decision\-Making\.prismestablishes a tight coupling between perception and decision\-making through adynamic question\-answering\(DQA\) mechanism\. The perception module \(VLM\) provides an initial scene description, which the decision module \(LLM\) critiques against the goal to identify and query missing information\. Once the VLM responds, the LLM synthesizes the interaction feedback into a compact, task\-driven description\. As illustrated in Figure[1](https://arxiv.org/html/2605.05407#S1.F1),prism’s interleaved reasoning yields superior task\-oriented descriptions compared to approaches decoupling VLM and LLM modules\(Zhouet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib34); Panet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib33); Aissiet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib48)\)\.

Our contributions can be summarized as follows:

∙\\bulletDynamic Question\-Answering \(DQA\):DQA tightly couples perception and decision\-making, and synthesizes interaction feedback into a compact textual scene description\. Crucially,prism’s DQA is fully automatic, eliminating the need for unrealistic handcrafted or oracle annotations\.

∙\\bulletEfficient training withprism’s architecture:Separating perception via DQA from decision via the LLM enables the use of established recipes for text\-based Reinforcement Learning \(RL\), while effectively grounding decisions through online interactions within the visual environment\.

∙\\bulletExtensive experimentson ALFWorld\(Shridharet al\.,[2021](https://arxiv.org/html/2605.05407#bib.bib41)\)and Room\-to\-Room \(R2R\)\(Andersonet al\.,[2018](https://arxiv.org/html/2605.05407#bib.bib10)\)show thatprismsignificantly outperforms current image\-based models\. We demonstrate consistent gains from DQA across both benchmarks and provide rich analyses validating the generated questions, answers, and descriptions\.

![Refer to caption](https://arxiv.org/html/2605.05407v1/x1.png)Figure 2:The PRISM Framework:The agent processes a goalggand image observationoto^\{t\}to execute an actionata^\{t\}\.\(a\) Interactive Goal\-Oriented Perception:A VLM and LLM collaboratethrough dynamic question answering \(DQA\)to refine scene understanding:\(1\)the VLM provides an initial descriptionditd\_\{i\}^\{t\};\(2\)the LLM generates goal\-relevant questions𝒬\\cal\{Q\}t;\(3\)the VLM provides a specific answerλit\\lambda\_\{i\}^\{t\}for each questionqit∈q\_\{i\}^\{t\}\\in𝒬\\cal\{Q\}t; and\(4\)the LLM synthesizes these into a final descriptiondftd\_\{f\}^\{t\}\.\(b\) Decision\-Making:The same LLM, augmented with LoRA adapters trained for action generation, takesdftd\_\{f\}^\{t\}, the goalgg, and the historyℋ\\cal\{H\}to generate the actionata^\{t\}executed in the environment\.
## 2Related Work

### 2\.1Visual sequential decision\-making

To reach goals from visual inputs, VLM\-based agents have been explored\. As zero\-shot generalist VLMs often lack sufficient grounding, recent research has shifted toward RL fine\-tuning,e\.g\.,RL4VLMZhaiet al\.\([2024](https://arxiv.org/html/2605.05407#bib.bib32)\)\. However, these agents significantly underperform LLMs provided with perfect textual descriptionsYanget al\.\([2024](https://arxiv.org/html/2605.05407#bib.bib31)\), as discussed in introduction and demonstrated in our experiments \(Section[4](https://arxiv.org/html/2605.05407#S4)\)\. This disparity highlights the limited decision\-making capacity of current VLMs and the difficulty of directly applying vanilla RL—highly effective for text\-based inputsCartaet al\.\([2023](https://arxiv.org/html/2605.05407#bib.bib45)\); Wanget al\.\([2023](https://arxiv.org/html/2605.05407#bib.bib57)\)—to visual sequential decision\-making\.

Further efforts have been made to design architectures combining VLMs and LLMs to leverage their complementary strengths\. Typically, the VLM describes the visual scene while the LLM selects actions\. Such systems have shown promise in navigation\(Andersonet al\.,[2018](https://arxiv.org/html/2605.05407#bib.bib10); Zhouet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib34); Panet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib33); Zhanget al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib55)\)and interactive environments,e\.g\.,VIPER\(Aissiet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib48)\)\. However, perception and decision\-making remain largely decoupled in these approaches: the VLM operates independently of the agent’s reasoning, often overlooking task\-relevant visual details \(see Figure[1](https://arxiv.org/html/2605.05407#S1.F1)\)\. We adopt this modular architecture with a VLM for perception and an LLM for action selection, but introduce a dynamic question\-answering \(DQA\) mechanism that tightly couples perception and decision\-making\.

### 2\.2Interactive question\-answering in embodied agents

Asking questions is a powerful mechanism to acquire missing information\(Testoni and Fernández,[2024](https://arxiv.org/html/2605.05407#bib.bib46)\), enabling agents to refine their environmental understanding for effective sequential decision\-making\. Prior work enabled agents to query external sources such as humans or oracles as in DialFRED\(Gaoet al\.,[2022](https://arxiv.org/html/2605.05407#bib.bib38)\)or\(Nguyenet al\.,[2022](https://arxiv.org/html/2605.05407#bib.bib44); Chenet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib11)\), but typically relies on predefined questions and access to accurate oracles, limiting adaptability and autonomy in real\-world settings\.

Recent works such asLonget al\.\([2023](https://arxiv.org/html/2605.05407#bib.bib36)\)and DiscussNAV\(Shenet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib37)\)allow agents to query VLM\-based perception modules\. However, these systems still rely on fixed, predefined questions\. Furthermore, in DiscussNAV, answers are merely concatenated with scene descriptions, creating long, noisy prompts that hinder decision\-making\. In contrast,prismleverages the LLM to synthesize DQA interactions into a compact, task\-driven prompt suitable for fine\-tuning, including with RL\. We demonstrate thatprismoutperforms DiscussNAV in R2R with a simpler architecture and without relying on oracle questions or answers\.

## 3Method

### 3\.1Problem Statement

We consider a goal\-conditioned partially observable Markov decision processM=\(S,A,T,R,G,O,γ\)M=\(S,~A,~T,~R,~G,~O,~\\gamma\), with state spaceSS, deterministic transition functionT:S×A↦ST:S\\times A\\mapsto S, goal\-conditioned rewardR:S×G↦ℝR:S\\times G\\mapsto\\mathbb\{R\}, and discount factorγ∈\[0,1\)\\gamma\\in\[0,1\)\. We assume a multimodal setting where actionsat∈Aa^\{t\}\\in Aand goalsg∈Gg\\in Gare textual commands and instructions\. Given a language vocabulary𝒱\\mathcal\{V\}and a maximum sequence lengthNN, we haveAAandG∈𝒱NG\\in\\mathcal\{V\}^\{N\}\. The state observationot=O​\(st\)o^\{t\}=O\(s^\{t\}\)is an RGB image inℝ3×H×W\\mathbb\{R\}^\{3\\times H\\times W\}, whereH×WH\\times Ware image dimensions\. Given an initial visual observationo0o^\{0\}and a goalgg, we want to find the policyπθ:ℝ3×H×W×G↦A\\pi\_\{\\theta\}:\\mathbb\{R\}^\{3\\times H\\times W\}\\times G\\mapsto Athat maximizes the discounted sum of rewards obtained along its trajectory\.

### 3\.2PRISM architecture

We proposeprism\(see Figure[2](https://arxiv.org/html/2605.05407#S1.F2)\), an architecture integrating a VLM and an LLM across two distinct phases:interactive goal\-oriented perceptionanddecision\-making\.

During the interactive goal\-oriented perception phase, the system generates an intermediate textual scene representationdftd\_\{f\}^\{t\}\. We design an interaction loop leveraging pre\-trained VLM and LLM knowledge to obtaindftd\_\{f\}^\{t\}with the necessary information for decision\-making\.

More precisely, to alleviate VLMs’ limited ability to provide descriptions for decision\-making\(Zhanget al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib55)\), we introduce an iterative communication process in which the LLM identifies informational gaps in the VLM initial description given the goal, and queries the VLM for specific details\. The LLM then synthesizes the responses to update the environmental context, producing a refined final representationdftd\_\{f\}^\{t\}after the feedback interaction steps\.

For decision\-making, the same LLM used in the previous step acts as the policy, augmented with LoRA adapter\(Huet al\.,[2021](https://arxiv.org/html/2605.05407#bib.bib13)\)trained for action generation\. Given the descriptiondftd\_\{f\}^\{t\}, goal, and interaction history, this module generates the next actionat∈𝒜a^\{t\}\\in\\mathcal\{A\}to achieve the goalgg\.

Interactive goal\-oriented perception with VLM and LLM:Standard modular perception\-decision approaches that strictly decouple perception from reasoning, such as the ones proposed byPanet al\.\([2024](https://arxiv.org/html/2605.05407#bib.bib33)\); Aissiet al\.\([2025](https://arxiv.org/html/2605.05407#bib.bib48)\)often produce generic scene descriptions with missing key information \(see Figure[2](https://arxiv.org/html/2605.05407#S1.F2)\(a\)\)\. Conversely, simply providing the goal to the perception module introduces a high risk of hallucinations, which significantly deteriorates downstream performance, as demonstrated in our experiments \(see Section[4\.4](https://arxiv.org/html/2605.05407#S4.SS4)\)\. To overcome this, we introduce a Dynamic Question\-Answering \(DQA\) phase between the VLM, acting as the perception module \(𝒱𝒫\\cal\{V\}\_\{P\}\) and the LLM, serving as the reasoning module \(ℛ\\cal\{R\}\), to retrieve missing, task\-critical information\. At each time steptt, the agent receives a visual observationoto^\{t\}\. The perception module𝒱p\{\\cal\{V\}\}\_\{p\}first produces an initial descriptionditd\_\{i\}^\{t\}by being prompted with a fixed goal\-agnostic instructiondpd\_\{p\}, specificallydp=”Describe the scene in detail”d\_\{p\}=\\textit\{"Describe the scene in detail"\}:

dit=𝒱p​\(dp,ot\)\.d\_\{i\}^\{t\}=\{\\cal\{V\}\}\_\{p\}\\big\(d\_\{p\},o^\{t\}\\big\)\.\(1\)
The reasoning moduleℛ\\mathcal\{R\}generate a set of questions𝒬t\{\\cal\{Q\}\}^\{t\}adapted to bridge the information gap between the requirements of goalggand the initial descriptionditd\_\{i\}^\{t\}:

𝒬t=\{q1t,…,qnt\}=ℛ​\(dit,g\)\.\{\\cal\{Q\}\}^\{t\}=\\\{\{q\_\{1\}^\{t\},\\dots,q\_\{n\}^\{t\}\}\\\}=\\mathcal\{R\}\\big\(d\_\{i\}^\{t\},g\\big\)\.\(2\)The perception module𝒱p\{\\cal\{V\}\}\_\{p\}is then queried with each questionqjt∈𝒬tq\_\{j\}^\{t\}\\in\{\\cal\{Q\}\}^\{t\}to extract additional details from the current observationoto^\{t\}\. Then, the VLM directly produces a corresponding set of answersΛt\\Lambda^\{t\}:

Λt=λjtj=1\|𝒬t\|,λjt=𝒱p​\(qjt,ot\)\.\\Lambda^\{t\}=\{\{\\lambda\_\{j\}^\{t\}\}\}\_\{j=1\}^\{\|\{\\cal\{Q\}\}^\{t\}\|\},\\qquad\\lambda\_\{j\}^\{t\}=\{\\cal\{V\}\}\_\{p\}\\big\(q\_\{j\}^\{t\},o^\{t\}\\big\)\.\(3\)This mechanism ensures a fully automated pipeline by removing the human from the perception loop\. The performance of the agent with VLM answers are further evaluated in Section[4\.3\.3](https://arxiv.org/html/2605.05407#S4.SS3.SSS3)\. Finally, rather than simply concatenating the raw QA pairs, as in\(Longet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib36)\)—which results in an overlong and noisy input—we perform a semantic synthesis\. The reasoning moduleℛ\\mathcal\{R\}integrates\(dit,𝒬t,Λt\)\(d\_\{i\}^\{t\},\{\\cal\{Q\}\}^\{t\},\\Lambda^\{t\}\)to produce a refined descriptiondftd\_\{f\}^\{t\}\. This approach leverages the LLM’s summarization capabilities to distill the visual feedback into a concise representation, filtering out irrelevant details while emphasizing goal\-critical elements, as shown in Figure[2](https://arxiv.org/html/2605.05407#S1.F2)\(b\), that is

dft=ℛ​\(dit,𝒬t,Λt\)\.d\_\{f\}^\{t\}=\\mathcal\{R\}\\big\(d\_\{i\}^\{t\},\{\\cal\{Q\}\}^\{t\},\\Lambda^\{t\}\\big\)\.\(4\)The final descriptiondftd\_\{f\}^\{t\}is then used for decision\-making to generate appropriate actionsat∈Aa^\{t\}\\in Ato achieve the goalgg\. Specific prompt templates for question generation, perception answers and final synthesis are detailed in Appendix[A](https://arxiv.org/html/2605.05407#A1)\.

Sequential decision\-making with an LLM:Building on the framework established byCartaet al\.\([2023](https://arxiv.org/html/2605.05407#bib.bib45)\), we define an LLM\-based policyπ\\pithat directly maps the target goalgg, the history of past transitionsℋ\\cal\{H\}, and the refined textual descriptiondf\(t\)d\_\{f\}^\{\(t\)\}to an actionat∈𝒜a^\{t\}\\in\\mathcal\{A\}\(see Figure[2](https://arxiv.org/html/2605.05407#S1.F2)\(b\)\)\. We compute an action probabilityπθ​\(ai\|g,ℋ,dft\)\\pi\_\{\\theta\}\(a\_\{i\}\|g,\\mathcal\{H\},d\_\{f\}^\{t\}\)as the probability assigned by the LLM of the action tokensai=\{w0,…,w\|ai\|\}a\_\{i\}=\\\{w\_\{0\},\.\.\.,w\_\{\|a\_\{i\}\|\}\\\}to follow the goalgg, the history of past transitionsℋ\\cal\{H\}and the final descriptiondftd\_\{f\}^\{t\}:

πθ​\(ai\|g,ℋ,dft\)=∏j=0\|ai\|PL​L​M​\(wj\|g,ℋ,dft,w<j\)\.\\pi\_\{\\theta\}\(a\_\{i\}\|g,\\mathcal\{H\},d\_\{f\}^\{t\}\)=\\prod^\{\|a\_\{i\}\|\}\_\{j=0\}\{P\}\_\{LLM\}\(w\_\{j\}\|g,\\mathcal\{H\},d\_\{f\}^\{t\},w\_\{<j\}\)\.\(5\)The most probable action is then executed in the environment, which provides the next state and observationst\+1s^\{t\+1\}andot\+1o^\{t\+1\}\. In practice, the interactive goal\-oriented perception and the decision\-making phases share the same LLM backbone\. However, the LoRA adapters are specifically trained for action generation and are only activated during the decision\-making phase\. The overall pipeline is summarized in Algorithm[1](https://arxiv.org/html/2605.05407#alg1)\.

Algorithm 1prism0:Environment with observations

oto^\{t\}Task goal

ggPerception module

𝒱p\{\\cal\{V\}\}\_\{p\}LLM reasoning module

ℛ\\mathcal\{R\}Policy LLM

πθ\\pi\_\{\\theta\}\.

1:Initialize interaction history

ℋ←∅\\mathcal\{H\}\\leftarrow\\emptyset
2:foreach time step

t=1,2,…t=1,2,\\dotsdo

3:

ot←O​\(st\)o^\{t\}\\leftarrow O\(s^\{t\}\)
4:Interactive goal\-oriented perception phase

5:

dit←𝒱p​\(dp,ot\)d\_\{i\}^\{t\}\\leftarrow\{\\cal\{V\}\}\_\{p\}\(d\_\{p\},o^\{t\}\)
6:

𝒬t←ℛ​\(dit,g\)\{\\cal\{Q\}\}^\{t\}\\leftarrow\\mathcal\{R\}\(d\_\{i\}^\{t\},g\)
7:

Λt←\{λjt=𝒱p​\(qjt,ot\)\}j=1\|𝒬t\|\\Lambda^\{t\}\\leftarrow\\\{\\lambda\_\{j\}^\{t\}=\{\\cal\{V\}\}\_\{p\}\(q\_\{j\}^\{t\},o^\{t\}\)\\\}\_\{j=1\}^\{\|\{\\cal\{Q\}\}^\{t\}\|\}
8:

dft←ℛ​\(dit,𝒬t,Λt\)d\_\{f\}^\{t\}\\leftarrow\\mathcal\{R\}\\big\(d\_\{i\}^\{t\},\\,\{\\cal\{Q\}\}^\{t\},\\,\{\\Lambda\}^\{t\}\\big\)
9:Sequential decision\-making phase

10:

at∼πθ\(⋅∣g,ℋ,dft\)a^\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid g,\\mathcal\{H\},d\_\{f\}^\{t\}\)
11:Execute

ata^\{t\}
12:Update interaction history:

ℋ←ℋ∪\{dft,at\}\\mathcal\{H\}\\leftarrow\\mathcal\{H\}\\cup\\\{d\_\{f\}^\{t\},a^\{t\}\\\}
13:endfor

Table 1:Success rate \(SR\) in ALFWorld:Results marked with \* are our reimplementation\. The highest score for each task within a category is highlighted in bold\.prismsignificantly outperforms methods with no access to the text environment and demonstrates more stable performance across all tasks\. Notably, it reduces the gap to methods using textual supervision with privileged information\.
### 3\.3Training Strategies

Recent studies suggest that despite their vast common\-sense knowledge, LLMs require fine\-tuning on downstream tasks to align their linguistic representations with specific environmental constraints when used as policy\(Cartaet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib45); Zhaiet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib32)\)\. We therefore fine\-tune the LLM\-based policy \(with LoRA adapters\) with the following two\-stage training pipeline: pre\-training via Behavioral Cloning \(BC\) followed by online Reinforcement Learning \(RL\)\.

Unlike methods that rely on data or supervision from parallel textual environments\(Yanget al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib31); Aoet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib54)\), our approach operates exclusively within the visual environment\. Let𝒟=\{ot,ℋ,g,at,rt\}\\mathcal\{D\}=\\\{o^\{t\},\\mathcal\{H\},g,a^\{t\},r^\{t\}\\\}be a demonstration dataset collected in a multimodal visual environment\. We generate a textual version of this corpus, denoted as𝒟text=\{dft,ℋ,g,at,rt\}\\mathcal\{D\}\_\{\\text\{text\}\}=\\\{d^\{t\}\_\{f\},\\mathcal\{H\},g,a^\{t\},r^\{t\}\\\}, by substituting the initial visual observationsoto^\{t\}with textual descriptorsdftd^\{t\}\_\{f\}obtained throughprism’s interactive goal\-oriented perception process\. The LoRA\-enhanced policyπθ\\pi\_\{\\theta\}is first optimized to mimic these expert demonstrations via a BC objective:

ℒB​C​\(θ\)=−𝔼\(dft,ℋ,g,at\)∼𝒟text​\[log⁡πθ​\(at\|dft,ℋ,g\)\]\.\\mathcal\{L\}\_\{BC\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(d^\{t\}\_\{f\},\\mathcal\{H\},g,a^\{t\}\)\\sim\\mathcal\{D\}\_\{\\text\{text\}\}\}\\left\[\\log\\pi\_\{\\theta\}\(a^\{t\}\|d^\{t\}\_\{f\},\\mathcal\{H\},g\)\\right\]\.\(6\)We then transition to online RL using Proximal Policy Optimization \(PPO\)\(Schulmanet al\.,[2017](https://arxiv.org/html/2605.05407#bib.bib58)\)\. Crucially, our dynamic QA\-based perception remains active during the RL phase\. Our LLM\-based policyπθ​\(at\|dft,ℋ,g\)\\pi\_\{\\theta\}\(a^\{t\}\|d^\{t\}\_\{f\},\\mathcal\{H\},g\)is optimized to minimize the PPO CLIP objective:

ℒP​P​O​\(θ\)=𝔼^t​\[min⁡\(rt​\(θ\)​A^t,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^t\)\]\\mathcal\{L\}\_\{PPO\}\(\\theta\)=\\hat\{\\mathbb\{E\}\}^\{t\}\\left\[\\min\(r^\{t\}\(\\theta\)\\hat\{A\}^\{t\},\\text\{clip\}\(r^\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}^\{t\}\)\\right\]\(7\)where the advantage estimateA^t\\hat\{A\}\_\{t\}is derived from a learned value functionV^​\(dft,ℋ,g\)\\hat\{V\}\(d^\{t\}\_\{f\},\\mathcal\{H\},g\), as inCartaet al\.\([2023](https://arxiv.org/html/2605.05407#bib.bib45)\)\.

## 4Experiments

### 4\.1Experimental setup

Environment:Our experiments are conducted in two distinct environments: the ALFWorld simulated environment\(Shridharet al\.,[2021](https://arxiv.org/html/2605.05407#bib.bib41)\), which focuses on sequential interaction with household objects, and the Room\-to\-Room \(R2R\) environment\(Andersonet al\.,[2018](https://arxiv.org/html/2605.05407#bib.bib10)\), dedicated to navigation within photo\-realistic scenes\. We perform all in\-depth analyses and ablation studies in ALFWorld, while R2R is used to evaluate the versatility of our method across different visual and task domains\. For each experiment, evaluation is performed on out\-of\-distribution test sets\. Details regarding the environments are provided in Appendix[B\.1](https://arxiv.org/html/2605.05407#A2.SS1)\.

Implementations:We implementprismwith Idefics2\-8B\(Laurençonet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib17)\)as the VLM for perception, as it offers the best trade\-off between efficiency and performance on POPE\(Liet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib15)\)\. For both reasoning and sequential decision\-making, we employ Mistral\-7B\(Jianget al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib14)\)\. While our main results \(Section[4\.2](https://arxiv.org/html/2605.05407#S4.SS2)\) focus on this configuration, we demonstrate the generalizability of our method to other LLM backbones in Section[4\.3](https://arxiv.org/html/2605.05407#S4.SS3)\. Specifically, we use the frozen base LLM to leverage its inherent common\-sense reasoning, while enabling fine\-tuned LoRA adapters—trained following the method in Section[3\.3](https://arxiv.org/html/2605.05407#S3.SS3)—exclusively for decision\-making\. Details regarding the LoRA finetuning can be found in Appendix[B\.2](https://arxiv.org/html/2605.05407#A2.SS2)\.

Baselines:We compareprismwith two families of approaches\. The first family consists of agents trained with supervision from the ALFWorld textual environment \(i\.e\., perfect description\)\. This oracle supervision provides agents with perfect perception and access to privileged information not present in the visual observation, such as object IDs, states of occulted objects, and global environment dynamics\. \(More details are provided in Appendix[B\.5\.1](https://arxiv.org/html/2605.05407#A2.SS5.SSS1)\.\) This includes LLM\-based agents that operate solely on textual observations, such as ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib2)\), Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib26)\), DEPS\(Wanget al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib52)\), and AutoGen\(Wuet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib53)\)\. It also includes VLM\-based agents that process visual observations but are still supervised using these oracle textual descriptions, such as EMMA\(Yanget al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib31)\)and EMAC\+\(Aoet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib54)\)\.

The second family comprises methods that do not rely on supervision from a textual environment or privileged information, learning instead directly from multimodal interactions\. This family includes VLM\-only agents, either zero\-shot \(MiniGPT\-4\(Zhuet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib24)\)\) or trained \(Idefics2 8B\(Laurençonet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib17)\)and RL4VLM\(Zhaiet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib32)\)\)\. It also includes VLM\+LLM agents, specifically VIPERBCand VIPERBC\+RL\(Aissiet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib48)\)\. Further implementation and training details are provided in Appendix[B\.2](https://arxiv.org/html/2605.05407#A2.SS2)\.

### 4\.2Main results on ALFWorld

We first study the evaluation results in ALFWorld for all approaches\. As Table[1](https://arxiv.org/html/2605.05407#S3.T1)shows,prismdemonstrates notable superiority over methods without access to oracle textual descriptions\. It achieves a considerable performance gain of approximately 60 pt compared to the MiniGPT\-4 and RL4VLM baselines\.prismBCandprismBC\+RLalso significantly outperform the Idefics2 8B model with respectively 9 pt and 12 pt gains on the average SR\.

Notably, the improvement over VIPER highlights the importance of our DQA mechanism in bridging the gap between perception and action\. The base configurationprismBCalready surpasses VIPERBC\+RL\(77% vs 75%\), effectively exceeding the performance of the latter even when it benefits from RL\. The integration of RL into our approach,prismBC\+RL, further enhances these results by providing consistent gains across 4 out of 6 tasks compared to the BC\-only version\. This demonstrates the ability of our approach to successfully deal with RL in visual environments—a domain where baselines like RL4VLM often struggle\. Specifically, RL fine\-tuning leads to improvements in the ”Look”, ”Clean”, ”Heat”, and ”Pick2” categories, resulting in an average SR of 80%\. This progression enablesprismto achieve performance levels comparable to EMMA \(82%\) and EMAC\+ \(86%\), despite those models benefiting supervision from oracle textual environment\.

Beyond these absolute scores,prismis distinguished by its operational consistency: while text\-based methods such as ReAct and DEPS exhibit highly heterogeneous results,prismmaintains strong homogeneity across different tasks\.

### 4\.3Ablation studies

To analyze the contribution of each component within our framework, we conduct a series of ablation studies evaluating the1\) impact of PRISM architecture compared to a monolithic VLM, the2\) impact of merging strategies for DQA, and the3\) impact of generated questions and answerson providing task\-relevant support for the decision\-making process\.

#### 4\.3\.1Impact of PRISM architecture

We compareprismto two baseline categories to assess the impact of decoupling perception from decision\-making\. First, we consider monolithic VLMs, namely Idefics2 8B\(Laurençonet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib17)\)and Qwen3\-VL 4B\(Baiet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib19)\), where perception and decision\-making are end\-to\-end\. Second, we compare with static modular VLM\-LLM architectures \(i\.e\.\(Aissiet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib48); Panet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib33)\)\), notedRaw perception, where the VLM naively describes the scene for the LLM without our interactive goal\-oriented perception mechanism\. To isolate the effects of modularity, we create comparable pairs based on the LLM used for decision\-making: \(1\)prism\-Mistral is compared to Idefics2 and Raw perception \(Idefics\-Mistral\), and \(2\)prism\-Qwen3 \(with Qwen3 4B\(Yanget al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib16)\)as LLM\) is compared to Qwen3\-VL and Raw perception \(Idefics\-Qwen3\)\. This ensures a fair comparison across models with the same number of trainable parameters\.

Results:As shown in Table[2](https://arxiv.org/html/2605.05407#S4.T2),prismachieves superior results compared to Raw perception and Monolithic VLM across both model variants, yielding an average improvement of 6 pt\. This gain is particularly pronounced in complex tasks, with an increase of 9 to 12 pt on Pick 2 and an SR of 87% on Heat\. A detailed comparison between Raw perception and Monolithic VLM reveals similar results between the two variants\. This demonstrates that, while the decomposition of perception and reasoning is necessary, the performance gain does not stem from this structure, but from the enhancement of perception through the extraction of task\-relevant information\. Finally, theprismarchitecture remains consistent even when changing the LLM, maintaining an average SR of 77% and 75% across tasks\.

Table 2:Impact of separating perception and decision\-making on success rates\.prismoutperforms monolithic baselines by 7–8% on average, with significant gains in complex tasks\.For the remaining ablation studies, we use the Idefic\-Qwen3 variant ofprism, trained with BC\.

#### 4\.3\.2Impact of merging strategy for DQA

To evaluate howprismintegrates additional information into the initial scene description \(ditd\_\{i\}^\{t\}\), we compare it with two alternative merging methods\. The first,Concat, appends the generated questions and answers to the initial description, mirroring the naive approach used by DiscussNav\(Longet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib36)\)\. The second, QA\-Only, discards the initial description entirely and relies solely on the QA pairs for decision\-making\.

Results:As shown in Table[3](https://arxiv.org/html/2605.05407#S4.T3), our method consistently outperforms both the QA\-only andConcatapproaches, achieving a performance gain of 3 to 5 pt over previous work\. This improvement is stronger in high\-complexity tasks such as Pick2, validating the robustness of our architecture\.

Our ablation confirms that while QA\-derived information is crucial, it is insufficient on its own, as the QA\-only variant plateaus at an average SR of 70%\. Furthermore, the naive concatenation of the original descriptions and QA pairs \(Concat\) proves suboptimal\. As shown in Appendix[B\.6\.4](https://arxiv.org/html/2605.05407#A2.SS6.SSS4),Concatgenerates significantly longer and more redundant inputs\. This verbosity not only increases computational cost but also introduces noise that obscures task\-critical information, complicating the decision\-making process\.

In contrast,prismperforms a targeted synthesis of both sources, extracting and merging their most pertinent elements\. This refined input structure—both concise and comprehensive—streamlines information processing and facilitates more effective reasoning by the model\.

Table 3:Impact of Information Merging Strategies\.LLM\-mergeprismoutperforms rigid concatenation and QA\-only baselines by up to 5% on average, demonstrating the necessity of synthesizing global context with task\-specific details\.
#### 4\.3\.3Impact of generated questions & answers

We evaluate the impact of generating questions and answers by comparing it with two reference configurations: Oracle questions \(Oracle Q\), which reproduces the DiscussNav setting\(Longet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib36)\)using predefined questions, and Oracle questions and Answers \(Oracle QA\), which provides ground\-truth pairs\. For these baselines, questions are drawn from DialFRED annotations\(Gaoet al\.,[2022](https://arxiv.org/html/2605.05407#bib.bib38)\)and answers are extracted from the textual environment\. We assess our approach based on the SRs in six tasks, see Table[4](https://arxiv.org/html/2605.05407#S4.T4)\.

Results:Based on Table[4](https://arxiv.org/html/2605.05407#S4.T4), Oracle Q and Oracle QA configurations achieve the same results across all tasks, suggesting that the quality of the responses provided by the VLM are highly effective for task completion\. However, a comparative analysis between Oracle Q andprismhighlights stable performance on four tasks, contrasted by a slight drop for the Heat and Cool\. This leads to a marginal average drop of 2pt on average, attributable to the varying relevance of the generated questions\. Although the qualitative analysis of Oracle andprismquestions in Appendix[8](https://arxiv.org/html/2605.05407#A2.F8)confirms that the generated questions are coherent with task goals, they rely on common sense rather than specific environment dynamics\. As shown in Example 1 of Appendix[B\.6\.4](https://arxiv.org/html/2605.05407#A2.SS6.SSS4), the Oracle questions are hardcoded with environment priors, specifically asking for a microwave, whereas the LLM asks about general heating devices such as an oven or stove\. This reliance on general knowledge instead of rigid, environment\-specific heuristics explains the marginal 2pt performance gap, but it highlights thatprismremains effective without being tethered to the predefined priors of ALFWorld\. Further analysis regarding the relevance of these generated questions can be found in Appendix[B\.6\.3](https://arxiv.org/html/2605.05407#A2.SS6.SSS3.Px1)

Table 4:Success rate comparison betweenprismand human\-annotated oracle questions and answers across tasks:prismachieves identical SRs on 4 out of 6 tasks with a marginal 2% overall difference compared to oracle questions and answers\.

### 4\.4Model analysis

To thoroughly evaluate interactive goal\-oriented perception, we compare two configurations:Raw perception\(see Section[4\.3\.1](https://arxiv.org/html/2605.05407#S4.SS3.SSS1)\), andGoal\-aware perception, where the goal is included within the VLM prompt to condition the description\. We evaluate these models on task performance and the accuracy of their generated descriptions against ground\-truth from the parallel textual environment\. This quantitative analysis uses lexical metrics \(RougeL\(Lin,[2004](https://arxiv.org/html/2605.05407#bib.bib6)\), METEOR\(Banerjee and Lavie,[2005](https://arxiv.org/html/2605.05407#bib.bib18)\)\), semantic evaluation with BERTScore\(Zhanget al\.,[2020](https://arxiv.org/html/2605.05407#bib.bib7)\)and an LLM\-as\-a\-judge approach using the GPT\-5 API \(the evaluation prompt is available in Appendix[A\.1](https://arxiv.org/html/2605.05407#A1.SS1)\)\.

Results: Task performance:As shown in Table[5](https://arxiv.org/html/2605.05407#S4.T5), the Interactive goal\-oriented perception approach \(prism\) significantly leads with a 75% average SR, outperforming Raw perception \(68%\) and the Goal\-aware perception baseline \(64%\)\. Interestingly, simply conditioning the VLM on the task goal without a structured QA loop degrades performance by 4%, likely because forcing goal alignment induces hallucinations or deviates from the VLM’s standard descriptive training\. In contrast, the interactive nature ofprismeffectively mitigates these errors, particularly in Clean \(\+10%\) and Heat \(\+8%\) tasks\. These results underscore that targeted, interactive information acquisition is far more robust than simple goal\-conditioning for complex sequential tasks\.

Table 5:Impact of the Interactive goal\-oriented perception on success rate:prismachieves the highest average performance \(75%\), consistently outperforming both baselines\.Results: accuracy of generated descriptionsAs illustrated in Figure[3](https://arxiv.org/html/2605.05407#S4.F3), our interactive goal\-oriented perception approach \(prism\) significantly outperforms both baselines, achieving substantial gains over Raw Perception in RougeL \(\+0\.082\), METEOR \(\+0\.052\), and BERTScore \(\+0\.068\)\. Specifically, in the LLM\-as\-judge evaluation, our method achieves a 46\.6% preference rate, representing a \+20\.7 pt lead over the Raw perception baseline \(25\.9%\)\. These statistically significant results \(p<0\.005p<0\.005\) demonstrate that the iterative interaction effectively enforces description accuracy\. In contrast, Goal\-aware perception consistently degrades performance and triggers hallucinations; as shown in Appendix Figure[10](https://arxiv.org/html/2605.05407#A2.F10), focusing strictly on the goal leads the VLM to describe non\-existent objects\. This highlights that goal\-conditioning requires a dedicated grounding mechanism—like our QA interaction—to prevent the model from prioritizing perceived task relevance over environmental reality, which is detrimental to reliable downstream execution \(as reflected in the lower task success rates in Table[5](https://arxiv.org/html/2605.05407#S4.T5)\)\. Further qualitative comparisons and additional results across alternative VLM backbones are provided in Appendix[6](https://arxiv.org/html/2605.05407#A2.F6)\.

![Refer to caption](https://arxiv.org/html/2605.05407v1/x2.png)Figure 3:Scene description quality compared to environment ground truth\.Our Interactive goal\-oriented perception significantly outperforms other perception methods, demonstrating that the LLM\-VLM Interactive goal\-oriented perception enhances description accuracy\. In contrast, Goal\-aware perception reduces description quality\. \(\*\*\* indicatesp<0\.005p<0\.005\)\.
### 4\.5Extension to vision\-and\-language navigation tasks

Baselines:To evaluate the versatility ofprism, we extend our analysis to the R2R navigation benchmark using BC on expert trajectories, comparing our architecture against LangNav\(Panet al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib33)\), NavGPT\(Zhouet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib34)\), and DiscussNav\(Longet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib36)\)\. We report results from the original paper and our own reimplementations, denoted as LangNav\* and DiscussNav\*, which are built using the same model as our proposed architecture\. For DiscussNav\*, since the original corpus of questions is unavailable and the full framework requires interaction with four distinct experts, we adapt the approach by using our own set of questions and concatenating the QA to the original scene description\. With this adaptation, we maintain a consistent backbone and a controllable setting across all evaluations, ensuring a fair comparison to a standard prompt\-augmentation approach, using identical visual information\.

Metrics:We evaluate performance using standard navigation metrics: Navigation Error \(NE\), measuring the average distance to the goal at the stop position; Success Rate \(SR\), requiring the agent to stop within 3m of the goal; Oracle Success Rate \(OSR\), which counts a success if the agent passes within 3m of the goal at any point; and SR penalized by Path Length \(SPL\), which weights the success rate by the efficiency of the path\.

Results:As shown in Table[6](https://arxiv.org/html/2605.05407#S4.T6),prismachieves an SR of 46\.08% and an OSR of 65\.38%, and consistently surpasses established models like LangNav, NavGPT, and the original DiscussNAV —despite its access to multiple expert models and oracle questions—, as detailed in the Appendix[B\.5\.2](https://arxiv.org/html/2605.05407#A2.SS5.SSS2)\. This superior navigation accuracy is further reflected in an SPL of 43\.14, the highest across all tested methods, indicating thatprismnavigates more successfully and with more efficient path execution\. The most telling result emerges from the comparison with our DiscussNAV\* reimplementation: despite both models using identical interactive questions,prismexceeds this baseline by over 10pt in SR and significantly reduces NE from 7\.66 to 6\.23\. This gap reveals a fundamental architectural advantage; standard prompt concatenation introduces excessive noise, forcing models to process unrefined data during decision\-making\. By contrast,prismeffectively structures sensory feedback, proving that the primary bottleneck in interactive navigation is not only the acquisition of information, but the ability to filter out noise and leverage task\-relevant cues for superior spatial reasoning and goal reaching\.

Table 6:Performance on R2R\.Results marked with \* are our reimplementation\.prismachieves the best results in OSR, SR, and SPL compared to other methods, demonstrating superior navigation accuracy and efficiency\.

## 5Conclusion

In this work, we introducedprism, a modular framework that addresses the perception\-decision\-making gap in embodied agents through a dynamic question–answer \(DQA\) pipeline\. By tightly coupling a Vision\-Language Model \(VLM\) for perception with a Large Language Model \(LLM\) for reasoning and decision\-making, our approach enables the agent to dynamically identify and extract task\-critical information that is often missed by standard goal\-agnostic scene descriptions\. Our evaluations on ALFWorld and Room\-to\-Room benchmarks show that guided perception outperforms monolithic VLMs and traditional systems where these modules remain isolated\. These results suggest that interleaved reasoning effectively compensates for perceptual limitations, offering a robust path toward autonomous embodied agents\. Future work will examine the challenge of integrating these interleaved perception\-reasoning capabilities with lower\-level control, such as robotics settings, to bridge the gap between decision\-making and physical action\.

## Impact Statement

This paper presents work aimed at advancing language\-conditioned reasoning from visual observation through VLM\-LLM architecture\. While our approach improves the interpretability and autonomy of general\-purpose robots, it also inherits the biases of large\-scale pre\-trained models\. Beyond these broader considerations for autonomous systems, there are no immediate societal consequences of this research that must be specifically highlighted here\.

## Acknowledgments

Experiments presented in this paper were carried out using the HPC resources of IDRIS under the allocations 2025\-\[AD011015093R1\] and 2025\-\[AD011016980R1\] made by GENCI\. This work was supported by the European Commission’s Horizon Europe Framework Programme under grant No 101070381 \(PILLAR\-robots\), by RODEO Project \(ANR\-24\-CE23\-5886\), by PEPR Sharp \(ANR\-23\-PEIA\-0008, ANR, FRANCE 2030\) and by Cluster PostGenAI@Paris \(ANR\-23\-IACL\-0007, FRANCE 2030\)\.

## References

- M\. S\. Aissi, C\. Grislain, M\. Chetouani, O\. Sigaud, L\. Soulier, and N\. Thome \(2025\)VIPER: Visual Perception and Explainable Reasoning for Sequential Decision\-Making\.arXiv\.Note:arXiv:2503\.15108 \[cs\]External Links:[Link](http://arxiv.org/abs/2503.15108),[Document](https://dx.doi.org/10.48550/arXiv.2503.15108)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p3.1),[§1](https://arxiv.org/html/2605.05407#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2605.05407#S3.SS2.p5.8),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p4.2),[§4\.3\.1](https://arxiv.org/html/2605.05407#S4.SS3.SSS1.p1.1)\.
- J\. Alayrac, J\. Donahue, P\. Luc, A\. Miech, I\. Barr, Y\. Hasson, K\. Lenc, A\. Mensch, K\. Millican, M\. Reynolds, R\. Ring, E\. Rutherford, S\. Cabi, T\. Han, Z\. Gong, S\. Samangooei, M\. Monteiro, J\. Menick, S\. Borgeaud, A\. Brock, A\. Nematzadeh, S\. Sharifzadeh, M\. Binkowski, R\. Barreira, O\. Vinyals, A\. Zisserman, and K\. Simonyan \(2022\)Flamingo: a visual language model for few\-shot learning\.External Links:2204\.14198,[Link](https://arxiv.org/abs/2204.14198)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p3.1)\.
- P\. Anderson, Q\. Wu, D\. Teney, J\. Bruce, M\. Johnson, N\. Sünderhauf, I\. Reid, S\. Gould, and A\. van den Hengel \(2018\)Vision\-and\-language navigation: interpreting visually\-grounded navigation instructions in real environments\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p9.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p1.1)\.
- S\. Ao, F\. D\. Salim, and S\. Khan \(2025\)EMAC\+: embodied multimodal agent for collaborative planning with vlm\+llm\.External Links:2505\.19905,[Link](https://arxiv.org/abs/2505.19905)Cited by:[§B\.5\.1](https://arxiv.org/html/2605.05407#A2.SS5.SSS1.p1.1),[§3\.3](https://arxiv.org/html/2605.05407#S3.SS3.p2.5),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p3.1)\.
- S\. Bai, Y\. Cai, R\. Chen, K\. Chen, X\. Chen, Z\. Cheng, L\. Deng, W\. Ding, C\. Gao, C\. Ge, W\. Ge, Z\. Guo, Q\. Huang, J\. Huang, F\. Huang, B\. Hui, S\. Jiang, Z\. Li, M\. Li, M\. Li, K\. Li, Z\. Lin, J\. Lin, X\. Liu, J\. Liu, C\. Liu, Y\. Liu, D\. Liu, S\. Liu, D\. Lu, R\. Luo, C\. Lv, R\. Men, L\. Meng, X\. Ren, X\. Ren, S\. Song, Y\. Sun, J\. Tang, J\. Tu, J\. Wan, P\. Wang, P\. Wang, Q\. Wang, Y\. Wang, T\. Xie, Y\. Xu, H\. Xu, J\. Xu, Z\. Yang, M\. Yang, J\. Yang, A\. Yang, B\. Yu, F\. Zhang, H\. Zhang, X\. Zhang, B\. Zheng, H\. Zhong, J\. Zhou, F\. Zhou, J\. Zhou, Y\. Zhu, and K\. Zhu \(2025\)Qwen3\-vl technical report\.External Links:2511\.21631,[Link](https://arxiv.org/abs/2511.21631)Cited by:[§4\.3\.1](https://arxiv.org/html/2605.05407#S4.SS3.SSS1.p1.1)\.
- S\. Banerjee and A\. Lavie \(2005\)METEOR: an automatic metric for MT evaluation with improved correlation with human judgments\.InProceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization,J\. Goldstein, A\. Lavie, C\. Lin, and C\. Voss \(Eds\.\),Ann Arbor, Michigan,pp\. 65–72\.External Links:[Link](https://aclanthology.org/W05-0909/)Cited by:[§4\.4](https://arxiv.org/html/2605.05407#S4.SS4.p1.1)\.
- T\. Carta, C\. Romac, T\. Wolf, S\. Lamprier, O\. Sigaud, and P\. Oudeyer \(2023\)Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning\.InProceedings of the 40th International Conference on Machine Learning,pp\. 3676–3713\(en\)\.Note:ISSN: 2640\-3498Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2605.05407#S3.SS2.p7.10),[§3\.3](https://arxiv.org/html/2605.05407#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2605.05407#S3.SS3.p2.8)\.
- X\. Chen, S\. Zhang, P\. Zhang, L\. Zhao, and J\. Chen \(2023\)Asking before acting: gather information in embodied decision making with language models\.arXiv preprint arXiv:2305\.15695\.Cited by:[§2\.2](https://arxiv.org/html/2605.05407#S2.SS2.p1.1)\.
- X\. Gao, Q\. Gao, R\. Gong, K\. Lin, G\. Thattai, and G\. S\. Sukhatme \(2022\)DialFRED: dialogue\-enabled agents for embodied instruction following\.IEEE Robotics and Automation Letters7\(4\),pp\. 10049–10056\.External Links:ISSN 2377\-3774,[Link](http://dx.doi.org/10.1109/LRA.2022.3193254),[Document](https://dx.doi.org/10.1109/lra.2022.3193254)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.05407#S2.SS2.p1.1),[§4\.3\.3](https://arxiv.org/html/2605.05407#S4.SS3.SSS3.p1.1)\.
- J\. J\. Gibson \(1986\)The Ecological Approach to Visual Perception\.Psychology Press\(en\)\.Note:Google\-Books\-ID: DrhCCWmJpWUCExternal Links:ISBN 978\-0\-89859\-959\-6Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p4.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. Chen \(2021\)LoRA: low\-rank adaptation of large language models\.External Links:2106\.09685,[Link](https://arxiv.org/abs/2106.09685)Cited by:[§3\.2](https://arxiv.org/html/2605.05407#S3.SS2.p4.3)\.
- B\. Ichter, A\. Brohan, Y\. Chebotar, C\. Finn, K\. Hausman, A\. Herzog, D\. Ho, J\. Ibarz, A\. Irpan, E\. Jang, R\. Julian, D\. Kalashnikov, S\. Levine, Y\. Lu, C\. Parada, K\. Rao, P\. Sermanet, A\. T\. Toshev, V\. Vanhoucke, F\. Xia, T\. Xiao, P\. Xu, M\. Yan, N\. Brown, M\. Ahn, O\. Cortes, N\. Sievers, C\. Tan, S\. Xu, D\. Reyes, J\. Rettinghouse, J\. Quiambao, P\. Pastor, L\. Luu, K\. Lee, Y\. Kuang, S\. Jesmonth, N\. J\. Joshi, K\. Jeffrey, R\. J\. Ruano, J\. Hsu, K\. Gopalakrishnan, B\. David, A\. Zeng, and C\. K\. Fu \(2023\)Do As I Can, Not As I Say: Grounding Language in Robotic Affordances\.InProceedings of The 6th Conference on Robot Learning,pp\. 287–318\(en\)\.Note:ISSN: 2640\-3498External Links:[Link](https://proceedings.mlr.press/v205/ichter23a.html)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p1.1)\.
- A\. Q\. Jiang, A\. Sablayrolles, A\. Mensch, C\. Bamford, D\. S\. Chaplot, D\. de las Casas, F\. Bressand, G\. Lengyel, G\. Lample, L\. Saulnier, L\. R\. Lavaud, M\. Lachaux, P\. Stock, T\. L\. Scao, T\. Lavril, T\. Wang, T\. Lacroix, and W\. E\. Sayed \(2023\)Mistral 7b\.External Links:2310\.06825,[Link](https://arxiv.org/abs/2310.06825)Cited by:[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p2.1)\.
- H\. Laurençon, L\. Tronchon, M\. Cord, and V\. Sanh \(2024\)What matters when building vision\-language models?\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 87874–87907\.External Links:[Document](https://dx.doi.org/10.52202/079017-2789),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/a03037317560b8c5f2fb4b6466d4c439-Paper-Conference.pdf)Cited by:[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p2.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p4.2),[§4\.3\.1](https://arxiv.org/html/2605.05407#S4.SS3.SSS1.p1.1)\.
- M\. Li, S\. Zhao, Q\. Wang, K\. Wang, Y\. Zhou, S\. Srivastava, C\. Gokmen, T\. Lee, L\. E\. Li, R\. Zhang, W\. Liu, P\. Liang, L\. Fei\-Fei, J\. Mao, and J\. Wu \(2025\)Embodied agent interface: benchmarking llms for embodied decision making\.External Links:2410\.07166,[Link](https://arxiv.org/abs/2410.07166)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p1.1)\.
- Y\. Li, Y\. Du, K\. Zhou, J\. Wang, W\. X\. Zhao, and J\. Wen \(2023\)Evaluating object hallucination in large vision\-language models\.External Links:2305\.10355,[Link](https://arxiv.org/abs/2305.10355)Cited by:[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p2.1)\.
- C\. Lin \(2004\)ROUGE: a package for automatic evaluation of summaries\.InText Summarization Branches Out,Barcelona, Spain,pp\. 74–81\.External Links:[Link](https://aclanthology.org/W04-1013/)Cited by:[§4\.4](https://arxiv.org/html/2605.05407#S4.SS4.p1.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.External Links:2304\.08485,[Link](https://arxiv.org/abs/2304.08485)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p3.1)\.
- Y\. Long, X\. Li, W\. Cai, and H\. Dong \(2023\)Discuss before moving: visual language navigation via multi\-expert discussions\.arXiv preprint arXiv:2309\.11382\.Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.05407#S2.SS2.p2.1),[§3\.2](https://arxiv.org/html/2605.05407#S3.SS2.p6.11),[§4\.3\.2](https://arxiv.org/html/2605.05407#S4.SS3.SSS2.p1.1),[§4\.3\.3](https://arxiv.org/html/2605.05407#S4.SS3.SSS3.p1.1),[§4\.5](https://arxiv.org/html/2605.05407#S4.SS5.p1.1)\.
- K\. X\. Nguyen, Y\. Bisk, and H\. D\. Iii \(2022\)A framework for learning to request rich and contextually useful information from humans\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 16553–16568\.External Links:[Link](https://proceedings.mlr.press/v162/nguyen22a.html)Cited by:[§2\.2](https://arxiv.org/html/2605.05407#S2.SS2.p1.1)\.
- B\. Pan, R\. Panda, S\. Jin, R\. Feris, A\. Oliva, P\. Isola, and Y\. Kim \(2024\)LangNav: language as a perceptual representation for navigation\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 950–974\.External Links:[Link](https://aclanthology.org/2024.findings-naacl.60/),[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.60)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p3.1),[§1](https://arxiv.org/html/2605.05407#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2605.05407#S3.SS2.p5.8),[§4\.3\.1](https://arxiv.org/html/2605.05407#S4.SS3.SSS1.p1.1),[§4\.5](https://arxiv.org/html/2605.05407#S4.SS5.p1.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal Policy Optimization Algorithms\.arXiv:1707\.06347 \[cs\]\.Note:arXiv: 1707\.06347External Links:[Link](http://arxiv.org/abs/1707.06347)Cited by:[§3\.3](https://arxiv.org/html/2605.05407#S3.SS3.p2.6)\.
- Y\. Shen, D\. Bis, C\. Lu, and I\. Lourentzou \(2024\)ELBA: learning by asking for embodied visual navigation and task completion\.External Links:2302\.04865,[Link](https://arxiv.org/abs/2302.04865)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p4.1),[§2\.2](https://arxiv.org/html/2605.05407#S2.SS2.p2.1)\.
- N\. Shinn, F\. Cassano, E\. Berman, A\. Gopinath, K\. Narasimhan, and S\. Yao \(2023\)Reflexion: language agents with verbal reinforcement learning\.External Links:2303\.11366,[Link](https://arxiv.org/abs/2303.11366)Cited by:[§B\.5\.1](https://arxiv.org/html/2605.05407#A2.SS5.SSS1.p1.1),[§1](https://arxiv.org/html/2605.05407#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p3.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2021\)ALFWorld: Aligning Text and Embodied Environments for Interactive Learning\.InProceedings of the International Conference on Learning Representations \(ICLR\),External Links:[Link](https://arxiv.org/abs/2010.03768)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p9.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p1.1)\.
- A\. Testoni and R\. Fernández \(2024\)Asking the right question at the right time: human and model uncertainty guidance to ask clarification questions\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 258–275\.External Links:[Link](https://aclanthology.org/2024.eacl-long.16/),[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.16)Cited by:[§2\.2](https://arxiv.org/html/2605.05407#S2.SS2.p1.1)\.
- G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023\)Voyager: An Open\-Ended Embodied Agent with Large Language Models\.Transactions on Machine Learning Research\(en\)\.External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=ehfRiF0R3a)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p2.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p1.1)\.
- Z\. Wang, S\. Cai, G\. Chen, A\. Liu, X\. Ma, and Y\. Liang \(2024\)Describe, explain, plan and select: interactive planning with large language models enables open\-world multi\-task agents\.External Links:2302\.01560,[Link](https://arxiv.org/abs/2302.01560)Cited by:[§B\.5\.1](https://arxiv.org/html/2605.05407#A2.SS5.SSS1.p1.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p3.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu, A\. H\. Awadallah, R\. W\. White, D\. Burger, and C\. Wang \(2023\)AutoGen: enabling next\-gen llm applications via multi\-agent conversation\.External Links:2308\.08155,[Link](https://arxiv.org/abs/2308.08155)Cited by:[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p3.1)\.
- A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv, C\. Zheng, D\. Liu, F\. Zhou, F\. Huang, F\. Hu, H\. Ge, H\. Wei, H\. Lin, J\. Tang, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Zhou, J\. Lin, K\. Dang, K\. Bao, K\. Yang, L\. Yu, L\. Deng, M\. Li, M\. Xue, M\. Li, P\. Zhang, P\. Wang, Q\. Zhu, R\. Men, R\. Gao, S\. Liu, S\. Luo, T\. Li, T\. Tang, W\. Yin, X\. Ren, X\. Wang, X\. Zhang, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Wang, Z\. Cui, Z\. Zhang, Z\. Zhou, and Z\. Qiu \(2025\)Qwen3 technical report\.External Links:2505\.09388,[Link](https://arxiv.org/abs/2505.09388)Cited by:[§4\.3\.1](https://arxiv.org/html/2605.05407#S4.SS3.SSS1.p1.1)\.
- Y\. Yang, T\. Zhou, K\. Li, D\. Tao, L\. Li, L\. Shen, X\. He, J\. Jiang, and Y\. Shi \(2024\)Embodied multi\-modal agent trained by an llm from a parallel textworld\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 26275–26285\.Cited by:[§B\.5\.1](https://arxiv.org/html/2605.05407#A2.SS5.SSS1.p1.1),[§1](https://arxiv.org/html/2605.05407#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2605.05407#S3.SS3.p2.5),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§B\.5\.1](https://arxiv.org/html/2605.05407#A2.SS5.SSS1.p1.1),[§1](https://arxiv.org/html/2605.05407#S1.p2.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p3.1)\.
- Y\. Zhai, H\. Bai, Z\. Lin, J\. Pan, S\. Tong, Y\. Zhou, A\. Suhr, S\. Xie, Y\. LeCun, Y\. Ma,et al\.\(2024\)Fine\-tuning large vision\-language models as decision\-making agents via reinforcement learning\.arXiv preprint arXiv:2405\.10292\.Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p1.1),[§1](https://arxiv.org/html/2605.05407#S1.p3.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2605.05407#S3.SS3.p1.1),[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p4.2)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2020\)BERTScore: evaluating text generation with bert\.External Links:1904\.09675,[Link](https://arxiv.org/abs/1904.09675)Cited by:[§4\.4](https://arxiv.org/html/2605.05407#S4.SS4.p1.1)\.
- Y\. Zhang, X\. Huang, J\. Ma, Z\. Li, Z\. Luo, Y\. Xie, Y\. Qin, T\. Luo, Y\. Li, S\. Liu, Y\. Guo, and L\. Zhang \(2023\)Recognize anything: a strong image tagging model\.External Links:2306\.03514,[Link](https://arxiv.org/abs/2306.03514)Cited by:[§B\.5\.2](https://arxiv.org/html/2605.05407#A2.SS5.SSS2.p1.1)\.
- Y\. Zhang, T\. Ma, Z\. Wang, Y\. Qiao, and P\. Kordjamshidi \(2025\)Vision\-and\-Language Navigation with Analogical Textual Descriptions in LLMs\.InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 15017–15025\.External Links:ISBN 979\-8\-89176\-332\-6,[Link](https://aclanthology.org/2025.emnlp-main.759/),[Document](https://dx.doi.org/10.18653/v1/2025.emnlp-main.759)Cited by:[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p2.1),[§3\.2](https://arxiv.org/html/2605.05407#S3.SS2.p3.1)\.
- G\. Zhou, Y\. Hong, and Q\. Wu \(2023\)NavGPT: explicit reasoning in vision\-and\-language navigation with large language models\.External Links:2305\.16986,[Link](https://arxiv.org/abs/2305.16986)Cited by:[§1](https://arxiv.org/html/2605.05407#S1.p3.1),[§1](https://arxiv.org/html/2605.05407#S1.p5.1),[§2\.1](https://arxiv.org/html/2605.05407#S2.SS1.p2.1),[§4\.5](https://arxiv.org/html/2605.05407#S4.SS5.p1.1)\.
- D\. Zhu, J\. Chen, X\. Shen, X\. Li, and M\. Elhoseiny \(2023\)MiniGPT\-4: enhancing vision\-language understanding with advanced large language models\.External Links:2304\.10592,[Link](https://arxiv.org/abs/2304.10592)Cited by:[§4\.1](https://arxiv.org/html/2605.05407#S4.SS1.p4.2)\.

## Appendix AMethod details

This section provides technical details regarding our pipeline to ensure reproducibility\. We describe the interactions between the perception and reasoning modules, as well as the communication protocols used during the Goal\-Oriented Perception phase\.

### A\.1Prompts

We employ structured templates to maintain consistency across different tasks and stages of the pipeline\. Below, we detail the specific instructions used for each phase\.

Initial description:prompt used by the VLM to generate the initial description of the observationoto^\{t\}\.

Prompt: Initial DescriptionDescribe the scene in detail

Question Generation:Prompts used by the reasoning module \(RR\) to identify missing information inditd\_\{i\}^\{t\}relative to goalgg\.

Prompt: Question GenerationYou are a robot and you need to achieve a task\.Your current observation is:ditd\_\{i\}^\{t\}Your goal is:ggTask:\- Generate a set of questions to obtain missing information and missing objects of the goal in the observation that is necessary to achieve the goal\. If the information is present in the observation, don’t ask question about it, ask only about missing informations in the observation\.\- If no information is missing, respond with: ”I have all the information\.”\- The question must: \* Be related only to the given observation, and goal \(not other parts of the house\)\. \* Begin with: ”Do you see …”Output format:A valid JSON dictionary like this and complete the questions:\{\{”question1”: ”Do you see … ?”,…”\}\}

Visual Question Answering \(VQA\):Instructions provided to the perception module \(V​PVP\) to answer questions𝒬t\{\\cal\{Q\}\}^\{t\}about missing or additional information \(i\.e\., extracting precise visual attributes from the image\)\.

Prompt: Visual Question AnsweringLook at the image and answer the question with a detailed explanation\. Do not answer with just yes/no:qitq\_\{i\}^\{t\}

Description Refinement:The template used to aggregate the initial descriptionditd\_\{i\}^\{t\}with the questions𝒬t\{\\cal\{Q\}\}^\{t\}and associated answersΛt\\Lambda^\{t\}into a final descriptiondftd\_\{f\}^\{t\}\.

Prompt: Description RefinementYou are an assistant that merges visual information into one coherent scene description\.Your task is to:1\. Take the initial description of the scene\.2\. Integrate all new information provided in the question–answer pairs\.3\. Remove contradictions and keep only information confirmed by the answers\.4\. Produce ONE final, clean, coherent, and natural description of the scene\.5\. Do NOT mention the questions, the Q/A process, or uncertainty\.6\. Only describe what is explicitly confirmed\.Here is the initial description:ditd\_\{i\}^\{t\}Here are the question–answer pairs:𝒬\\cal\{Q\}t,Λ,\\LambdatNow rewrite the scene into a SINGLE enriched description that combines all confirmed details\. Make it concise, factual, and fluent\. Do not invent new objects or properties\.

In addition to our main pipeline, we also provide the specific prompts used for the Goal\-Aware Perception baseline from Section[4\.4](https://arxiv.org/html/2605.05407#S4.SS4)—where the goal is directly injected into the perception module—and the LLM\-as\-a\-Judge template, used to evaluate how well the generated description aligns with the ground truth of the environment \(from Section[4\.4](https://arxiv.org/html/2605.05407#S4.SS4)\)\.

Baseline: Goal\-Aware PerceptionYou are a robot and you need to accomplish the following goal:gg\.Task: Identify the key elements to reach the goal\. Then, describe the image\.

Evaluation prompt: LLM as judgeYour task is to determine which of three candidate descriptions more close to the GT\.\#\#\# Instructions\- Carefully compare each candidate with the GT\.\- Focus on:1\. The presence of all objects mentioned in the GT\.\- Ignore minor stylistic or grammatical differences\.\- Choose \*\*one\*\* candidate that best matches the GT\.\- If two or more descriptions are equally aligned with the GT, output ”tie”\.\- Do not provide any reasoning or explanation\.\#\#\# Instructions1\. Compare each detail in Text A, Text B and Text C against the GT\.2\. Do not return empty output3\. Only output one of these options exactly:\- ”Text A is more aligned”\- ”Text B is more aligned”\- ”Text C is more aligned”Goal Text: GTText A: Raw perceptionText B: Goal Aware perceptionText C: Interactive goal\-oriented perceptionBest aligned description \[Text A/Text B/Text C\]:

## Appendix BExperiments

### B\.1Environment

Room\-to\-Room \(R2R\):The R2R dataset is a benchmark for Vision\-and\-Language Navigation \(VLN\) based on the Matterport3D simulator\. It uses real\-world indoor panoramas where an agent must follow natural language instructions\. The action space is discrete, consisting of: Turn Left, Turn Right, Move Forward, and Stop\. Success is measured by the agent’s ability to reach the target location within a specific distance threshold\.

ALFWorld:ALFWorld is a synthetic environment combining AI2\-THOR and TextWorld for multi\-step household tasks\. It requires agents to perform object manipulation \(e\.g\., cleaning, heating\) through a text\-based action space\. Success depends on the agent’s ability to identify task\-relevant objects and execute a sequence of interactions in the correct order\.

Table 7:Episode counts for Train and Val Unseen splits\.
### B\.2Implementations details

##### Training dataset

For data collection in ALFWorld, we use the native rule\-based bot provided by the environment\. Given this bot’s success rate of 70% over the training tasks, we ensure the quality of demonstrations by training our agents exclusively on successful trajectories\. During data collection, we runprisminteractive goal\-oriented perception for each transition to turn the environment observations into textual descriptions\. We then train the decision\-making component with BC using this data \(thus generating textual descriptions only once\)\. Following this initial supervised phase, we perform RL across the entire training set, starting from the BC weights\.

For the R2R environment, we use the bot provided by R2R\. This bot uses the Dijkstra shortest\-path algorithm to find the optimal path from the initial position to the goal\. We use the same BC training scheme as in ALFWorld: we compute textual descriptions once over the entire training set and then train the decision\-making component\.

##### Implementation Details

We update the LLM used as sequential decision\-making component by applying LoRA adapters exclusively to the attention layers \(Wq,WvW\_\{q\},W\_\{v\}\)\. During BC training, we use a learning rate of10−410^\{\-4\}\. For online RL with PPO, we reduce the learning rate to5×10−65\\times 10^\{\-6\}to prevent catastrophic forgetting\. More details about these hyperparameters can be found in Table[8](https://arxiv.org/html/2605.05407#A2.T8)\. Following training, we keep the checkpoint with highest SR over training tasks as our final checkpoint used for evaluation\.

Table 8:Hyperparameters for LoRA adaptation, BC pre\-training, and PPO fine\-tuning\.CategoryParameterValueLoRA ConfigurationTarget LayersWq,WvW\_\{q\},W\_\{v\}Rank \(rr\)32Alpha \(α\\alpha\)16ContextHistory Transition Length \(\|ℋ\|\|\\mathcal\{H\}\|\)5BC pre\-trainingLearning Rate10−410^\{\-4\}OptimizerAdamWRL fine\-tuningLearning Rate5×10−65\\times 10^\{\-6\}PPO Clip \(ϵ\\epsilon\)0\.1Entropy Coefficient0\.001Gamma \(γ\\gamma\)0\.99

### B\.3RL fine\-tuning efficiency

We track the training progress by monitoring the SR across different tasks\. Figure[4](https://arxiv.org/html/2605.05407#A2.F4)shows the evolution of SR during the RL fine\-tuning on ALFWorld\.

![Refer to caption](https://arxiv.org/html/2605.05407v1/x3.png)Figure 4:Evolution of the success rate \(SR\) during RL fine\-tuning across the six ALFWorld tasks\.The x\-axis represents the number of training episodes, while the y\-axis indicates the SR achieved\.The training curves show that applying PPO consistently improves SR\. However, the scale of this improvement varies significantly between tasks\. These curves corroborate the results from Table[1](https://arxiv.org/html/2605.05407#S3.T1), particularly regarding the Pick task, where only little improvement is observed\. For this specific task, the SR stagnates, hovering around 0\.80 for a significant portion of the training episodes before a late increase\.

### B\.4Inference Latency

We report in Table[9](https://arxiv.org/html/2605.05407#A2.T9)the per\-step inference latency ofprismand Raw Perception \(No QA\), alongside their SR and average number of steps executed per episode on ALFWorld\. All measurements are collected on a single NVIDIA H100 GPU\.

Table 9:Inference latency, SR and average steps on ALFWorld\.prismincurs a per\-step latency approximately3×3\\timeshigher than Raw Perception, due to the three sequential forward passes involved at each step: question generation, visual question answering, and description refinement\. This overhead is however offset at the episode level:prismexecutes 18% fewer environment steps on average \(17\.1 vs\. 20\.9\), resulting in fewer redundant or erroneous actions\. Combined with a 6\-point gain in SR, these results show that the additional per\-step cost is compensated by more reliable action selection across the full episode\.

### B\.5Baselines

#### B\.5\.1Baselines on ALFWorld

This section details the ALFWorld baselines evaluated in our experiments \(see Section[4](https://arxiv.org/html/2605.05407#S4)\) that utilize ground\-truth supervision from the textual environment\. This category includes methods such as ReAct\(Yaoet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib2)\), Reflexion\(Shinnet al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib26)\), DEPS\(Wanget al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib52)\), EMMA\(Yanget al\.,[2024](https://arxiv.org/html/2605.05407#bib.bib31)\), and EMAC\+\(Aoet al\.,[2025](https://arxiv.org/html/2605.05407#bib.bib54)\)\. Unlike approaches that operate on visual input, these agents are trained and evaluated with access to privileged state information provided by the simulator in textual form\. This supervisory signal includes perfect symbolic metadata—such as the unique internal IDs of all objects, the presence and states of occluded items not in the agent’s view, and explicit goal destination IDs—which is impossible to infer from pixel\-level observations alone\.

An example of this informational advantage is illustrated in Figure[5](https://arxiv.org/html/2605.05407#A2.F5)\. The textual observation for the scene lists objects such as spraybottle 1, toiletpaper 1, plunger 1, and cloth 1\. In the corresponding visual scene, only the spraybottle is directly visible\. The toiletpaper is occluded behind it, and the plunger and cloth are not present in the image at all\. Critically, the textual supervision not only reveals the existence and precise identifiers of these non\-visible objects, but also provides their exact labels \(e\.g\., 1\)\. This information creates a perfect, lossless mapping to the simulator’s discrete action space \(e\.g\., ake plunger 1 from toilet 1\)\. A purely visual agent cannot perceive these absent or hidden objects, nor can it infer their abstract IDs from pixels\. This fundamental disparity in environmental access—where text\-supervised agents receive privileged symbolic truth—explains the substantial performance gap between these baselines andprism, which must reason and act based solely on partial, egocentric visual observations\.

![Refer to caption](https://arxiv.org/html/2605.05407v1/x4.png)Figure 5:An example of a textual observation returned by ALFWorld Textual Environment
#### B\.5\.2Baselines on R2R

In R2R, existing baselines primarily address environmental ambiguity through static perception pipelines\. For instance, NavGPT employs a static perception module, using InstructBLIP as its VLM and GPT\-3\.5 via an API for decision\-making\. It thus relies on the powerful but generic reasoning of a large, zero\-shot LLM\. In a parallel architectural approach, LangNav also decouples perception from decision\-making, but opts for a lightweight LLM specifically fine\-tuned for navigation\. Alternatively, DiscussNav queries a predefined corpus of oracle questions, creating a separation where perception \(using a VLM and an object\-detection model\(Zhanget al\.,[2023](https://arxiv.org/html/2605.05407#bib.bib9)\)\) gathers information, and a static set of questions retrieves answers\. While distinct in implementation, these methods share a core limitation: their perception\-and\-information\-gathering mechanisms are fixed and non\-adaptive, operating independently from the navigation agent’s immediate needs or the evolving context\.

### B\.6Additional results

#### B\.6\.1Description Quality

We extend our analysis of scene description quality by evaluating theprismframework with alternative VLM backbones, specifically employing InternVL for perception while maintaining Mistral as the reasoning module\. This complements the main results where Idefics served as the primary VLM\. As illustrated in Figure[6](https://arxiv.org/html/2605.05407#A2.F6), our Interactive goal\-oriented perception approach \(prism\) significantly outperforms both Raw and Goal\-aware perception baselines even when using this alternative backbone, achieving a dominant 66\.9% preference in the LLM\-as\-a\-judge evaluation and superior scores across RougeL, METEOR, and BERTScore \(p<0\.005p<0\.005\)\. While the Goal\-aware baseline using InternVL shows a marginal improvement in semantic alignment \(BERTScore\) over the raw output, it suffers from a decrease in description quality \(RougeL\), highlighting that a simple goal\-directed prompt is insufficient\. These findings confirm that the performance gains ofprismare consistent across different VLM backbones\.

![Refer to caption](https://arxiv.org/html/2605.05407v1/x5.png)Figure 6:Scene description quality compared to environment ground truth with InternVL\.Our Interactive goal\-oriented perception approach significantly outperforms other perception methods\. In contrast, Goal\-aware perception reduces description quality\. \(\*\*\* indicatesp<0\.005p<0\.005\)\.
#### B\.6\.2VLM Answer Accuracy

In this section, we evaluate the accuracy of Idefics2 responses to goal\-directed questions over 1,000 transitions sampled from ALFWorld\. Ground\-truth answers are derived from the symbolic state of the textual environment\. We report precision, recall, and F1\-score per task category in Table[10](https://arxiv.org/html/2605.05407#A2.T10)\.

Table 10:VLM answer accuracy per task category on ALFWorld\.Metrics are computed by comparing Idefics2 responses against ground\-truth answers from the textual environment over 1,000 transitions\.Idefics2 achieves an average F1\-score of 0\.92\. Performance is strongest on Heat and Clean, where task\-relevant objects are spatially prominent, and slightly lower on Cool and Pick due to greater visual ambiguity and object occlusion\. The average error rate of 8% does not translate into proportional degradation in task SR, consistent with the Oracle comparisons in Table[1](https://arxiv.org/html/2605.05407#S3.T1)\. This robustness is attributable to the description refinement step, which filters inconsistent information before it reaches the decision\-making module, and to the sequential structure of the tasks, which allows the agent to recover from isolated perceptual errors over subsequent steps\.

#### B\.6\.3Question Generation

##### Impact of the number of questions:

In this section, we evaluate the impact of the number of questions used in our Interactive goal\-oriented perception approach \(using the Idefics\-Qwen variant\)\.

We first report in Figure[7](https://arxiv.org/html/2605.05407#A2.F7)the mean number of questions asked byprismover the different environments and tasks\. Results indicateprismasks an average of 2 to 3 questions\.

We then evaluateprismwhen forcing the number of questions that must be generated\. We do so by specifying the maximum number of questions that can be asked at each step\. The results, detailed in Table[11](https://arxiv.org/html/2605.05407#A2.T11), demonstrate that increasing the QA Budget from 1 to 3 leads to a consistent performance gain, with the average SR increasing from 70% to 74%\. This trend suggests that the ability to ask multiple questions allows the LLM to better disambiguate task requirements, particularly in tasks such as Heat, where improvement is 8%\.

Importantly, results show thatprismstill obtains the highest performance with dynamic question generation\.

Table 11:Impact of number of Questions on Performance![Refer to caption](https://arxiv.org/html/2605.05407v1/x6.png)Figure 7:Average number of questions asked by PRISM across different tasks and environments\.
##### Relevance of generated questions:

![Refer to caption](https://arxiv.org/html/2605.05407v1/sec/Assets/Section4/scatter_plot.png)Figure 8:Semantic alignment between Oracle and questions generated by PRISM\.T\-SNE visualization of BERT embeddings along with lines between two questions associated with the same environment state\. Results demonstrate thatprismproduces questions similar to the ones authored by humans\.To evaluate the quality of the generated questions, we perform a comparative analysis with the human\-authored Oracle questions described in section[4\.3\.3](https://arxiv.org/html/2605.05407#S4.SS3.SSS3)\. We first compute BERT embeddings for both the Oracle and questions generated byprismand visualize their distributions using t\-SNE\. As shown in Figure[8](https://arxiv.org/html/2605.05407#A2.F8), the two sets are closely aligned within the semantic space, indicating a high degree of distributional similarity\. Furthermore, by connecting points belonging to the same environment state with arrows, we observe that the generated questions remain close to their corresponding oracle reference\. This consistent state\-level alignment confirms thatprismaccurately identifies the same information needs as the human\-authored questions\.

#### B\.6\.4Qualitative examples

In this section, we provide qualitative results and examples ofprism\.

##### Generated questions vs Oracle questions:

We observe in Example 1 that the generated questions are remarkably close to the Oracle ones, as they successfully identify the missing information indid\_\{i\}that is required to solve goalgg\. However, environment understanding remains limited\. While the generated questions demonstrate some common\-sense understanding from the LLM, the Oracle ones benefit from a clearer prior knowledge of the environment\. For instance, in the heating task, the Oracle focuses on a ”Microwave” and even queries its state \(e\.g\., ”is it empty?”\), showing it already knows which specific heating device is available in that scene\. In contrast, the generated questions remain broader, asking for a ”stove or heating source” in general\. This same distinction appears in cleaning tasks; where the Oracle might ask about a specific sink based on its prior mapping of the room, the LLM asks about ”cleaning devices” generally\. Ultimately, while the LLM correctly infers the necessary steps through logic and identifies what is missing from the initial description, the Oracle questions are more precise because they are tailored to the known affordances of that specific space\.

Example 1did\_\{i\}:In this picture we can see a table, there are some plants, bottles, bowls and other things present on the table, in the background we can find a wall\.Goalgg:Heat some egg and put it in the dining tableOracle questions:\[Do you see a egg in the image?, Do you see a Microwave in the image?, Do you see a dining table in the image?’, ’is the Microwave empty?\]Generated questions:\[’Do you see a stove or heating source?’, ’Do you see an egg?’, ’Do you see a plate or container to place the cooked egg on?’, ’Do you see a dining table?’\]

Example 2did\_\{i\}:In this picture we can see a table, there is a toaster, bread, an apple and some other things present on the table, in the background we can find cupboards\.Goalgg:Clean some tomato and put it in the microwave\.Oracle questions:\[Do you see a plate in the image?, Do you see a dining table in the image?, Do you see a fridge in the image?’, ’Is the fridge open?’\]Generated questions:\[’Do you see a plate that needs to be cooled?’, ’Do you see a cooling source or method available, such as ice or cooler?’, ’Do you see a dining table?’\]

Example 3did\_\{i\}:In this picture we can see a table, there are some bottles, bowls, spoons and other things present on the table, at the bottom there is floor\.Goalgg:Cool some plate and put it in the dining table\.Oracle questions:\[Do you see a egg in the image?, Do you see a Microwave in the image?, Do you see a dining table in the image?’, ’Is the Microwave empty?\]Generated questions:\[’Do you see a stove or heating source?’, ’Do you see an egg?’, ’Do you see a plate or container to place the cooked egg on?’, ’Do you see a dining table?’\]

##### Merging strategies:

In this analysis, we compare the merging strategy used byprismand theConcatstrategy \(see Section[4\.3\.2](https://arxiv.org/html/2605.05407#S4.SS3.SSS2)\), notably used by DiscussNav\. At an equivalent operational state, theConcatapproach exhibits a significantly higher token volume, often doubling the prompt length; this makes processing the textual observation harder, as the model must disentangle the relationship between raw QA pairs and initial descriptions\. In contrast,prismemploys a merging mechanism that synthesizes these elements into a more compact and accurate representation\. By eliminating semantic redundancy and maintaining a lower token count without sacrificing critical information\.

prismprompt in R2RYou are an agent and you need to accomplish a task Your task is to: Walk down one flight of stairs and stop on the landing\.Current Observation:Front: The scene consists of a pillar, floor mat, various objects on it, doors, windows, a chandelier, walls, and a railing on the left side\. Additionally, there’s a staircase visible, with landings between each set of steps, all stairs either lead to other floors or upward, and they have a handrail on both sides\. At the top right corner of the image, there is a ceiling with lights\.Left: The scene consists of pillars, carpeted floor, various objects, and a fire extinguisher located on the left side\. Above, lights illuminate the space, but there is no sign of a staircase or a handrail within view\.Right: The scene features a pillar, carpet on the floor, various objects, and a set of stairs with a handrail on one side, descending from the area where these items are located, under a lit ceiling\.Next action of the agent:\(Number of token= 967\)

CONCAT prompt in R2RYou are an agent and you need to accomplish a task\. Your task is to: Walk down one flight of stairs and stop on the landing\.Current Observation:Front: In this picture, we can see a pillar, floor mat and some objects on it\. We can see doors, windows, chandelier and wall\. On the left side of the picture we have railing\. At the top right corner of the image, we can also see ceiling with lights\.QA: \[\(’Do you see any staircase leading downwards from where I am currently located?’, ’ yes, there is a visible staircase that leads downward from my current location\.’\), \(’Do you see a handrail along the staircase that leads downwards?’, ’ Yes, there is a hand railing on both sides of the stairs\.’\), \(’Is there a landing area visible between two consecutive steps while looking at the downward staircase?’, ’ Yes, there is a landing in\-between the stairs as we can see from this picture\.’\)\]Left: In this picture, we can see pillars, carpet on the floor and some objects\. On the left side of the image there is a fire extinguisher\. At the top of the ceiling there are lights\.QA: \[\(’Do you see any staircase leading down from where you currently are?’, ’ No, I do not see any stairs in this room\.’\), \(’Do you see a handrail or anything similar along the staircase?’, ’ No, there is no handrails on the stairs\.’\)\]Right: In this picture, we can see a pillar, carpet on the floor and some objects\. On top of the image there is ceiling with lights\.QA: \[\(’Do you see any staircase leading down from the location where the pillar, carpet, and objects are?’, ’ Yes\.’\), \(’Do you see a handrail or anything similar along the potential staircase?’, ’ Yes, there is a hand railing on one side of the stairs\.’\)\]Next action of the agent:\(Number of token=1764\)

##### Interactive goal\-oriented perception

Finally, we provide a qualitative analysis of the whole Interactive goal\-oriented perception in Figure[9](https://arxiv.org/html/2605.05407#A2.F9)and Figure[10](https://arxiv.org/html/2605.05407#A2.F10)\.

prismdemonstrates a sophisticated ability to generate relevant, goal\-oriented questions, particularly within the R2R dataset\. As seen in Examples \(1\) and \(2\) of Figure[9](https://arxiv.org/html/2605.05407#A2.F9), the model proactively inquires about the presence of a staircase and potential obstacles, which are essential pieces of information to complete a navigation task safely\.

Examining ALFWorld reveals the limitations of Goal\-aware perception\. In several instances, this approach generates descriptions that are nearly identical to Raw perception, effectively ignoring the goal\. Furthermore, this perception approach is prone to hallucinations when under pressure to find goal\-related items; for instance, in Example \(4\) of Figure[10](https://arxiv.org/html/2605.05407#A2.F10), the Goal\-aware description falsely claims that there is ”cd on the table,” whileprism’s interactive perception later correctly identifies that no CDs are actually visible in the scene\.

![Refer to caption](https://arxiv.org/html/2605.05407v1/sec/Assets/Appendix/Exemple1r.png)Figure 9:Qualitative examples ofprismgenerated descriptions on R2R\.![Refer to caption](https://arxiv.org/html/2605.05407v1/sec/Assets/Appendix/Exemple2r.png)Figure 10:Qualitative examples ofprismgenerated descriptions on ALFWorld\.