Advancing DialNav through Automatic Embodied Dialog Augmentation

# Advancing DialNav through Automatic Embodied Dialog Augmentation
Source: [https://arxiv.org/html/2606.19948](https://arxiv.org/html/2606.19948)
Leekyeung Han1Sangwon Jung2Hyunji Min1 Jinseong Jeong1Minyoung Kim1Paul Hongsuck Seo1 1Korea University2Trillion Labs \{happilee12, daream2, dw9030, omniverse186, phseo\}@korea\.ac\.kr sangwon\.jung@trillionlabs\.co

###### Abstract

For embodied agents capable of physical interaction, the capability to create and understand dialog is crucial to ensure both safety and effectiveness\. While DialNavHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)provides a framework for holistic evaluation of the dialog–execution loop in photorealistic indoor navigation, its performance remains limited by a critical scarcity of training data \(2K episodes\)\. To address this, we propose an automatic generation pipeline, and construct theRAINbowdataset, a large\-scale training dataset with 238K episodes for DialNav\. Our pipeline converts existing VLN datasets into multi\-turn dialog and creates cost\-efficient and high\-quality dataset\. Then, we introduce two additional complementary advances to unlock the data’s full potential: \(1\) Dual\-Strategy Training, a navigation training scheme to align the navigation training with the dynamic dialog\-navigation loop, and \(2\) a localization model that leverages VLN knowledge\. By combining these complementary solutions, our model substantially outperforms the baseline in success rate on bothVal Seen\(58\.24,\+89%\) andVal Unseen\(29\.05,\+100%\) splits, establishing a new state of the art\. Our code and dataset are available at:[https://happilee12\.github\.io/RAINbow](https://happilee12.github.io/RAINbow)

Advancing DialNav through Automatic Embodied Dialog Augmentation

Leekyeung Han1Sangwon Jung2Hyunji Min1Jinseong Jeong1Minyoung Kim1Paul Hongsuck Seo11Korea University2Trillion Labs\{happilee12, daream2, dw9030, omniverse186, phseo\}@korea\.ac\.krsangwon\.jung@trillionlabs\.co

![Refer to caption](https://arxiv.org/html/2606.19948v1/x1.png)Figure 1:Overview of the DialNav task\.Top:The Navigator starts at an initial nodebband navigates to reach the goal regionRR\. Since the initial instruction is underspecified, the Navigator engages in multi\-turn dialog with a remote Guide to acquire additional guidance along the navigation\.Bottom:At each step, the Navigator follows a modular decision process: it either proceeds autonomously \(*Navigation*\) or requests help \(*Question Generation*\)\. When a question is asked, the remote Guide localizes the Navigator and provides an answer describing the next path to the goal\. This forms an alternating loop of dialog and action that continues until the goal region is reached\.## 1Introduction

Embodied agents must operate with high reliability, since misinterpreting instructions or executing incorrect actions can cause physical harm\. Enabling dialog improves both safety and task effectiveness: by asking questions and refining their understanding before acting, agents can resolve ambiguity and adapt to dynamic situations\. Learning dialog\-enabled agents, however, remains highly challenging\. Collecting datasets is costly, as it requires two people to coordinate in real time while grounding their conversations in the given task context\. Furthermore, even with such data, training remains difficult because if either the action trajectory or the dialog deviates from the collected annotations, the supervision becomes invalid\.

We study these challenges inDialNav\(Hanet al\.,[2025](https://arxiv.org/html/2606.19948#bib.bib1)\), a cooperative dialog\-based vision\-and\-language navigation \(VLN\)\(Andersonet al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib22)\)task\. VLN is an embodied navigation problem in which an agent follows a fixed natural language instruction given at the start of an episode to reach a goal\. DialNav extends this into a dialog\-based setup, where navigation unfolds through an interactive dialog exchange\.

In DialNav, two agents, aNavigatorand a remoteGuide, collaborate through dialog to accomplish the navigation task in a photo\-realistic indoor environment \(Matterport3D\(Changet al\.,[2017](https://arxiv.org/html/2606.19948#bib.bib21)\)\)\. The Navigator starts with only an ambiguous and underspecified instruction \(e\.g\.,“the target room has a plant”\), which is insufficient to reach the goal; success therefore hinges on the following dialog\. To this end, the Navigator proactively asks the Guide for additional guidance during navigation whenever uncertainty arises\. Due to the remote Guide setup in DialNav, the Guide first has tolocalizethe Navigator based on this question, which makes the question critical\. The Guide then answers with context\-specific instructions that help subsequent navigation decisions\. To train such agents,Hanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)first collected the RAIN dataset, a human\-human interaction dataset for DialNav\. Then, the Navigator and Guide are modeled by separated neural modules specialized for their constituent capabilities \(e\.g\.,question generation, answer generation, and navigation\)\. These modules are trained separately on RAIN and later integrated forholisticnavigation\-dialog loop of interactive navigation\.

WhileHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)introduced the first dataset and baseline models for DialNav, their performance remains limited\. A key reason is the severe scarcity of training data: RAIN contains only 2K episodes, which is insufficient to support the high complexity of dialog\-based navigation\. Yet collecting human annotated DialNav dataset is exceptionally expensive \(e\.g\.,7,500 USD for 2K episodes\), as it requires two people to coordinate in real time while grounding their conversations\. Although dataset augmentation techniques to mitigate the cost of human annotation have been explored in VLN\(Friedet al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib29); Fanet al\.,[2024](https://arxiv.org/html/2606.19948#bib.bib49); Wanget al\.,[2025b](https://arxiv.org/html/2606.19948#bib.bib50)\), they are restricted to single\-turn settings and are not directly applicable to DialNav\.

To address this data scarcity challenge, we propose an automatic DialNav episode generation pipeline\. Our pipeline leverages existing fine\-grained VLN datasets by concatenating their paths, generating scene captions to act as questions, and using an LLM to reformat these into natural dialog\. Through this process, we yield theRAINbowdataset, which is more than two orders of magnitude larger than the original RAIN\. Despite this substantial data scaling, our preliminary experiments reveal that data augmentation alone is not effective when paired with the existing training framework inHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)\. We identify that the training scheme fromHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)fails to effectively leverage the augmented data\. Therefore, we adopt Dual\-Strategy Training, a training scheme better aligned with DialNav task that fully exploits our scaled multi\-turn dialog\-based episodes\. We additionally improve the localization, an essential subtask to DialNav which remains underexplored\.

Together, these advances improve performance of the holistic navigation\-dialog loop by more than two\-fold: success rate increases from 30\.77 to 58\.24 \(\+89%\) onVal Seenand from 14\.52 to 29\.05 \(\+100%\) onVal Unseen\. We will publicly release our dataset, code, and models\.

In summary, our contributions are threefold:

- •We construct RAINbow, a large\-scale dataset, expanding the training data by more than two orders of magnitude beyond RAIN\.
- •We adopt Dual\-Strategy Training to better exploit large\-scale dialog data and improve localization, an underexplored but critical subtask in DialNav\.
- •We deliver a new state of the art performance on DialNav, doubling success rate over the previous baseline\.

## 2Related Work

Vision and Language Navigation\.Vision\-and\-Language Navigation \(VLN\) is an embodied AI task in which an agent navigates through visual environments by following natural language instructions\. Instructions are either coarse\-grained\(Zhuet al\.,[2021a](https://arxiv.org/html/2606.19948#bib.bib26); Qiet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib25)\), often too ambiguous for reliable navigation, or fine\-grained\(Andersonet al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib22); Kuet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib24); Chenet al\.,[2019](https://arxiv.org/html/2606.19948#bib.bib37)\), often overly detailed and unnatural for human interaction\. Some prior works have explored incorporating dialog or interaction in navigation\(Thomasonet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib23); Romanet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib8); de Vrieset al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib7); Banerjeeet al\.,[2021](https://arxiv.org/html/2606.19948#bib.bib9); Nguyen and Daumé III,[2019](https://arxiv.org/html/2606.19948#bib.bib10); Zhuet al\.,[2021b](https://arxiv.org/html/2606.19948#bib.bib38)\)\. The recently proposed DialNav task\(Hanet al\.,[2025](https://arxiv.org/html/2606.19948#bib.bib1)\)places particular emphasis on the dialog through its remote guide setup and dialog\-navigation loop evaluation\. Although DialNav provides a valuable benchmark, its performance is limited primarily due to its small dataset size\. In this work, we improve DialNav performance by large\-scale data augmentation, complemented by enhancements in training\.

VLN dataset augmentation\.Data scarcity is a well\-known challenge in VLN\. To address this, various data augmentation approaches have been explored\. Some works focused on training a text generation model to generate new instructions for unannotated paths\(Friedet al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib29); Zhang and Kordjamshidi,[2023](https://arxiv.org/html/2606.19948#bib.bib39); Wanget al\.,[2024a](https://arxiv.org/html/2606.19948#bib.bib40)\)\. Other works scaled environments, either by editing existing scenes\(Tanet al\.,[2019](https://arxiv.org/html/2606.19948#bib.bib44); Liuet al\.,[2021](https://arxiv.org/html/2606.19948#bib.bib45); Liet al\.,[2022](https://arxiv.org/html/2606.19948#bib.bib46)\)or leveraging large sets of additional environments\(Wanget al\.,[2023c](https://arxiv.org/html/2606.19948#bib.bib33); Chenet al\.,[2022a](https://arxiv.org/html/2606.19948#bib.bib28); Kohet al\.,[2022](https://arxiv.org/html/2606.19948#bib.bib43); Guhuret al\.,[2021](https://arxiv.org/html/2606.19948#bib.bib53); Linet al\.,[2023](https://arxiv.org/html/2606.19948#bib.bib52)\)\. Recently, more advanced methods have emerged, adopting LLM generators or applying additional filtering or refinement\(Zenget al\.,[2023](https://arxiv.org/html/2606.19948#bib.bib47); Fanet al\.,[2024](https://arxiv.org/html/2606.19948#bib.bib49); Konget al\.,[2024](https://arxiv.org/html/2606.19948#bib.bib48); Wanget al\.,[2025b](https://arxiv.org/html/2606.19948#bib.bib50),[2024b](https://arxiv.org/html/2606.19948#bib.bib51)\)\. However, these methods are exclusively focused on generatingsingle\-turn instructions\. Our work proposes a simple yet powerful pipeline that converts existing single\-turn VLN datasets into multi\-turn DialNav episodes\. Through this pipeline, we generate RAINbow dataset, which expands the original RAIN datasetHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)by more than two orders of magnitude\.

Embodied Dialog\.Embodied dialog presents challenges in both dataset and training\. To mitigate the challenge of data scarcity, some works explore data augmentation leveraging large language models\(Padmakumaret al\.,[2023](https://arxiv.org/html/2606.19948#bib.bib12)\)or template\-based procedures\(Gaoet al\.,[2022](https://arxiv.org/html/2606.19948#bib.bib3)\)\. For training, many of existing works primarily train and evaluate models on static dialog histories\(Thomasonet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib23); Padmakumaret al\.,[2022](https://arxiv.org/html/2606.19948#bib.bib11); Hahnet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib16)\)\. Some works propose training scheme for holistic execution\-dialog loop setups, using human correction\(de Vrieset al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib7)\), reinforcement learning\(Romanet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib8)\), or training with all possible dialog cases\(Gaoet al\.,[2022](https://arxiv.org/html/2606.19948#bib.bib3)\)\. The prior work on DialNav\(Hanet al\.,[2025](https://arxiv.org/html/2606.19948#bib.bib1)\)trains agents in a static setup, which creates a train\-test mismatch\. In this work, motivated by recent findings that large\-scale synthetic training data can substantially improve model capabilitiesWanget al\.\([2024b](https://arxiv.org/html/2606.19948#bib.bib51)\); Minet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib58)\), we generate a high\-quality, well\-grounded dataset at low cost, and adopt a training scheme to fully utilize this large\-scale data\.

## 3Preliminaries

Notation\.In DialNavHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\), theNavigatoragent traverses toward a goal with the assistance of a remoteGuide, as illustrated in Figure[1](https://arxiv.org/html/2606.19948#S0.F1)\(top\)\. Formally, the environment is represented as a connectivity graphG=\(V,E\)G=\(V,E\), wherev∈Vv\\in Vis a navigable node, andE⊆V×VE\\subseteq V\\times Vis the set of navigable edges between nodes\. A DialNav episodeℰ\\mathcal\{E\}is defined asℰ=\(G,b,R,I,TJ,DJ\)\\mathcal\{E\}=\(G,b,R,I,T\_\{J\},D\_\{J\}\), whereb∈Vb\\in Vis the initial node,R⊆VR\\subseteq Vis the goal region spanning multiple adjacent nodes, andIIis an initial instruction\. The Navigator’s complete trajectory𝒯\\mathcal\{T\}is a sequence of traversed nodes overKKsteps𝒯=\(v0,v1,…,vK\)\\mathcal\{T\}=\(v\_\{0\},v\_\{1\},\\ldots,v\_\{K\}\), wherev0=bv\_\{0\}=bandvK∈Rv\_\{K\}\\in R\. During navigation,JJ\-turn dialog occurs, denoted asDJ=\(\(q1,a1\),…,\(qJ,aJ\)\)D\_\{J\}=\(\(q\_\{1\},a\_\{1\}\),\\ldots,\(q\_\{J\},a\_\{J\}\)\), with each\(qj,aj\)\(q\_\{j\},a\_\{j\}\)exchanged at a dialog pointvdj∈𝒯v\_\{d\_\{j\}\}\\in\\mathcal\{T\}\. TheseJJdialog points segment the full trajectory𝒯\\mathcal\{T\}intoJ\+1J\+1sub\-trajectories:\(T\(0\),T\(1\),…,T\(J\)\)\(T^\{\(0\)\},T^\{\(1\)\},\\ldots,T^\{\(J\)\}\), whereT\(j\)=\(vdj,…,vdj\+1−1\)T^\{\(j\)\}=\(v\_\{d\_\{j\}\},\\ldots,v\_\{d\_\{j\+1\}\-1\}\)\.

![Refer to caption](https://arxiv.org/html/2606.19948v1/x2.png)Figure 2:Overview of the dataset generation pipeline\.\(Left\)We start from existing single\-turn fine\-grained VLN datasets, where each path is paired with its instructionFjF\_\{j\}\.\(Middle\)Multiple sub\-trajectories are concatenated into an extended trajectory\. The starting node of each sub\-trajectory becomes a dialog pointvdjv\_\{d\_\{j\}\}, and at each dialog point, a panoramic captionCjC\_\{j\}is generated using a vision–language model\. The original fine\-grained instructionsFjF\_\{j\}are repurposed as dialog answers, while the final node defines the goal regionRR\.\(Right\)Caption–instruction pairs\(Cj,Fj\)\(C\_\{j\},F\_\{j\}\)are then reformatted into natural multi\-turn dialogs using a large language model, producing large\-scale dialog\-style data for DialNav training\.Navigator and Guide Workflow\.The DialNav task unfolds as a holistic workflow integrating all subtasks \(Figure[1](https://arxiv.org/html/2606.19948#S0.F1)bottom\)\. The episode begins with the Navigator atbbwith the initial instructionII, and the remote Guide with knowledge of the full environmentGG, the goalRR, andII\. The process then unfolds in rounds\. At each steptt, the Navigator first decides whether to ask for help\. If the Navigator chooses toproceed without help, it autonomously selects its next action and advances to nodevt\+1v\_\{t\+1\}\. If the Navigator chooses toask for help, it generates a natural language questionqjq\_\{j\}, and its current node is denoted as a dialog pointvdjv\_\{d\_\{j\}\}\. Upon receiving the question, the remote Guide first localizes the Navigator by estimating its dialog point asv^dj\\hat\{v\}\_\{d\_\{j\}\}\. Then, it generates a natural language answeraja\_\{j\}, typically describing the path from the inferred positionv^dj\\hat\{v\}\_\{d\_\{j\}\}toRR\. The Navigator continues navigation using this exchange\(qj,aj\)\(q\_\{j\},a\_\{j\}\)as additional guidance\. This process repeats until the Navigator determines it has reached the goalRR\.

## 4Method

In this section, we first present our automatic episode generation pipeline \(§[4\.1](https://arxiv.org/html/2606.19948#S4.SS1)\)\. This pipeline leverages existing VLN datasets to synthesize a large\-scale dataset,RAINbow, expanding the available training data by over two orders of magnitude\. We then introduce complementary enhancements to the navigation training \(§[4\.2\.1](https://arxiv.org/html/2606.19948#S4.SS2.SSS1)\) and localization model \(§[4\.2\.2](https://arxiv.org/html/2606.19948#S4.SS2.SSS2)\), enabling the model to better exploit this large\-scale augmented dataset\.

![Refer to caption](https://arxiv.org/html/2606.19948v1/x3.png)Figure 3:Qualitative example of RAINbow\.This figure shows a 2\-turn dialog episode in RAINbow\. The left column shows the first dialog exchange, and the right column shows the second\. The generated dialogs exhibit a natural conversational flow across turns and are well\-grounded in both the Navigator’s visual observations \(e\.g\.,“fireplace”, “pool table”\) and the route the Guide describes \(e\.g\.,“leave this room”, “past the pool table”\)\.### 4\.1RAINbow Dataset Generation

Collecting dialog\-based navigation datasets is prohibitively expensive, as it requires two people to coordinate in real time while grounding their conversations in the given task context\. Consequently, the human\-annotated RAIN datasetHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)contains only 2K episodes, which is insufficient to capture the complexity of dialog\-conditioned navigation and thus hinders generalization\. To overcome this limitation, we automatically generate large\-scale DialNav episodes by utilizing VLN datasets\. We observe that, in DialNavHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\), dialog unfolds in turns along a long trajectory, where each question–answer exchange guides a subsequent portion of the path\. To simulate this characteristic, we designe our pipeline in three stages: \(1\) concatenating paths from existing VLN datasets; \(2\) generating scene captions for each dialog point \(to serve as question content\); and \(3\) rewriting the caption\-instruction pairs into natural dialog using an LLM\. Figure[2](https://arxiv.org/html/2606.19948#S3.F2)illustrates the RAINbow data generation pipeline\. We describe each stage in detail below\.

Trajectory Concatenation\.From three widely used VLN datasets, R2R\(Andersonet al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib22)\), RxR\(Kuet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib24)\)and CVDN111Although CVDN is a dialog dataset, its answers describe the 5 next nodes along the shortest path to the goal, which we utilize as fine\-grained instructions\.\(Thomasonet al\.,[2020](https://arxiv.org/html/2606.19948#bib.bib23)\), we concatenate 2–4 paths to form a new full trajectory𝒯\\mathcal\{T\}\. Each path, which becomes a sub\-trajectoryT\(j\)T^\{\(j\)\}, is paired with its original fine\-grained instructionFjF\_\{j\}\. We then form the initial exploration pathT\(0\)T^\{\(0\)\}by prepending up to five navigation steps to the start node ofT\(1\)T^\{\(1\)\}\. To better reflect the noise encountered in real dialog\-based navigation, we additionally inject*mislocalization*,*misnavigation*, and*exploration*scenarios into 10% of the constructed trajectories and their corresponding dialogs\. From the resulting trajectory𝒯=\(T\(0\),…,T\(J\)\)\\mathcal\{T\}=\(T^\{\(0\)\},\\ldots,T^\{\(J\)\}\), the initial node ofT\(0\)T^\{\(0\)\}becomes the episode’s start nodebb, and the region containing the final node ofT\(J\)T^\{\(J\)\}becomes the goal regionRR\. We then generate an initial instructionIIthat indirectly specifies the goal regionRRby describing an object located within it\. This region and object information is based on the metadata inChanget al\.\([2017](https://arxiv.org/html/2606.19948#bib.bib21)\)\. Further implementation details are provided in Supp\. Mat\.[C\.1](https://arxiv.org/html/2606.19948#A3.SS1)\.

Primitive Q&A for Dialog Points\.Given the concatenated trajectoryTT, the initial nodevdjv\_\{d\_\{j\}\}of each sub\-trajectoryT\(j\)T^\{\(j\)\}\(forj≥1j\\geq 1\) is treated as a*dialog point*where a dialog exchange occurs\. In this step, we prepare the primitive content for the dialog exchange \(qjq\_\{j\},aja\_\{j\}\) at each dialog pointvdjv\_\{d\_\{j\}\}\. Leveraging the question style of RAINHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\), where questions typically describe the visual scene to aid localization, we first generate the question content,CjC\_\{j\}, by captioning the observation at the dialog point via a large vision–language model \(e\.g\., LLaVA\-1\.5\-7B\(Liuet al\.,[2023](https://arxiv.org/html/2606.19948#bib.bib34)\)\)\. For the answer content, we repurpose the original fine\-grained instructionFjF\_\{j\}associated with the sub\-trajectoryT\(j\)T^\{\(j\)\}\. The resulting\(Cj,Fj\)\(C\_\{j\},F\_\{j\}\)thus form primitive question–answer pairs aligned with the dialog point of the concatenated trajectory\. As these raw pairs are not yet in natural dialog form and the flow across turns is often awkward, we refine them in the following step\. Details are provided in the Supp\. Mat\.[C\.2](https://arxiv.org/html/2606.19948#A3.SS2)\.

Reformatting into Multi\-turn Dialog\.Finally, we reformat the caption–instruction pairs\(Cj,Fj\)\(C\_\{j\},F\_\{j\}\)obtained from the previous steps into coherent multi\-turn dialogs\. While the captionCjC\_\{j\}describes the local scene and the instructionFjF\_\{j\}gives a specific direction for the next sub\-trajectoryT\(j\)T^\{\(j\)\}, these sentences are originally written as independent sentences and therefore do not form a natural conversation\. To bridge this gap, we employ an LLM \(e\.g\.,GPT\-4o\-mini\(OpenAI,[2024](https://arxiv.org/html/2606.19948#bib.bib35)\)\) to rewrite the raw\(Cj,Fj\)\(C\_\{j\},F\_\{j\}\)pairs into fluent question–answer exchanges\(qj,aj\)\(q\_\{j\},a\_\{j\}\)\. To reduce hallucination and ensure faithfulness to the intended goal, we first refine each instruction before converting it into dialog, and further validate this process through human evaluation and closed\-source VLMs\. Details of the prompting and validation are provided in the Supp\. Mat\.[C\.3](https://arxiv.org/html/2606.19948#A3.SS3),[C\.4](https://arxiv.org/html/2606.19948#A3.SS4),[C\.5](https://arxiv.org/html/2606.19948#A3.SS5)\.

The RAINbow Dataset\.Through this process, we synthesize theRAINbow\(RAINbuiltonwide set\) dataset\. The resulting RAINbow is a 238K episode dataset that is over two orders of magnitude larger than the RAIN dataset\. Figure[3](https://arxiv.org/html/2606.19948#S4.F3)shows an example of the final generated dialog data\. The key characteristics in RAINbow remain comparable to the RAIN dataset\. In Rainbow, the average numbers of dialog turns, trajectory nodes, question words, and answer words are 2\.71, 19\.54, 30\.43, and 43\.17, respectively \(compared to 1\.88, 26\.03, 27\.48, and 42\.53 in RAIN\)\. The reformatting step using GPT\-4o\-mini incurred a total cost of about USD 400, equating to USD 0\.0016 per episode\. This makes our generation pipeline about 2,000 times more cost\-effective than manual annotation in RAIN \(USD 3\.75 per episode\)\. Statistics and more examples of RAINbow are provided in Supp\. Mat\.[A](https://arxiv.org/html/2606.19948#A1)\.

### 4\.2Training the Navigator and Guide

To train the Navigator and the Guide, we follow the modular design introduced inHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)\. As described in the workflow in Figure[1](https://arxiv.org/html/2606.19948#S0.F1)\(bottom\), each agent is decomposed into task\-specific modules—whether\-to\-ask,question generation, andnavigationfor the Navigator, andlocalizationandanswer generationfor the Guide\. For the whether\-to\-ask module, we adopt a confidence thresholding strategy, prompting the agent to initiate a question whenever the model confidence falls below a predefined threshold\. For the question and answer generation modules, we follow the training scheme ofHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)\.

#### 4\.2\.1Dual\-Strategy Training for Navigation

The segment\-based training in prior workHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)trains the navigator without previous navigation or future dialog updates, not taking the evolving dialog nature of DialNav into account\. While training on the full episode with dialog updates along the path seems to solve this, it presents a fundamental dilemma: the agent must strictly follow the annotated path to receive valid dialog updates\(qj,aj\)\(q\_\{j\},a\_\{j\}\)at the correct nodesvdjv\_\{d\_\{j\}\}\. However, this strict adherence prevents the agent from learning to recover from its own errors\.

To this end, we adopt aDual\-Strategy Trainingfor navigation that combines two types of rollouts: data\-guided and on\-policy \(Figure[4\(b\)](https://arxiv.org/html/2606.19948#S4.F4.sf2)\)\. Thedata\-guided rollout\(Figure[4\(b\)](https://arxiv.org/html/2606.19948#S4.F4.sf2), white nodes\) constrains the Navigator to follow the dataset trajectory, allowing it to arrive at each dialog nodevdjv\_\{d\_\{j\}\}and receive the ground\-truth dialog exchange\(qj,aj\)\(q\_\{j\},a\_\{j\}\)as a dynamic instruction update\. This is complemented by anon\-policy rollout\(Figure[4\(b\)](https://arxiv.org/html/2606.19948#S4.F4.sf2), blue nodes\), forked at each dialog pointvdjv\_\{d\_\{j\}\}\(Figure[4\(b\)](https://arxiv.org/html/2606.19948#S4.F4.sf2), red nodes\), where the agent follows its own policy without receiving further dialog updates, supervised to predict the next node on the shortest path to the goal regionRR\. \(Figure[4\(b\)](https://arxiv.org/html/2606.19948#S4.F4.sf2), red arrows\)\. By jointly training on both data\-guided and on\-policy rollouts, the navigation module learns to leverage full episode while remaining robust to its own prediction errors\.

![Refer to caption](https://arxiv.org/html/2606.19948v1/x4.png)\(a\)DialNav training episodeℰ\\mathcal\{E\}
![Refer to caption](https://arxiv.org/html/2606.19948v1/x5.png)\(b\)Dual\-Strategy Training

Figure 4:Comparison of navigation training strategies\.\(a\)DialNav training episodeℰ\\mathcal\{E\}: An episode unfolds from an initial node \(markedII\), with dialog dynamically updated at each dialog points \(red nodes\)\. \(b\)Dual\-Strategy Training: Thedata\-guided rollout, follows the annotated path in dataset \(white nodes\), updating dialog at each dialog point \(red nodes\)\. This is supplemented withon\-policy rollouts\(blue nodes\) forked at each dialog point\. The model is supervised to predict the next node along the dataset path for data\-guided rollouts and the shortest path to the goal for on\-policy rollouts, respectively \(red arrows\)\.The loss for each dialog point,vdjv\_\{d\_\{j\}\}is defined as followings:

ℒ=∑i=djdj\+1−1log⁡p​\(vi\+1\|v1:i,Dj\)⏟data\-guided rollout\+∑t=djK−1log⁡p​\(v^t\+1\|v1:dj,v¯dj\+1:t,Dj\)⏟on\-policy rollout,\{\\begin\{aligned\} \\mathcal\{L\}=&\\underbrace\{\\sum\_\{i=d\_\{j\}\}^\{d\_\{j\+1\}\-1\}\\log p\(v\_\{i\+1\}\|v\_\{1:i\},D\_\{j\}\)\}\_\{\\text\{data\-guided rollout\}\}\\\\ &\+\\underbrace\{\\sum\_\{t=d\_\{j\}\}^\{K\-1\}\\log p\(\\hat\{v\}\_\{t\+1\}\|v\_\{1:d\_\{j\}\},\\bar\{v\}\_\{d\_\{j\}\+1:t\},D\_\{j\}\)\}\_\{\\text\{on\-policy rollout\}\},\\end\{aligned\}\}\(1\)in whichv^\\hat\{v\}is the next step on the shortest path andv¯\\bar\{v\}denotes the next step sampled the own policy of the navigation model\.

#### 4\.2\.2Transferring VLN Knowledge for Localization

Localization is a subtask arising from DialNav’s remote Guide setup\. The Guide, who knows the entire mapGG, must predict the dialog pointvdjv\_\{d\_\{j\}\}based on the Navigator’s questionqjq\_\{j\}to provide accurate guidance\. Localization tasks share similarities with VLN in its environment and task, as both operate in the Matterport3DChanget al\.\([2017](https://arxiv.org/html/2606.19948#bib.bib21)\)and are text\-grounded node selection tasks\. This suggests that localization can benefit from models and pretraining strategies originally developed for VLNHahn and Rehg \([2022](https://arxiv.org/html/2606.19948#bib.bib18)\)\. Motivated by this, we adopt the graph\-based Transformer architecture ofChenet al\.\([2022b](https://arxiv.org/html/2606.19948#bib.bib27)\)for localization\. As shown in our experiments, this significantly improves localization performance\. Details of the model are provided in Supp\. Mat\.[E](https://arxiv.org/html/2606.19948#A5)\.

Table 1:Performance gains from progressively adding components\.\+RAINbow: training with RAINbow dataset \(§[4\.1](https://arxiv.org/html/2606.19948#S4.SS1)\)\.\+DST: training navigation module with Dual\-Strategy Training \(§[4\.2\.1](https://arxiv.org/html/2606.19948#S4.SS2.SSS1)\)\.\+GTL: adopting Graph\-based Transformer Localization model \(§[4\.2\.2](https://arxiv.org/html/2606.19948#S4.SS2.SSS2)\)\. RAINbow alone \(Row 2\) offers performance gains, but its potential is unlocked when combined with our proposed DST \(Row 4\), which significantly amplifies its effect\. GTL \(Row 5\) provides further improvement, leading to the best overall performance\.Table 2:Dataset augmentation ablations\.RAIN: training only with the RAIN dataset\.\+ VLN instructions: adding VLN instructions\.\+ Traj\. Concat: concatenating trajectories from VLN instructions\.\+ Primitive Q&A: using primitive QA content\.\+ Reformat: reformatting into natural multi\-turn QA dialogs\.

## 5Experiments

### 5\.1Experimental Settings

Implementation of Navigator and Guide\.Following prior workHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\), we use DUET\(Chenet al\.,[2022b](https://arxiv.org/html/2606.19948#bib.bib27)\)fornavigationand LANA\(Wanget al\.,[2023a](https://arxiv.org/html/2606.19948#bib.bib36)\)for thequestion/answer generationmodules\. For thelocalizationmodule, we compare the GCN fromHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)with Graph\-based Transformer Localization \(§[4\.2\.2](https://arxiv.org/html/2606.19948#S4.SS2.SSS2)\)\. All models are initialized from publicly available pre\-trained weights\. We then train the modules using a 1:9 mixture of RAIN and RAINbow data\. For thewhether\-to\-askmodule, we fix the confidence threshold of0\.90\.9\. Details of each model architecture are provided in Supp\. Mat\.[E](https://arxiv.org/html/2606.19948#A5)\.

Evaluation\.We evaluate our method on the RAIN validation dataset, which is split intoVal SeenandVal Unseen\(Andersonet al\.,[2018](https://arxiv.org/html/2606.19948#bib.bib22)\)\. TheVal Seensplit consists of environments seen during training, albeit with novel paths, whereasVal Unseencontains entirely novel environments\. We report performance under the holistic dialog–navigation loop setting, where the Navigator starts with an initial instruction and must proactively ask questions to receive guidance and reach the goal\. We measure performance using standard VLN metrics: Success Rate \(SR\), Oracle Success Rate \(OSR\), Success weighted by Path Length \(SPL\), and Navigation Error \(NE, in meters\), which capture navigation quality and efficiency\. We also report Navigation Step Count \(NSC\) and Dialog Turn Count \(DTC\) to measure exploration behavior and dialog usage\.

### 5\.2Results

Impact of the proposed components\.Table[1](https://arxiv.org/html/2606.19948#S4.T1)presents the impact of our proposed components: adding the RAINbow dataset, Dual\-Strategy Training \(DST\), and Graph\-based Transformer Localization \(GTL\)\. RAINbow alone \(Row 2\) provides a modest improvement over the baseline \(Row 1\) in SR under both setups \(Val Seen \+3\.30, Val Unseen \+3\.74\)\. However, its true potential is realized when paired with DST \(Row 4\), which yields substantially larger gains \(Val Seen \+12\.09, Val Unseen \+9\.96, compared to Row 3\)\. This result confirms our hypothesis: while the baseline segment\-based trainingHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)cannot fully leverage RAINbow, DST effectively utilizes the large\-scale, multi\-turn episodes and unlocks their full potential\. Finally, adding GTL \(Row 5\) produces the best performance: SR reaches 58\.24 \(\+89%\) on Val Seen and 29\.05 \(\+100%\) on Val Unseen, roughly doubling the baseline success rate on both splits\.

Ablations of the Data Augmentation Pipeline\.Table[2](https://arxiv.org/html/2606.19948#S4.T2)analyzes the effect of each stage in our augmentation pipeline\. For fair comparison, we applied Dual\-Strategy Training and Graph\-based Transformer Localization across all setups\. Naively adding publicly available fine\-grained VLN instructionsAndersonet al\.\([2018](https://arxiv.org/html/2606.19948#bib.bib22)\); Kuet al\.\([2020](https://arxiv.org/html/2606.19948#bib.bib24)\)\(Row 2\), providing a baseline for data augmentation, improved navigation SR \+3\.30 \(\+9%\) in Val Seen and \+2\.69 \(\+11%\) in Val Unseen\. This gain from simply exposing the Navigator to more environments highlights the importance of data diversity for improving performance\. With trajectory concatenation \(Row 3\), the performance drops compared to Row 2\. We attribute this to data reduction; the constraints for connecting paths \(§[4\.1](https://arxiv.org/html/2606.19948#S4.SS1)\) necessarily discard some source trajectories, providing the navigator with less environmental coverage than the full VLN instruction set\. For Rows 2 and 3, only the navigation module can use the augmented data, because these settings lack the dialog content required to train the localization, question generation, and answer generation modules\. Introducing Primitive Q&A \(Row 4\) addresses this issue by providing supervision across all modules, resulting in a substantial improvement in SR on Val Seen: \+18\.68 \(\+41\.66%\) compared to Row 1\. However, this setting still struggles in Val Unseen\. We conjecture this is due to a style mismatch, as modules are trained on a mixture of human\-annotated RAIN dialogs and the unformatted, raw content from the pipeline, which hinders generalization\. Finally, Reformat \(Row 5\) converts this raw content into natural, contextually coherent dialogs, yielding the best results \(Seen SR: 58\.24, Unseen SR: 29\.05\)\. Crucially, reformatting significantly boosted SR in Val Unseen \(compared to Row 4, \+6\.23\)\. Overall, these results demonstrate that while raw VLN fine\-grained data or naively concatenated trajectories provide limited benefits, our pipeline’s full process of structuring and naturalizing content into consistent dialog is effective for DialNav\.

Table 3:Navigation training methods ablation\.Comparison of baseline and Dual\-Strategy Training \(DST\) with and without On\-policy Rollout\.Table 4:
textbfDetour analysis on Val Unseen\. Comparison of Success Rate between DST and the variant without on\-policy rollout across different detour ratio\.Ablations of Navigation Training\.Table[3](https://arxiv.org/html/2606.19948#S5.T3)ablates Dual\-Strategy Training \(§[4\.2\.1](https://arxiv.org/html/2606.19948#S4.SS2.SSS1)\) on both the Val Seen and Val Unseen splits\. For fair comparison, we applied the RAINbow data and Graph\-based Transformer Localization across all setups\. Comparing Dual\-Strategy Training \(DST\) \(Row 2 and 5\) to the Baseline \(Row 1 and 4\), we see a dramatic improvement in SR across both splits \(Val Seen \+76\.6%, Val Unseen \+27\.3%\)\. This confirms that Dual\-Strategy Training \(DST\) that leverages the full, dynamic dialog loop is critical\. Row 3 and 6 ablates our training by removing the on\-policy rollout\. While this maintains high SR in Val Seen, there is a severe drop in Val Unseen \(\-7\.89\)\. Table[4](https://arxiv.org/html/2606.19948#S5.T4)further analyzes SR by detour ratio \(path length / shortest path\) on Val Unseen\. When the detour is small \(1\.0–1\.3\), DST and the variant without on\-policy rollout achieve comparable SR \(11\.20 vs\. 10\.79\)\. However, when the detour is larger \(≥\\geq1\.3\), DST yields substantially higher SR \(17\.84\) than the variant without on\-policy rollout \(10\.37\)\. By explicitly incorporating exploration through on\-policy rollout during training, our model learns to recover from trajectory errors and make more robust predictions\.

Table 5:Localization model ablation\.AC: Architecture change to Graph\-based Transformer Localization\.VT: VLN pretrained knowledge transfer\. All together, the new localization model reduces localization error \(LE\) by 2\.56m in Val Seen and 3\.02m in Val Unseen compared to the baseline\.Table 6:Human\-agent cooperation results on DialNav\.SR: Navigation Success Rate \(%\)\.HS: Human Score, representing the perceived helpfulness of the agent counterpart, rated by human participants on a scale from 1 to 5\. For each setting, either the human or the agent takes the Guide role, while the other takes the Navigator role\.![Refer to caption](https://arxiv.org/html/2606.19948v1/x6.png)\(a\)BaselineHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)
![Refer to caption](https://arxiv.org/html/2606.19948v1/x7.png)\(b\)Ours

Figure 5:Qualitative comparison on the same task instance between the baseline\(Hanet al\.,[2025](https://arxiv.org/html/2606.19948#bib.bib1)\)\(left\) and Ours \(right\)\.The baseline agent produces broken language with wrong details \(marked inred\), likely due to dataset scarcity, leading to high localization errors and navigation failure\. In contrast, our agent provides richer, well\-grounded descriptions \(marked inbold\), yielding accurate localization and reliable instructions, ultimately leading to successful navigation\.Ablations of Localization Model\.Table[5](https://arxiv.org/html/2606.19948#S5.T5)ablates the Graph\-based Transformer Localization model\. To isolate the localization performance, we evaluate it outside the holistic dialog\-navigation loop\. Instead, we use the human\-annotated questions from the validation set \(the human\-human dialog for validation episodes in RAIN\); the model is tasked to predict the node from which that question was asked\. We report Localization Error \(LE\) in meters and 3m accuracy \(A@3m\), the percentage of predictions within 3m of the ground truth\. Compared to the GCN baseline from prior workHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)\(Row 1\), changing the architecture to Graph\-based Transformer Localization alone \(Row 2\) degrades performance, highlighting the difficulty of training this complex model from scratch on scarce data\. However, initializing it with VLN pretrained weights \(Row 3\) yields a dramatic improvement, substantially surpassing the baseline \(Row 1\) in both environments\. This confirms the efficacy of transferring VLN knowledge for localization\. These results demonstrate that GTL, enhanced by knowledge transfer and the RAINbow dataset, effectively improves localization\.

Qualitative Examples\.Figure[5](https://arxiv.org/html/2606.19948#S5.F5)and Supp\. Mat\. Sec\.[H](https://arxiv.org/html/2606.19948#A8)provide qualitative comparisons between the baseline Navigator–Guide agent pair\(Hanet al\.,[2025](https://arxiv.org/html/2606.19948#bib.bib1)\)and ours\.

Human\-Agent Cooperation\.To further validate the effectiveness of our approach, we evaluate it in a human–agent cooperation setup \(Table[6](https://arxiv.org/html/2606.19948#S5.T6)\)\. We build a simulator in which a human participant collaborates with an agent: one takes the role of the Guide, while the other acts as the Navigator\. Our method consistently outperforms the baseline across all roles and environments, demonstrating that it substantially improves human–agent collaboration\. Details are provided in Supp\. Mat\. Sec[J](https://arxiv.org/html/2606.19948#A10)\.

## 6Conclusion

In this work, we addressed the key limitations of DialNav, a holistic dialog\-based navigation task whose performance has been constrained by critical data scarcity\. We first proposed an automatic data generation pipeline to create RAINbow, a large\-scale \(238K\) dataset that expands the available training data by utilizing existing VLN instructions, addressing the core data scarcity issue\. Second, we adopted the Dual\-Strategy Training scheme for the navigator, aligning the training process with the dynamic, multi\-turn nature of the task by combining data\-guided rollout and on\-policy rollout\. Finally, we adapted a graph\-based transformer for the localization, enabling us to leverage powerful VLN pretraining for localization task\. Our experiments demonstrated that these components are highly synergistic\. Integrating these solutions, our final model establishes a new state\-of\-the\-art, more than doubling the baseline success rate on bothVal Seen\(30\.77 to 58\.24, \+89%\) andVal Unseen\(14\.52 to 29\.05, \+100%\) splits\. By providing a large\-scale, high\-quality dataset and adopting training and model to leverage it, this work lays a stronger foundation for future research into more complex, interactive embodied agents\.

## Limitations

While our work substantially improves DialNav through large\-scale automatic data generation and training enhancements, several limitations remain\. Although RAINbow significantly increases data scale, it is built upon existing VLN trajectories, which may limit environmental and behavioral diversity\. In addition, the generated dialogs, while generally natural and goal\-consistent, may not fully capture the full richness of human embodied communication\. Our evaluation is primarily conducted in indoor navigation settings, and further validation in broader embodied domains would be beneficial\.

## References

- P\. Anderson, Q\. Wu, D\. Teney, J\. Bruce, M\. Johnson, N\. Sünderhauf, I\. Reid, S\. Gould, and A\. Van Den Hengel \(2018\)Vision\-and\-language navigation: interpreting visually\-grounded navigation instructions in real environments\.InProceedings of the IEEE conference on computer vision and pattern recognition,pp\. 3674–3683\.Cited by:[item 1](https://arxiv.org/html/2606.19948#A3.I1.i1.p1.1),[§1](https://arxiv.org/html/2606.19948#S1.p2.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p2.12),[§5\.1](https://arxiv.org/html/2606.19948#S5.SS1.p2.1),[§5\.2](https://arxiv.org/html/2606.19948#S5.SS2.p2.1)\.
- The robotslang benchmark: dialog\-guided robot localization and navigation\.InConference on Robot Learning,pp\. 1384–1393\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.8.6.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1)\.
- A\. Chang, A\. Dai, T\. Funkhouser, M\. Halber, M\. Niessner, M\. Savva, S\. Song, A\. Zeng, and Y\. Zhang \(2017\)Matterport3d: learning from rgb\-d data in indoor environments\.arXiv preprint arXiv:1709\.06158\.Cited by:[item 5](https://arxiv.org/html/2606.19948#A3.I1.i5.p1.1),[§1](https://arxiv.org/html/2606.19948#S1.p3.1),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p2.12),[§4\.2\.2](https://arxiv.org/html/2606.19948#S4.SS2.SSS2.p1.3)\.
- H\. Chen, A\. Suhr, D\. Misra, N\. Snavely, and Y\. Artzi \(2019\)Touchdown: natural language navigation and spatial reasoning in visual street environments\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12538–12547\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p1.1)\.
- S\. Chen, P\. Guhur, M\. Tapaswi, C\. Schmid, and I\. Laptev \(2022a\)Learning from unlabeled 3d environments for vision\-and\-language navigation\.InECCV,Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- S\. Chen, P\. Guhur, M\. Tapaswi, C\. Schmid, and I\. Laptev \(2022b\)Think global, act local: dual\-scale graph transformer for vision\-and\-language navigation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 16537–16547\.Cited by:[§E\.2](https://arxiv.org/html/2606.19948#A5.SS2.p1.1),[§E\.4](https://arxiv.org/html/2606.19948#A5.SS4.p1.1),[§4\.2\.2](https://arxiv.org/html/2606.19948#S4.SS2.SSS2.p1.3),[§5\.1](https://arxiv.org/html/2606.19948#S5.SS1.p1.1)\.
- H\. de Vries, K\. Shuster, D\. Batra, D\. Parikh, J\. Weston, and D\. Kiela \(2018\)Talk the walk: navigating new york city through grounded dialogue\.External Links:1807\.03367,[Link](https://arxiv.org/abs/1807.03367)Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.9.7.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- S\. Fan, R\. Liu, W\. Wang, and Y\. Yang \(2024\)Navigation instruction generation with bev perception and large language models\.InEuropean Conference on Computer Vision,pp\. 368–387\.Cited by:[§1](https://arxiv.org/html/2606.19948#S1.p4.1),[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- Y\. Fan, W\. Chen, T\. Jiang, C\. Zhou, Y\. Zhang, and X\. E\. Wang \(2023\)Aerial vision\-and\-dialog navigation\.InFindings of the Association for Computational Linguistics: ACL 2023,Toronto, Canada,pp\. 3043–3061\.External Links:[Link](https://aclanthology.org/2023.findings-acl.190),[Document](https://dx.doi.org/10.18653/v1/2023.findings-acl.190)Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.10.8.1)\.
- D\. Fried, R\. Hu, V\. Cirik, A\. Rohrbach, J\. Andreas, L\. Morency, T\. Berg\-Kirkpatrick, K\. Saenko, D\. Klein, and T\. Darrell \(2018\)Speaker\-follower models for vision\-and\-language navigation\.Advances in neural information processing systems31\.Cited by:[§1](https://arxiv.org/html/2606.19948#S1.p4.1),[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- X\. Gao, Q\. Gao, R\. Gong, K\. Lin, G\. Thattai, and G\. S\. Sukhatme \(2022\)Dialfred: dialogue\-enabled agents for embodied instruction following\.IEEE Robotics and Automation Letters7\(4\),pp\. 10049–10056\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.7.5.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- P\. Guhur, M\. Tapaswi, S\. Chen, I\. Laptev, and C\. Schmid \(2021\)Airbert: in\-domain pretraining for vision\-and\-language navigation\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 1634–1643\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- M\. Hahn, J\. Krantz, D\. Batra, D\. Parikh, J\. M\. Rehg, S\. Lee, and P\. Anderson \(2020\)Where are you? localization from embodied dialog\.arXiv preprint arXiv:2011\.08277\.Cited by:[Appendix F](https://arxiv.org/html/2606.19948#A6.p1.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- M\. Hahn and J\. M\. Rehg \(2022\)Transformer\-based localization from embodied dialog with large\-scale pre\-training\.arXiv preprint arXiv:2210\.04864\.Cited by:[§E\.4](https://arxiv.org/html/2606.19948#A5.SS4.p1.1),[Table 11](https://arxiv.org/html/2606.19948#A6.T11.4.4.8.4.1),[Table 11](https://arxiv.org/html/2606.19948#A6.T11.4.4.9.5.1),[§4\.2\.2](https://arxiv.org/html/2606.19948#S4.SS2.SSS2.p1.3)\.
- L\. Han, H\. Min, G\. Hwangbo, J\. Choi, and P\. H\. Seo \(2025\)DialNav: multi\-turn dialog navigation with a remote guide\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 8514–8523\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.12.10.1),[§E\.2](https://arxiv.org/html/2606.19948#A5.SS2.p1.1),[Table 12](https://arxiv.org/html/2606.19948#A7.T12.15.15.15.6),[Table 12](https://arxiv.org/html/2606.19948#A7.T12.5.5.5.6),[§1](https://arxiv.org/html/2606.19948#S1.p2.1),[§1](https://arxiv.org/html/2606.19948#S1.p3.1),[§1](https://arxiv.org/html/2606.19948#S1.p4.1),[§1](https://arxiv.org/html/2606.19948#S1.p5.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1),[§2](https://arxiv.org/html/2606.19948#S2.p2.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1),[§3](https://arxiv.org/html/2606.19948#S3.p1.22),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p3.11),[§4\.2\.1](https://arxiv.org/html/2606.19948#S4.SS2.SSS1.p1.2),[§4\.2](https://arxiv.org/html/2606.19948#S4.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.19948#S4.T1.8.8.10.1.2),[Figure 5](https://arxiv.org/html/2606.19948#S5.F5.1.1),[Figure 5](https://arxiv.org/html/2606.19948#S5.F5.4.1),[5\(a\)](https://arxiv.org/html/2606.19948#S5.F5.sf1),[§5\.1](https://arxiv.org/html/2606.19948#S5.SS1.p1.1),[§5\.2](https://arxiv.org/html/2606.19948#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2606.19948#S5.SS2.p4.1),[§5\.2](https://arxiv.org/html/2606.19948#S5.SS2.p5.1),[Table 3](https://arxiv.org/html/2606.19948#S5.T3.2.2.4.2.2),[Table 3](https://arxiv.org/html/2606.19948#S5.T3.2.2.8.6.2)\.
- W\. Hao, C\. Li, X\. Li, L\. Carin, and J\. Gao \(2020\)Towards learning a generic agent for vision\-and\-language navigation via pre\-training\.InProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pp\. 13137–13146\.Cited by:[Appendix D](https://arxiv.org/html/2606.19948#A4.p1.1)\.
- J\. Kiseleva, Z\. Li, M\. Aliannejadi, S\. Mohanty, M\. ter Hoeve, M\. Burtsev, A\. Skrynnik, A\. Zholus, A\. Panov, K\. Srinet,et al\.\(2022\)Interactive grounded language understanding in a collaborative environment: iglu 2021\.InNeurIPS 2021 Competitions and Demonstrations Track,pp\. 146–161\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.4.2.1)\.
- J\. Y\. Koh, H\. Agrawal, D\. Batra, R\. Tucker, A\. Waters, H\. Lee, Y\. Yang, J\. Baldridge, and P\. Anderson \(2022\)Simple and effective synthesis of indoor 3d scenes\.InAAAI Conference on Artificial Intelligence,External Links:[Document](https://dx.doi.org/10.48550/arXiv.2204.02960)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- X\. Kong, J\. Chen, W\. Wang, H\. Su, X\. Hu, Y\. Yang, and S\. Liu \(2024\)Controllable navigation instruction generation with chain of thought prompting\.InEuropean Conference on Computer Vision,External Links:[Document](https://dx.doi.org/10.1007/978-3-031-73397-0%5F3)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- A\. Ku, P\. Anderson, R\. Patel, E\. Ie, and J\. Baldridge \(2020\)Room\-across\-room: multilingual vision\-and\-language navigation with dense spatiotemporal grounding\.arXiv preprint arXiv:2010\.07954\.Cited by:[item 1](https://arxiv.org/html/2606.19948#A3.I1.i1.p1.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p2.12),[§5\.2](https://arxiv.org/html/2606.19948#S5.SS2.p2.1)\.
- J\. Li, H\. Tan, and M\. Bansal \(2022\)Envedit: environment editing for vision\-and\-language navigation\.InComputer Vision and Pattern Recognition,External Links:[Document](https://dx.doi.org/10.1109/CVPR52688.2022.01497)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- K\. Lin, P\. Chen, D\. Huang, T\. H\. Li, M\. Tan, and C\. Gan \(2023\)Learning vision\-and\-language navigation from youtube videos\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 8317–8326\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- C\. Liu, F\. Zhu, X\. Chang, X\. Liang, and Y\. Shen \(2021\)Vision\-language navigation with random environmental mixup\.InIEEE International Conference on Computer Vision,External Links:[Document](https://dx.doi.org/10.1109/ICCV48922.2021.00167)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- H\. Liu, C\. Li, Q\. Wu, and Y\. J\. Lee \(2023\)Visual instruction tuning\.NeurIPS\.Cited by:[§C\.2](https://arxiv.org/html/2606.19948#A3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p3.11)\.
- H\. Min, S\. Jung, J\. Sung, D\. Lee, L\. Han, and P\. H\. Seo \(2025\)GOAT: a training framework for goal\-oriented agent with tools\.arXiv preprint arXiv:2510\.12218\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- K\. Nguyen and H\. Daumé III \(2019\)Help, anna\! visual navigation with natural multimodal assistance via retrospective curiosity\-encouraging imitation learning\.arXiv preprint arXiv:1909\.01871\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p1.1)\.
- OpenAI \(2024\)GPT\-4o mini: advancing cost\-efficient intelligence\.Note:OpenAI blog postAvailable at[https://openai\.com/index/gpt\-4o\-mini\-advancing\-cost\-efficient\-intelligence/](https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/)\(accessed DATE\)Cited by:[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p4.6)\.
- A\. Padmakumar, M\. Inan, S\. Gella, P\. Lange, and D\. Hakkani\-Tur \(2023\)Multimodal embodied plan prediction augmented with synthetic embodied dialogue\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 6114–6131\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.374/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.374)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- A\. Padmakumar, J\. Thomason, A\. Shrivastava, P\. Lange, A\. Narayan\-Chen, S\. Gella, R\. Piramuthu, G\. Tur, and D\. Hakkani\-Tur \(2022\)Teach: task\-driven embodied agents that chat\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 2017–2025\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.6.4.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- Y\. Qi, Q\. Wu, P\. Anderson, X\. Wang, W\. Y\. Wang, C\. Shen, and A\. v\. d\. Hengel \(2020\)Reverie: remote embodied visual referring expression in real indoor environments\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 9982–9991\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p1.1)\.
- H\. R\. Roman, Y\. Bisk, J\. Thomason, A\. Celikyilmaz, and J\. Gao \(2020\)Rmm: a recursive mental model for dialog navigation\.arXiv preprint arXiv:2005\.00728\.Cited by:[Table 10](https://arxiv.org/html/2606.19948#A4.T10.1.1.1.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- A\. Suhr, C\. Yan, J\. Schluger, S\. Yu, H\. Khader, M\. Mouallem, I\. Zhang, and Y\. Artzi \(2019\)Executing instructions in situated collaborative interactions\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing \(EMNLP\-IJCNLP\),pp\. 2119–2130\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.3.1.1)\.
- H\. Tan, L\. Yu, and M\. Bansal \(2019\)Learning to navigate unseen environments: back translation with environmental dropout\.InNorth American Chapter of the Association for Computational Linguistics,External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1268)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- J\. Thomason, M\. Murray, M\. Cakmak, and L\. Zettlemoyer \(2020\)Vision\-and\-dialog navigation\.InConference on Robot Learning,pp\. 394–406\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.11.9.1),[item 1](https://arxiv.org/html/2606.19948#A3.I1.i1.p1.1),[Table 10](https://arxiv.org/html/2606.19948#A4.T10),[§2](https://arxiv.org/html/2606.19948#S2.p1.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1),[§4\.1](https://arxiv.org/html/2606.19948#S4.SS1.p2.12)\.
- H\. Wang, L\. Wang, S\. Zhou, J\. Tian, Z\. Qin, Y\. Wang, G\. Hua, and W\. Tang \(2025a\)Towards precise embodied dialogue localization via causality guided diffusion\.InProceedings of the Computer Vision and Pattern Recognition Conference,pp\. 13350–13360\.Cited by:[Table 11](https://arxiv.org/html/2606.19948#A6.T11.4.4.11.7.1)\.
- L\. Wang, Z\. He, R\. Dang, M\. Shen, C\. Liu, and Q\. Chen \(2024a\)Vision\-and\-language navigation via causal learning\.InComputer Vision and Pattern Recognition,External Links:[Document](https://dx.doi.org/10.1109/CVPR52733.2024.01248)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- X\. Wang, W\. Wang, J\. Shao, and Y\. Yang \(2023a\)Lana: a language\-capable navigator for instruction following and generation\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 19048–19058\.Cited by:[§E\.3](https://arxiv.org/html/2606.19948#A5.SS3.p1.1),[§5\.1](https://arxiv.org/html/2606.19948#S5.SS1.p1.1)\.
- X\. Wang, T\. Kwon, M\. Rad, B\. Pan, I\. Chakraborty, S\. Andrist, D\. Bohus, A\. Feniello, B\. Tekin, F\. V\. Frujeri,et al\.\(2023b\)Holoassist: an egocentric human interaction dataset for interactive ai assistants in the real world\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 20270–20281\.Cited by:[Table 8](https://arxiv.org/html/2606.19948#A2.T8.1.1.5.3.1)\.
- Z\. Wang, Y\. Zhu, G\. H\. Lee, and Y\. Fan \(2025b\)Navrag: generating user demand instructions for embodied navigation through retrieval\-augmented llm\.arXiv preprint arXiv:2502\.11142\.Cited by:[§1](https://arxiv.org/html/2606.19948#S1.p4.1),[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- Z\. Wang, J\. Li, Y\. Hong, S\. Li, K\. Li, S\. Yu, Y\. Wang, Y\. Qiao, Y\. Wang, M\. Bansal,et al\.\(2024b\)Bootstrapping language\-guided navigation learning with self\-refining data flywheel\.arXiv preprint arXiv:2412\.08467\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1),[§2](https://arxiv.org/html/2606.19948#S2.p3.1)\.
- Z\. Wang, J\. Li, Y\. Hong, Y\. Wang, Q\. Wu, M\. Bansal, S\. Gould, H\. Tan, and Y\. Qiao \(2023c\)Scaling data generation in vision\-and\-language navigation\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 12009–12020\.Cited by:[§E\.2](https://arxiv.org/html/2606.19948#A5.SS2.p3.1),[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- H\. Zeng, X\. Wang, W\. Wang, and Y\. Yang \(2023\)Kefa: a knowledge enhanced and fine\-grained aligned speaker for navigation instruction generation\.InarXiv\.org,External Links:[Document](https://dx.doi.org/10.48550/arXiv.2307.13368)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- C\. Zhang, M\. Li, I\. Budvytis, and S\. Liwicki \(2024\)DiaLoc: an iterative approach to embodied dialog localization\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12585–12593\.Cited by:[Table 11](https://arxiv.org/html/2606.19948#A6.T11.4.4.10.6.1)\.
- Y\. Zhang and P\. Kordjamshidi \(2023\)VLN\-trans: translator for the vision and language navigation agent\.InAnnual Meeting of the Association for Computational Linguistics,External Links:[Document](https://dx.doi.org/10.48550/arXiv.2302.09230)Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p2.1)\.
- F\. Zhu, X\. Liang, Y\. Zhu, Q\. Yu, X\. Chang, and X\. Liang \(2021a\)Soon: scenario oriented object navigation with graph\-based exploration\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 12689–12699\.Cited by:[§2](https://arxiv.org/html/2606.19948#S2.p1.1)\.
- Y\. Zhu, Y\. Weng, F\. Zhu, X\. Liang, Q\. Ye, Y\. Lu, and J\. Jiao \(2021b\)Self\-motivated communication agent for real\-world vision\-dialog navigation\. in 2021 ieee\.InCVF International Conference on Computer Vision \(ICCV\)\. IEEE, Montreal, QC, Canada,pp\. 1574–1583\.Cited by:[Table 10](https://arxiv.org/html/2606.19948#A4.T10.1.1.4.2.1),[§2](https://arxiv.org/html/2606.19948#S2.p1.1)\.

## Appendix ARAINbow Dataset

### A\.1Qualitative Examples

We present qualitative examples from the RAINbow dataset in Figures[8](https://arxiv.org/html/2606.19948#A13.F8)and[9](https://arxiv.org/html/2606.19948#A13.F9)\. The dialogs in RAINbow arewell\-groundedin the environment and the interactions exhibit anatural flow\. The multi\-turn exchanges maintain contextual coherence and conversational fluency, closely mimicking the dynamics of human\-to\-human collaboration\.

### A\.2RAIN vs\. RAINbow

We compare the originalRAINdataset \(2,233 episodes\) with our automatically generatedRainbowdataset \(238,028 episodes\)\. Table[7](https://arxiv.org/html/2606.19948#A1.T7)reports mean, median, maximum, and minimum values across episodes for key statistics\. Figure[6](https://arxiv.org/html/2606.19948#A1.F6)compares dialog style between human annotated RAIN dataset and generated RAINbow dataset\.

Table 7:Comparison of dataset statistics between RAIN and Rainbow\.Rainbow is more than two orders of magnitude larger, with richer dialogs and more diverse language\.\(a\)RAIN
\(b\)RAINbow

Figure 6:Qualitative comparison of RAIN \(Human\) and RAINbow \(Generated\) dialogs\.The figure presents examples of complex, multi\-turn dialogs\. The RAIN example \(top\) illustrates the necessity of detailed scene descriptions for successful remote localization\. The RAINbow example \(bottom\) demonstrates that our automatic generation pipeline successfully replicates this essential dialog structure\.
### A\.3CVDN vs\. RAINbow

We provide a qualitative comparison with CVDN\. Their interaction dynamics differ due to task setups: CVDN \(Guide knows Navigator’s position\) vs\. DialNav \(Guide must infer Navigator’s location\)\. In CVDN, the Navigator’s questions often serve as a mere trigger for the next instruction \(e\.g\.,“hello", “where now?"\), and each QA turn tends to resemble a sequence of independentinstruction\-followinginstances\. Conversely, DialNav requires the Navigator to share context for localization\. This involves potential misalignments, such as mislocalization or misnavigation, which RAINbow captures in its pipeline \(details in Supp\. Sec\. C\.1\.7\) exhibitingdecision\-drivenactions, such as re\-querying to resolve uncertainty\.

RAINbowCVDNN:I’m in a bathroom with green tile walls and a painting over the …
G:Got it\! From the washroom, turn a bit right towards the drawers and then …
N:I think I might have gone off track a little\.Right now, I’m in a bright …
G:No problem, thanks for letting me know\. Head out the open door next …N:hello
G:hi, you need to leave the room and go to the room that has four two\-seat sofas in it\.
N:where now?
G:go up the stairs located beside the table in the room you are at now\.

## Appendix BComparison of Embodied Dialog Dataset

Table[8](https://arxiv.org/html/2606.19948#A2.T8)presents a comparison between RAINbow and existing embodied dialog datasets\. Prior datasets typically rely on direct human annotation, which creates a trade\-off between quality and cost, resulting in limited scale\. In contrast, RAINbow is constructed by repurposing existing high\-quality, human\-annotated VLN datasets\. By employing a simple yet effective pipeline to reformat these resources into dialog, we achieve a massive scale\-up at a negligible cost per episode\.

Table 8:Comparison with existing embodied dialog datasets\.Most existing datasets relying on human annotation are costly, limited in scale, or restricted to synthetic environments\.Source: ‘Human’ denotes human\-annotated datasets, while ‘Auto’ denotes automatically generated ones\.
## Appendix CData Augmentation Details

In this section, we describe our scheme to generate realistic, visually well\-grounded, and natural\-flowing dialog in detail\. Our pipeline is designed around three stages: \(1\) trajectory concatenation, \(2\) detailed caption generation for each node, and \(3\) reformatting into a natural dialog flow\. This process ensures the resulting dialog is contextually grounded while preserving the fine\-grained details from the well\-formed captions and original VLN descriptions\.

### C\.1Trajectory Concatenation Details

For trajectory concatenation, we applied the following constraints to ensure natural continuity:

1. 1\.We concatenated 2\-4 trajectories from R2RAndersonet al\.\([2018](https://arxiv.org/html/2606.19948#bib.bib22)\), RxRKuet al\.\([2020](https://arxiv.org/html/2606.19948#bib.bib24)\), CVDNThomasonet al\.\([2020](https://arxiv.org/html/2606.19948#bib.bib23)\)answer trajectories into a single episode\. For CVDN dataset, we use its answer and pair its path with the next 5 shortest path from the question node based on its data collecton description\.
2. 2\.The endpoint of one trajectory and the start of the next must be within 1 meter in the navigation graph\.
3. 3\.To prevent overly circuitous paths, the detour ratio, the concatenated path length divided by the shortest path length between the start and end nodes, was constrained to be less than 1\.3\.
4. 4\.Episodes in which the goal region contained no selectable object were discarded\.
5. 5\.The ambiguous instructionIIwas derived from the Matterport3DChanget al\.\([2017](https://arxiv.org/html/2606.19948#bib.bib21)\)metadata by randomly selecting one visible object in the goal region\. To avoid overly generic references, we excluded a predefined set of objects \(e\.g\.,wall,floor,ceiling, etc\.\)\.
6. 6\.Since goal regions in DialNav correspond to rooms rather than single nodes, we excluded cases where the agent had already reached the goal room before subsequent dialog turns were appended, avoiding unnatural “post\-goal” interactions\.
7. 7\.To further increase diversity, we additionally introduced variations in 10% of the constructed episodes, simulating potential deviations in real dialog navigation\. We consider three types of variations:*mislocalization*,*misnavigation*, and*exploration*\. In the case of mislocalization, the Guide intentionally provides an incorrect path description that does not match the Navigator’s true position; the Navigator then proceeds by moving 1–2 nodes randomly from the original location rather than following the erroneous instruction\. For misnavigation, the Navigator deviates from the instructed path and follows a randomly chosen alternative route\. In exploration, the Navigator continues beyond the instructed trajectory by taking an additional 2–5 random steps after completing the suggested path\. By incorporating these cases, we were able to include instructions that were previously filtered out by the strict trajectory\-connection criteria, thereby maximizing the utilization of existing VLN data for dialog augmentation\.

### C\.2Caption Generation Details

To produce natural and visually grounded questions, we prompted LLaVA\-1\.5\-7BLiuet al\.\([2023](https://arxiv.org/html/2606.19948#bib.bib34)\)for visual description and Llama\-3\.1\-8B for text synthesis with different visual/textual contexts\. Our aim was to ensure that the resulting utterances contained sufficient local detail for Guide\-side localization, while generating diverse content even for the same node\. We experimented with three variants and randomly used one of these schemes for caption generation for each episode\. The prompt for each variant is provided in Figure[10](https://arxiv.org/html/2606.19948#A13.F10)and the resulting captions are provided in Figure[11](https://arxiv.org/html/2606.19948#A13.F11)\.

\(A\) Simple panoramic caption: The panoramic observation at a node is directly provided to LLaVA with a short instruction to generate a navigation\-oriented question\. This produces concise questions\.

\(B\) Region\-grounded caption: Here, we first get the room type of each node from Matterpot3D metadata\. Then, we instructed LLaVA to provide a description of the panoramic scene, explicitly mentioning the room type\. The room types are provided in Matterport3D metadata paired to each navigation node\. This yields richer, more descriptive questions\.

\(C\) Region and object\-grounded panoramic caption: For a more detailed and visually grounded caption, we first generate captions for visible objects in the given node\. Based on Matterport3D metadata, we first extract image patches of all visible objects at a given node\. Then, we prompt LLaVA to generate a description of the given object image\. To avoid hallucination or errors from wrong metadata annotations, we prompt it to generate ’none’ if the given object is not visible in the image\. We then jointly supply three inputs to LLaMA\-3\.1\-8B: a general panoramic caption and a randomly sampled subset of the valid object\-level captions\. LLaMA is then prompted to generate a single detailed caption that integrates both global layout and salient object cues\. To increase diversity, we vary the sampling ratio and the number of in\-context examples\. For in\-context samples, we find similar samples in the RAIN dataset based on room types and visible objects\.

### C\.3Dialog Reformat Details

The primitive Q&A content, derived from raw captions and VLN instructions, is less natural than human conversation\. To reformat this content into a natural multi\-turn dialog, we employ GPT\-4o\-mini\. To ensure formatting consistency and a natural flow, we perform this in two steps\. The prompts for these steps are provided in Figure[12](https://arxiv.org/html/2606.19948#A13.F12)\. The result of each steps are provided in Figure[13](https://arxiv.org/html/2606.19948#A13.F13)\. Through this 2 step reformatting, the resulting dialog is natural, visually well grounded and better reflects realistic Navigator\-Guide interactions\.

1. 1\.Instruction Refining\.First, we refine the raw VLN instructions\. We prompt the LLM to make each instruction concise while preserving key spatial references, preventing overly verbose text that is unnatural in conversation\. Moreover, since the original VLN instructions are for single\-turn tasks, they often prematurely mention reaching the final goal\. To prevent this, we also instruct the LLM to generate two versions of each refined instruction: a ‘goal\-oriented’ variant \(e\.g\., "You have reached your destination"\) and a ‘neutral’ variant \(which does not mention the goal\)\. We use the goal\-oriented variant only for the final dialog turn and the neutral version for all intermediate turns\. This explicit branching strategy prevents ambiguous or inconsistent goal mentions\.
2. 2\.Conversational Smoothing\.We then construct a draft dialog by sequencing the original scene captions and their corresponding refined, goal\-conditioned instructions \(from Step 1\)\. This entire sequence is then paraphrased by GPT\-4o\-mini into fluent, conversational language, resulting in the final multi\-turn dialog\. This step retains all navigation\-critical details while simplifying overly formal phrases, removing redundancies, and inserting subtle acknowledgments where appropriate to enhance conversational flow\.

### C\.4Dataset Evaluation\.

To validate our final dataset we conducted a human evaluation\. 30 samples each from RAIN and RAINbow were randomly shuffled and evaluated by 6 annotators\.\(1\) Goal Alignment:When verifying if the dialog remains faithful to the initial goal, RAINbow achieved 90\.0% accuracy \(comparable to RAIN’s 93\.3%\)\. This confirms that ourInstruction Refiningstep \(Supp\. Sec\. C\.3\.1\) successfully preserves goal consistency\.\(2\) Naturalness:In terms of linguistic flow, RAINbow achieved a score of 4\.76/5\.0, closely mirroring the human\-written RAIN baseline 4\.83/5\.0\.

Table 9:Average number of objects and hallucinations per turn\.
### C\.5Hallucination Analysis

To further assess the reliability of our pipeline, we conduct a hallucination evaluation on a sampled subset of RAINbow\.

Evaluation protocol\.We randomly sampled 100 episodes \(247 dialog turns in total\) and used a closed\-source VLM \(Claude Opus 4\.7\) to verify each object mentioned in the dialog against the actual observation\. The table[9](https://arxiv.org/html/2606.19948#A3.T9)reports the average number of hallucinated objects per\-turn\.

Results\.Our pipeline directly repurposes the human\-annotated VLN instructions as answers, which results in low hallucination ratio \(2\.0%\)\. The hallucination rate in questions is 10\.7%\. This low hallucination rate confirms the effectiveness of our pipeline\.

## Appendix DResults on CVDN

Table 10:Evaluation on the CVDN Dialog Setup\.The table compares our method against established baselines on the CVDN datasetThomasonet al\.\([2020](https://arxiv.org/html/2606.19948#bib.bib23)\)\. The performance isGoal Progress \(m\), where higher values indicate better performance\.We evaluate our agents on the established Vision\-and\-Dialog Navigation \(CVDN\) datasetHaoet al\.\([2020](https://arxiv.org/html/2606.19948#bib.bib32)\)\. Table[10](https://arxiv.org/html/2606.19948#A4.T10)presents a comparison against methods evaluated under the dialog in a loop setup\. The CVDN evaluation uses a simplified remote Guide configuration: the Guide is provided with the Ground Truth localization and limits its guidance to 5 future navigation steps\. Our approach achieves superior performance over all listed methods, demonstrating the strong generalization effect of our large\-scale RAINbow dataset and training scheme to this external domain\.

## Appendix EModel Details

### E\.1Agent Capabilities and Information Flow

The overall DialNav task relies on the specialized capabilities and distinct information access of the two agents\. The integrated agents \(Navigator and Guide\) possess the full knowledge boundaries listed below\.Navigator Agent\(Navigation, Question, Whether\-to\-ask Decision\): The Navigator has access to the accumulated dialog history, its navigational history \(the subgraph of visited nodes and adjacent locations\), and the initial instruction \(II\)\.Guide Agent\(Localization, Answer\): The remote Guide has knowledge of the full environment graph \(GG\), the initial instruction \(II\), and the goal region \(RR\)\. Since the Guide knows the full environmentGG, it can calculate the shortest path between any two nodes\. However, i this work, for implementation and training convenience, each submodule utilized only the specific input features that its model architecture was designed to accept\.

### E\.2Navigation Model

For the core action prediction module \(Navigator\), we adopt the Dual\-scale graph Transformer \(DUET\)Chenet al\.\([2022b](https://arxiv.org/html/2606.19948#bib.bib27)\)architecture, following established practices in DialNav researchHanet al\.\([2025](https://arxiv.org/html/2606.19948#bib.bib1)\)\. DUET is a strong VLN backbone designed for joint long\-term action planning and fine\-grained cross\-modal understanding\.

Architecture and Reasoning\.DUET explicitly builds a topological map and dynamically combines two scales of encoding via graph transformers: a fine\-scale encoding of local observations and a coarse\-scale encoding of the global map\. This dual\-scale approach allows the Navigator to reason efficiently over a large action space\.

Pretraining\.We leverage publicly available weights from large\-scale VLN pretraining frameworks, such as ScaleVLNWanget al\.\([2023c](https://arxiv.org/html/2606.19948#bib.bib33)\)\. This initialization is vital because the pretraining process exposes the model to millions of diverse instruction\-trajectory pairs across various environments, imparting robust environmental knowledge that is crucial for significantly reducing the generalization gap between navigating in seen and unseen environments\.

- •Input:Accumulated visual and spatial history, initial instruction and the latest navigation instruction appended as single instruction\.
- •Output:Probability distribution over the next optimal actionvt\+1v\_\{t\+1\}\(move or stop\)\.

### E\.3Question and Answer Generation Model

For both question and answer generation, we adopt theLANAmodelWanget al\.\([2023a](https://arxiv.org/html/2606.19948#bib.bib36)\)as our baseline module\. LANA is a vision\-and\-language model originally designed to enhance navigation performance by jointly learning to describe past and upcoming paths\.

- •Input:Panoramic observation \(XvX\_\{v\}\) for question generation; or a sequence of panoramic images for the next subpath for answer generation\.
- •Output:Natural language question \(qq\) or guiding answer \(aa\)\.

In our setup, LANA is initialized with its proposed pretraining and fine\-tuned on the RAINbow and RAIN datasets to adapt navigation\-style instructions into dialog\-style questions and answers\.

### E\.4Localization Model

For localization, we use a lightweight cross\-modal architecture, the Graph Convolutional Network \(GCN\) LocalizationHahn and Rehg \([2022](https://arxiv.org/html/2606.19948#bib.bib18)\), as our reproducible baseline\. The GCN model formulates localization as a*node\-selection*problem\. We introduce our graph\-based localization, adopting a transformer inspired byDUETChenet al\.\([2022b](https://arxiv.org/html/2606.19948#bib.bib27)\)to improve position inference from dialog\.

Repurposing the VLN Architecture\.We adapt the DUET architecture to leverage knowledge from VLN pretraining\. For localization, we repurpose the architecture by substituting the inputs and reinterpreting the output:

- •VLN Input→\\rightarrowLocalization Input:The navigation instructionIIis replaced with theNavigator’s questionqq, and the partial navigation mapG^t\\hat\{G\}\_\{t\}is replaced with theentire house graphGG\.
- •VLN Output→\\rightarrowLocalization Output:The model is tasked to select the most probable node corresponding to the Navigator’s current positionvqv\_\{q\}from all nodes inGG:vq=DUETLOC​\(G,q\)v\_\{q\}=\\text\{DUET\}\_\{\\text\{LOC\}\}\(G,q\)\.

This adaptation, leveraging large\-scale pretrained weights, provides robust perceptual grounding and improves localization accuracy\.

### E\.5Whether\-to\-Ask

Thewhether\-to\-askmodule governs the interactive nature of the Navigator, deciding when to request external guidance\. This module uses a confidence thresholding strategy, where the decision to ask or proceed autonomously is determined by the model’s confidence in its immediate next action\.

- •Input:The Navigator’s probability distribution over the next possible actionsp​\(vt\+1\|context\)p\(v\_\{t\+1\}\|\\text\{context\}\)\.
- •Mechanism:The most confident action in next possible actions is treated as our candidate action\. If the candidate action’s probability falls below a fixed thresholdτ\\tau, the Navigator initiates a questionqjq\_\{j\}\.
- •Output:Binary decision \(Ask=1 or Proceed=0\)\.

## Appendix FLocalization Model on WAY Task

TheWhere Are You? \(WAY\)Hahnet al\.\([2020](https://arxiv.org/html/2606.19948#bib.bib16)\)is a foundational benchmark for embodied localization, requiring a model to predict its current positionvqv\_\{q\}within a visual environment \(Matterport3D\) based solely on a textual query\. This task is crucial as it isolates the language grounding ability required by the remote Guide setup in DialNav\.

Our model, which integrates the effects of large\-scale pretraining from ScaleVLN and the data diversity provided by the RAINbow dataset, has achieved strong performance in the holistic DialNav setup\.

We further fine\-tune our graph\-based transformer localization model on the WAY benchmark to demonstrate that our architecture generalizes well to existing localization tasks\. Table[11](https://arxiv.org/html/2606.19948#A6.T11)provides the results of our model against established single\-shot baselines for the WAY localization task, where our model shows superior performance\.

Table 11:Localization Performance on the WAY Dataset \(Single\-Shot\)\.Comparison of our graph\-based transformer model against established single\-shot baselines\. We report accuracy for exact node prediction \(Acc@0m\) and Localization Error \(LE\) in meters across seen and unseen environments\.
## Appendix GFurther Analysis on Table[1](https://arxiv.org/html/2606.19948#S4.T1)

Table 12:Trajectory statistics across success cases\.Comparison of Success Rate \(SR\), Navigation Step Count \(NSC\), and Dialog Turn Count\(DTC\) for all episodes \(All\) and successful episodes only \(Succ\.\)\.Overall, SPL, NE, and OSR exhibit trends consistent with SR\. Separately, NSC and DTC demonstrate that RAINbow and DST encourage exploration in uncertain trajectories\. To verify this, we measured NSC and DTC specifically within successful episodes for both the baseline and the models equipped with RAINbow and DST\. In the Val Seen split, both scores remain nearly identical to the baseline \(DTC: 3\.57 → 3\.49; NSC: 19\.89 → 19\.66\), indicating that the overall increases in these metrics stem primarily from exploratory behavior in uncertain cases\. Building on this, the Val Unseen results show that this uncertainty\-driven exploration leads to improved generalization to unseen environments: the models achieve nearly double the success rate while exhibiting increased exploration even within successful episodes \(DTC: 5\.03 → 8\.23; NSC: 14\.29 → 20\.26\)\.

## Appendix HQualitative Results

### H\.1Holistic Navigation

Success with Self\-Recovery Despite Localization Failure\.Figure[14](https://arxiv.org/html/2606.19948#A13.F14)illustrates a successful DialNav episode that strongly demonstrates the self\-recovery capability afforded by our integrated framework\. Initially, the Navigator faced a complex environment where the Guide failed to accurately determine the current position\. Despite receiving this falsely grounded instruction, the Navigator successfully overrode the erroneous localization estimate and dynamically reconstructed its trajectory \(self\-recovery\) based on goal\-relevant information within the dialog\.

Failure Due to Instruction\-Execution Inconsistency\.Conversely, Figure[15](https://arxiv.org/html/2606.19948#A13.F15)presents a critical failure case that underscores the remaining challenge in seamlessly integrating the conversational policy with the motion execution policy\. In this episode, the Guide successfully localized the Navigator and provided a well\-grounded answer\. However, the Navigator failed to accurately execute the trajectory dictated by the dialogue’s instructions, resulting in navigation failure\.

### H\.2Generated Dialog

We provide a qualitative comparison between dialogs generated by the baseline \(trained only on RAIN\) and our enhanced model \(trained on RAINbow\) against the human\-annotated Ground Truth \(GT\) in Table[13](https://arxiv.org/html/2606.19948#A13.T13)\. All generated samples were produced under the identical preceding dialog context and current panoramic viewpoint\. This analysis clearly demonstrates the improvement afforded by large\-scale data augmentation\.

The baseline model trained solely on the small RAIN dataset often suffers from redundancy and fragmentation\. For instance, in the Answer Example, the RAIN model enters a circular loop \("you have to go to the dining room"\) and omits essential path information\. Similarly, in the Question Example, the generated question is highly repetitive \("a fireplace, a fireplace, and a fireplace"\), failing to provide sufficient, diverse visual cues for the Guide to localize the agent\. In contrast, the model fine\-tuned onRAINbowproduces significantly more fluent and descriptive instructions\. The RAINbow answers include detailed navigational steps grounded in specific objects \("exit the room and turn right to find a staircase"\) and generally maintain a coherent conversational flow\. The RAINbow questions successfully utilize rich visual features \("a wooden table right in the middle," "a large, sliding glass door to the outside"\) to generate contextually relevant descriptions, which is vital for localization\.

However, a qualitative gap still exists compared to the human\-written Ground Truth\. While RAINbow greatly improves fluency, it sometimes lacks the fine\-grained object\-level details\. Nonetheless, the substantial gain in fluency and coherence achieved by RAINbow validates the effectiveness of our automatic generation pipeline for boosting dialog quality\.

## Appendix IAnalysis of Confidence Threshold

![Refer to caption](https://arxiv.org/html/2606.19948v1/images/cf_3.png)\(a\)Dialog Turn Count vs\. Confidence Threshold
![Refer to caption](https://arxiv.org/html/2606.19948v1/images/cf_1.png)\(b\)Success Rate vs\. Confidence Threshold
![Refer to caption](https://arxiv.org/html/2606.19948v1/images/cf_2.png)\(c\)Navigation Steps vs\. Confidence Threshold

Figure 7:Impact of Confidence Threshold \(τ\\tau\) on performance\.Adjusting the threshold value allows the Navigator to balance autonomous navigation with dialog requests, showing the trade\-off between task success and exploration efficiency\.We analyze the impact of the Confidence Threshold \(τ\\tau\), the hyperparameter governing the Navigator’s decision to ask for help\. As the thresholdτ\\tauincreases, the agent triggers a question more often when facing uncertainty\. This phenomenon is clearly demonstrated by the Dialog Turn Count \(DTC\), which shows a consistent increase across both splits asτ\\tauapproaches0\.90\.9\(Figure[7\(a\)](https://arxiv.org/html/2606.19948#A9.F7.sf1)\)\.

The increase in dialog frequency directly correlates with the final navigation Success Rate \(SR\)\(Figure[7\(b\)](https://arxiv.org/html/2606.19948#A9.F7.sf2)\)\. InVal Seen, SR shows a sustained increase as the threshold rises\. This indicates that dialog is effective at resolving uncertainty and guiding the agent to success in familiar environments\. InVal Unseen, SR increases up toτ≈0\.5\\tau\\approx 0\.5and then saturates\. This contrast suggests that while dialog provides essential initial information, the model reaches a performance ceiling in novel environments, possibly limited by the quality of answer in complex, unseen settings\. The difference in saturation points confirms that the benefit of dynamic dialog is substantially greater in known environments, where the learned policy can best leverage the updated instruction for planning\.

Furthermore, we observe that dialog significantly impacts exploration efficiency \(Figure[7\(c\)](https://arxiv.org/html/2606.19948#A9.F7.sf3)\)\. Across both Val Seen and Val Unseen, the Navigation Step Count \(NSC\) decreases up toτ≈0\.5\\tau\\approx 0\.5before stabilizing\. This demonstrates that proactive dialog is highly effective at reducing redundant exploration, as the agent quickly seeks guidance instead of wandering when uncertain\. The stable reduction in NSC followed by stabilization in SR \(afterτ=0\.5\\tau=0\.5in Val Unseen\) shows a clear correlation between reducing navigational uncertainty via dialog and achieving task success\.

## Appendix JDetails on Human\-Agent Cooperation

Participants\.Ten participants were recruited internally from our research group\. All participants signed consent form consent prior to the evaluation\.

Task protocol\.Each participant interacted with our agent through a GUI simulator that supported mouse\-based navigation in the Matterport3D environment paired with a text\-based dialog window\. Participants took the role of either Navigator \(sending questions and navigating based on the agent’s answers\) or Guide \(receiving the participant Navigator’s questions and providing localization\-grounded answers\), while the agent took the complementary role\. The four conditions in Table 5 — Navigator, Guide × Baseline, Ours — were presented in randomized order across both Val Seen and Val Unseen splits\. Each participant completed 20–30 episodes in total\. After each episode, participants rated the agent counterpart’s helpfulness on a 1–5 Likert scale, which we report as Human Score \(HS\) in Table 5; Success Rate \(SR\) was computed from task outcomes\.

Ethics/IRB\.Our human evaluation qualifies for IRB exemption under our institution’s policy, as it does not identify individual participants and does not collect or record sensitive personal information as defined by applicable data protection regulations\. The evaluation collected only task outcomes \(navigation success\) and helpfulness ratings; no personal identifiers, demographic information, or sensitive data were recorded\.

## Appendix KPotential Risks

Our work studies dialog\-enabled embodied navigation, where language directly affects action decisions in a physical environment\. As such systems become more capable, incorrect or misleading dialog may lead to unsafe or inefficient actions\. This is particularly important in embodied settings, where execution errors may carry physical consequences\.

In addition, our automatic data generation pipeline uses large language and vision–language models to synthesize dialog\. Although we validate faithfulness to the intended goal and manually inspect examples, generated data may still contain occasional inaccuracies, biases, or unnatural interaction patterns\. If used without careful validation, such artifacts could propagate into trained embodied agents\.

We mitigate these risks in three ways\. First, our task is evaluated in simulation rather than physical deployment\. Second, we validate generated dialog quality and goal faithfulness through both automatic design constraints and human evaluation\. Future work should further investigate robustness to adversarial, ambiguous, or misleading dialog in real\-world embodied settings\.

## Appendix LData Safety, Privacy, and Content Checks

We discuss here whether the data used or created in this work contains personally identifying information \(PII\) or offensive content\.

Our dataset is built by repurposing existing VLN resources grounded in Matterport3D indoor environments and by automatically generating navigation dialog from those trajectories\. The resulting data consists of navigation paths, scene\-grounded object references, and generated question–answer pairs about indoor environments\. It is not designed to include names, personal identities, contact information, or other uniquely identifying information about individuals\.

## Appendix MUse of AI Assistants

AI assistants were used in two ways in this work\. First, they were used in the research pipeline for automatic data generation, including dialog reformatting and caption\-based synthesis, as described in the main paper\. Second, they were used during manuscript preparation for writing support, such as grammar correction and improving clarity and flow\. All scientific decisions, experimental design, interpretations, and final wording were reviewed and verified by the authors\.

![Refer to caption](https://arxiv.org/html/2606.19948v1/x8.png)\(a\)
![Refer to caption](https://arxiv.org/html/2606.19948v1/x9.png)\(b\)

Figure 8:Examples of RAINbow data \(1\)![Refer to caption](https://arxiv.org/html/2606.19948v1/x10.png)\(a\)
![Refer to caption](https://arxiv.org/html/2606.19948v1/x11.png)\(b\)

Figure 9:Examples of RAINbow data \(2\)Prompt for Simple Panoramic CaptionGenerate a sentence that describes the given panoramic image\. Start your question with "I’m in \.\.\." or "I can see \.\.\."\.\(a\)Prompt for Simple Panoramic CaptionPrompt for Region Grounded CaptionPlease describe the \{region\} in this image\.\(b\)Prompt for region grounded captionPrompt for Object CaptionPlease describe the \{obj\} in this image\. If there’s no \{obj\}, response ’None’\(c\)Prompt for object captionPrompt for Region and Object Grounded CaptionYou will be provided with a panoramic image caption describing an entire space and detailed captions of key objects visible in that image\. Based on the input, describe the surrounding environment clearly and concisely\. Your description should include the structure, layout, and the locations of key objects\. Do not add anything, but just the sentence describing around\.\(d\)Prompt for region and object grounded caption
Figure 10:Prompts used to generation captions for navigation questions\.![Refer to caption](https://arxiv.org/html/2606.19948v1/images/ad409728ae8b4beb8d01580b3570bbdd.png)

Figure 11:Example of generated captions from three different captioning schemes\.Prompt for Goal\-conditioned ReformatYou will be provided with a sequence of sentences describing a path\.Your task is to rewrite the path description in two versions:1\. with\_goal: A version that includes reaching the final destination2\. without\_goal: A version that excludes any mention of reaching the destinationBoth versions should:\- Be concise while preserving key spatial references\- Focus on objects, rooms, and layout\- Use directions only when necessary\- Remove terms implying current situation \(Start facing , You’re in a , You’re facing\)For with\_goal version, Always mention that you’ve arrived at the destination at the end of the description with expressions like: That’s the goal, You have arrived, That’s your destination, The goal, You’ve reached your destination, This is the final spot, You’re at the destination, Mission accomplished, End point reached, You’ve made it\.For without\_goal version, Remove any mentions of arriving at or reaching the final destination\.\(a\)Prompt template for goal\-conditioned path reformatting\.Prompt for Natural Dialog ReformatYou will rewrite a dialog between a Navigator and a Guide in an indoor navigation task\.TASK CONTEXT:\- The Navigator and Guide work together to reach a goal room\.\- Both Navigator and Guide know about a shared object in the goal room\.\- The Guide knows the goal room but the Navigator does not\.\- The object may appear in other rooms, so the Navigator must ask clarifying questions\.\- The Guide cannot see the Navigator’s position but knows the environment layout\.\- Dialog alternates turns: Navigator → Guide → Navigator … \(no limit on turns\)\.\- Dialog always starts with the Navigator\.OBJECTIVE:\- Rephrase the dialog to sound like natural conversation\.\- Style:\* Use casual, friendly language \("Ok", "Alright", "Got it"\)\.\* Add acknowledgments \("I see", "Understood"\)\.\* Smooth transitions between turns\.\* Vary vocabulary, keep meaning the same\.\- Actively incorporate mentions of the shared object in the conversation\.\- The Guide’s final response should mention reaching the goal room\.RULES:1\. Keep the same number of turns and same speakers \(Navigator/Guide\)\.2\. Do not add or remove turns\.3\. Do not add new objects, rooms, or details not in input\.4\. Never use phrases like "You’re looking at…" or "You are facing…"\. \- Instead: ask questions \("Do you see…?"\) or omit\.5\. Do not drop or shorten navigation instructions\.6\. Preserve all key details:\- Objects and attributes \(color, shape, material\)\.\- Room names and types\.\- Spatial relations and directions\.7\. Do not use em dashes \(—\)\. Use commas, periods, or "and"\.OUTPUT FORMAT:Return ONLY valid JSON:"reformatted": \["Navigator": "\.\.\.", "Guide": "\.\.\." ,"Navigator": "\.\.\.", "Guide": "\.\.\."\]No extra text, no markdown, no explanations\.\(b\)Prompt template for natural dialog reformatting\.
Figure 12:Prompt for dialog reformatting\.\(a\)Instruction Refining\.

\(b\)Conversational Smoothing \(Q: Question, A: Answer\)\.

Figure 13:Example of dialog reformatting\.\(a\) Examples ofStep 1 \(Instruction Refining\), where raw VLN instructions are reformatted into ‘without goal‘ and ‘with goal‘\. \(b\) Examples ofStep 2 \(Conversational Smoothing\), where the sequence of captions and refined instructions is paraphrased into fluent dialog\.![Refer to caption](https://arxiv.org/html/2606.19948v1/x12.png)Figure 14:Successful example of DialNav episode of our Navigator and Guide agent in Val Unseen environment\.Although the agent failed to localize its current position in the complex environment, it successfully acquired crucial information about the target room through dialogue\. This allowed the agent to self\-recover its trajectory and successfully navigate to the destination\.![Refer to caption](https://arxiv.org/html/2606.19948v1/x13.png)Figure 15:Failure case of DialNav episode of our Navigator and Guide agent in Val Unseen environment\.Despite achieving successful localization and receiving a well\-grounded answer regarding the path, the navigation agent failed to follow the instruction derived from the dialogue, resulting in a failure to reach the destination\.\(a\)Generated Question
\(b\)Generated Answer

Table 13:Qualitative Comparison of Generated Dialog\.Comparison of Ground Truth \(GT\) questions and answers versus models trained on RAIN and RAINbow data\. Bold text indicates essential content in the Ground Truth \(GT\),blue textdenotes overlapping content found in the generated outputs, andred texthighlights non\-fluent content\. RAINbow significantly improves fluency and reduces redundancy compared to the RAIN baseline\.