GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topology

# GIST: Multimodal Knowledge Extraction and Spatial Grounding via Intelligent Semantic Topology
Source: https://arxiv.org/html/2604.15495

###### Abstract

Navigating and searching complex, densely packed environments such as retail stores, warehouses, libraries, and hospitals poses a great spatial grounding challenge for both humans and embodied AI. In these environments, dense visual features become stale quickly given the quasi-static nature of items, and long-tail semantic distributions pose a challenge for dedicated computer vision methods. Vision-Language Models (VLMs) have emerged as a popular paradigm to help assistive systems and robots understand and navigate semantically-rich spaces, yet they still struggle with spatial grounding in dense, cluttered, quasi-static environments. We present **GIST** (Grounded Intelligent Semantic Topology), a multimodal knowledge extraction pipeline that transforms a consumer-grade mobile point cloud into a semantically annotated navigation topology. Our architecture distills the scene into a 2D occupancy map, extracts its topological layout, and overlays a lightweight semantic layer via intelligent keyframe and semantic selection. We demonstrate the versatility of this structured, semantically-rich spatial knowledge through critical downstream Human-AI interaction tasks: (1) an intent-driven **Semantic Search** engine that actively infers categorical alternatives and zones when exact matches fail; (2) a one-shot **Semantic Localizer** achieving 1.04 m top-5 mean translation error; (3) a **Zone Classification** module that segments the walkable floor plan into high-level semantic regions; and (4) a **Visually-Grounded Instruction Generator** that synthesizes optimal paths into egocentric, landmark-rich natural language routing. Evaluated via an independent multi-criteria LLM evaluation protocol, GIST outperforms navigation instruction generation baselines. Finally, an in-situ formative evaluation (N=5) yields an 80% navigation success rate relying solely on verbal cues, validating the system's capacity for universal design.

**Keywords:** Navigation, Topology, Semantic Map, Multimodal Data, Universal Design, Interactive systems

---

A block diagram showing mobile RGB-D data processed through object extraction and a Vision-Language Model to create a semantic topology map, which subsequently powers intent-aware search, semantic localization, and natural language routing.

**Figure 1.** The **GIST** Multimodal Knowledge Extraction Architecture. Raw multimodal inputs (RGB-D and mobile odometry) are distilled via intelligent keyframe selection, representative object selection, and VLM labeling into Structured Spatial Knowledge. This shared representation enables robust downstream Human-AI interaction and autonomous system tasks, including intent-aware semantic search, global pose localization, and spatially grounded natural language routing.

## 1. Introduction

Navigating and searching dense, cluttered environments poses a significant spatial grounding challenge for both humans and autonomous embodied AI. Conventional mapping techniques capture geometric free space, but they do not represent the semantic structure people rely on for search, orientation, and route understanding.

In dense quasi-static environments, navigation depends heavily on semantic landmarks and topological transitions rather than solely metric commands. Prior work in spatial cognition shows that people interpret routes more effectively when directions are chunked into meaningful segments and anchored to recognizable landmarks (Lovelace et al., 1999; Klatzky, 1998). For example, "walk past the lentils on your left, then turn right" is more actionable and lower-burden than "walk 8.6 meters forward and then turn right."

These spaces are inherently **quasi-static**—environments where the spatial distribution, orientation, and quantity of objects evolve over extended timescales, such as fluctuating product inventory on grocery shelves or moving books in a library. Furthermore, as demonstrated by our 3,500 sq. ft. international grocery store testbed, they exhibit extreme long-tail distributions of culturally specific, visually similar items that induce severe perceptual aliasing. Recent Vision-and-Language Navigation (VLN) frameworks perform well in benchmarked and synthetic domains (Krantz et al., 2020; Chen et al., 2024), but consistently underperform in real-world settings (Aghzal et al., 2025; Wang et al., 2025a; Windecker et al., 2025) where target semantics may be partially occluded, visually repetitive, or absent from the immediate view.

Recent instruction generation frameworks, such as NavComposer (He et al., 2025), are an important step toward scalable data generation, but they are optimized for visual sequence modeling rather than human-centered spatial communication. In dense environments, this leads to directions that often depend on transient scene content and under-specify stable route geometry, definitive turn structure, and durable landmarks.

To bridge this gap, we introduce **GIST** (Grounded Intelligent Semantic Topology), an end-to-end framework for extracting a semantically grounded navigation representation from a single consumer mobile scan (Figure 1). GIST converts RGB-D data and odometry into a 2D occupancy map, derives a topology graph over walkable space, and anchors semantically labeled products to that graph through keyframe selection, representative object selection, and VLM-based annotation. The resulting semantic topology separates deterministic geometric structure from higher-level semantic reasoning while keeping both in a shared coordinate frame.

This structured representation supports critical Human-AI interaction tasks: (1) **Intent-Driven Search & Missing Goal Estimation** to estimate zone locations when exact matches are not available on the semantic map; (2) **Semantic Localization**, a one-shot localizer estimating an agent's discrete pose (x, y, θ) from a single smartphone image using pure text-embeddings; (3) **Zone Classification** over walkable space; and (4) **Visually-Grounded Spatial Routing** translating spatial paths into view-independent, natural language directions. Our key contribution is the shared intermediate representation that supports search, localization, and communication via a coordinated, human-centered approach.

Across 15 real-world scenarios, we demonstrate that GIST's explicit topological structures significantly outperform raw RGB sequence baselines in multi-criteria LLM evaluations, particularly as route complexity increases. Finally, a formative in-situ ecological probe (N=5) yields an 80% navigation success rate using generated verbal cues alone, providing encouraging initial evidence of real-world utility.

## 2. Related Work

**GIST** facilitates several critical human-AI and embodied AI tasks; the following subsections present related work foundational to these domains.

### 2.1. Vision-and-Language Navigation (VLN)

VLN requires an embodied agent to interpret natural language instructions and navigate 3D environments solely from visual observations (Anderson et al., 2018). Early benchmarks operated in discrete graph-based settings where agents teleport between panoramic viewpoints (Qi et al., 2020). While VLN-CE (Krantz et al., 2020) pushed toward continuous environments requiring low-level motor control, these systems still assume high-quality instructions are already provided.

Automated instruction generation has emerged to scale VLN training data. Early speaker models used LSTM-based networks to translate panoramic sequences into directions (Fried et al., 2018), lacking spatial specificity. Recent models like NavRAG (Wang et al., 2025b) utilize scene description trees to provide fine-grained context. Goal-conditioned generation models like GoViG (Wu et al., 2025) attempt to produce instructions from only a first-person observation and a goal image. NavComposer (He et al., 2025) generates instructions from RGB image sequences combined with discrete action traces. However, because sequence-to-sequence methods bypass explicit topological reasoning, they struggle to infer path geometry from images alone. This results in instructions that reference visually salient but transient objects (e.g., red crates, shopping carts) and cannot reliably specify definitive turn sequences, exact metric distances, or macro-level navigation instructions.

### 2.2. Localization in Quasi-Static Environments

Particle filter-based localization methods (e.g., AMCL (D. Fox, 2001)) remain standard for onboard robotics. However, their reliance on static geometric maps makes them notoriously brittle in quasi-static environments like warehouses and retail stores, where local semantics fluctuate even when global geometry remains constant (Yin et al., 2024).

Semantic SLAM systems (Goswami et al., 2023; Zimmerman et al., 2023) often assume landmark stability, an assumption that fails in real-world quasi-static environments where objects frequently shift or fluctuate in quantity. Recent probabilistic approaches like **ShelfAware** (Agrawal et al., 2025) model shelf semantics as statistical distributions over object counts to handle these changes. However, they rely strictly on low-level visual features, which often fail to distinguish semantically distinct but visually similar items (e.g., bags of sugar, salt, or flour). Deep visual localization methods (Dong et al., 2025; Loiseau et al., 2025) and implicit neural representations (Kuang et al., 2023) demonstrate impressive accuracy but require server-grade compute, precluding real-time use on wearable edge devices. GIST overcomes these limitations by utilizing one-shot, text-embedding-based semantics to enable instant global localization on lightweight hardware.

### 2.3. Assistive Guidance Systems

Navigating large retail spaces or warehouses presents a critical challenge for people with visual impairments, involving macro-navigation in the locomotor space and micro-navigation in the haptic space (Gharpure and Kulyukin, 2008).

**Macro-Navigation and Product Search:** Recent work focuses largely on navigation inside the store (Kulyukin and Kutiyanawala, 2010; Kulyukin et al., 2005; Kulyukin and Gharpure, 2006). Solutions that rely on environmental augmentation, such as RFID tags (López-de-Ipiña et al., 2011) or Bluetooth beacons, introduce high maintenance overhead and barriers to adoption. For product identification, existing techniques using fixed-class object detectors (Feng et al., 2020) or barcode scanning (Google, 2022; Nicholson et al., 2009) are impractical for the sheer volume of long-tail retail items (NielsenIQ, 2019). While researchers have explored conversational agents (Kamikubo et al., 2024; Kaniwa et al., 2024) and the "last few meters" way-finding problem (Saha et al., 2019), these systems often still rely on BLE beacons, underscoring a critical need for uninstrumented, scalable semantic search.

**Manipulation Guidance and Social Dynamics:** Once a user is in the vicinity of a product, manipulation guidance helps them find regions of interest (Vázquez and Steinfeld, 2014; Bonani et al., 2018). Crowdsourced assistance like Be My Eyes (Eyes, 2022) or VizWiz (Bigham et al., 2010) relies on human availability. Some researchers have focused on the product retrieval sub-task by providing verbal manipulation guidance for product retrieval off the shelf (Agrawal et al., 2023; Agrawal, 2023). However, these systems inherently assume the user is already perfectly localized in front of the correct shelf (Zientara et al., 2017). GIST bridges this gap by creating intermediate spatial knowledge that focuses on solving the prerequisite locomotor navigation problem by searching for desired goals and estimating the current pose of the system.

## 3. Multimodal Knowledge Extraction

Our offline pipeline converts consumer mobile RGB-D data into a reusable semantic topology without manual annotation.

### 3.1. Mobile Data to 2D Occupancy

We captured a real-world, 3,500 sq ft international grocery store using a consumer mobile device equipped with LiDAR. A 14.5-minute traversal yielded 52,006 RGB-D frames, which were subsampled 6×6 to 8,668 frames (~10 fps) alongside 6-DoF ARKit visual-inertial odometry. To ensure lightweight processing and downstream scalability, we explicitly avoid computationally heavy classical SLAM backends. Instead, we project a 2D occupancy grid (0.05 m/pixel) by applying a height-slice to the raw mobile point cloud.

### 3.2. Informative Keyframe & Object Extraction

To reduce multimodal input processing and annotation cost, we first perform keyframe selection. Each full RGB frame is embedded via DINOv3 (vitb16, 768-dim). A sequential cosine filter retains only frames whose similarity to the previously accepted keyframe falls below a 0.85 threshold, condensing the sequence to 660 visually distinct keyframes (a 92.4% frame reduction) to eliminate redundancy.

Each keyframe is processed by a YOLOv9 model fine-tuned on the SKU-110K dataset (Goldman et al., 2019) for dense shelf products. Because tightly-packed retail shelves generate an extreme density of bounding boxes for repetitive products, we filter the detections to maintain only crops that maximize image quality and semantic diversity. Specifically, we compute the Laplacian variance (sharpness) for every crop