LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
Summary
This paper proposes a modality transfer task for Large Multimodal Models (LMMs) in GIS workflows and finds that current OpenAI LMMs struggle to transfer spatial information between image and text modalities, highlighting a critical bottleneck for autonomous GIS agents.
arXiv:2608.06948v1 Announce Type: new
Abstract: AI models are becoming increasingly adept at understanding and processing spatial information, thereby facilitating agentic problem-solving in spatial tasks and workflows. However, most of the research on their spatial capabilities (e.g., spatial reasoning) has focused on the textual modality as input and output. This contrasts with the human approach to GIS workflows, where text and visual modalities are often used together, interchangeably, and in a complementary manner. Thus, to truly achieve an automated GIS analysis pipeline or carry out human-designed GIS workflows, AI models --- Large Multimodal Models (LMMs) in particular --- need to be able to seamlessly transition between image- and text-based modalities that are traditionally used in such workflows. We present a modality transfer task that (1) asks an LMM to first describe an input image of colored squares in a regular grid, and (2) asks a new LMM instance to re-generate an image of the original spatial scene using the textual description output by the former model. This task quantifies the ability of LMMs to transfer spatial information between image and text modalities. Ultimately, by examining the modality transfer capability of LMMs through the lens of spatial information theory, this work highlights a critical bottleneck: achieving strong and robust geospatial understanding in LMMs requires rigorous, multi-modal alignment. Our results indicate that recent LMMs (here from OpenAI) still struggle with modality transfer, when tasked with re-generating an image of a simple spatial grid of color squares.
View Cached Full Text
Cached at: 08/10/26, 07:59 AM
# LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
Source: [https://arxiv.org/html/2608.06948](https://arxiv.org/html/2608.06948)
Graz University of Technology, Austria and University of Vienna, Austriamajic@tugraz\.athttps://orcid\.org/0000\-0002\-0834\-3791 Graz University of Technology, Austria zexian\.huang@tugraz\.athttps://orcid\.org/0000\-0003\-0270\-1604 Graz University of Technology, Austria franziska\.huebl@tugraz\.athttps://orcid\.org/0000\-0002\-3548\-2455 University of Vienna, Austriakrzysztof\.janowicz@univie\.ac\.athttps://orcid\.org/0009\-0003\-1968\-887X University of Liverpool, UKmeilin\.shi@liverpool\.ac\.ukhttps://orcid\.org/0000\-0001\-6039\-7810 University of Vienna, Austriamina\.karimi@univie\.ac\.athttps://orcid\.org/0000\-0003\-2521\-8164 University of Vienna, Austriazilong\.liu@univie\.ac\.athttps://orcid\.org/0000\-0002\-7699\-3366 University of Vienna, Austriaalexandra\.fortacz@univie\.ac\.athttps://orcid\.org/0009\-0007\-8370\-1572\\CopyrightIvan Majic, Zexian Huang, Franziska Huebl, Krzysztof Janowicz, Meilin Shi, Mina Karimi, Zilong Liu, and Alexandra Fortacz\\supplementdetails\[subcategory=Source Code\]Softwarehttps://github\.com/Geoinfo\-TUGraz/COSIT2026\_LMM\_modality\_transfer\\ccsdesc\[500\]Computing methodologies Artificial intelligence\\ccsdesc\[500\]Computing methodologies Intelligent agents\\ccsdesc\[500\]Information systems Geographic information systems\\EventEditorsSabine Timpf, Gabriele Filomena, Armand Kapaj, Rui Zhu, Nicholas Giudice, and Ed Manley\\EventNoEds6\\EventLongTitle17th International Conference on Spatial Information Theory \(COSIT 2026\)\\EventShortTitleCOSIT 2026\\EventAcronymCOSIT\\EventYear2026\\EventDateSeptember 22–25, 2026\\EventLocationYork, UK\\EventLogo\\SeriesVolume393\\ArticleNo14
Zexian HuangFranziska HüblKrzysztof JanowiczMeilin ShiMina KarimiZilong LiuAlexandra Fortacz\-Lazan
###### Abstract
AI models are becoming increasingly adept at understanding and processing spatial information, thereby facilitating agentic problem\-solving in spatial tasks and workflows\. However, most of the research on their spatial capabilities \(e\.g\., spatial reasoning\) has focused on the textual modality as input and output\. This contrasts with the human approach to GIS workflows, where text and visual modalities are often used together, interchangeably, and in a complementary manner\. Thus, to truly achieve an automated GIS analysis pipeline or carry out human\-designed GIS workflows, AI models — Large Multimodal Models \(LMMs\) in particular — need to be able to seamlessly transition between image\- and text\-based modalities that are traditionally used in such workflows\. We present a modality transfer task that \(1\) asks an LMM to first describe an input image of colored squares in a regular grid, and \(2\) asks a new LMM instance to re\-generate an image of the original spatial scene using the textual description output by the former model\. This task quantifies the ability of LMMs to transfer spatial information between image and text modalities\. Ultimately, by examining the modality transfer capability of LMMs through the lens of spatial information theory, this work highlights a critical bottleneck: achieving strong and robust geospatial understanding in LMMs requires rigorous, multi\-modal alignment\. Our results indicate that recent LMMs \(here from OpenAI\) still struggle with modality transfer, when tasked with re\-generating an image of a simple spatial grid of color squares\.
###### keywords:
Large Multimodal Model \(LMM\), Spatial Reasoning, GIS Agent, Modality Transfer, GeoAI
## 1Introduction
Typical GIS workflows inherently use both text and image modalities interchangeably, depending on the situation and the communication needs\. For example, two human GIS analysts who work side by side on a joint project may exchange textual information like feature attributes, but also discuss the semantic meaning of certain colored polygons by asking questions like*“what does this elongated purple shape represent”*\. If one or both of these GIS analysts were to be replaced by autonomous GIS agents, we argue that the same \(human\-expert\) level of understanding and seamless transition between both modalities of geographic information is a pre\-requisite for smooth, reliable, and deterministically repeatable operation\[scheider2015talk\]\.
Development of such intelligent GIS agents is a continuation of the long\-standing desire and need to automate GIS workflows\. Tasks such as information retrieval, data preparation, and data cleaning have always been resource\-hungry\. This is why many computational workflows and task\-specific GIS automations have been developed as plugins to GIS environments such as ArcGIS or QGIS\[Majic2019b\], or as standalone software tools\[Majic2019,Majic2021\]\. With the advent of generative AI and foundation models that are capable of addressing a variety of downstream tasks without specific pre\-training, the aspirations of autonomous GIS agents have also risen\[Li2025\]\. An intelligent GIS agent that can understand the data it is working with and autonomously adjust the workflows to the specific situation would be of great value\. Hence, such aGeoMachinahas been declared one of the potential common moonshots for the field of GeoAI\[Janowicz2020\]\. While there is substantial progress\[Li2025,akinboyewa2025gis\], some fundamental considerations regarding the multi\-modality of spatial information remain unaddressed\.
Thus, we bring forth an essential question regarding the underlying foundation model infrastructure and its readiness to fully autonomously handle inherently multi\-modal GIS workflows:*?how well can large multimodal models \(LMMs\) understand each other and can they transfer spatial information between the two modalities \(image and text\) without losing information and quality??*To this end, we have designed a modality transfer task that takes images of \(spatially arranged\) grids of colored cells \(the likes of which can be found in typical land cover maps\), prompts the LMM to textually describe the spatial configuration visible in the image, and then asks another model to generate an image based on this textual output\. In other words, the proposed task assesses the ability of LMMs to performimage→\\rightarrowtext→\\rightarrowimagemodality transfer\. Our case study applies this task on recent\-generation OpenAI models with different grid sizes and numbers of colors\.Thus, the contributions of this study are two\-fold:
- •We identify the lossless modality transfer as a fundamental requirement for many GIS tasks to be solved by autonomous GIS agents\.
- •We define a modality transfer task that can be used to benchmark the progress of spatial capabilities of LMMs in this context\.
Thereby, our work addresses the important challenge of aligning the progress of the state\-of\-the\-art foundational AI models with the context of GIScience by testing their ability to comprehend geographical information\.
The remainder of this paper is organized as follows\. In Section[2](https://arxiv.org/html/2608.06948#S2), we review the literature related to our study\. We propose our approach in Section[3](https://arxiv.org/html/2608.06948#S3)\. Then, the results are explained in Section[4](https://arxiv.org/html/2608.06948#S4)\. We discuss our finding in Section[5](https://arxiv.org/html/2608.06948#S5)\. Finally, Section[6](https://arxiv.org/html/2608.06948#S6)concludes the paper\.
## 2Related work
The rapid evolution of GeoAI has significantly advanced geographic knowledge discovery and spatial analysis\. Previous foundational work has highlighted the transformative potential of spatially explicit AI techniques in addressing complex geographic problems, moving beyond traditional analysis to intelligent, agentic workflows\[Janowicz2020,mai2023opportunities\]\. While they outline various fundamental challenges in GeoAI, in particular, spatial heterogeneity, scale dependencies, and the need for spatially explicit models, the specific challenges of*large multi\-modal*geographic reasoning remain largely unaddressed\. In contrast, single\-mode spatial reasoning has received increasing attention\[cohn\_et\_al2024,ji2025foundation,cohn2024can,gardelakosCanLargeReasoning2025\]\. Work has also focused on vision language models and their ability to answer spatial reasoning questions\[cheng2024spatialrgpt\]\. Recently, Xie et al\. demonstrated that LLMs still lack a deeper understanding of spatial relations but rely on linguistic patterns instead\[xie2025evaluating\]\.
Traditional geographic information and human\-driven GIS workflows are inherently multi\-modal\. Human analysts seamlessly blend visual information \(e\.g\., maps, satellite imagery, and topological configurations\) with textual information \(e\.g\., attributes, spatial queries, and descriptive metadata\)\. However, even recent GeoAI and foundation model research evaluates these modalities independently\[ji2025foundation\]\. Recent work has also shown that foundation models yield uneven success depending on which of these modalities the spatial reasoning task is presented in\[Majic2024\], showcasing there is a difference in spatial reasoning capabilities in different modalities\.Our work directly addresses this gap\.By introducing a framework to explicitly test the multi\-modal spatial reasoning capabilities of LMMs, we study the lossless transfer of spatial information between visual and textual modalities as a critical, unaddressed prerequisite for the development of fully autonomous GeoAI agents\.
Evaluating the true representation and reasoning capabilities of AI systems requires benchmarks that isolate genuine problem\-solving from mere pattern matching or data memorization\. A prominent example is the Abstract and Reasoning Corpus for Artificial General Intelligence \(ARC\-AGI\) introduced by Chollet et al\.\[chollet2024arc\]\. A guiding design philosophy of ARC\-AGI is to formulate tasks that are intuitive for humans yet highly challenging for AI models, thereby exposing gaps and weaknesses in an AI’s capacity to generalize abstract rules from limited examples without prior training\. Our proposed modality transfer task shares this core principle but targets a different aspect\. While ARC\-AGI focuses on abstract logical reasoning and skill acquisition within a single visual\-symbolic domain, our modality transfer task explicitly evaluates the*transferability and preservation of spatial information across disparate data modalities*\.
Human spatial cognition can effortlessly translate visual cues into language descriptions and vice versa\. For instance, one person can look at a visual grid and verbally describe a?red square situated in the top\-left corner?, and another one can accurately reproduce those visual cues solely based on the description\. Ourground\-truth image→\\rightarrowgenerated text→\\rightarrowgenerated imagepipeline isolates this exact translation\. By abstracting the task away from heavily prior GIS knowledge \(such as specific coordinate reference systems or complex vector geometries\) and utilizing colored spatial grids, we ensure a fair assessment\. Similar to ARC\-AGI, our task relies on basic spatial core priors \(grids, colors, and relative positioning\)\. However, instead of asking AI models to deduce a hidden logical rule, we test LMM’s capability to accurately translate spatial reality from one modality to another\.
## 3Approach
In order to assess the ability of the LMM to understand spatial information, we propose an approach that transfers the modality of a given spatial land\-cover\-like example from an image representation – into a textual representation – and back into an image representation \(i\.e\.,image→\\rightarrowgenerated text→\\rightarrowimagemodality transfer\)\. We call this the*modality transfer task*\. By performing it, we can get an estimation of how well the model really understands the provided spatial configuration\. We can also directly compare the final output \(generated image\) to the original input \(image\), as they are both in the same modality and, when the system performs completely correctly, should be identical\.
The overall approach to the LMM modality transfer task is carried out as a multi\-stage workflow designed to isolate, control, and quantify each step of the task\. As shown in Figure[1](https://arxiv.org/html/2608.06948#S3.F1), the workflow consists of the generation of input ground\-truth images \(i\.e\.,N×NN\\times Ngrids of colored rectangles\), an LMM prompting sequence \(image\-to\-text followed by text\-to\-image\), the systematic extraction of generated images into discrete matrices, and a quantitative evaluation utilizing perceptual color thresholding of color matrices, text\-based and image\-based Levenshtein Distance\[levenshtein1966binary\]and, Earth Mover’s Distance \(EMD\)\[rubner2000earth\]\.
Figure 1:A diagram showing the flow of the proposed modality transfer task\. Input images are generated in different configurations and batch sizes\. They are then used as an input to the Prompt 1, along with a textual prompt that directs the AI how to generate a suitable textual description\. The outputs of the Prompt 1 are used as inputs for the Prompt 2 where textual descriptions are used to generate new images\. Finally, the generated outputs of both Prompt 1 \(text\) and Prompt 2 \(images\) are evaluated against the original input images\.While Figure[1](https://arxiv.org/html/2608.06948#S3.F1)shows the logical flow of the workflow, the approach is implemented in Python programming language as a series of interactive Python notebooks\. The repository containing the code, input images, and LMM prompt outputs \(text and images\) is published in a freely accessible repository222[https://github\.com/Geoinfo\-TUGraz/COSIT2026\_LMM\_modality\_transfer\.git](https://github.com/Geoinfo-TUGraz/COSIT2026_LMM_modality_transfer.git)\.
### 3\.1Generation of input \(baseline\) images
The ground\-truth images consisting ofN×NN\\times Ngrids of colored rectangles are generated procedurally based on the Tableau color palette available in the Matplotlib Python library\. The five selected colors from this 10\-color palette are depicted in Figure[2](https://arxiv.org/html/2608.06948#S3.F2), while the perceptual differences between them are expressed using the CIEDE2000 \(ΔE00\\Delta E\_\{00\}\)\[Luo2001\]color difference formula which can express the perceived difference between two colors better than any measure based on the RGB or HSL color values\[melgosaTestingCIELABbasedColordifference2000\]\. These distinct colors can serve as semantic labels for discrete categorical GIS data, such as land use classes, ensuring that different categories are visually distinguishable and minimizing perceptual ambiguity\.
To evaluate varying degrees of spatial complexity, the grid sizeNNis incrementally scaled from55to1010, and the number of colors used was increased from33to44and55\(i\.e\., red, green, blue, orange, and cyan\), where each cell was randomly assigned one of the possible color values \(Figure[3](https://arxiv.org/html/2608.06948#S3.F3)\)\. Both the images and their corresponding exact spatial matrices are recorded to serve as the definitive baseline for all subsequent comparisons\.
Figure 2:The five selected named colors of the Tableau Palette available in the Python Matplotlib library and the differences between them expressed asΔE00\\Delta E\_\{00\}\.\(a\)5×5×35\\times 5\\times 3
\(b\)6×6×36\\times 6\\times 3
\(c\)7×7×37\\times 7\\times 3
\(d\)8×8×38\\times 8\\times 3
\(e\)9×9×39\\times 9\\times 3
\(f\)10×10×310\\times 10\\times 3
\(g\)5×5×45\\times 5\\times 4
\(h\)6×6×46\\times 6\\times 4
\(i\)7×7×47\\times 7\\times 4
\(j\)8×8×48\\times 8\\times 4
\(k\)9×9×49\\times 9\\times 4
\(l\)10×10×410\\times 10\\times 4
\(m\)5×5×55\\times 5\\times 5
\(n\)6×6×56\\times 6\\times 5
\(o\)7×7×57\\times 7\\times 5
\(p\)8×8×58\\times 8\\times 5
\(q\)9×9×59\\times 9\\times 5
\(r\)10×10×510\\times 10\\times 5
Figure 3:Examples of baseline images generated for this study\. The grid sizes range from N=5 on the left to N=10 on the right, and the number of colors randomly assigned to each cell range from N\_colors=3 in the first row to N\_colors=5 in the third row\.
### 3\.2Prompting LMMs
#### 3\.2\.1Prompt 1: Image\-to\-Text
As depicted in Figure[1](https://arxiv.org/html/2608.06948#S3.F1), generated input images are provided to an LMM\. The accompanying prompt directs the LMM to act as a spatial extraction agent, tasking it to generate a machine\-readable textual representation of the spatial layout that details the domain size, indexing system, and the coordinate mapping of the colored cells\. The exact prompt was written as follows:
Theimageshowscoloredsquaresina
gridpattern\.Colorsrepresentlanduseclasses\.Explainthisimage
inatextthatcanbeusedbyotherLMMsforspatialrepresentation
andreasoningtasks\.
As a response, the LMM would return the generated textual description of the input image\. Figure[4](https://arxiv.org/html/2608.06948#S3.F4)shows an example of the prompt response when image shown in Figure[3\(a\)](https://arxiv.org/html/2608.06948#S3.F3.sf1)was used as an input\.
Figure 4:Textual description of the image shown in Figure[3\(a\)](https://arxiv.org/html/2608.06948#S3.F3.sf1), generated with the OpenAI’s GPT5 model\.
#### 3\.2\.2Prompt 2: Text\-to\-Image
In the subsequent step, the textual description output by Prompt11is used as an input for Prompt22where an image\-generation variant of an LMM is prompted\. No additional system prompts or directions were used, as the input textual description is usually already quite detailed and the image\-generation LMM understands that by default its task is to generate an image output based on the textual input\. Figure[5](https://arxiv.org/html/2608.06948#S3.F5)shows the generated output images that correspond to the input images from Figure[3](https://arxiv.org/html/2608.06948#S3.F3)\.
\(a\)5×5×35\\times 5\\times 3
\(b\)6×6×36\\times 6\\times 3
\(c\)7×7×37\\times 7\\times 3
\(d\)8×8×38\\times 8\\times 3
\(e\)9×9×39\\times 9\\times 3
\(f\)10×10×310\\times 10\\times 3
\(g\)5×5×45\\times 5\\times 4
\(h\)6×6×46\\times 6\\times 4
\(i\)7×7×47\\times 7\\times 4
\(j\)8×8×48\\times 8\\times 4
\(k\)9×9×49\\times 9\\times 4
\(l\)10×10×410\\times 10\\times 4
\(m\)5×5×55\\times 5\\times 5
\(n\)6×6×56\\times 6\\times 5
\(o\)7×7×57\\times 7\\times 5
\(p\)8×8×58\\times 8\\times 5
\(q\)9×9×59\\times 9\\times 5
\(r\)10×10×510\\times 10\\times 5
Figure 5:Examples of LMM generated output images based on input baseline images from Figure[3](https://arxiv.org/html/2608.06948#S3.F3)after they have been through the image\-to\-text transfer with Prompt 1 and subsequently through the text\-to\-image transfer with Prompt 2\.
### 3\.3Processing of LMM outputs
To be able to compare the outputs of Prompt 1 and Prompt 2 to the input images, some preprocessing steps are required to sample and extract the relevant information and convert it into the format that allows the evaluation measures to be calculated\. For this purpose, we converted all three color grid depictions \- input images, generated texts, and generated images \- into 2\-dimensional arrays of color names\. This required separate processing steps for the textual and image outputs of the LMM prompts\.
The textual description outputs of Prompt 1 \(image\-to\-text\) usually contained longer natural language descriptions of the grids visible in the input images that explained the size, format, colors used, color meanings, clusters of the same colors, and sometimes even topological relations between individual clusters\. However, the majority of outputs also included some kind of matrix encoding of the input grid and this is the information that we have extracted and standardized into 2\-dimensional arrays of colors where each color is represented by a single letter\.
The outputs of Prompt 2 \(text\-to\-image\) are in an image modality and require a different kind of processing to convert the information into a 2D array\. Although it is possible to perform this step with yet another LMM prompt, this would be subject to the same uncertainty as Prompt 1, as it would require another image\-to\-text prompt and would not seem reasonable in a workflow that is intended to evaluate LMMs in the first place\. Thus, we have created a small application that allows user to sample colors from the cells of output images using a graphical user interface \(Figure[6](https://arxiv.org/html/2608.06948#S3.F6)\)\. It should be noted that output images can also contained highly distorted grids that cannot be sampled and have to be omitted completely \(see Table[1](https://arxiv.org/html/2608.06948#S3.T1)\)\. After all output images from one directory have been sampled, the application converts the sampled points and colors into 2D arrays\.
Figure 6:Graphical user interface of the output image color\-sampling application\. Each point represents a location where the user has sampled the cell color in order to capture the grid layout and colors of an image generated by an LMM\.
### 3\.4Evaluation of LMM spatial hallucination
Evaluating the accuracy of the modality transfer requires quantifying both spatial and attribute degradation\. Generative image models rarely output exact, deterministic hex color codes, often producing slight gradients or perceptual variations\. Facing this challenge, we apply a strict CIEDE2000 \(ΔE00\\Delta E\_\{00\}\) thresholding function to rigorously evaluate the extracted visual matrices against the discrete ground\-truth grids\. Let𝒫=\{p1,p2,…,pk\}\\mathcal\{P\}=\\\{p\_\{1\},p\_\{2\},\\dots,p\_\{k\}\\\}denote the ground\-truth color palette\. For each generated pixel colorcc, this heuristic calculates the perceptual distanceΔE00\(c,pi\)\\Delta E\_\{00\}\(c,p\_\{i\}\)to every palette color\. Generated colors falling within a defined perceptual thresholdτ\\tauare mapped to their nearest ground\-truth categorical class, while those exceeding the threshold are penalized and registered as out\-of\-bounds attribute hallucinations \(hh\)\. The assigned classllfor a generated colorccis defined as:
l=\{argminpi∈𝒫ΔE00\(c,pi\)ifminΔE00\(c,pi\)≤τhotherwisel=\\begin\{cases\}\\arg\\min\_\{p\_\{i\}\\in\\mathcal\{P\}\}\\Delta E\_\{00\}\(c,p\_\{i\}\)&\\text\{if \}\\min\\Delta E\_\{00\}\(c,p\_\{i\}\)\\leq\\tau\\\\ h&\\text\{otherwise\}\\end\{cases\}
#### 3\.4\.11D Levenshtein Distance
Following their categorical normalization into discrete matrices, letMMrepresent the ground\-truth matrix andM^\\hat\{M\}represent the generated matrix\. To facilitate the evaluation of spatial determinism, these 2D spatial structures are flattened into 1D sequential strings,SSandS^\\hat\{S\}, such thatS=\(s1,s2,…,sN2\)S=\(s\_\{1\},s\_\{2\},\\dots,s\_\{N^\{2\}\}\)\. Subsequently, the Levenshtein Distance\[levenshtein1966binary\]is computed between the generated sequences and the ground truth to strictly quantify spatial loss\. LetℒS,S^\(\|S\|,\|S^\|\)\\mathcal\{L\}\_\{S,\\hat\{S\}\}\(\|S\|,\|\\hat\{S\}\|\)denote the distance between the two sequences, defined recursively to calculate the minimum number of mathematical edit operations required to reconstruct the original input sequence:
ℒS,S^\(i,j\)=\{max\(i,j\)ifmin\(i,j\)=0min\{ℒS,S^\(i−1,j\)\+1ℒS,S^\(i,j−1\)\+1ℒS,S^\(i−1,j−1\)\+𝕀\(si≠s^j\),otherwise\\mathcal\{L\}\_\{S,\\hat\{S\}\}\(i,j\)=\\begin\{cases\}\\max\(i,j\)&\\text\{if \}\\min\(i,j\)=0\\\\ \\min\\begin\{cases\}\\mathcal\{L\}\_\{S,\\hat\{S\}\}\(i\-1,j\)\+1\\\\ \\mathcal\{L\}\_\{S,\\hat\{S\}\}\(i,j\-1\)\+1\\\\ \\mathcal\{L\}\_\{S,\\hat\{S\}\}\(i\-1,j\-1\)\+\\mathbb\{I\}\(s\_\{i\}\\neq\\hat\{s\}\_\{j\}\),\\end\{cases\}&\\text\{otherwise\}\\end\{cases\}where𝕀\\mathbb\{I\}is the indicator function that equals11when the characters differ and0when they match\. Within this spatial context, the operations directly translate to model errors: insertions \(ℒS,S^\(i,j−1\)\+1\\mathcal\{L\}\_\{S,\\hat\{S\}\}\(i,j\-1\)\+1\) represent hallucinated blocks, deletions \(ℒS,S^\(i−1,j\)\+1\\mathcal\{L\}\_\{S,\\hat\{S\}\}\(i\-1,j\)\+1\) represent missing blocks, and substitutions \(𝕀\(si≠s^j\)=1\\mathbb\{I\}\(s\_\{i\}\\neq\\hat\{s\}\_\{j\}\)=1\) represent incorrectly placed or colored blocks\. If the input and output images are identical \(given the allowed perceptual difference threshold for colorsτ\\tau\), the Levenshtein Distance between them is zero, i\.e\.,ℒS,S^\(\|S\|,\|S^\|\)=0\\mathcal\{L\}\_\{S,\\hat\{S\}\}\(\|S\|,\|\\hat\{S\}\|\)=0\.
#### 3\.4\.22D Spatial Earth Mover’s Distance
While the 1D Levenshtein sequence evaluation effectively captures topological errors of generated images and texts, this evaluation is inherently insensitive to the geometric magnitude of spatial displacements\. To quantify the true 2D geographic loss and penalize severe spatial hallucinations, we formulate a discrete, class\-wise Spatial Earth Mover’s Distance \(EMD\)\[rubner2000earth\]as an optimal transport problem\.
Let𝒰\\mathcal\{U\}denote the union of all categorical labels present in both the ground\-truth matrixMMand the generated matrixM^\\hat\{M\}\. For each labell∈𝒰l\\in\\mathcal\{U\}, we extract the sets of 2D Cartesian coordinates representing the spatial locations of those specific class blocks:Cl=\{\(x,y\)∣Mx,y=l\}C\_\{l\}=\\\{\(x,y\)\\mid M\_\{x,y\}=l\\\}andC^l=\{\(x,y\)∣M^x,y=l\}\\hat\{C\}\_\{l\}=\\\{\(x,y\)\\mid\\hat\{M\}\_\{x,y\}=l\\\}\. Letnl=\|Cl\|n\_\{l\}=\|C\_\{l\}\|andn^l=\|C^l\|\\hat\{n\}\_\{l\}=\|\\hat\{C\}\_\{l\}\|denote the cardinality of these sets\.
To account for unmapped features, in particular deletions where the LMM fails to generate a required block, or insertions where the model hallucinates a new, out\-of\-bounds color, a maximum spatial penaltyρ=H2\+W2\\rho=\\sqrt\{H^\{2\}\+W^\{2\}\}is defined, which represents the maximum diagonal distance of theH×WH\\times Wspatial grid, whereHHandWWstand for height and width of the grid\.
In particular, for a given classll, the spatial transport costW\(Cl,C^l\)W\(C\_\{l\},\\hat\{C\}\_\{l\}\)computes the minimum Euclidean distance required to align the generated blocks with the ground truth, penalized by any cardinality mismatch:
W\(Cl,C^l\)=minπ∑i=1min\(nl,n^l\)‖ci−c^π\(i\)‖2\+\|nl−n^l\|ρ,W\(C\_\{l\},\\hat\{C\}\_\{l\}\)=\\min\_\{\\pi\}\\sum\_\{i=1\}^\{\\min\(n\_\{l\},\\hat\{n\}\_\{l\}\)\}\\\|c\_\{i\}\-\\hat\{c\}\_\{\\pi\(i\)\}\\\|\_\{2\}\+\|n\_\{l\}\-\\hat\{n\}\_\{l\}\|\\rho,whereπ\\pirepresents the optimal injective mapping from the smaller coordinate set to the larger coordinate set, solved via the Hungarian algorithm \(linear sum assignment\), and∥⋅∥2\\\|\\cdot\\\|\_\{2\}denotes the standardL2L\_\{2\}Euclidean distance\. Finally, the aggregated Spatial EMD is divided by the total number of valid ground\-truth blocks,Ngt=∑l∈𝒰nlN\_\{gt\}=\\sum\_\{l\\in\\mathcal\{U\}\}n\_\{l\}, which ensures that the final metric represents the average physical pixel shift per correct block:
EMD=1Ngt∑l∈𝒰W\(Cl,C^l\)\.\\text\{EMD\}=\\frac\{1\}\{N\_\{gt\}\}\\sum\_\{l\\in\\mathcal\{U\}\}W\(C\_\{l\},\\hat\{C\}\_\{l\}\)\.
#### 3\.4\.3Interpretation of Metrics
To systematically evaluate the modality transfer pipeline and diagnose the exact nature of spatial degradation, we analyze the interplay between the 1D sequence error \(i\.e\., 1D Levenshtein\) and the 2D geometric error \(i\.e\., 2D spatial EMD\)\. While the Levenshtein distance effectively measures whether the generative model understood the topological shape and categorical sequence of the prompt, it is highly sensitive to minor structural shifts and heavily penalizes simple translations\. Conversely, the spatial EMD quantifies the absolute geographic magnitude of these displacements, indicating whether the model adheres to strict coordinate geometry\.
By evaluating these two metrics, as summarized in Table[1](https://arxiv.org/html/2608.06948#S3.T1), we establish a diagnostic framework to classify the spatial reasoning failure of LMMs\. This decoupled approach allows us to definitively distinguish between minor spatial coordinate drifts, where the model maintains relative topology but fails at absolute positioning and catastrophic generative breakdowns, where all spatial context is completely hallucinated\.
Table 1:Spatial Error Diagnostic Matrix based on Levenshtein Distance and Spatial EMD\.
## 4Results
We performed an experiment where recent generation OpenAI LMMs were evaluated with the modality transfer task proposed in Section[3](https://arxiv.org/html/2608.06948#S3)\. We first generated input images with the grid sizes \(N×NN\\times N\) ranging fromN=5N=5toN=10N=10, and the number of colors ranging fromNcolors=3N\_\{colors\}=3toNcolors=5N\_\{colors\}=5\. For each parameter combination \(e\.g\.,5×5×35\\times 5\\times 3\),5050distinct baseline images were created, resulting in a total of900900input images\. For Prompt 1 \(image\-to\-text\), we utilize OpenAI’sgpt\-5model333[https://developers\.openai\.com/api/docs/models/gpt\-5](https://developers.openai.com/api/docs/models/gpt-5)to generate textual descriptions of input images withtemperature=1temperature=1and default token size\. For Prompt 2 \(text\-to\-image\), we use OpenAI’sgpt\-image\-1model444[https://developers\.openai\.com/api/docs/models/gpt\-image\-1](https://developers.openai.com/api/docs/models/gpt-image-1)to generate new images based on the outputs of Prompt 1\. We set the parameters for Prompt 2 asimage\_size=1024×1024image\\\_size=1024\\times 1024\(pixels\) andquality=highquality=high\. It is important to note that OpenAI’s API is a paid service and image generation prompts may accumulate particularly high costs\. We executed prompts forNcolors=3N\_\{colors\}=3cases by sending API calls directly from our Python scripts\. For all other cases, we executed prompts in batches directly on the OpenAI’s API dashboard in order to lower the costs and save resources\. Total cost of the entire experiment —900900gpt\-5prompts,900900gpt\-image\-1prompts, as well as a number of test prompts — was approximately200200USD\.
\(a\)Evaluation metrics for spatial grids utilizing a3\-color palette\(i\.e\., red, green, blue\)\. At this lower number of grid and color complexity, the models maintain relatively stable topological and geometric coherence at smaller grid sizes \(N≤7N\\leq 7\)\. While slight misalignments occur, catastrophic geometric failures \(high EMD\) remain infrequent until the grid size reachesN≥9N\\geq 9\.
\(b\)Evaluation metrics for spatial grids utilizing a4\-color palette\(i\.e\., red, green, blue, orange\)\. The introduction of a fourth categorical color significantly destabilizes the modality transfer\. Notably, atN≥8N\\geq 8, the models exhibit simultaneous, severe spikes in both sequence error \(Img Levenshtein\) and geometric displacement \(Spatial EMD\), indicating complete losses of spatial context within individual samples\.
\(a\)Evaluation metrics for spatial grids utilizing a5\-color palette\(i\.e\., red, green, blue, orange, cyan\)\. Maximum visual complexity results in immediate and significant modality transfer breakdowns across nearly all grid sizes\. The baseline vision\-to\-text parsing \(Txt Levenshtein\) exhibits frequent failures early on, subsequently corrupting the text\-to\-image generation phase and driving continuous catastrophic geometric errors\.Figure 8:Detailed evaluation of modality transfer performance across varying grid sizes \(N=5N=5toN=10N=10\) and color palette complexities \(3, 4, and 5 colors\)\. The primary y\-axis \(left\) displays the normalized 1D sequence error \(Levenshtein distance divided byN2N^\{2\}\), contrasting the intermediate text description error \(blue dotted line\) against the final generated image error \(red shaded line\)\. The secondary y\-axis \(right\) displays the 2D geometric error via normalized Spatial EMD \(green solid line\)\. Across all configurations, scaling the grid dimensions and increasing the number of categorical colors correlates with an exponential rise in spatial hallucinations and loss of geometric determinism\.Figure[8](https://arxiv.org/html/2608.06948#S4.F8)shows the evaluation of each individual input image \(Image ID axis\) with the Normalized Sequential Error \(Levenshtein Distance divided by grid sizeN2N^\{2\}\) and the Normalized Geometric Error \(Spatial EMD divided by grid diagonalN2N\\sqrt\{2\}\) measures\. The tested models demonstrate a high degree of variability dependent on both the scale of the grid \(NN\) and the color complexity\. In the simplest configuration, testing with 3 color palette \(Figure[7\(a\)](https://arxiv.org/html/2608.06948#S4.F7.sf1)\), the vision\-to\-text sequence error remains relatively low forN≤6N\\leq 6\. However, it exhibits a sharp capability drop atN=7N=7, driving downstream errors that peak atN=9N=9\. Expanding the palette to 4 colors \(Figure[9\(b\)](https://arxiv.org/html/2608.06948#S4.F9.sf2)\) complicates the representational mapping and accelerates this breakdown; baseline vision parsing degrades much earlier, compounding sequence penalties and geometric misalignments\. At maximum complexity with 5 colors \(Figure[9\(c\)](https://arxiv.org/html/2608.06948#S4.F9.sf3)\), the tested LMMs experience immediate catastrophic failures, where converging high sequence penalties and very high Spatial EMD values indicate a total loss of topological and geometric structure\. Across all configurations, scaling spatial density consistently triggers an abrupt loss of geometric determinism\.
\(a\)3\-color palette
\(b\)4\-color palette
\(c\)5\-color palette
Figure 9:Scaling behavior of average spatial reasoning errors across varying grid dimensions \(N=5N=5toN=10N=10\)\. The left y\-axis displays the normalized 1D sequence errors \(Levenshtein distance divided byN2N^\{2\}\), tracking topological failures in the text description phase \(blue dashed line\) and image generation phase \(red dashed line\)\. The right y\-axis tracks 2D geometric error via normalized Spatial EMD \(green solid line\)\. Individual figures show different trends in these measures when performing modality transfer for cases with 3\(a\), 4\(b\), and 5 colors\(c\)\.Figure[9](https://arxiv.org/html/2608.06948#S4.F9)shows the aggregated average measures for each grid size and color configuration to highlight the effect of increasing spatial complexity on the performance of the LMM transfer task\. Overall, the average scaling trends also identify a capability drop after a certain point\. For the 3\-color palette \(Figure[9\(a\)](https://arxiv.org/html/2608.06948#S4.F9.sf1)\), this threshold occurs abruptly atN=7N=7, where the textual encodings begin to fail, cascading into significantly higher image generation errors\. This phenomenon is exacerbated in the 4\-color palette \(Figure[9\(b\)](https://arxiv.org/html/2608.06948#S4.F9.sf2)\), where the modality transfer success destabilizes earlier and more severely, resulting in a steady, proportional increase in both sequence and geometric errors as the grid size scales\. At the maximum complexity of 5\-color palette \(Figure[9\(c\)](https://arxiv.org/html/2608.06948#S4.F9.sf3)\), the models exhibit systemic failure\. The initial image\-to\-text extraction phase \(Txt Levenshtein\) incurs massive errors at grid size atN=7N=7\. As the foundational spatial description is intrinsically corrupted, the subsequent text\-to\-image generation is fed hallucinated or highly disjointed spatial coordinates\. This results in maximum\-penalty geometric displacements \(Spatial EMD\) peaking consistently acrossN≥7N\\geq 7, demonstrating that the models entirely lost the capacity to deterministically map or reconstruct the original spatial context\.
Figure[10](https://arxiv.org/html/2608.06948#S4.F10)places the individual cases into four main evaluation categories introduced in Table[1](https://arxiv.org/html/2608.06948#S3.T1), namely the Minor Attribute Error, Slight Misalignment \(Topology Failure\), Severe Hallucination \(Catastrophic Failure\), and Geometric Outlier \(Shape Distortion\)\. Each instance of an input image \- generated text \- and generated image \(e\.g\.,5×5×35\\times 5\\times 3\- img\_1\) is plotted as a single point using the Normalizes Sequence Error and Normalized Geometric Error as the axes\. For the 3\-color palette, the majority of points cluster in the bottom\-left quadrant \(Minor Attribute Error\), indicating successful modality transfer at smaller grid scales\. The 4\-color palette \(Figure[10\(b\)](https://arxiv.org/html/2608.06948#S4.F10.sf2)\) induces broader dispersion, with a noticeable migration ofN≥8N\\geq 8data points into the right hemisphere \(Slight Misalignment\), reflecting topological failures where sequences are corrupted despite partially intact geometries\. At 5 colors \(Figure[10\(c\)](https://arxiv.org/html/2608.06948#S4.F10.sf3)\), the distribution is severely skewed toward the top\-right quadrant \(Severe Hallucination\)\. This dense clustering highlights a catastrophic failure in modality transfer, where the model simultaneously loses both sequential topology and geometric precision\. Overall, the progression visually demonstrates the LMM’s failure trajectory as both grid scale and categorical color density overwhelm its spatial reasoning capabilities\.
\(a\)3\-color palette
\(b\)4\-color palette
\(c\)5\-color palette
Figure 10:Diagnostic scatter matrices mapping 1D sequence error \(Normalized Levenshtein, x\-axis\) against 2D geometric error \(Normalized Spatial EMD, y\-axis\)\. The spatial distribution patterns directly correspond to the diagnostic categories defined in Table[1](https://arxiv.org/html/2608.06948#S3.T1)\. Individual figures show diagnostic matrices for different cases with 3\(a\), 4\(b\), and 5 colors\(c\)\.
## 5Discussion
While the overall results of the study are presented in Section[4](https://arxiv.org/html/2608.06948#S4), some interesting insights that were observed during the experiments are not visible from the figures and information presented there\. In the following, we discuss our observations and thoughts regarding the proposed approach and the LMM modality transfer\. Namely, we discuss their issues in performing image\-to\-text prompts and the anomalies we observed in the results of text\-to\-image prompts\. We also provide further interpretation of the diagnostic analysis of modality transfer and lay out the limitations of our approach and case study\.
### 5\.1Issues in Prompt 1 \(Image\-to\-Text\) outputs
We intentionally defined the instructions to LMM in Prompt 1 loosely, not to affect the spatial comprehension of the LMM with our prompt engineering\. That said, the outputs for our initial input images of5×55\\times 5grids with33colors seemed quite uniform as the model would briefly describe the overall task in natural language and then provide some kind of matrix representation of the grid in the input image \- often with single letters representing different colors \(see the example response in Figure[4](https://arxiv.org/html/2608.06948#S3.F4)\)\. However, when the complexity of the input grid would increase through higher cell or color count, we noticed that textual responses would sometimes not include the full grid representation at all and sometimes only include the first few rows with the note such as*“If you need a lossless, per\-cell matrix for this exact image, ask for per\-cell encoding and I will return a row\-major 2D array”*\. We noticed that this effect started occuring more often from theN=7N=7and upwards – Figure[11](https://arxiv.org/html/2608.06948#S5.F11)show the number of Prompt 1 responses per configuration \(out of 50\) where the explicit grid is missing\.
Figure 11:Bars show the number of invalid Prompt 1 \(img\-to\-text\) responses where the explicitly defined grid of colors is missing from the generated textual description\. Each bar shows this value for oneN×N×NcolorsN\\times N\\times N\_\{colors\}configuration where a total of 50 prompts were executed\.If we compare this with the evaluation of the higher dimension grid cases \(N=8,9,10N=8,9,10\) in Figure[8](https://arxiv.org/html/2608.06948#S4.F8), we can notice that the images generated using such textual descriptions \(i\.e\., cases where Txt Levenshtein=1=1and Img Levevenshtein<1<1\) are still able to better represent the input image\. This is because the LMM is able to use the other natural language description from the Prompt 1 output to somewhat successfully recreate the input image, even if the grid configuration and colors are not excplicitly specified\.
### 5\.2Generative Anomalies in Text\-to\-Image Generation
After the LMM has generated output images with Prompt 2 \(text\-to\-image\), manual processing of these images was needed to sample their grids and colors \(see Section[3\.3](https://arxiv.org/html/2608.06948#S3.SS3)\)\. Our goal in sampling was to rely on clearly defined criteria but also include as many grids as possible, where feasible\. Thus, most grids with valid color schemes could be sampled and included in further evaluation\.
In our criteria, we assume that a valid color scheme is given when one clearly defined and closed cell \(no matter what shape\) is filled with a single color or uniform coloring\. Therefore, grids with white or unfilled cells were also taken into account \(see Figures[12\(f\)](https://arxiv.org/html/2608.06948#S5.F12.sf6)and[12\(g\)](https://arxiv.org/html/2608.06948#S5.F12.sf7)\)\. This does not affect the evaluation, as such cases appear to be incorrect anyway\. Examples where this is not the case \(i\.e\., color scheme is invalid\) are shown in Figures[12\(d\)](https://arxiv.org/html/2608.06948#S5.F12.sf4),[12\(e\)](https://arxiv.org/html/2608.06948#S5.F12.sf5),[12\(h\)](https://arxiv.org/html/2608.06948#S5.F12.sf8), and[12\(j\)](https://arxiv.org/html/2608.06948#S5.F12.sf10)\. Such cases were ommitted from further processing as cell colors could not be sampled and they were assigned maximum errors in the evaluation\.
Borderline cases where grids are cut off \(see Figure[12\(f\)](https://arxiv.org/html/2608.06948#S5.F12.sf6)\) have been included, as well as grids with merged cells that follow the lines of the otherwise valid rectangular grid \(see Figure[12\(g\)](https://arxiv.org/html/2608.06948#S5.F12.sf7)\), as long as cells are closed and clearly defined\. In contrast to grids with merged cells of other shapes like shown in Figure[12\(c\)](https://arxiv.org/html/2608.06948#S5.F12.sf3)or grids with interrupted grid lines as shown in Figure[12\(h\)](https://arxiv.org/html/2608.06948#S5.F12.sf8)\. If a grid appears to have no visible lines as shown in Figure[12\(b\)](https://arxiv.org/html/2608.06948#S5.F12.sf2)it is a borderline case as well\. However, grids that do not have neighbouring cells of equal color and that do not have grid lines are still sampled as valid grids\. Nevertheless, the proposed solution is to treat neighbouring cells with the same color as merged cells in order to reflect this in the sampled data\. Merged cells are generally sampled once, with the cells being captured in the direction from top left to bottom right at the time of their initial appearance\. Grids like shown in Figure[12\(g\)](https://arxiv.org/html/2608.06948#S5.F12.sf7)are therefore treated as invalid in the evaluation and receive higher error scores\. However, grids with irregularly shaped cells could not be taken into account within the sampling process, as no clear division into rows and columns is possible here\. Figure[12\(a\)](https://arxiv.org/html/2608.06948#S5.F12.sf1)shows an example of an invalid grid with valid colors which had to be omitted for further evaluation\.
Clearly, grids lacking meaningful semantic geometry, valid grid structures, or consistent coloring could not be encoded into discrete 2D grids\. As these severe generative failures represent a complete loss of spatial determinism, they were deemed entirely invalid and had to be omitted from the quantitative analysis \(see Figures[12\(i\)](https://arxiv.org/html/2608.06948#S5.F12.sf9)and[12\(j\)](https://arxiv.org/html/2608.06948#S5.F12.sf10)\)\.
\(a\)target grid size:9×9×49\\times 9\\times 4; invalid grid with valid colors: different shapes;omitted\.
\(b\)target grid size:10×10×410\\times 10\\times 4; invalid grid with valid colors: no grid lines;included\(as merged cells\)\.
\(c\)target grid size:10×10×410\\times 10\\times 4; invalid grid with valid colors: merged and covered cells, curved grid line;omitted\.
\(d\)target grid size:6×6×46\\times 6\\times 4; valid grid with invalid colors: diagonal color mix;omitted\.
\(e\)target grid size:7×7×47\\times 7\\times 4; valid grid with invalid colors: river\-like structure and dots;omitted\.
\(f\)target grid size:8×8×48\\times 8\\times 4; valid grid with invalid colors: empty/white cells;included\.
\(g\)target grid size:6×6×36\\times 6\\times 3; invalid grid and colors: merged and empty/white cells;included\.
\(h\)target grid size:6×6×56\\times 6\\times 5; invalid grid and colors: interrupted grid and colored symbols within cells;omitted\.
\(i\)target grid size:9×9×59\\times 9\\times 5; invalid grid and colors: missing grid visualisation;omitted\.
\(j\)target grid size:9×9×59\\times 9\\times 5; invalid grid and colors: different shapes and color blurring;omitted\.
Figure 12:The following examples illustrate the invalid results of text\-to\-image generation\. The target grid sizes, validity, and decision of inclusion or omittance in the process of image sampling \(manual color capturing\) are indicated below each image\.
### 5\.3Diagnostic Analysis of Modality Transfer
By analysing the LMM performance through the lens of Spatial Error Diagnostic Matrix \(Figure[10](https://arxiv.org/html/2608.06948#S4.F10), and Table[1](https://arxiv.org/html/2608.06948#S3.T1)\), we successfully categorize the nature of these breakdowns\. In low\-complexity scenarios \(Figure[10\(a\)](https://arxiv.org/html/2608.06948#S4.F10.sf1)\), the grid samples mostly aggregate in the bottom left quadrant \(Minor Attribute Eorror, Low Lev\. and Low EMD\)\. This establishes an acceptable baseline understanding of spatial arrangement, committing several minor localized color substitutions while preserving the overarching geometric shape\.
As the grid and color complexity increases, in a 4\-color palette \(Figure[10\(b\)](https://arxiv.org/html/2608.06948#S4.F10.sf2)\), the grid samples disperse toward the first \(High Lev\. and Low EMD\) and fourth \(High Lev\. and High EMD\) quadrants, implying a failure of topology and showing hints of model hallucination\. In this stage, while the LMMs can grasp the general colors of grid but fail to anchor them into correct, absolute coordinate geometries\. The transition to the 5\-color palette \(Figure[10\(c\)](https://arxiv.org/html/2608.06948#S4.F10.sf3)\) shows a descent into catastrophic spatial failure\. The dense grid clustering extending into the top\-right quadrant \(Severe Hallucination, High Lev\. and High EMD\) indicates a complete collapse of cross\-modal translation\. In these cases, the models fail to preserve both the 1D sequential order and the 2D geometric boundaries, resulting in images where blocks are either entirely omitted, scattered randomly, or drawn using out\-of\-bounds categorical colors\.
### 5\.4Limitations
While our experimental design establishes a quantifiable baseline for spatial modality transfer, several limitations need to be discussed\. Currently, our empirical evaluation is limited to a narrow selection of proprietary flagship models, specifically recent iterations from OpenAI \(gpt\-5andgpt\-image\-1\)\. While there is little reason to expect another LMM would suddenly perform this task perfectly, other models need to be tested to gain a wholistic overview of the current state\-of\-the\-art\. Additionally, the prompts in this study were executed strictly under zero\-shot conditions, leaving the potential performance gains of few\-shot prompting or domain\-specific fine\-tuning unexplored\. It is thus possible that a few\-shot performance would show significant improvements, but it is questionable whether it would allow for a seamless experience of working with GIS agents if every communication step between the user and the agent has to be repeated several times — at least when it involves different modalities\.
In addition to being tested in a zero\-short setting, our prompts have intentionally only vaguely defined the task at hand\. This has possibly led to some of the observed issues such as incomplete or incorrectly structured grids\. The model would likely perform better with well defined prompts where the valid output format and execution steps are explicitly defined\. We believe that future research on agentic GeoAI calls for a systematic investigation of the effects of vague versus well\-structured prompts on the agents’ success\.
Furthermore, although the procedurally generatedN×NN\\times Ncategorical grids offer a grounded and robust baseline, they represent a highly abstracted and relatively simplistic spatial reasoning task\. This works well to expose the fundamental issue of current LMMs which lose spatial information through modality transfer, but is not a good representation of a full\-scale GIS workflow\. Because foundation models are trained on real\-world data \(i\.e\., photographs as well as textual descriptions of spatial scenes\) and not grids of colored squares, they seem to struggle to generalize from context\-rich real world vision\-reasoning tasks\[cheng2024spatialrgpt\]to simpler tasks like the one presented here\. Thus, it remains to be investigated if LMMs would perform better if they were tested on more complex and well contextualized workflows embedded into a full\-scale GIS project environment\. If the modality transfer was not performed in isolation on simple grid images, but on the real\-world land cover maps in a software like ESRI ArcGIS, irregular shapes and additional information about them would make the task more complex, but the added context may also improve the comprehension of the LMM\.
Another limitation within our current approach is the GUI\-based sampling procedure in the evaluation of the generated images\. In its current format, this step is not scalable to larger sets of generated images and remains vulnerable to human errors and subjective interpretation of the operator\. For those reasons, it would be better to replace this step with an automated computer vision approach in the future to improve reproducibility and usability\. We also only provide the joint evaluation of both modality transfer steps, while it may be informative to consider them in isolation – i\.e\., use ground truth input text for the text\-to\-image prompt\. This would allow for a more detailed and objective evaluation of individual modality transfers without allowing the errors in the output of the first step to influence the second modality transfer step\.
## 6Conclusion
This study focuses on the fundamental issue of LMMs losing geographical information simply due to transferring it from textual to image modality or vice\-versa\. Anecdotally, this can often be observed when an LMM is asked to generate an image based on a textual description or to textually describe an input image \(e\.g\., a map\), and the results returned are imperfect\. We argue that a \(near\) lossless transfer of geographical information between text and image modalities is a pre\-requisite for enabling the development of autonomous GIS agents that can adequately perform GIS workflows\.
To this end, we propose an LMM modality transfer task where a set of input images, here regular grids of colored squares, are to be transferred from image to text modality with Prompt 1 and then, based on the textual description outputs of Prompt 1, again be transferred from text to image modality in Prompt 2\. This represents a simplified example of working with a land\-cover dataset in a GIS workflow\. Both outputs of Prompt 1 \(generated text\) and Prompt 2 \(generated image\) are then evaluated against the input image to quantify the amount of information lost during modality transfer\.
We executed the modality transfer task on two recent OpenAI models \(gpt\-5 for Prompt 1 and gpt\-image\-1 for Prompt 2\) using 900 input image with increasing grid sizes \(5×55\\times 5to10×1010\\times 10\) and number of colors \(3 to 5\)\. Our results demonstrate that, while LMMs may process vast prior core knowledge learned from large image or language corpus, they still suffer from spatial information loss and hallucination during cross\-modal translation, especially on inputs with larger grids, proving that lossless modality transfer remains a task that is yet profoundly difficult for state\-of\-the\-art LMMs\.
Future work should first apply this task to a wider array of state\-of\-the\-art LMMs from different producers\. It should also consider performing this task in a few\-shot manner to avoid the issue where explicit full\-grid textual representations are not generated for inputs with larger grid sizes\. We also discussed the possibility that tasks using real\-world geographic information instead of synthetic examples as input may, even though they seem more complex at first, yield better comprehension in LMMs as they would provide more context and are more similar to the raw training data that the LMMs are trained on\.
## ReferencesSimilar Articles
Spatial Reasoning via Modality Switching Between Language and Symbolic Representation
This paper explores grounding multi-hop textual-spatial stories into geometry-aware modalities like grids, showing a 42% performance improvement when switching from language-only to grid-based reasoning, and introduces a switching metric for modality selection in LLMs.
GeoArbiter: Verifiability-Guided Grounding for Remote-Sensing Multimodal LLMs
GeoArbiter proposes a training-free pipeline that selectively injects image-unverifiable geographic facts into remote-sensing multimodal LLMs to reduce knowledge hallucinations while preserving retrieval accuracy gains.
Evaluating Multimodal LLMs across Text and Audio Modalities for Accessible Disaster Assistance
This paper evaluates multimodal LLMs for disaster risk communication, finding that current models lack consistency across text and audio modalities, which undermines accessibility for vulnerable populations.
Large Language Model Teaches Visual Students: Cross-Modality Transfer of Fine-Grained Conceptual Knowledge
This paper introduces LaViD, a framework that transfers semantic knowledge from a language-only LLM to a vision student model by generating multiple-choice questions as conceptual signatures, achieving superior fine-grained classification performance and robustness.
Do Multimodal LLMs See Before They Read? Diagnosing Contextual Sycophancy
The paper introduces multimodal contextual sycophancy in large language models, where external text overrides visual evidence, and proposes a diagnostic method using System-2 Visual Arbitration to improve performance.