注意空间差距：地理空间推理与受约束空间编辑的基准

# Mind the Spatial Gap: A Benchmark for Geospatial Reasoning and Constrained Spatial Editing
Source: [https://arxiv.org/html/2610.12881](https://arxiv.org/html/2610.12881)
Zheng Ren†Yanning Wang†Ziheng Huang†Fei HuangShiqi Wang Ziqian Ye Yang Yue\* Urban Governance and Design \(UGOD\) ThrustThe Hong Kong University of Science and Technology \(Guangzhou\)Guangzhou, Chinayangyue@hkust\-gz\.edu\.cn †Equal contribution\.\*Corresponding author\.

###### Abstract

Understanding and manipulating spatial relations is fundamental to AI systems that reason about physical environments and construct grounded world models\. However, existing evaluations of multimodal large language models \(MLLMs\) for spatial reasoning rely predominantly on static question\-answering \(QA\) over small\-scale or synthetic scenes, rarely probing whether models can modify spatial configurations while satisfying explicit geometric constraints\. We introduce GeoSpatialBench, a benchmark designed to evaluate an escalating progression of spatial cognition, ranging from spatial relation understanding and geometric reasoning to constrained spatial editing\. Grounded in real\-world urban data across six geographically diverse cities, GeoSpatialBench evaluates a comprehensive taxonomy of 28 topological, directional, metric, and ordering relations across structured GeoJSON and map image modalities, with ground\-truth references and constraints verified via GIS operations\. Our empirical evaluation across frontier MLLMs uncovers a stark spatial gap: structured GeoJSON consistently outperforms map images, with multimodal combinations offering no reliable gain over GeoJSON alone, and while models show reasonable competence on qualitative topological and ordering, performance drops on quantitative metric relations\. Crucially, editing performance on ordering relations declines relative to QA, and most failed edits violate at least one specified spatial constraint\. These results highlight a gap between passively recognizing spatial relations and actively manipulating geometric configurations\. GeoSpatialBench provides an open, rigorous diagnostic testbed to benchmark these spatial limitations, laying essential empirical groundwork for physically grounded world models and embodied spatial AI\.

## 1Introduction

The ability to understand spatial relations and use them to manipulate spatial configurations is fundamental to AI systems that reason about physical environments and construct world models\([Ha and Schmidhuber, 2018](https://arxiv.org/html/2610.12881#bib.bib20);[Assran et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib17)\)\. Consider planning a road across a region: it needs to avoid mountains and rivers, cross a densely populated district, remain connected, and meet a distance limit\. Such reasoning underlies navigation, autonomous driving, spatial planning, embodied interaction, and world models\([Zhou et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib28);[Hong et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib22)\)\. Yet existing work on world models, embodied AI, and multimodal spatial reasoning is primarily evaluated in confined indoor or local scenes and over short trajectories\([Cheng et al\., 2024](https://arxiv.org/html/2610.12881#bib.bib21);[Yang et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib27);[Hong et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib22)\)\. Geospatial reasoning at city and regional scales introduces a different challenge: models must reason over heterogeneous geographic features and their topological, directional, metric, and ordering relations across larger spatial extents and multiple scales\.

Recent multimodal large language models \(MLLMs\) have achieved impressive results on visual/geometric benchmarks\. The proprietary GPT\-6 Astra scores 95\.9% with tools on BenchCAD, which evaluates programmatic CAD reasoning\([Zhang et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib15);[OpenAI, 2026](https://arxiv.org/html/2610.12881#bib.bib25)\)\. The open\-weight Kimi K3 scores 81\.6% without tools on MMMU\-Pro, a benchmark of multidisciplinary visual understanding and reasoning\([Yue et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib29);[Kimi Team, 2026](https://arxiv.org/html/2610.12881#bib.bib23)\)\. These results demonstrate strong general spatial and visual reasoning capabilities but provide limited evidence about geospatial reasoning over real\-world geographic configurations\. Recent studies have begun to explore this problem using different forms of geospatial representation, including maps\([Dihan et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib5);[Ung et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib7)\), imagery\([Xu et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib2)\), vector geometry\([Bao et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib3)\), and structured urban data\([Ji et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib1)\)\. Across these settings, evaluations cover spatial relation families such as topological, directional, metric and ordering\. However, the extent to which models can consistently recognize these relations and use them to modify spatial configurations remains unclear\.

Most existing benchmarks use question\-answering \(QA\) to assess whether models can understand and apply the underlying spatial relations\. A correct answer, however, may not show whether that relation can guide a change to the scene\([Vafa et al\., 2024](https://arxiv.org/html/2610.12881#bib.bib16)\)\. More broadly, benchmark answers need not reflect coherent use of the underlying concepts\([Mancoridis et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib14)\)\. Constrained spatial editing provides a complementary solution: object placements, geometric constructions, and urban layouts can be directly verified against explicit constraints\([Abdelreheem et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib13);[Huang et al\., 2025b](https://arxiv.org/html/2610.12881#bib.bib18);[Kim et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib19);[Liu et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib6)\)\. Yet, existing studies usually evaluate editing separately from QA, rather than testing whether models can both recognize and apply the same spatial relations\.

We introduce GeoSpatialBench, a benchmark that combines QA tasks on spatial relations with constrained spatial editing \(Figure[1](https://arxiv.org/html/2610.12881#S3.F1)\)\. By evaluating both tasks against the same set of spatial relations, GeoSpatialBench provides a stronger test of whether models can apply the relations they identify\. It covers 28 formally defined relations in four families \(topological, directional, metric, and ordering\), drawing on established Geographic Information Science \(GIS\) and spatial\-database concepts\([Egenhofer and Herring, 1991](https://arxiv.org/html/2610.12881#bib.bib9);[Open Geospatial Consortium, 2011](https://arxiv.org/html/2610.12881#bib.bib12)\)\. The benchmark contains 8,400 instances derived from real OpenStreetMap data across six cities\. QA and editing use separate instance pools but share the same set of relations\. Each scene is available as GeoJSON or as a map rendered from the same geometry\. Our results show that GeoJSON generally supports stronger geospatial reasoning, whereas map images remain substantially weaker\. This gap highlights an underexplored challenge in using maps as a visual medium for precise geospatial reasoning\.

Our contributions are threefold\. First, we introduce GeoSpatialBench, a comprehensive geospatial relation benchmark covering 28 spatial relations recognized in spatial database and GIS, with instances drawn from six cities worldwide\. Second, we propose spatial editing with QA tasks to verify models’ genuine spatial reasoning abilities\. Third, we systematically compare leading models, including GPT\-6 Astra, across input modalities using GIS\-derived answers and deterministic checks of edited geometry\. Together, these contributions enable more accurate measurement of spatial\-relation understanding and reveal where current models still need improvement\. Section[2](https://arxiv.org/html/2610.12881#S2)reviews related work, Section[3](https://arxiv.org/html/2610.12881#S3)describes the benchmark and evaluation protocol, Section[4](https://arxiv.org/html/2610.12881#S4)presents the results and analyses, and Section[5](https://arxiv.org/html/2610.12881#S5)concludes\.

## 2Related Work

### 2\.1Spatial relations and spatial reasoning of AI systems

Spatial relations describe how points, lines, and regions are arranged and relate to one another\. GIS and qualitative spatial reasoning formalize a broad range of such relations and provide geometric predicates for evaluating them\([Egenhofer and Franzosa, 1991](https://arxiv.org/html/2610.12881#bib.bib8);[Frank, 1992](https://arxiv.org/html/2610.12881#bib.bib10);[Cohn and Renz, 2008](https://arxiv.org/html/2610.12881#bib.bib11);[Open Geospatial Consortium, 2011](https://arxiv.org/html/2610.12881#bib.bib12)\)\. Recent work has extended spatial reasoning to MLLMs\. SpatialRGPT uses region and depth information to reason about relative direction and distance, SpatialReasoner incorporates explicit 3D representations, and SpatialScore evaluates a wider range of visual spatial abilities\([Cheng et al\., 2024](https://arxiv.org/html/2610.12881#bib.bib21);[Ma et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib24);[Wu et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib26)\)\. Related work on world models uses learned representations for prediction and planning, while VSI\-Bench and ESI\-Bench examine spatial recall and evidence gathering through action\([Ha and Schmidhuber, 2018](https://arxiv.org/html/2610.12881#bib.bib20);[Assran et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib17);[Zhou et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib28);[Yang et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib27);[Hong et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib22)\)\. Together, these studies establish the importance of spatial knowledge across visual and embodied settings, but their evaluations largely center on local scenes with limited spatial extent\. It remains less explored whether current MLLMs can reason over a broad set of formal spatial relations among georeferenced geographic features, derive geometric quantities from their configurations, and use those relations to modify spatial configurations\.

### 2\.2Benchmarks for geospatial reasoning in MLLMs

Recent benchmarks have extended spatial reasoning to larger\-scale geographic settings and real\-world geospatial data\. MapEval and CartoMapQA assess map reading and route reasoning, while FRIEDA evaluates multi\-step reasoning over map images\([Dihan et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib5);[Ung et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib7);[Pyo et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib4)\)\. EarthSpatialBench combines Earth imagery with explicit geographic coordinates to evaluate distance, direction, and topology, while UrbanGeoEval provides raw city\-scale geometry for spatial inference\([Xu et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib2);[Bao et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib3)\)\.[Ji et al\. \(2025\)](https://arxiv.org/html/2610.12881#bib.bib1)further study topological inference from WKT and relation\-conditioned geometry generation\. These studies demonstrate that geospatial reasoning can be evaluated using real geographic data and a range of representations, including map images, imagery, and structured or vector geometry\. However, existing benchmarks differ substantially in their representations, relation sets, and task formulations, making it difficult to compare how models reason across spatial modalities and relation families\. In particular, few systematically compare GeoJSON with a map image rendered from the same underlying geometry while evaluating a broad and formally defined set of spatial relations\.

### 2\.3Spatial editing and relation\-grounded evaluation

Most existing geospatial benchmarks evaluate models via QA tasks, testing their ability to recognize spatial relationships and compute geometric quantities\. However, obtaining correct answers does not prove that a model can leverage these relationships to modify spatial configurations\. Benchmark success can mask gaps in a model’s underlying understanding\([Vafa et al\., 2024](https://arxiv.org/html/2610.12881#bib.bib16);[Mancoridis et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib14)\)\. Spatial editing provides a complementary test because the resulting configuration can be checked against explicit spatial constraints\. Recent work has explored language\-guided object placement in 3D scenes, executable CAD editing, and iterative modification of urban GeoJSON\([Abdelreheem et al\., 2025](https://arxiv.org/html/2610.12881#bib.bib13);[Zhang et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib15);[Liu et al\., 2026](https://arxiv.org/html/2610.12881#bib.bib6)\)\. These studies show the feasibility of evaluating whether models can translate spatial instructions into valid geometric changes\. However, spatial editing has generally been studied separately from geospatial QA, rather than evaluating whether models can recognize and then apply the same spatial relations\.

GeoSpatialBench brings the two together, scoring each task by relation family on separate instance pools and verifying answers and edits with GIS operations\. This design provides a direct test of whether such spatial reasoning ability transfers to editing geographic geometry so that the models actually understand the underlying spatial relations\.

## 3GeoSpatialBench

GeoSpatialBench evaluates spatial reasoning through spatial question answering and constrained geometric editing\. Figure[1](https://arxiv.org/html/2610.12881#S3.F1)summarizes the benchmark construction, model inputs, and evaluation procedures\.

![Refer to caption](https://arxiv.org/html/2610.12881v1/geofm_pipeline_evaluation.png)Figure 1:Overview of GeoSpatialBench\. QA and spatial editing are constructed from urban vector data across six cities and 28 spatial relations\. Both tasks support GeoJSON, image, and combined inputs\. QA is evaluated by answer accuracy\. GeoJSON and Together editing use the Constraint Satisfaction Rate \(CSR\), while map\-image editing uses a raster\-tolerant success rate\.### 3\.1Task Definition

GeoSpatialBench covers 28 geospatial relations over points, lines, and polygons at the city scale\. These relations are categorized into four groups: topology, direction, metric, and order\. Instances of these relations include touching, relative direction, distance, and connectivity\. Each relation is defined by a precise geometric or attribute\-based rule\. For instance, two roads are considered connected only if they share an endpoint, while vertical order is ascertained by road\-layer attributes\.

In QA, models are tasked with providing an answer based on a specific scene and an associated question\. Depending on the types of relations involved, the answer may take the form of a category, a number, a feature ID, or an ordering\. Each QA instance is presented in both open\-ended and four\-option multiple\-choice formats\.

Spatial editing requires the model to modify the scene instead of describing it\. Given an initial scene, an instruction, and a target featureF0F\_\{0\}, the model is expected to make edits exclusively to the specified featureF0F\_\{0\}, while leaving other features unchanged\. Editing accuracy is assessed by whether the scene resulting from the model’s edit satisfies the requested spatial constraints\. The aim of the editing task is to evaluate whether the models truly comprehend spatial relations\.

### 3\.2Benchmark Construction

The benchmark covers six cities with diverse geographic contexts: Stockholm, Paris, Hong Kong, Wuhan, Detroit, and Nairobi\. Each city contributes 700 QA and 700 editing instances, giving 8,400 canonical instances in total\.

We use OpenStreetMap data for roads, buildings, land\-use regions, and city\-level points of interest \(POIs\)\. All features are projected to a city\-specific metric coordinate system before distance and area calculations\. We then filter them by geometry type and city\-specific length or area thresholds\. Road\-layer attributes are retained for vertical\-order questions\. QA and editing use separate feature pools, so their instances are not paired\.

For each geospatial relation, we construct a canonical scene from the filtered features\. The scene records the required geometries, roles, source attributes, and construction provenance\. It may use observed geometries or controlled constructions when the source data do not provide enough cases\. A geometric oracle evaluates the scene and produces the reference answer\. Source identities and construction records are kept for auditing but excluded from model inputs\.

Each canonical scene is serialized as a GeoJSON FeatureCollection and also rendered as an annotated map\. GeoJSON preserves the task\-coordinate geometry and uses anonymous feature identifiers, with only task\-required properties\. For the map, the same scene is converted to WGS84 and rendered in a square Web Mercator view\. Task features are highlighted and labeled, with a legend, detail insets, and attribute cards added when needed\. All images use a1,536×1,5361\{,\}536\\times 1\{,\}536pixel canvas\.

QA questions are generated from the oracle output\. Open\-ended questions ask the model to return the answer in the required form\. Multiple\-choice distractors depend on the relation\. Among all, directional relations use nearby bearings, ordering relations use alternative sequences, and numerical relations use adjacent intervals\. Topological relations and feature\-selection tasks use plausible competing labels or feature IDs\. Each item is checked before use\. We verify the geometry and reference answer, then check that the options are distinct\.

Editing instances are constructed separately from QA\. Candidates are rejected if the spatial constraints specified by the instruction are already satisfied in the initial scene, and retained only when a reference edit confirms that a valid solution exists\. The reference edit is used only to verify feasibility; any valid edit that satisfies the instruction is accepted\. Geometric edits must change the target featureF0F\_\{0\}, while attribute edits may leave its geometry unchanged\.

### 3\.3Input Modalities

We evaluate three input modalities: GeoJSON, map image, and their combination \(Together\)\. GeoJSON provides explicit coordinates and anonymous feature identifiers, while map image shows the same scene as an annotated map\. Together pairs the annotated map with the same GeoJSON used in the GeoJSON condition\. For QA, all three conditions use the same scene, question, and reference answer, with both open\-ended and four\-option multiple\-choice forms\. The answer format remains unchanged across conditions, allowing us to compare the effect of input representation on spatial reasoning\.

For editing, GeoJSON uses a structured interface\. Models receive the scene, target featureF0F\_\{0\}, and instruction, and return a transformation and its parameters \(e\.g\., translation, rotation, scaling, or attribute editing\)\. An executor applies the transformation toF0F\_\{0\}, and the evaluator checks the resulting scene against the required spatial constraints\. Map image editing instead returns edited geometry in pixel coordinates with any required attributes\. The evaluator maps the prediction back to the task coordinate system before checking the same constraints\. All images use a1,536×1,5361\{,\}536\\times 1\{,\}536pixel canvas\.

Together follows the same structured editing interface as GeoJSON while additionally providing the corresponding annotated map\. This allows a direct comparison with GeoJSON to test whether visual information improves structured editing\. By contrast, map image editing changes both the input representation and the output interface because the model must generate pixel\-coordinate geometry\. Differences between GeoJSON and map image editing should therefore not be interpreted as a pure input\-only ablation\.

### 3\.4Evaluation Protocol

#### Question answering\.

We use deterministic scoring rather than a learned judge\. Multiple\-choice outputs are parsed as option letters, while open\-ended outputs use answer\-specific normalization\. Distance and perimeter answers allow a one\-meter tolerance, bearing answers a one\-degree circular tolerance, and density answers a tolerance of 0\.001 in the stated units\. Unparseable answers are counted as incorrect, and open\-ended and multiple\-choice accuracy are reported separately\.

GeoJSON, map image, and Together use the same canonical scene, question, and reference answer\. GeoJSON exposes the exact task\-coordinate geometry, map image shows its annotated rendering, and Together provides both\. For geometry\-only conditions, we also report an information\-sufficient score that removes the 25 vertical\-order questions per city whose required layer attributes are not provided\. Annotated maps may provide these attributes through attribute cards, but raster measurements still limit visual precision\.

#### Structured editing\.

GeoJSON and Together share the same structured editing protocol\. Given an initial scene, target featureF0F\_\{0\}, and instruction, the model returns a transformation and its parameters\. LetSiS\_\{i\}andSi′S^\{\\prime\}\_\{i\}denote the initial and edited scenes, and let𝒞i\\mathcal\{C\}\_\{i\}be the task constraints\. LetViV\_\{i\}indicate a parseable, nonempty, valid prediction\. LetUiU\_\{i\}indicate that a geometric edit changesF0F\_\{0\}; for attribute\-editing tasks, we setUi=1U\_\{i\}=1\. We define

Ei=Vi​Ui​∏c∈𝒞i\[c⁡\(Si,Si′\)\],CSR=1N​∑i=1NEi\.E\_\{i\}=V\_\{i\}U\_\{i\}\\prod\_\{c\\in\\mathcal\{C\}\_\{i\}\}\\mathbf\{1\}\\\!\\left\[c\(S\_\{i\},S^\{\\prime\}\_\{i\}\)\\right\],\\qquad\\mathrm\{CSR\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}E\_\{i\}\.\(1\)Here,EiE\_\{i\}is a binary indicator of whether editiiis valid and satisfies all task constraints, and Constraint Satisfaction Rate \(CSR\) is the proportion of successful edits over the evaluation set\. With one corrective retry, an undecodable or unchanged edit triggers a second attempt requesting a valid, nonzero edit\. We score only the final attempt; the feedback does not reveal whether the task constraints were satisfied\.

#### Map image editing\.

The evaluator first maps the predicted pixel geometry back to the task coordinate system\. It then accepts either metric\-space constraint satisfaction or pixel geometry within four pixels of the reference under Hausdorff distance, provided that required attributes match\. We report this raster\-tolerant success rate separately from strict CSR\. The reference geometry is used for feasibility verification and image\-level scoring\.

We aggregate scores by task, input modality, and QA form, and report results by city, relation family, and individual relation\. Comparisons across GeoJSON, map image, and Together use matched scenes and instructions whenever the model supports the corresponding input modality\.

## 4Evaluation on GeoSpatialBench

### 4\.1Evaluation Setup

We evaluate nine models across the six cities, including four open\-weight models \(Qwen3\.5\-9B, Qwen3\.8\-27B, Qwen3\-VL\-8B\-Instruct, and InternVL3\-8B\) and five API\-accessed models \(GPT\-6 Astra, GPT\-5\.6 Sol, KIMI K3, MiMo\-V2\.6\-Pro, and DeepSeek\-V4\.1\-Flash\)\.

For question answering, each model is evaluated in both open\-ended and four\-option multiple\-choice settings\. We consider three input settings: GeoJSON, map image, and Together, where Together combines the GeoJSON representation with the corresponding annotated map\. Spatial editing is evaluated under the same three input settings using the task\-specific output interfaces described in Section[3](https://arxiv.org/html/2610.12881#S3)\. We also report performance separately for open\-ended QA, multiple\-choice QA, and spatial editing\. Additional model\- and setting\-specific results are provided in Appendix[A](https://arxiv.org/html/2610.12881#A1)\.

### 4\.2Overall Performance

Figure[2](https://arxiv.org/html/2610.12881#S4.F2)summarizes overall QA and editing performance across models and input modalities\. QA accuracy is averaged over the open\-ended and multiple\-choice settings\. Under GeoJSON, GPT\-6 Astra achieves the strongest overall performance, with 96\.5% QA accuracy and 97\.6% editing success\. KIMI K3 follows at 90\.7% and 85\.1%, respectively\. Performance then declines across the remaining models, although the relative order is not identical between QA and editing\.

![Refer to caption](https://arxiv.org/html/2610.12881v1/overall.png)Figure 2:Average QA and spatial editing performance across models and input modalities\. \(a\) QA accuracy averaged over open\-ended and multiple\-choice settings\. \(b\) Spatial editing success\. GeoJSON and Together editing use strict CSR, whereas Image editing uses raster\-tolerant success\.QA and editing performance are broadly aligned under GeoJSON\. Models with higher QA accuracy generally also achieve higher editing success, but several exceptions remain\. MiMo\-V2\.6\-Pro reaches 67\.3% QA accuracy, above Qwen3\.8\-27B at 55\.5%, whereas Qwen3\.8\-27B performs better on editing \(54\.4% vs\. 43\.0%\)\. Thus, strong relation understanding does not always translate into equally strong constrained editing\.

Performance also differs across input modalities\. Map image scores are generally lower than GeoJSON scores for both QA and spatial editing\. For MiMo\-V2\.6\-Pro, mean QA accuracy decreases from 67\.3% with GeoJSON to 45\.2% with map image, while Together reaches 65\.2%\. In spatial editing, the same model achieves 43\.0% CSR with GeoJSON and 51\.4% with Together, whereas its raster\-tolerant success rate with map image is 16\.5%\. Together therefore remains close to GeoJSON in QA, but its effect on spatial editing varies across models\. Because map image editing uses a different output interface and evaluation protocol, its score is not directly comparable to strict CSR\.

### 4\.3Performance across Relations and Cities

As shown in Figure[3](https://arxiv.org/html/2610.12881#S4.F3)\(a\), performance varies substantially across the four relation families under GeoJSON\. In open\-ended QA, topology and ordering achieve 74\.4% and 71\.9% accuracy, followed by direction at 55\.3% and metric at 38\.6%\. Multiple\-choice QA shows a similar pattern, with 77\.9% for topology, 69\.4% for ordering, 62\.0% for direction, and 53\.4% for metric\. Spatial editing shows a different distribution\. Topology and direction achieve the highest CSR at 65\.1% and 63\.4%, whereas ordering falls to 47\.3% and metric remains the lowest at 42\.9%\.

![Refer to caption](https://arxiv.org/html/2610.12881v1/relation.png)Figure 3:Performance across spatial relations under GeoJSON input\. \(a\) Mean performance across the four relation families for open\-ended QA, multiple\-choice QA, and spatial editing\. \(b\) Relation\-level multiple\-choice QA accuracy for each model across the 28 spatial relations\. Metric relations are generally more challenging, while QA and spatial editing show different patterns across relation families\.Figure[3](https://arxiv.org/html/2610.12881#S4.F3)\(b\) further shows substantial variation among individual relations in multiple\-choice QA\. Relation codes used in the figure are listed in Appendix B, Table 4\. Averaged across models, Equality \(T01\) achieves 98\.4% accuracy and Touching \(T04\) reaches 92\.7%, whereas Boundary Coverage \(T09\) drops to 46\.4%\. Metric relations are generally more difficult: POI Density \(M18\) achieves 37\.6%, and Hausdorff Distance \(M08\) reaches 46\.8%\. Ordering relations also vary widely, from 89\.2% for Along\-Line Order \(O01\) to 23\.7% for Layer Hierarchy \(O06\)\. Layer Hierarchy is attribute\-dependent, and the required road\-layer attribute is omitted from the geometry\-only GeoJSON QA input\. Its raw accuracy therefore reflects information availability in addition to geometric reasoning difficulty\.

Performance varies relatively little across the six evaluated cities\. For open\-ended GeoJSON QA, the within\-model range across cities is 1\.00–5\.71 percentage points\. By comparison, pooled accuracy differs by 67\.03 percentage points between the highest\- and lowest\-performing models\. The observed variation across models is therefore substantially larger than the within\-model variation across these six cities\. These results do not establish generalization to unseen cities\.

### 4\.4Error Analysis and Discussion

We further examine whether spatial\-relation recognition transfers to spatial intervention, which relations remain difficult, and how representation affects spatial reasoning\. In GeoSpatialBench, this capability is operationalized through spatial editing\. Figures[4](https://arxiv.org/html/2610.12881#S4.F4)and[5](https://arxiv.org/html/2610.12881#S4.F5)show representative QA and editing cases, while complete relation\-level statistics are provided in Appendix[B](https://arxiv.org/html/2610.12881#A2)\.

The representative cases reveal distinct sources of error across tasks and representations\. In QA, some relations remain stable across inputs, while precise metric estimates are more fragile\. In the Touching case, DeepSeek\-V4\.1\-Flash is correct with GeoJSON but predicts*intersects*from the map image\. For Boundary Distance, KIMI K3 gives an accurate open\-ended value with GeoJSON but fails under the map image, while still selecting the correct multiple\-choice interval\. Spatial editing introduces a different class of errors\. In the Bearing case, KIMI K3 succeeds under both inputs, whereas InternVL3\-8B moves the target in the wrong direction with GeoJSON\. In the Euclidean Distance case, GPT\-6 Astra succeeds under both inputs, while KIMI K3 succeeds with GeoJSON but produces a 42\.8 m distance from the map image, outside the required 8–22 m range\.

#### Relation recognition can approach saturation before spatial intervention\.

Several basic topological relations are already recognized reliably in GeoJSON MCQ\-4: Equality, Touching, and Disjointness have error rates of only 1\.6%, 7\.3%, and 8\.4%\. However, this does not consistently transfer to editing\. Touching and Disjointness increase to 71\.2% and 47\.3% editing error, whereas Equality remains easy at 3\.3%\. Thus, recognizing a relation and constructing a geometry that satisfies it are distinct capabilities\. Figure[5](https://arxiv.org/html/2610.12881#S4.F5)also illustrates this gap: models may choose a plausible operation but still produce an incorrect direction, distance, or final geometry\. Editing therefore remains discriminative even when some recognition tasks approach saturation\. The gap is also relation\-dependent\. For direction relations, GeoJSON editing reaches 63\.4% CSR, close to the 62\.0% MCQ\-4 accuracy\.

#### Metric reasoning remains the most persistent bottleneck\.

Metric relations remain the most difficult family under both GeoJSON and map\-image inputs\. Even with explicit GeoJSON geometry, open\-ended metric error reaches 61\.4%\. Several numerical relations are particularly difficult: Minimum Distance, Hausdorff Distance, Total Perimeter, and POI Density exceed 70% error in open\-ended QA\. Figure[4](https://arxiv.org/html/2610.12881#S4.F4)shows a related pattern\. Models can sometimes identify the correct distance interval while failing to produce an accurate numerical value\. These results point to a persistent weakness in quantitative geometric reasoning\. Access to explicit coordinates reduces representational ambiguity, but does not by itself ensure reliable computation of continuous spatial quantities\.

#### Explicit geometry better supports precise spatial reasoning\.

For QA, GeoJSON and map\-image conditions use the same scene, question, and reference answer, enabling a direct comparison of representations\. GeoJSON has lower family\-level error across all four relation families, with the largest gap in metric reasoning\. Figure[4](https://arxiv.org/html/2610.12881#S4.F4)shows the same tendency: coarse directional or categorical judgments may remain correct from maps, while precise boundary and distance estimates are less reliable\. Together inputs also do not consistently improve over GeoJSON alone, suggesting that additional visual context does not automatically strengthen an explicit geometric representation\. For editing, the GeoJSON–image gap should be interpreted cautiously because image editing also uses a different output interface\.

![Refer to caption](https://arxiv.org/html/2610.12881v1/qualitative_cases.png)Figure 4:Representative QA cases across the four spatial relation families\. Map images often preserve coarse spatial judgments but are less reliable for precise boundary and metric reasoning\.![Refer to caption](https://arxiv.org/html/2610.12881v1/editing_error_analysis.png)Figure 5:Representative spatial editing cases across four relation families, illustrating failures in geometric execution and constraint satisfaction\.
#### Toward explicit spatial representations\.

These patterns suggest that precise spatial reasoning may require representations that preserve geometric and relational structure more explicitly\. Metric reasoning depends on continuous quantities, while rasterization requires exact geometry to be recovered indirectly from a finite\-resolution image\. Our experiments do not isolate a specific architectural cause, but they suggest that additional multimodal visual information alone may be insufficient\. Spatial tokenization methods such as SPOK\([Huang et al\., 2025a](https://arxiv.org/html/2610.12881#bib.bib30)\)provide one possible direction by representing geographic space and spatial relations explicitly\. Such representations may help bridge spatial recognition, geometric reasoning, and spatial intervention\.

## 5Conclusion

We introduced GeoSpatialBench, a benchmark for evaluating spatial relation understanding, geometric computation, and constrained spatial editing in urban geographic scenes using structured GeoJSON and rendered map images\. Across nine models, GeoJSON generally yields stronger performance than map images, while combining the two inputs does not consistently improve performance\. Precise metric computation and boundary relations remain challenging, and constraint violations account for most editing failures\. By evaluating spatial reasoning and editing within a common, automatically verifiable framework, GeoSpatialBench shows that recognizing a spatial relation does not necessarily mean a model can construct geometry that satisfies it\. A promising next step is to test whether dedicated, relation\-aware representations can help bridge this gap\. Difficulty with basic 2D geometric constraints also highlights the challenge of more complex spatial generation and provides a baseline for future work\.

## Reproducibility Statement

Section[3](https://arxiv.org/html/2610.12881#S3)specifies the six OpenStreetMap source cities, instance counts, GeoJSON and map\-image construction, task formats, and GIS\-based procedures for generating answers and checking edited geometry\. It also defines the parsing rules, numerical tolerances, corrective retry, and separate raster\-tolerant criterion for image editing\. The evaluation code and supporting benchmark materials will be provided by the authors upon request\.

## AI Use Statement

AI\-based tools were used to assist with language editing andLaTeXformatting\. The authors used OpenAI’s ChatGPT \(5\.6 Sol\) to assist with code development, data processing, and language editing of the manuscript\. The authors reviewed and tested the AI\-assisted code, checked the data\-processing outputs, and reviewed and revised the AI\-edited text\. The authors take full responsibility for the accuracy and integrity of the methods, results, and final manuscript\.

## References

- A\. Abdelreheem, F\. Aleotti, J\. Watson, Z\. Qureshi, A\. Eldesokey, P\. Wonka, G\. Brostow, S\. Vicente, and G\. Garcia\-HernandoPlaceIt3D: language\-guided object placement in real 3d scenes\.InProceedings of the IEEE/CVF International Conference on Computer Vision,pp\. 6645–6655\.External Links:[Document](https://dx.doi.org/10.1109/ICCV51701.2025.00626),2505\.05288,[Link](https://openaccess.thecvf.com/content/ICCV2025/html/Abdelreheem_PlaceIt3D_Language-Guided_Object_Placement_in_Real_3D_Scenes_ICCV_2025_paper.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.12881#S2.SS3.p1.1)\.
- Assranet al\.\(2025\)M\. Assran, A\. Bardes, D\. Fan, Q\. Garrido, R\. Howes, M\. Komeili, M\. Muckley, A\. Rizvi, C\. Roberts, K\. Sinha, A\. Zholus, S\. Arnaud, A\. Gejji, A\. Martin, F\. R\. Hogan, D\. Dugas, P\. Bojanowski, V\. Khalidov, P\. Labatut, F\. Massa, M\. Szafraniec, K\. Krishnakumar, Y\. Li, X\. Ma, S\. Chandar, F\. Meier, Y\. LeCun, M\. Rabbat, and N\. BallasV\-JEPA 2: self\-supervised video models enable understanding, prediction and planning\.Note:PreprintExternal Links:2506\.09985,[Link](https://arxiv.org/abs/2506.09985)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Baoet al\.\(2026\)M\. Bao, Q\. Qi, T\. Liang, J\. Zhang, W\. Zhou, M\. Kong, L\. Mo, and Q\. ZhuUrbanGeoEval: a city\-scale benchmark for evaluating large language models in geospatial reasoning\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),San Diego, California, United States,pp\. 40183–40223\.External Links:[Document](https://dx.doi.org/10.18653/v1/2026.acl-long.1867),[Link](https://aclanthology.org/2026.acl-long.1867/)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12881#S2.SS2.p1.1)\.
- Chenget al\.\(2024\)A\. Cheng, H\. Yin, Y\. Fu, Q\. Guo, R\. Yang, J\. Kautz, X\. Wang, and S\. LiuSpatialRGPT: grounded spatial reasoning in vision\-language models\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 135062–135093\.External Links:[Document](https://dx.doi.org/10.52202/079017-4293),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/f38cb4cf9a5eaa92b3cfa481832719c6-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Cohn and Renz \(2008\)A\. G\. Cohn and J\. RenzQualitative spatial representation and reasoning\.InHandbook of Knowledge Representation,F\. van Harmelen, V\. Lifschitz, and B\. Porter \(Eds\.\),Foundations of Artificial Intelligence, Vol\.3,pp\. 551–596\.External Links:[Document](https://dx.doi.org/10.1016/S1574-6526%2807%2903013-1),[Link](https://doi.org/10.1016/S1574-6526(07)03013-1)Cited by:[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Dihanet al\.\(2025\)M\. L\. Dihan, M\. T\. Hassan, M\. T\. Parvez, M\. H\. Hasan, M\. A\. Alam, M\. A\. Cheema, M\. E\. Ali, and M\. R\. ParvezMapEval: a map\-based evaluation of geo\-spatial reasoning in foundation models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 13774–13813\.External Links:[Link](https://proceedings.mlr.press/v267/dihan25a.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12881#S2.SS2.p1.1)\.
- Egenhofer and Franzosa \(1991\)M\. J\. Egenhofer and R\. D\. FranzosaPoint\-set topological spatial relations\.International Journal of Geographical Information Systems5\(2\),pp\. 161–174\.External Links:[Document](https://dx.doi.org/10.1080/02693799108927841),[Link](https://doi.org/10.1080/02693799108927841)Cited by:[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Egenhofer and Herring \(1991\)M\. J\. Egenhofer and J\. R\. HerringCategorizing binary topological relations between regions, lines, and points in geographic databases\.Technical reportDepartment of Surveying Engineering, University of Maine,Orono, Maine\.Note:Reprinted in NCGIA Technical Report 94\-1External Links:[Link](https://escholarship.org/uc/item/5nj6647c)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p4.1)\.
- Frank \(1992\)A\. U\. FrankQualitative spatial reasoning about distances and directions in geographic space\.Journal of Visual Languages & Computing3\(4\),pp\. 343–371\.External Links:[Document](https://dx.doi.org/10.1016/1045-926X%2892%2990007-9),[Link](https://doi.org/10.1016/1045-926X(92)90007-9)Cited by:[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Ha and Schmidhuber \(2018\)D\. Ha and J\. SchmidhuberWorld models\.arXiv preprint arXiv:1803\.10122\.External Links:1803\.10122,[Link](https://arxiv.org/abs/1803.10122)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Honget al\.\(2026\)Y\. Hong, J\. Liu, H\. Yin, M\. Li, L\. Guibas, F\. Li, J\. Wu, and Y\. ChoiESI\-Bench: towards embodied spatial intelligence that closes the perception\-action loop\.External Links:2605\.18746,[Document](https://dx.doi.org/10.48550/arXiv.2605.18746),[Link](https://arxiv.org/abs/2605.18746)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Huanget al\.\(2025a\)F\. Huang, J\. Lv, G\. Li, and Y\. YueSPOK: tokenizing geographic space for enhanced spatial reasoning in geoai\.International Journal of Geographical Information Science39\(12\),pp\. 2768–2808\.External Links:[Document](https://dx.doi.org/10.1080/13658816.2025.2497810)Cited by:[§4\.4](https://arxiv.org/html/2610.12881#S4.SS4.SSS0.Px4.p1.1)\.
- Huanget al\.\(2025b\)I\. Huang, Y\. Bao, K\. Truong, H\. Zhou, C\. Schmid, L\. Guibas, and A\. FathiFirePlace: geometric refinements of llm common sense reasoning for 3d object placement\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 13466–13476\.Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p3.1)\.
- Jiet al\.\(2025\)Y\. Ji, S\. Gao, Y\. Nie, I\. Majić, and K\. JanowiczFoundation models for geospatial reasoning: assessing the capabilities of large language models in understanding geometries and topological spatial relations\.International Journal of Geographical Information Science39\(9\),pp\. 1866–1903\.External Links:[Document](https://dx.doi.org/10.1080/13658816.2025.2511227),[Link](https://doi.org/10.1080/13658816.2025.2511227)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12881#S2.SS2.p1.1)\.
- Kimet al\.\(2026\)J\. Kim, R\. Yang, and H\. ZhangGeoBuildBench: a benchmark for interactive and executable geometry construction from natural language\.External Links:2605\.13167,[Document](https://dx.doi.org/10.48550/arXiv.2605.13167),[Link](https://arxiv.org/abs/2605.13167)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p3.1)\.
- Kimi Team \(2026\)Kimi TeamKimi K3: open frontier intelligence\.External Links:2607\.24653,[Document](https://dx.doi.org/10.48550/arXiv.2607.24653),[Link](https://arxiv.org/abs/2607.24653)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1)\.
- Liuet al\.\(2026\)R\. Liu, S\. J\. Quan, Z\. Peng, Z\. Yao, H\. Wang, P\. Wang, K\. Liu, Y\. Fu, and D\. WangCity editing: hierarchical agentic execution for dependency\-aware urban geospatial modification\.Note:Preprint; accepted by ACM SIGSPATIAL 2026External Links:2602\.19326,[Link](https://arxiv.org/abs/2602.19326)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.12881#S2.SS3.p1.1)\.
- Maet al\.\(2025\)W\. Ma, Y\. Chou, Q\. Liu, X\. Wang, C\. de Melo, J\. Xie, and A\. YuilleSpatialReasoner: towards explicit and generalizable 3d spatial reasoning\.InAdvances in Neural Information Processing Systems,Vol\.38,pp\. 140751–140774\.External Links:[Document](https://dx.doi.org/10.52202/085713-4703),[Link](https://proceedings.neurips.cc/paper_files/paper/2025/hash/cdd63c6c813a6ec826051b6a49f11b5e-Abstract-Conference.html)Cited by:[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Mancoridiset al\.\(2025\)M\. Mancoridis, B\. Weeks, K\. Vafa, and S\. MullainathanPotemkin understanding in large language models\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 42857–42881\.External Links:[Link](https://proceedings.mlr.press/v267/mancoridis25a.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.12881#S2.SS3.p1.1)\.
- Open Geospatial Consortium \(2011\)Open Geospatial ConsortiumJ\. R\. Herring \(Ed\.\)OpenGIS implementation standard for geographic information – simple feature access – part 1: common architecture\.OpenGIS Implementation StandardTechnical ReportOGC 06\-103r4,Open Geospatial Consortium\.Note:Version 1\.2\.1External Links:[Link](https://www.ogc.org/standards/sfa/)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p4.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- OpenAI \(2026\)OpenAIGPT\-6 Astra: a new generation of intelligence\.Note:OpenAI research releaseExternal Links:[Link](https://openai.com/index/gpt-6-astra/)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1)\.
- Pyoet al\.\(2026\)J\. Pyo, Y\. Jiao, D\. Jung, Z\. Li, L\. Jang, S\. Kirsanova, J\. Kim, Y\. Lin, Q\. Liu, J\. Xie, H\. Askari, N\. Xu, M\. Chen, and Y\. ChiangFRIEDA: benchmarking multi\-step cartographic reasoning in vision\-language models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=QQCadccQqU),2512\.08016Cited by:[§2\.2](https://arxiv.org/html/2610.12881#S2.SS2.p1.1)\.
- Unget al\.\(2025\)H\. Q\. Ung, G\. Habault, Y\. Nishimura, H\. Niu, R\. Legaspi, T\. Oya, R\. Kojima, M\. Taya, C\. Ono, A\. Minamikawa, and Y\. LiuCartoMapQA: a fundamental benchmark dataset evaluating vision\-language models on cartographic map understanding\.InProceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems,pp\. 440–453\.External Links:[Document](https://dx.doi.org/10.1145/3748636.3762755),[Link](https://doi.org/10.1145/3748636.3762755),2512\.03558Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12881#S2.SS2.p1.1)\.
- Vafaet al\.\(2024\)K\. Vafa, J\. Y\. Chen, A\. Rambachan, J\. Kleinberg, and S\. MullainathanEvaluating the world model implicit in a generative model\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 26941–26975\.External Links:[Document](https://dx.doi.org/10.52202/079017-0846),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/2f6a6317bada76b26a4f61bb70a7db59-Abstract-Conference.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p3.1),[§2\.3](https://arxiv.org/html/2610.12881#S2.SS3.p1.1)\.
- Wuet al\.\(2026\)H\. Wu, X\. Huang, Y\. Chen, Y\. Zhang, Y\. Wang, and W\. XieSpatialScore: towards comprehensive evaluation for spatial intelligence\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 31029–31041\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2026/html/Wu_SpatialScore_Towards_Comprehensive_Evaluation_for_Spatial_Intelligence_CVPR_2026_paper.html)Cited by:[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Xuet al\.\(2026\)Z\. Xu, Y\. Zhang, S\. Adhikari, S\. Islam, T\. Xiao, Z\. Liu, S\. Chen, D\. Yan, and Z\. JiangEarthSpatialBench: benchmarking spatial reasoning capabilities of multimodal LLMs on earth imagery\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=mR9YFoMHBK),2602\.15918Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12881#S2.SS2.p1.1)\.
- Yanget al\.\(2025\)J\. Yang, S\. Yang, A\. W\. Gupta, R\. Han, L\. Fei\-Fei, and S\. XieThinking in space: how multimodal large language models see, remember, and recall spaces\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 10632–10643\.External Links:[Document](https://dx.doi.org/10.1109/CVPR52734.2025.00994),[Link](https://openaccess.thecvf.com/content/CVPR2025/html/Yang_Thinking_in_Space_How_Multimodal_Large_Language_Models_See_Remember_CVPR_2025_paper.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.
- Yueet al\.\(2025\)X\. Yue, T\. Zheng, Y\. Ni, Y\. Wang, K\. Zhang, S\. Tong, Y\. Sun, B\. Yu, G\. Zhang, H\. Sun, Y\. Su, W\. Chen, and G\. NeubigMMMU\-pro: a more robust multi\-discipline multimodal understanding benchmark\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 15134–15186\.External Links:[Link](https://aclanthology.org/2025.acl-long.736/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.736),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1)\.
- Zhanget al\.\(2026\)H\. Zhang, K\. Liu, M\. Chen, L\. Li, S\. Yang, C\. Peng, and H\. ChenBenchCAD: a comprehensive, industry\-standard benchmark for programmatic CAD\.Note:Preprint; supplied version v2External Links:2605\.10865,[Link](https://arxiv.org/abs/2605.10865)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p2.1),[§2\.3](https://arxiv.org/html/2610.12881#S2.SS3.p1.1)\.
- Zhouet al\.\(2025\)G\. Zhou, H\. Pan, Y\. LeCun, and L\. PintoDINO\-wm: world models on pre\-trained visual features enable zero\-shot planning\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 79115–79135\.External Links:[Link](https://proceedings.mlr.press/v267/zhou25t.html)Cited by:[§1](https://arxiv.org/html/2610.12881#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12881#S2.SS1.p1.1)\.

## Appendix AAdditional Results

Table[1](https://arxiv.org/html/2610.12881#A1.T1)jointly reports GeoJSON, Image, and Together performance for all nine models, ordered by six\-city GeoJSON open\-ended QA\. Table[2](https://arxiv.org/html/2610.12881#A1.T2)shows the six\-city variation in GeoJSON open\-ended QA\. The differences between cities range from 1\.00% to 5\.71%\.

Table 1:Overall performance \(%; higher is better\), ordered by GeoJSON open\-ended QA\. Each model uses 4,200 instances across six cities\. Image editing reports raster\-tolerant success; GeoJSON and Together editing report strict CSR\. Bold marks the highest reported score in each column\.Table 2:Open\-ended GeoJSON QA across six cities \(%; higher is better\), ordered by pooled accuracy\. Each city has 700 questions per model\. Overall pools 4,200 questions; Range is the maximum minus minimum city score in percentage points\.
## Appendix BAdditional Error Analysis

### B\.1Distribution by Relation

Figure[6](https://arxiv.org/html/2610.12881#A2.F6)and Tables[3](https://arxiv.org/html/2610.12881#A2.T3)and[4](https://arxiv.org/html/2610.12881#A2.T4)summarize relation\-level error rates for the nine reported models\.

![Refer to caption](https://arxiv.org/html/2610.12881v1/overall_error_distribution_six_city.png)Figure 6:Error rates by spatial relation, input, and task \(%; lower is better\)\.Table 3:Error rates by relation family \(%; lower is better\), pooled over the nine reported models\. GeoJSON and Map Image use six cities for every model\.Table 4:Error rates for the 28 spatial relations \(%; lower is better\)\.