GeoOutageBench：面向多模态电力中断与韧性分析的歧义感知、基于本体的地理时空知识图谱问答基准

# GeoOutageBench: Benchmarking Ambiguity-aware, Ontology-grounded Geospatiotemporal KGQA for Multimodal Power Outage and Resilience Analysis
Source: [https://arxiv.org/html/2609.36082](https://arxiv.org/html/2609.36082)
Conference:The 34th ACM International Conference on Advances in Geographic Information Systems; November 03–06, 2026; Riverside, CA, USAThe 34th ACM International Conference on Advances in Geographic Information Systems \(SIGSPATIAL ’26\), November 03–06, 2026, Riverside, CA, USADOI:[10\.1145/3841645\.3842982](https://doi.org/10.1145/3841645.3842982)ISBN:979\-8\-4007\-2950\-8/2026/11CCS:Information systems Geographic information systemsCCS:Information systems Spatial\-temporal systemsCCS:Information systems Question answeringCCS:Information systems Graph\-based database modelsCCS:Computing methodologies Natural language processingCCS:Computing methodologies Knowledge representation and reasoningEthan D\. Frakes[https://orcid.org/0009-0008-8869-5703](https://orcid.org/0009-0008-8869-5703)Note:Corresponding author\.Affiliation:University of Central Florida,Orlando,Florida,USAemail:[ethan\.frakes@ucf\.edu](mailto:[email protected])Amy Kvien[https://orcid.org/0000-0003-4546-1858](https://orcid.org/0000-0003-4546-1858)Affiliation:University of Central Florida,Orlando,Florida,USAemail:[amy\.kvien@ucf\.edu](mailto:[email protected]),Rishabh Kundu[https://orcid.org/0000-0002-0719-3939](https://orcid.org/0000-0002-0719-3939)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[rxk857@case\.edu](mailto:[email protected]),Redad Mehdi[https://orcid.org/0000-0002-0593-5222](https://orcid.org/0000-0002-0593-5222)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[mxm1684@case\.edu](mailto:[email protected]),Van D\. Tran[https://orcid.org/0009-0008-4355-0543](https://orcid.org/0009-0008-4355-0543)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[vxt101@case\.edu](mailto:[email protected]),Vibha S\. Mandayam[https://orcid.org/0009-0008-8628-9904](https://orcid.org/0009-0008-8628-9904)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[vsm21@case\.edu](mailto:[email protected]),Kristopher O\. Davis[https://orcid.org/0000-0002-5772-6254](https://orcid.org/0000-0002-5772-6254)Affiliation:University of Central Florida,Orlando,Florida,USAemail:[kristopher\.davis@ucf\.edu](mailto:[email protected]),Erika I\. Barcelos[https://orcid.org/0000-0002-9273-8488](https://orcid.org/0000-0002-9273-8488)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[eib14@case\.edu](mailto:[email protected]),Roger H\. French[https://orcid.org/0000-0002-6162-0532](https://orcid.org/0000-0002-6162-0532)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[roger\.french@case\.edu](mailto:[email protected]),Yinghui Wu[https://orcid.org/0000-0003-3991-5155](https://orcid.org/0000-0003-3991-5155)Affiliation:Case Western Reserve University,Cleveland,Ohio,USAemail:[yinghui\.wu2@case\.edu](mailto:[email protected])andMengjie Li[https://orcid.org/0000-0002-9531-2474](https://orcid.org/0000-0002-9531-2474)Affiliation:University of Central Florida,Orlando,Florida,USAemail:[mengjie\.li@ucf\.edu](mailto:[email protected])

© cc

###### Abstract\.

We introduceGeoOutageBench, a benchmark for assessing LLM\-based geospatiotemporal KGQA for multimodal outage and resilience analysis\. Unlike existing KGQA benchmarks for Web knowledge,𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}considers a spatiotemporal KG that integrates visual, textual, and structured data from outage records, remote sensing, weather observations, storm and power events, geographic entities, and domain ontologies\. It provides a competency query taxonomy at different difficulty levels from spatiotemporal containment and proximity, spatiotemporal co\-occurrence analysis, multimodal evidence, to hypothetical evaluation\. Over multimodal KG and query classes,𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}provides user\-configurable evaluation of three important, highly coherent yet less studied tasks: \(1\) LLMs’ understanding for ambiguous geospatiotemporal questions in terms of NL to SPARQL interpretation, \(2\) query\-driven assessment of ontology utility, and \(3\) answer accuracy of multimodal KGQA retrieval\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}provides a design principle and foundation for assessing LLM\-KG systems that support real\-world infrastructure resilience analysis\. Our benchmark, source code, data, results, and other documentation are available at[https://github\.com/UCF\-SAGE/GeoOutageBench](https://github.com/UCF-SAGE/GeoOutageBench)\.

###### Keywords:

Geospatiotemporal knowledge graphs, KGQA, Benchmark datasets, Multimodal retrieval, Outage resilience

††cc\-license:by## 1\.Introduction

Question Answering over knowledge graphs \(KGQA\), which increasingly leverages Large Language Models \(LLMs\), has significantly advanced knowledge exploration and spatial intelligence via natural language \(NL\) questions\. Meanwhile, geospatiotemporal KGs\([Yang et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib3);[Böckling et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib4);[Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\)have become critical for modeling dynamic spatial systems, including mobility, administrative boundaries, sensor observations, and event dynamics\. This calls for effective integration of diverse, dynamic multimodal geospatiotemporal data \(*e\.g\.,*localized storm events\([NOAA National Centers for Environmental Information, 2026](https://arxiv.org/html/2609.36082#bib.bib13)\), hurricane tracks\([National Hurricane Center, 2026](https://arxiv.org/html/2609.36082#bib.bib14)\), and nighttime light remote sensing\([Román et al\., 2018](https://arxiv.org/html/2609.36082#bib.bib16)\)\) to understand evolving geographic phenomena\([Yang et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib3);[Böckling et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib4);[Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\)\. For example, GeoOutageKG\([Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\)curates factual spatiotemporal data for power outage analysis, remote sensing observations, and energy resilience\.

Effective KGQA requires benchmarking tools to understand how to best exploit LLMs, ontologies, and KGs\. Existing KGQA benchmarks\([Auer et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib31);[Kosten et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib32);[Dubey et al\., 2019](https://arxiv.org/html/2609.36082#bib.bib38);[Usbeck et al\., 2018](https://arxiv.org/html/2609.36082#bib.bib37)\)provide valuable resources for evaluating answer correctness, formal query generation, and execution accuracy over scholarly, encyclopedic, or heterogeneous linked\-data KGs\. However, they provide limited support for assessing how ontologies, geospatial KGs, and multimodal evidence jointly affect the usability and accuracy of spatiotemporal KGQA for outage and storm\-resilience analysis\.

Figure 1\.Abstract view of how one question can support multiple KGQA interpretations\. The same term can bind the state, year, SVI threshold, and event\-count semantics differently, while alternative ontology\-compatible traversals can reach the same answer through different paths in KG\.A one\-column abstract branching diagram for a GeoOutageBench natural\-language question\. The question points to four underspecified slots labeled as spatial, temporal, semantic, and semantic ambiguity: state, year, SVI threshold, and the meaning of dozen or so storm events\. The first expanded interpretation box shows a query graph bound to Florida, year 2022, SVI at least 0\.75, Thunderstorm Wind events, and count at least 12\. The second interpretation keeps the Louisiana 2018 SVI at least 0\.85, Wind\-event count between 10 and 14 semantics, but grounds the county path in a nighttime\-light image, acquisition year, and VIIRS sensor before joining with SVI and storm\-event records\.###### Example 1\.1\.

Consider the question“Which counties in the hurricane\-prone state had high socioeconomic vulnerability in a recent year and a dozen or so wind\-related storm events recorded that same year?”Figure[1](https://arxiv.org/html/2609.36082#S1.F1)abstracts the interpretations licensed by this question, with blue elements denoting query variables and green elements denoting ontology classes\. For outage and storm analysis, the answer depends on how a KGQA system binds underspecified state, year, SVI threshold, and event\-count semantics\. In the left query reading \(Q1\), ontology classes ground county, SVI\-record, and storm\-event nodes in a direct county\-year query; in the right reading \(Q2\), an NTL Image class mediates image coverage and acquisition year to the county answer and associated constraints\. These choices preserve the same county\-year co\-occurrence structure but lead to different executable queries and resilience evidence\.

Example[1\.1](https://arxiv.org/html/2609.36082#S1.Thmtheorem1)highlights three limitations of general\-purpose KGQA benchmarks that are especially consequential for geospatiotemporal resilience analysis\. First, current benchmarks provide limited evidence about whether modern LLMs can perform domain\-specific spatiotemporal understanding\. Real\-world NL questions are often underspecified not only in their entity references, but also in the joint spatial and temporal constraints they imply; as Example[1\.1](https://arxiv.org/html/2609.36082#S1.Thmtheorem1)shows, a single outage\-and\-storm question can support multiple valid interpretations over state, year, and event\-count semantics\. Second, existing evaluations rarely measure the practicalusabilityof ontologies and KGs for KGQA: given the same NL workload, how much do a particular ontology and KG improve schema grounding, query generation, and answer accuracy? Third, benchmarks provide limited insight into the role ofmultimodality\. Additional modalities such as outage records, storm reports, and remote\-sensing imagery may enrich answers but also introduce irrelevant evidence or conflicting signals that degrade KGQA quality\.

The need for such efforts is evident to support accurate KGQA for location\- and temporal\-sensitive power event and resilience analysis\. Indeed, power outages have become more frequent, more intense, and more prolonged over the past decade, driven by factors including the frequency and severity of extreme weather events, deferred maintenance, and insufficient investment in modernizing energy infrastructure\([Li et al\., 2026](https://arxiv.org/html/2609.36082#bib.bib12)\)\. We advocate a benchmark that can provide all the above assessments, beyond ranking LLMs over static knowledge, hence helping users to best exploit LLMs, ontologies, and multimodal data holistically to support geospatial and temporal retrieval and reasoning\.

Contribution\. We introduce𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}, a domain\-specific benchmark for ontology\-grounded geospatiotemporal KGQA over multimodal power infrastructure outage data\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}advances geospatiotemporal KGQA benchmarking for outage and storm resilience through the following primary contributions\.

\(1\)A multimodal geospatiotemporal KG, built on top of the GeoOutageKG\([Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\)architecture with multimodal outage records including administrative geographies, storm and hurricane data, social vulnerability indicators, nighttime\-light imagery, and outage\-severity map metadata\.

\(2\)A taxonomy of competency queries, spanning seven categories—spatial containment, spatial proximity, temporal interval, event sequence, spatiotemporal co\-occurrence, multimodal evidence, and scenario\-based evaluation—with complexity levels ranging from one\-hop to multi\-hop reasoning\.

\(3\)Three benchmark taskswith evaluation metrics and algorithms\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}evaluates ontology\-grounded NL2SPARQL generation, spatiotemporal KGQA over GeoOutageKG, and query\-driven ontology utility\. Its metrics capture schema\-valid query generation and ambiguity handling, answer quality for multimodal and cross\-modal questions, and GeoOutageOnto coverage through workload\-vocabulary relevance and schema\-utilization density\.

\(4\)A configurable sandbox platformthat integrates GeoOutageKG and ontologies\([Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\), KG hosting through partners such as GraphDB\([Ontotext, 2026](https://arxiv.org/html/2609.36082#bib.bib27)\), SPARQL engines, and spatial and time\-series retrieval tools\.

Figure 2\.GeoOutageBench framework\. Dataset assets are linked through GeoOutageOnto and instantiated in GeoOutageKG; natural\-language questions are mapped to ambiguity\-aware SPARQL interpretations; and LLMs, GraphDB\([Ontotext, 2026](https://arxiv.org/html/2609.36082#bib.bib27)\), and OntoCheck\([Kundu et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib36)\)support the benchmark tasks and custom metrics𝒫\\mathcal\{P\}\.Top\-down sketch of the GeoOutageBench framework\. An orange GeoOutageBench box containing the KGQA profile tuple is centered above three aligned process panels connected by arrows\. The Dataset panel contains a Multimodal Data group with text\-only boxes for Tabular Data, Remote Sensing Data, and Outage Maps, followed by GeoOutageOnto and GeoOutageKG boxes\. The Query Construction panel contains Natural Language Questions and ambiguity\-aware SPARQL Interpretations grounded in the ontology, including semantic, spatial, temporal, and spatiotemporal readings with one\-hop, two\-hop, and multi\-hop graph traversals\. The Tools and Benchmark Tasks panel contains a Tools group with LLMs, GraphDB, and OntoCheck, a raised Benchmark Tasks group with NL2SPARQL, spatiotemporal KGQA, and ontology\-utility assessment, and a custom evaluation metrics box for $\\mathcal\{P\}$ covering ambiguity, schema\-linking, spatiotemporal, answer\-quality, and ontology\-utility metrics\.Related Work\. We summarize related work below\.

Geospatial Ontologies and KGs\. The W3C Geospatial Ontologies Incubator report\([Lieberman et al\., 2007](https://arxiv.org/html/2609.36082#bib.bib34)\)provides an early foundation for representing geospatial concepts on the Web, including features, feature types, spatial relationships, coordinate reference systems, metadata, and geospatial services\. The W3C Basic Geo/WGS84 vocabulary\([Brickley, 2003](https://arxiv.org/html/2609.36082#bib.bib35)\)offers a lightweight RDF namespace for WGS84 latitude, longitude, and altitude, supporting simple point\-based geolocation but not richer geometric or topological reasoning\. Related spatial and spatiotemporal ontologies include the Time ontology\([Cox and Little, 2017](https://arxiv.org/html/2609.36082#bib.bib28)\), GeoSPARQL\([Battle and Kolas, 2012](https://arxiv.org/html/2609.36082#bib.bib8)\), and SOSA/SSN\([Janowicz et al\., 2019](https://arxiv.org/html/2609.36082#bib.bib9)\)for sensor observations, sampled features, and measurement processes relevant to remote sensing and meteorological data\. Although widely adopted, these ontologies remain only partially interoperable because their terms are not fully mapped to ISO\-standard ontologies and they often separate spatial and temporal domains\. The Common Core Ontologies \(CCO\)\([Jensen et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib30);[Rudnicki, 2019](https://arxiv.org/html/2609.36082#bib.bib29)\)address this gap by integrating spatial and temporal concepts in a Basic Formal Ontology\-mapped framework\([International Organization for Standardization and International Electrotechnical Commission, 2021](https://arxiv.org/html/2609.36082#bib.bib26)\)\. As a mid\-level ontology, CCO links top\-level abstractions to specialized domains, but practical deployment still requires low\-level ontologies such as MDS\-Onto\([Rajamohan et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib7)\)and GeoOutageOnto\([Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\)\.

KGQA Benchmarks\. Standard benchmarks such as LC\-QuAD 2\.0 and QALD\-9 focus on open\-domain factual questions over encyclopedic knowledge bases such as DBpedia and Wikidata\([Dubey et al\., 2019](https://arxiv.org/html/2609.36082#bib.bib38);[Usbeck et al\., 2018](https://arxiv.org/html/2609.36082#bib.bib37)\)\. However, their pattern\-based generation may not capture real\-world linguistic diversity\. Spider4SPARQL\([Kosten et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib32)\)addresses this limitation with a complex benchmark derived from 200 databases, where state\-of\-the\-art models reach only about 45 percent execution accuracy for complex SPARQL queries\.

Power System KGQA and Benchmarks\. Research on KGQA and benchmarking for power outages and resilience analysis remains limited\. BuildingQA\([Mulayim et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib33)\), which provides NL interfaces for building metadata, identifies a significant lexical gap between practitioner terminology and formal graph classes\. More broadly, NL ambiguity affects KGQA due to ambiguous schema, vague value mappings, and underspecified language\([Floratou et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib5)\)\. Other domain benchmarks include SciQA\([Auer et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib31)\)for scholarly knowledge graphs\. Unlike benchmarks over simpler encyclopedic sources, SciQA uses the Open Research Knowledge Graph \(ORKG\) to capture concept drift and varied granularities of scientific literature\. While SciQA provides a useful framework for text\-rich KGs and some geographic concepts, its geospatial coverage is limited: most of its 2,465 automatically generated questions are restricted to Computer Science\.

We remark that our work is not to develop a new KGQA system, but to advocate and demonstrate enhanced assessment capability to enable finer\-grained, in\-depth experimental analysis; to provide evaluation infrastructure and assessment tools for LLM\-based spatiotemporal analysis of power\-system resilience\.

## 2\.GeoOutageBench Overview

Figure 3\.A fraction of GeoOutageOnto\. Data classes are displayed in blue, geospatial classes are in yellow, temporal properties are in orange, and miscellaneous instrument/superclass classes are in gray\.goostands for GeoOutageOnto,mdsstands for MDS\-Onto\([Rajamohan et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib7)\), andxsdstands for XML Schema Definition\([World Wide Web Consortium, 2004](https://arxiv.org/html/2609.36082#bib.bib40)\)\.Ontology diagram showing the GeoOutageOnto subset\. Data classes are displayed in blue, geospatial classes are in yellow, temporal properties are in orange, and miscellaneous instrument/superclass classes are in gray\. \\texttt\{goo\} stands for GeoOutageOnto, \\texttt\{mds\} stands for MDS\-Onto, and \\texttt\{xsd\} stands for XML Schema Definition\. Edges represent semantic relationships among outage, storm, remote sensing, administrative region, and observation entities\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}characterizes KGQA assessment using a unified, user\-specifiedconfiguration profile𝒞=\(𝒬,𝒪,ℛ​𝒬,𝒢,ℳ,𝒫\)\{\\mathcal\{C\}\}=\(\{\\mathcal\{Q\}\},\{\\mathcal\{O\}\},\{\\mathcal\{RQ\}\},\{\\mathcal\{G\}\},\{\\mathcal\{M\}\},\{\\mathcal\{P\}\}\), where𝒬=\{qi\}i=1N\{\\mathcal\{Q\}\}=\\\{q\_\{i\}\\\}\_\{i=1\}^\{N\}is a set of NL questions,ℛ​𝒬\{\\mathcal\{RQ\}\}is the corresponding reference SPARQL\-interpretation workload,𝒪\{\\mathcal\{O\}\}and𝒢\{\\mathcal\{G\}\}are the ontology and spatiotemporal KG,ℳ\{\\mathcal\{M\}\}is a set of language models or agents, and𝒫\{\\mathcal\{P\}\}is a set of evaluation metrics\. To support the benchmark tasks \(see Section[3](https://arxiv.org/html/2609.36082#S3)\),𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}provides a libraryℒ\{\\mathcal\{L\}\}of primitive operators summarized below, where\_′′\{\}^\{\\prime\}\\\_^\{\\prime\}denotes an optional wildcard input\.

- •Multimodal Information Extractor: this operator performs data integration and extraction to curate and maintain𝒢\{\\mathcal\{G\}\}from raw multimodal data, supported by LLMs, graph data systems, and IE tools\.
- •NL query interpreter𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}\(qi,𝒪,∅,𝒢,m,𝒫\)\(q\_\{i\},\{\\mathcal\{O\}\},\\emptyset,\{\\mathcal\{G\}\},m,\{\\mathcal\{P\}\}\): takes as input an NL questionqiq\_\{i\}posed on𝒢\{\\mathcal\{G\}\}, and leverages a language modelm∈ℳm\\in\{\\mathcal\{M\}\}to generate structured SPARQL interpretationsℛ​𝒬^im\\widehat\{\\mathcal\{RQ\}\}\_\{i\}^\{m\}that can interpretqiq\_\{i\}; if𝒪\{\\mathcal\{O\}\}is specified\(𝒪≠∅\)\(\{\\mathcal\{O\}\}\\neq\\emptyset\),𝒪\{\\mathcal\{O\}\}is included in promptingmmto generateℛ​𝒬^im\\widehat\{\\mathcal\{RQ\}\}\_\{i\}^\{m\}that aligns with the classes, properties, and constraints in𝒪\{\\mathcal\{O\}\}\.
- •Graph Query Evaluator𝖰𝟤𝖠\\mathsf\{Q2A\}\(\_,\_,ℛ​𝒬,𝒢,\_,𝒫\)\(\\\_,\\\_,\{\\mathcal\{RQ\}\},\{\\mathcal\{G\}\},\\\_,\{\\mathcal\{P\}\}\): this operator calls a SPARQL engine to deterministically compute the answers of structured interpretationsℛ​𝒬​\(𝒢\)\{\\mathcal\{RQ\}\}\(\{\\mathcal\{G\}\}\)\.
- •Ontology\-based Evaluator𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{OntoQ2A\}\(qi,𝒪,ℛ​𝒬,𝒢,\_,𝒫\)\(q\_\{i\},\{\\mathcal\{O\}\},\{\\mathcal\{RQ\}\},\{\\mathcal\{G\}\},\\\_,\{\\mathcal\{P\}\}\): this operator extends𝖰𝟤𝖠\\mathsf\{Q2A\}by using𝒪\{\\mathcal\{O\}\}to identify ontology\-compatible labels and relations that satisfy the conditions expressed inℛ​𝒬\{\\mathcal\{RQ\}\}\.
- •Multimodal Evaluators𝗆𝗆𝖰𝟤𝖠\\mathsf\{mmQ2A\}\(resp\.𝗆𝗆𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{mmOntoQ2A\}\), a variant of𝖰𝟤𝖠\\mathsf\{Q2A\}\(resp\.𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{OntoQ2A\}\) that supports multimodal retrieval; rather than only querying and returning entities in𝒢\{\\mathcal\{G\}\}, it performs retrieval over multimodal data and returns a wikipage\-style document to include all relevant answers\.
- •User\-defined Functions \(UDFs\) and Tools: users register task\-specific operators withℒ\{\\mathcal\{L\}\}to extend query semantics beyond what the SPARQL engine evaluates natively\. Registered operators include a geodesic buffer for resolving vague radius expressions, spatiotemporal join, spatial indexing, kNN search, and raster\-tile access forNTLImageandOutageMapretrieval\.

These operators serve as algorithmic building blocks and are synthesized for automated assessment pipelines in𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}\.

###### Example 2\.1\.

For the ambiguous question in Example[1\.1](https://arxiv.org/html/2609.36082#S1.Thmtheorem1),𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}\(qi,𝒪,∅,𝒢,m,𝒫\)\(q\_\{i\},\{\\mathcal\{O\}\},\\emptyset,\{\\mathcal\{G\}\},m,\{\\mathcal\{P\}\}\)producesℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}, binding the underspecified state, year, SVI, and event\-count phrases to concrete entities and constraints\.𝖰𝟤𝖠\\mathsf\{Q2A\}then returns county\-level bindings \(county, SVI record, vulnerability score, storm\-event count\);𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{OntoQ2A\}checks these against ontology labels and relations, and𝗆𝗆𝖰𝟤𝖠\\mathsf\{mmQ2A\}and𝗆𝗆𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{mmOntoQ2A\}attach multimodal evidence to the answers\.

We next elaborate details of𝒪\{\\mathcal\{O\}\}and𝒢\{\\mathcal\{G\}\}\(Section[2\.1](https://arxiv.org/html/2609.36082#S2.SS1)\), competency NL questions𝒬\{\\mathcal\{Q\}\}and reference SPARQL interpretationsℛ​𝒬\{\\mathcal\{RQ\}\}\(Section[2\.2](https://arxiv.org/html/2609.36082#S2.SS2)\), benchmark tasks with matching evaluation metrics𝒫\{\\mathcal\{P\}\}\(Section[3](https://arxiv.org/html/2609.36082#S3)\) and evaluation methods built on operator libraryℒ\{\\mathcal\{L\}\}\. Figure[2](https://arxiv.org/html/2609.36082#S1.F2)illustrates the general framework\. While we make a case for power outage and storm event analysis,𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}readily extends to other geospatiotemporal KGQA query scenarios\.

### 2\.1\.GeoOutageBench Ontologies and KGs

Multimodal GeoOutageOnto𝒪\{\\mathcal\{O\}\}\. GeoOutageOnto provides the schema for GeoOutageKG and the𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}KGQA dataset\. The ontology𝒪\{\\mathcal\{O\}\}is designed as a robust representation for integrating heterogeneous geospatial and temporal resources, including satellite\-derived nighttime light images\([Román et al\., 2018](https://arxiv.org/html/2609.36082#bib.bib16)\), meteorological remote\-sensing products\([GOES\-R Series Program, 2019](https://arxiv.org/html/2609.36082#bib.bib15);[NASA Earthdata, 2026a](https://arxiv.org/html/2609.36082#bib.bib17);[NASA Earthdata, 2026b](https://arxiv.org/html/2609.36082#bib.bib18)\), county\-level customer outage records\([Tansakul et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib19)\), and storm event reports\([NOAA National Centers for Environmental Information, 2026](https://arxiv.org/html/2609.36082#bib.bib13)\)\. By defining a unified semantic layer over these resources, GeoOutageOnto supports systematic ingestion, alignment, and reasoning for power outage analysis and storm resilience applications\.

###### Example 2\.2\.

Figure[3](https://arxiv.org/html/2609.36082#S2.F3)illustrates a representative subset of GeoOutageOnto\. Classes associated with ingested datasets are spatially grounded through their administrative divisions, such as counties and states in the U\.S\. These classes are further organized by modality, distinguishing between quantitative tabular datasets and geospatial raster imagery\. GeoOutageOnto is curated to be conceptually well aligned with the Materials Data Science Ontology \(MDS\-Onto\)\([Rajamohan et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib7)\)to promote the standardization of location\- and time\-sensitive concepts\. MDS\-Onto is aligned with the Basic Formal Ontology \(BFO\)\([International Organization for Standardization and International Electrotechnical Commission, 2021](https://arxiv.org/html/2609.36082#bib.bib26)\), which justifies the well\-groundedness of GeoOutageOnto\.

Multimodal Spatiotemporal KG𝒢\{\\mathcal\{G\}\}\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}specifies𝒢\{\\mathcal\{G\}\}by default as GeoOutageKG\([Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2)\), enriched with multimodal data sources typed by𝒪\{\\mathcal\{O\}\}\. Its semantically enriched entities are grounded by real\-world geographic location, acquisition time, and record\-time metadata, forming the geospatial, temporal, and multimodal evidence base for𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}\. Full statistics on GeoOutageKG classes and instance counts are provided in Table[1](https://arxiv.org/html/2609.36082#S2.T1)\.

Table 1\.GeoOutageKG instance and triple counts by class\.ClassInstancesTriplesCustomerOutageRecord11 733 06411\\,733\\,064117 330 640117\\,330\\,640NTLImage335 469335\\,4697 907 4447\\,907\\,444OutageMap191 661191\\,6612 108 2672\\,108\\,267StormEventRecord2 012 8562\\,012\\,85636 211 66836\\,211\\,668HurricaneEventRecord3 2663\\,266128 831128\\,831SVIRecord21 99721\\,997395 940395\\,940Total14 298 31314\\,298\\,313164 082 790164\\,082\\,790Raster layer\. The raster layer uses NASA Black Marble nighttime light \(NTL\) observations\([Román et al\., 2018](https://arxiv.org/html/2609.36082#bib.bib16)\), which provide nightly imagery from 2012 to 2025 at 500 meters per pixel; because artificial illumination is a spatially explicit proxy for electricity availability, infrastructure functionality, and human activity, departures from historical radiance can indicate outage impacts when interpreted with administrative boundaries, outage records, and storm metadata\.

Raw rasters are partitioned into U\.S\. counties using county\-level bounding boxes and coordinate polygons, then transformed into outage severity maps by comparing daily radiance against the preceding three\-month average\([Frakes et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib2);[Aparcedo et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib1);[Cole et al\., 2017](https://arxiv.org/html/2609.36082#bib.bib6);[Cui et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib10);[Kalb et al\., 2018](https://arxiv.org/html/2609.36082#bib.bib11)\)\. These maps encode anomalous radiance loss as outage severity, allowing𝒢\{\\mathcal\{G\}\}to represent the spatial extent, intensity, and temporal evolution of disruption and recovery; daily NTL images and outage severity maps are serialized asNTLImageandOutageMapinstances, while the observing Suomi\-NPP satellite and VIIRS sensor are represented as instances ofSatelliteandSensor, respectively\.

![One-column multimodal data workflow for GeoOutageBench. Remote-sensing, geographic, and tabular sources feed a shared county key. The raster branch clips nighttime-light imagery by county, places the NTL image stack and outage-map stack side by side with a rightward arrow between them, and shows radiance loss below the stacks. The tabular branch uses one combined normalization and county-date alignment block for outage, storm, hurricane, and vulnerability records, then passes through overlapping cylindrical Counties boxes. Both branches instantiate county-level GeoOutageKG evidence bundles.](https://arxiv.org/html/2609.36082v1/GeoOutageBenchData.png)Figure 4\.Multimodal data construction for GeoOutageBench\. County geography provides the shared spatial key used to segment nighttime\-light imagery, derive radiance\-loss outage maps, and ground outage, storm, hurricane, and vulnerability records as county\-level GeoOutageKG evidence\.One\-column multimodal data workflow for GeoOutageBench\. Remote\-sensing, geographic, and tabular sources feed a shared county key\. The raster branch clips nighttime\-light imagery by county, places the NTL image stack and outage\-map stack side by side with a rightward arrow between them, and shows radiance loss below the stacks\. The tabular branch uses one combined normalization and county\-date alignment block for outage, storm, hurricane, and vulnerability records, then passes through overlapping cylindrical Counties boxes\. Both branches instantiate county\-level GeoOutageKG evidence bundles\.Tabular Layer\. The tabular layer comprises quantitative datasets aligned by observation time and grounded through coordinate pairs or administrative divisions, most commonly U\.S\. counties; currently, the multimodal geospatiotemporal KG𝒢\{\\mathcal\{G\}\}integrates:

\(1\) The Department of Energy’s EAGLE\-I Customer Outage Record\([Tansakul et al\., 2023](https://arxiv.org/html/2609.36082#bib.bib19);[Tansakul et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib20);[Tansakul et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib21);[Tansakul et al\., 2026](https://arxiv.org/html/2609.36082#bib.bib22);[Brelsford et al\., 2024](https://arxiv.org/html/2609.36082#bib.bib23)\), which provides county\-level outage observations at 15\-minute intervals \(2014\-2025\) and the primary temporal signal for modeling outage severity and persistence\.

\(2\) NOAA’s Storm Events Database\([NOAA National Centers for Environmental Information, 2026](https://arxiv.org/html/2609.36082#bib.bib13)\), which records county\-level storm events and damaging episodes from 1950 to 2025, contextualizing outages through meteorological events, affected locations, severity, and infrastructure disruption\.

\(3\) NOAA’s Hurricane Databases \(HURDAT\)\([National Hurricane Center, 2026](https://arxiv.org/html/2609.36082#bib.bib14)\), which contain metadata for tropical cyclones in the North Atlantic Ocean from 1851 to 2025 and the Northeastern Pacific Ocean from 1949 to 2025, including name, maximum wind strength, barometric pressure, and six\-hour status updates with location and wind strength\.

\(4\) The CDC/ATSDR Social Vulnerability Index \(SVI\)\([Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry/Geospatial Research, Analysis, and Services Program, 2022](https://arxiv.org/html/2609.36082#bib.bib24)\), which provides spatially resolved measures of relative social vulnerability across socioeconomic status, household characteristics, racial and ethnic minority status, etc\.; its theme\-specific and overall percentile rankings allow GeoOutageKG to incorporate community vulnerability when analyzing outage exposure, disaster preparedness, and post\-event recovery capacity\.

Geographic Metadata\.𝒢\{\\mathcal\{G\}\}also serializes metadata for all 3,144 U\.S\. counties and county\-equivalent entities, together with all U\.S\. states and territories, using the U\.S\. Census Bureau’s TIGER digital mapping and shapefile database\([U\.S\. Census Bureau, 2025](https://arxiv.org/html/2609.36082#bib.bib25)\)\. County and state instances include centroid coordinates, Federal Information Processing Standards \(FIPS\) codes, geohashes, and bounding\-box geometries\. We note that Storm Events, SVI, HURDAT, and TIGER geometries are ingested for all counties, while Black Marble and EAGLE\-I derived assets are currently sampled within Florida due to raster volume\. Because all layers are partitioned by the same shared county key, extending coverage requires re\-running the ingestion pipeline over additional TIGER geometries rather than revising GeoOutageOnto or the query templates\.

Table 2\.Query classification by primary category\. Coverage entries aretemplate instances / SPARQL interpretationsand are inclusive: an interpretation at a given hop depth also satisfies shallower depths\. Ambiguous spans in the example questions are shown assurface form\[binding type\]\.Coverage by hop depthCategory1\-hop2\-hopMulti\-hopTotalQ / IExample questionUnderlying operationSpatial containment12 / 137 / 87 / 812 / 13WhichU\.S\. county\[jurisdiction type\] is depicted in thepower outage severity map\[map type\] corresponding toa specific customer outage record\[outage record\]?Region containment, jurisdiction lookup, or spatial aggregationSpatial proximity4 / 94 / 94 / 94 / 9OnIan’s Florida landfall day\[event date\], what was the peak outage count inthe southwest Florida county centered on Fort Myers\[place descriptor\], and were any hurricanes active withinthe general area of the county\[distance/radius abstraction\] that day?Buffer, distance filter, spatial joinTemporal interval11 / 143 / 32 / 211 / 14How many and whichhurricanes\[cyclone type\] made landfall inLouisiana\[jurisdiction\] inAugust\[month\] across all years?Temporal filtering, aggregation, and interval comparisonEvent sequence3 / 73 / 73 / 73 / 7Whichstorm events\[event type\] inFlorida\[state\] started before same\-countyEAGLE\-I records\[outage record source\] that later exceeded50,000 outages\[outage threshold\] onthe same day\[temporal alignment\]?Temporal precedence over event graphSpatiotemporal co\-occurrence3 / 73 / 72 / 43 / 7Whichcounties\[jurisdiction type\] inthe hurricane\-prone state\[state descriptor\] hadhigh socioeconomic vulnerability\[vulnerability descriptor\] ina recent year\[relative year descriptor\] anda dozen or so wind\-related storm events\[event\-count condition\] recordedthat same year\[temporal alignment\]?Spatial–temporal joinMultimodal evidence13 / 186 / 102 / 413 / 18Whichoutage map\[map type\] corresponds tothis customer outage record\[outage record\]?KG retrieval plus file, text, or image lookupScenario / what\-if2 / 42 / 42 / 42 / 4IfFort Myers’s county\[focal place\] outage impact extended toneighboring counties\[spatial relation\] duringHurricane Ian\[event window\], which neighboring counties hadhigh social vulnerability\[vulnerability condition\] andhigh recorded outages\[outage condition\]?Constrained scenario retrieval and impact explorationOverall48 / 7228 / 4822 / 3848 / 7248 benchmark questions drawn from 11 template families, with 72 reference SPARQL interpretations\. Per\-category counts are illustrative; corpus\-level metrics are the reportable figures\.Multimodal Data Integration\. Figure[4](https://arxiv.org/html/2609.36082#S2.F4)summarizes the multimodal data\-processing workflow \(conducted by multimodal information extractor; Section[2](https://arxiv.org/html/2609.36082#S2)\): geographic boundary data partitions remote\-sensing imagery into county\-level NTL images and derived outage severity maps while also spatially grounding tabular records as geospatially segmented county\-level data\. By linking county\-partitioned raster imagery and temporally aligned tabular records through shared administrative geographies,𝒢\{\\mathcal\{G\}\}provides the semantic substrate for GeoOutageBench questions that require multimodal and geospatiotemporal reasoning over power outages, environmental hazards, and community resilience\.

### 2\.2\.Competency Query Classes

𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}considers a set of built\-in competency NL questions𝒬\{\\mathcal\{Q\}\}for evaluation tasks\. Questions in𝒬\{\\mathcal\{Q\}\}are constructed through human authoring and LLM\-assisted generation\.

NL query classes\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}characterizes the query complexity of NL questions in terms of the common practice of “hops” as in Web KGQA \(length of traversal across entities via relations in KG\), and spatiotemporal semantics\. \(1\) One\-hop queries involve direct retrieval or filtering over a single class or relation, while two\-hop and multi\-hop queries require the composition of multiple relations across linked entities\. \(2\) Each NLq∈𝒬q\\in\{\\mathcal\{Q\}\}is further assigned to one of seven categories\. Together, these categories address a useful fragment of reasoning needed for outage and storm resilience analysis, from basic spatial or temporal filtering to complex linking of administrative geographies, hazard events, outage records, remote\-sensing imagery, and vulnerability indicators\.

NL query generation\. The built\-in competency question set𝒬\{\\mathcal\{Q\}\}contains manually designed questions and LLM\-generated ones\. It starts with a set of parameterized NL templates that specify hop and query category, with ‘‘placeholders’’ for specific targeted entities and predicates\. \(1\) The human expert team consists of 11 researchers, with overlapping expertise spanning 4 environmental and energy system experts, 4 computer scientists, 2 geospatial experts, and 3 power domain experts\.111Some researchers occupy multiple positions\.The geospatial and power experts are requested to propose practical NL questions, and the full team works together to make ground\-truth interpretations\. \(2\) For LLM\-assisted question generation, an initial “draft” is generated from GeoOutageBench metadata and ontology conforming to the target query categories and template, and then manually augmented to introduce varied ambiguity and query\-generation difficulty by human experts\.

NL Query Interpretation\. For the generated competency NL questions, the CS experts provide validated ground\-truth \(“gold”\) SPARQL interpretation setsℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}for each NL questionqi∈𝒬q\_\{i\}\\in\{\\mathcal\{Q\}\}\. The resulting reference query workload isℛ​𝒬=⋃i=1Nℛ​𝒬i\{\\mathcal\{RQ\}\}=\\bigcup\_\{i=1\}^\{N\}\{\\mathcal\{RQ\}\}\_\{i\}, and the ground\-truth answers𝒜=ℛ​𝒬​\(𝒢\)\{\\mathcal\{A\}\}=\{\\mathcal\{RQ\}\}\(\{\\mathcal\{G\}\}\)are derived by executing these queries on supporting multimodal and NoSQL data engines such as graph and document data systems \(*e\.g\.,*GraphDB, MongoDB\), with their answers passing the manual validation of domain experts\.

Table[2](https://arxiv.org/html/2609.36082#S2.T2)summarizes the query categories, hop\-complexity, question and interpretation counts, representative questions, and underlying operations used to process the query workload\.

Table 3\.Major benchmark tasks and Evaluation metricsBenchmark TasksEvaluation objectExample Metrics in𝒫\{\\mathcal\{P\}\}What the metrics measureTask 1: Geospatiotemporal NL to SPARQLPredicted interpretationsℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}compared with reference interpretationsℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}Syntax; Compat\.; Exact; Alignment; Class F1; Prop\. F1; Schema F1; Spatial F1; Temporal F1; ST F1Whethermmgenerates executable, ontology\-grounded SPARQL and preserves schema, spatial, and temporal constraints under ambiguity\.Task 2: Ontology UtilityWorkload term setTaT\_\{a\}compared with ontology term setToT\_\{o\}Relevance / Query Coverage\|Ta∩To\|\|Ta\|\\frac\{\|T\_\{a\}\\cap T\_\{o\}\|\}\{\|T\_\{a\}\|\}; Accuracy / Query Closeness\|Ta∩To\|\|To\|\\frac\{\|T\_\{a\}\\cap T\_\{o\}\|\}\{\|T\_\{o\}\|\}Whether𝒪\{\\mathcal\{O\}\}declares the GeoOutageOnto vocabulary required by the SPARQL workload and how densely each query activates that vocabulary\.Task 3: Multimodal Spatiotemporal KGQAPredicted answer𝒜^im\\widehat\{\{\\mathcal\{A\}\}\}\_\{i\}^\{\\,m\}vs\. ground\-truth answer𝒜i\{\\mathcal\{A\}\}\_\{i\}over𝒢\{\\mathcal\{G\}\}Prec\.; Rec\.; Answer F1; Sel\.; Hits@kk; MRR; nDCG@1010; spatial correctnessSiS\_\{i\}; temporal correctnessTiT\_\{i\}; SRSWhether the executable query returns correct entities, values, records, images, or aggregates and preserves the question’s spatiotemporal intent\.

## 3\.Benchmark Tasks and Evaluation Metrics

Given the benchmark\-level configuration\(𝒬,𝒪,ℛ​𝒬,𝒢,ℳ,𝒫\)\(\{\\mathcal\{Q\}\},\{\\mathcal\{O\}\},\{\\mathcal\{RQ\}\},\{\\mathcal\{G\}\},\{\\mathcal\{M\}\},\{\\mathcal\{P\}\}\), an ML oracle \(*e\.g\.,*an LLM\)m∈ℳm\\in\{\\mathcal\{M\}\}deterministically maps each NL questionqi∈𝒬q\_\{i\}\\in\{\\mathcal\{Q\}\}to a generated set ofinterpretations\(*i\.e\.,*syntactically validated structured SPARQL queries\)ℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\. The corresponding generated query workload over𝒬\{\\mathcal\{Q\}\}isℛ​𝒬^m=⋃i=1Nℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}^\{\\,m\}=\\bigcup\_\{i=1\}^\{N\}\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\. Let the ground\-truth interpretations \(resp\. answers\) ofqi∈𝒬q\_\{i\}\\in\{\\mathcal\{Q\}\}beℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}\(resp\.𝒜i=ℛ​𝒬i​\(𝒢\)\{\\mathcal\{A\}\}\_\{i\}=\{\\mathcal\{RQ\}\}\_\{i\}\(\{\\mathcal\{G\}\}\),*i\.e\.,*the union of the answers of each ground\-truth SPARQL query inℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}\)\. The ground\-truth queries for𝒬\{\\mathcal\{Q\}\}and answers, denoted asℛ​𝒬=⋃i=1Nℛ​𝒬i\{\\mathcal\{RQ\}\}=\\bigcup\_\{i=1\}^\{N\}\{\\mathcal\{RQ\}\}\_\{i\}and𝒜=ℛ​𝒬​\(𝒢\)=⋃i=1N𝒜i\{\\mathcal\{A\}\}=\{\\mathcal\{RQ\}\}\(\{\\mathcal\{G\}\}\)=\\bigcup\_\{i=1\}^\{N\}\{\\mathcal\{A\}\}\_\{i\}, are defined similarly\. Table[3](https://arxiv.org/html/2609.36082#S2.T3)summarizes the metrics used by the three benchmark tasks \(see formal definitions in Appendix[A\.2](https://arxiv.org/html/2609.36082#A1.SS2)\.\)

### 3\.1\.Task 1: Assessing LLM’s ability in Geospatiotemporal Understanding

The first benchmark task assesses LLMsℳ\{\\mathcal\{M\}\}’s ability of geospatiotemporal understanding, by quantifying the quality of its query interpretation\. We introduce three groups of metrics below, in terms of interpretation ambiguity, query accuracy, and accuracy of answers from interpreted queries compared with ground\-truth SPARQL queries and answers\.

Interpretation Ambiguity\. For each questionqi∈𝒬q\_\{i\}\\in\{\\mathcal\{Q\}\}, the evaluator records the number of ground\-truth interpretations\|ℛ​𝒬i\|\|\{\\mathcal\{RQ\}\}\_\{i\}\|and generated interpretations\|ℛ​𝒬^im\|\|\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\|; questions with multiple interpretations indicate ambiguity in the NL\-to\-SPARQL mapping\.

Query Accuracy\. We measure the accuracy of modelmmby comparing generated SPARQL interpretationsℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}with ground\-truth interpretationsℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}\. The evaluator reports normalized exact original\-query match, interpretation alignment, syntactic validity, and endpoint compatibility with𝒢\{\\mathcal\{G\}\}\. In the Task 1 workflow, interpretation alignment is the structural pair\-selection score used for each candidate SPARQL interpretation generated by𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}perqiq\_\{i\}\. For each generated–gold pair\(r^,r\)∈ℛ​𝒬^im×ℛ​𝒬i\(\\hat\{r\},r\)\\in\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\\times\{\\mathcal\{RQ\}\}\_\{i\}, the evaluator extracts class, property, spatial\-constraint, and temporal\-constraint feature sets, computes the corresponding F1 scores, and takes the maximum average score over all pairs \(interpretations\) forqiq\_\{i\}\. Thus, alignment measures whether an interpretation preserves the ontology grounding and spatiotemporal constraints of a gold counterpart, rather than whether its syntax exactly matches a gold counterpart\.

Schema\-level F1 measures\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}also provides schema\-level query accuracy by extracting semantic components from interpreted and ground\-truth SPARQL queries, and reporting F1 scores for classes, properties, schema\-linking, spatial constraints, temporal constraints, and spatiotemporal constraints\. For ambiguous questions, every model\-generated interpretation is scored against its ground\-truth counterpart, and the best aligned generated–gold pair provides the schema\-level scores\. This group includes three metrics: class F1, property F1, and schema\-linking F1\. Class and property F1 compare the classes and object or datatype properties in ground\-truth and interpreted queries\. Schema\-linking F1 is the harmonic mean of class and property F1, penalizing outputs that recover only entities or relations\.

Quality of Query Answers\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}provides a rich set of evaluation metrics for quality of query answers, including \(1\) commonly used precision, recall, and F1 measures, and \(2\) Hits@11, Hits@55, Hits@1010, MRR, nDCG@1010, spatial correctness, temporal correctness, and SRS, to distinguish the assessment of entity\-level correctness and spatiotemporal\-focused interpretation\.

𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}can also be easily configured to perform in\-depth, finer\-grained assessment for spatiotemporal reasoning, with performance metric𝒫\{\\mathcal\{P\}\}specified as follows\.

Spatiotemporal\-focused Metrics\. For spatial and temporal reasoning,𝒫\{\\mathcal\{P\}\}specifies three F1 scores for individual competency query classes\. This includes spatial\-constraint, temporal\-constraint, and spatiotemporal\-constraint\. Spatial\-constraint F1 captures constraints such as geometry vectors, coordinates, distance functions, and spatial predicates; temporal\-constraint F1 captures temporal literals, datatype use, date operators, and interval filters\. Spatiotemporal\-constraint F1 is the harmonic mean of spatial and temporal F1\.

The benchmark reports the macro\-average ofS​T​F​1iSTF1\_\{i\}across questions and the corpus\-level harmonic mean of macro spatial\- and temporal\-constraint F1\. The question\-level score is stricter because it requires both constraint types to be present within the same selected interpretation, rather than aggregating them independently across the benchmark\. We provide a detailed description of the metrics in Appendix[A\.2](https://arxiv.org/html/2609.36082#A1.SS2)\.

Evaluation method\. The assessment is triggered by a configuration\(𝒬,𝒪,∅,𝒢,ℳ,𝒫\)\(\{\\mathcal\{Q\}\},\{\\mathcal\{O\}\},\\emptyset,\{\\mathcal\{G\}\},\{\\mathcal\{M\}\},\{\\mathcal\{P\}\}\), with𝒫\{\\mathcal\{P\}\}specified as ambiguity and accuracy \(a vector of accuracy metrics\), by default\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}invokes operator𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}\(qi,𝒪,∅,𝒢,m,𝒫\)\(q\_\{i\},\{\\mathcal\{O\}\},\\emptyset,\{\\mathcal\{G\}\},m,\{\\mathcal\{P\}\}\)\(see Appendix[A\.1](https://arxiv.org/html/2609.36082#A1.SS1)\) withqiq\_\{i\}ranging over𝒬\{\\mathcal\{Q\}\}andmmoverℳ\{\\mathcal\{M\}\}to produce an assessment report\.

### 3\.2\.Task 2: Ontology Utility Evaluation

Task 2 aims to evaluate whether an ontology𝒪\{\\mathcal\{O\}\}\(e\.g\., GeoOutageOnto\) is usable and useful by measuring how well it supplies the domain vocabulary required by the reference interpretationsℛ​𝒬\{\\mathcal\{RQ\}\}or model\-generated interpretationsℛ​𝒬^m\\widehat\{\\mathcal\{RQ\}\}^\{m\}\.

Evaluation Metric\. Given an interpretation workloadℛ​𝒬\{\\mathcal\{RQ\}\}\(orℛ​𝒬^m\\widehat\{\\mathcal\{RQ\}\}^\{m\}\), letTaT\_\{a\}denote the ontology terms occurring in the supplied SPARQL interpretations\. For a scope radiushh, we construct the scoped ontology term setToT\_\{o\}as follows: for each class labell∈Tal\\in T\_\{a\}with an instancev∈𝒢v\\in\{\\mathcal\{G\}\}typed byll,ToT\_\{o\}includes all ontology terms in𝒪\{\\mathcal\{O\}\}associated with nodes reachable withinhhhops ofvv\. Intuitively,ToT\_\{o\}includes all the concept classes that may contribute in answering𝒬\{\\mathcal\{Q\}\}within a scope specified byhh\-hop neighbors\. We report two complementary metrics:Query Coverage, defined as\|Ta∩To\|\|Ta\|\\frac\{\|T\_\{a\}\\cap T\_\{o\}\|\}\{\|T\_\{a\}\|\}, andQuery Closeness, defined as\|Ta∩To\|\|To\|\\frac\{\|T\_\{a\}\\cap T\_\{o\}\|\}\{\|T\_\{o\}\|\}\. The coverage measure quantifies the fraction of workload terms that are also mentioned in the ontology; higher values are preferable\. The closeness penalizes irrelevant concepts not seen in𝒬\{\\mathcal\{Q\}\}; the larger, the better\.

Evaluation Method\. Task 2 is triggered by configuration\(𝒬,𝒪,ℛ​𝒬,𝒢,ℳ,𝒫\)\(\{\\mathcal\{Q\}\},\{\\mathcal\{O\}\},\{\\mathcal\{RQ\}\},\{\\mathcal\{G\}\},\{\\mathcal\{M\}\},\{\\mathcal\{P\}\}\)\. Ifℛ​𝒬=∅\{\\mathcal\{RQ\}\}=\\emptyset, it invokes𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}to generateℛ​𝒬^m\\widehat\{\\mathcal\{RQ\}\}^\{m\}from𝒬\{\\mathcal\{Q\}\}and uses the generated interpretations as the query workload\. It then computes Query Coverage and Query Closeness and may call𝖰𝟤𝖠\\mathsf\{Q2A\}or𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{OntoQ2A\}for answer\-accuracy comparison\. The computation pipeline adapts OntoCheck\([Kundu et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib36)\), an ontology\-assessment package, to support ontology\-grounded SPARQL processing\.

### 3\.3\.Task 3: Multimodal Spatiotemporal KGQA

Task 3 evaluates the impact of multimodal data and queries on answer quality by comparing structured KGQA workflows \(𝖰𝟤𝖠\\mathsf\{Q2A\}or𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{OntoQ2A\}\) with multimodal counterparts that explicitly retrieve multimodal information \(𝗆𝗆𝖰𝟤𝖠\\mathsf\{mmQ2A\}or𝗆𝗆𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{mmOntoQ2A\}\)\. Table[3](https://arxiv.org/html/2609.36082#S2.T3)summarizes the answer\-quality metrics\.

Evaluation Metric\. Task 3 uses the answer\-quality metrics listed in Task 1, with a few KGQA\-specific additions\. Sel\. denotes the selected interpretation\-pair score used when multiple generated and reference interpretations are available\. Spatial correctnessSiS\_\{i\}and temporal correctnessTiT\_\{i\}check whether the returned answers preserve the expected spatial and temporal constraints, and the Spatiotemporal Relevance Score \(SRS\) summarizes this spatiotemporal agreement at the answer level\.

Evaluation Method\. Task 3 is triggered by a configuration\(𝒬,𝒪,ℛ​𝒬,𝒢,ℳ,𝒫\)\(\{\\mathcal\{Q\}\},\{\\mathcal\{O\}\},\{\\mathcal\{RQ\}\},\{\\mathcal\{G\}\},\{\\mathcal\{M\}\},\{\\mathcal\{P\}\}\)\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}executes each reference or generated interpretation over𝒢\{\\mathcal\{G\}\}with𝖰𝟤𝖠\\mathsf\{Q2A\},𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{OntoQ2A\},𝗆𝗆𝖰𝟤𝖠\\mathsf\{mmQ2A\}, or𝗆𝗆𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{mmOntoQ2A\}, compares𝒜^im\\widehat\{\{\\mathcal\{A\}\}\}\_\{i\}^\{\\,m\}with𝒜i\{\\mathcal\{A\}\}\_\{i\}, and reports macro\-averages over the best selected answer pair for eachqi∈𝒬q\_\{i\}\\in\{\\mathcal\{Q\}\}\.

Table 4\.Task 1 macro results for NL2SPARQL over 48 GeoOutageBench questions\.ModelSyntaxCompat\.ExactAlign\.Class F1Prop\. F1Schema F1Spatial F1Temporal F1ST F1GPT\-5\.51\.0001\.0000\.0000\.7450\.7160\.6790\.6780\.8700\.7150\.662Gemini 3\.1 Pro1\.0000\.9790\.0210\.7490\.8340\.7030\.7370\.7830\.6750\.578Claude Opus 4\.71\.0001\.0000\.0000\.8050\.8070\.7740\.7720\.8860\.7540\.711Table 5\.Assessment of LLMs in Task 1, with finer\-grained analysis over spatiotemporal axis\. S abbreviates Spatial, T abbreviates Temporal, and ST abbreviates Spatiotemporal\.ModelAxisAlign\.S F1T F1ST F1GPT\-5\.5Non\-ST0\.5851\.0000\.2500\.250GPT\-5\.5S0\.6990\.9690\.6250\.607GPT\-5\.5T0\.7511\.0000\.6260\.676GPT\-5\.5ST0\.7760\.8120\.8070\.725Gemini 3\.1 ProNon\-ST0\.6881\.0000\.2500\.250Gemini 3\.1 ProS0\.6970\.8750\.5000\.375Gemini 3\.1 ProT0\.7340\.7500\.6090\.485Gemini 3\.1 ProST0\.7710\.7380\.7800\.682Claude Opus 4\.7Non\-ST0\.6881\.0000\.2500\.250Claude Opus 4\.7S0\.7270\.8750\.6250\.500Claude Opus 4\.7T0\.7850\.7500\.6390\.485Claude Opus 4\.7ST0\.8420\.8910\.8630\.850Table 6\.Task 3 results over 48 questions \(Sel\.: interpretation selection; SRS: Spatiotemporal Relevance Score\)ModelPrec\.Rec\.Answer F1Sel\.Hits@1Hits@5Hits@10MRRnDCG@10SRSGPT\-5\.50\.5290\.5820\.5220\.7540\.5420\.5830\.5830\.5530\.5300\.785Gemini 3\.1 Pro0\.3230\.4600\.3140\.6690\.4170\.4380\.4380\.4240\.3870\.739Claude Opus 4\.70\.5460\.5450\.5400\.7920\.5210\.5420\.5420\.5310\.5300\.850

## 4\.Experimental Evaluation

We next make a case to demonstrate𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}in assessing spatiotemporal KGQA for power outage event analysis\. We initialize the configuration𝒞\{\\mathcal\{C\}\}by default, where \(1\)𝒪\{\\mathcal\{O\}\}and𝒢\{\\mathcal\{G\}\}refer to GeoOutageOnto and GeoOutageKG, respectively, \(2\)𝒬\{\\mathcal\{Q\}\}is a sampled competency query workload spanning all the categories, \(3\)ℳ\{\\mathcal\{M\}\}includes three frontier LLMs: GPT\-5\.5 with Extended Thinking, Gemini 3\.1 Pro with Extended Thinking, and Claude Opus 4\.7 with Extra Effort and Adaptive Thinking; and \(4\)𝒫\{\\mathcal\{P\}\}is selected to match the benchmark tasks\.

Environment\.𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}is developed in Python with the RDFLib\([Krech et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib39)\)library\. All tests are conducted on Intel\(R\) Core\(TM\) i5\-14400F CPU @ 2\.50GHz, 32 GB Memory, 10 cores, and 1 8GB NVIDIA\(R\) RTX\(TM\) 4060 GPU\. Our source code and datasets are made available222https://github\.com/UCF\-SAGE/GeoOutageBench\. A live demo of GeoOutageBench, as well as additional resources of the GeoResilience project, will be available on our sandbox website333https://purl\.archive\.org/georesilience\.

Evaluation\-1: Assessing LLM for Power Outage KGQA\. The first evaluation measures whether a modelm∈ℳm\\in\{\\mathcal\{M\}\}can translate an ambiguous NL questionqiq\_\{i\}about power\-outage and storm\-event analysis into ontology\-grounded SPARQL\. We evaluate𝒬eval⊆𝒬\{\\mathcal\{Q\}\}\_\{\\mathrm\{eval\}\}\\subseteq\{\\mathcal\{Q\}\}with\|𝒬eval\|=48\|\{\\mathcal\{Q\}\}\_\{\\mathrm\{eval\}\}\|=48, where eachqiq\_\{i\}is associated with at most 3 valid SPARQL interpretations inℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}\. Table[4](https://arxiv.org/html/2609.36082#S3.T4)summarizes corpus\-level results, and Table[5](https://arxiv.org/html/2609.36082#S3.T5)reports a finer\-grained ablation assessment over spatial \(S\), temporal \(T\), spatiotemporal \(ST\), and non\-spatiotemporal \(Non\-ST\)\.

Interpretation ambiguity\. The 48\-question evaluation set contains 72 reference SPARQL interpretations, with eachqiq\_\{i\}associated with at most three valid readings\. Gemini 3\.1 Pro generated the fewest interpretations \(51\), while GPT\-5\.5 generated 60 and Claude Opus 4\.7 generated 61\. We treat these counts as an ambiguity diagnostic: fewer generated interpretations reduce downstream execution cost, whereas larger interpretation sets can cover more alternative readings of the same NL question\.

Query accuracy\. We next report the query accuracy \(the first three columns of Table[4](https://arxiv.org/html/2609.36082#S3.T4)\)\. All three LLMs inℳ\{\\mathcal\{M\}\}generate syntactically valid SPARQL interpretations\. GPT\-5\.5 and Claude Opus 4\.7 also achieve full endpoint compatibility with𝒢\{\\mathcal\{G\}\}, while Gemini 3\.1 Pro achieves 0\.979 compatibility\. Exact query\-text match is near zero, which is expected because valid interpretations may differ in variable names, projections, join order, filters, or the selected reading of an ambiguous NL question\.

Alignment & schema performance\.Interpretation alignment, defined in Section[3\.1](https://arxiv.org/html/2609.36082#S3.SS1), is therefore a more informative Task 1 signal than exact match\. Claude Opus 4\.7 obtains the highest alignment score \(0\.805\), followed by Gemini 3\.1 Pro \(0\.749\) and GPT\-5\.5 \(0\.745\)\. At the schema level, Gemini achieves the highest class F1 \(0\.834\), but Claude leads on property F1 \(0\.774\) and schema\-linking F1 \(0\.772\), showing stronger joint recovery of classes and properties\.

Spatiotemporal performance\.Claude Opus 4\.7 also leads the corpus\-level spatial\-, temporal\-, and spatiotemporal\-constraint F1 scores, with 0\.886, 0\.754, and 0\.711, respectively\. GPT\-5\.5 is close on spatial grounding \(0\.870\) and reaches 0\.662 ST F1, while Gemini 3\.1 Pro is lower on the same spatiotemporal measures \(0\.783 spatial F1, 0\.675 temporal F1, and 0\.578 ST F1\)\. The results in Table[5](https://arxiv.org/html/2609.36082#S3.T5)reinforce this pattern: Claude performs best on explicitly spatiotemporal questions, with 0\.842 alignment and 0\.850 ST F1 on the ST axis\.

Overall performance\. Overall, Claude Opus 4\.7 provides the strongest Task 1 performance for NL spatiotemporal understanding because it combines the best alignment, schema\-linking, and spatiotemporal\-constraint scores\. GPT\-5\.5 is the most competitive alternative on execution reliability, matching Claude on syntax and endpoint compatibility and remaining close on spatial F1\. Gemini 3\.1 Pro is strongest at class recovery and is the only model with a nonzero exact\-match score, but its lower compatibility and spatiotemporal\-constraint scores reduce its overall performance on ontology\-grounded spatiotemporal interpretation\. We report details \(*e\.g\.,*prompts\) in Appendix[A\.1](https://arxiv.org/html/2609.36082#A1.SS1)\.

Evaluation\-2: Ontology Utility Evaluation\. We evaluated the coverage of𝒪\{\\mathcal\{O\}\}over the SPARQL workload using OntoCheck’s task\-based framework\([Kundu et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib36)\)\. The query workload contains 72 query interpretations\. For the case of GeoOutageOnto𝒪\{\\mathcal\{O\}\}, the Query Coverage is 1\.0: every extractedgoo:domain term in the query workload has been declared in𝒪\{\\mathcal\{O\}\}, leaving no extracted GeoOutageOnto term unanchored in the schema\. This result is expected given the co\-design of GeoOutageOnto and the benchmark workload\. The Query Closeness, which measures schema\-utilization, ranges from 0\.0196 for simple single\-predicate lookups to 0\.1373 for the most compositionally complex queries, with a mean of 0\.0673\. This range reflects benchmark heterogeneity rather than an ontology weakness: simple queries activate a small subset of the vocabulary, whereas multi\-hop, spatiotemporal, and aggregative queries require broader schema coverage\.

Case study on competency queries\. We present three query\-case analyses of LLM\-generated interpretations and compare their answers\. \(1\) Queries 70–72 \(Average Query Closeness = 0\.1078\) are different interpretations for the same NL question “Which counties in the hurricane\-prone state had high socioeconomic vulnerability in a recent year, and a dozen or so wind\-related storm events recorded that year?” with different predicates posed on state, year, vulnerability threshold, event type, and count; all conform to the same GeoOutageOnto vocabulary\. \(2\) Queries 22–24 \(Average Query Closeness = 0\.0490\) resolve “around Hurricane Ian” as a 15\-day impact window \(2022\-09\-23 to 2022\-10\-08\), a two\-month disruption period \(2022\-08\-28 to 2022\-10\-28\), or the Florida landfall date \(2022\-09\-28 to 2022\-09\-29 UTC\), while consistently grounding Lee County severity maps through ontology\-defined imagery classes and spatial, temporal, and map\-association properties\. \(3\) Queries 37–39 \(Average Query Closeness = 0\.0686\) resolve “within the general area of Cleveland County” to within a 10 km radius, a 50 km radius, and a buffered Moore tornado event window, respectively\.

These observations support two conclusions\. First, the consistently high query coverage indicates that𝒪\{\\mathcal\{O\}\}covers all extractedgoo:domain terms in the query workload, suggesting that GeoOutageOnto provides a grounded schema reference for power\-outage analysis over the spatiotemporal GeoOutageKG𝒢\{\\mathcal\{G\}\}\. Second, query\-level closeness variation shows that different workload types activate different portions of𝒪\{\\mathcal\{O\}\}: simple lookups use small schema subsets, while multi\-hop, spatiotemporal, multimodal, and aggregative queries use broader vocabulary\. The per\-query scores make this variation visible, including the multimodal evidence interpretations in queries 46–48 and the alternative interpretations of the ambiguous outage\-and\-storm question in queries 70–72\. A full list of per\-query scores is reported in Table[7](https://arxiv.org/html/2609.36082#A1.T7), Appendix[A\.2](https://arxiv.org/html/2609.36082#A1.SS2)\.

Evaluation\-3: Multimodal KGQA\. The third evaluation executesℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}against𝒢\{\\mathcal\{G\}\}and compares the generated answer set𝒜^im\\widehat\{\{\\mathcal\{A\}\}\}\_\{i\}^\{\\,m\}against ground\-truth answers𝒜i\{\\mathcal\{A\}\}\_\{i\}\. This task differs from query accuracy in Task 1, because a query can be syntactically valid and structurally plausible yet still return an irrelevant answer\.

Table[6](https://arxiv.org/html/2609.36082#S3.T6)emphasizes that answer\-level evaluation over𝒜i\{\\mathcal\{A\}\}\_\{i\}is more discriminative than syntax or executability\. Claude Opus 4\.7 obtains the highest answer F1 and SRS, while GPT is comparable in answer F1 and slightly higher on Hits@1; Gemini 3\.1 Pro lags in answer F1 despite strong Task 1 syntax and class\-recovery scores\. This gap shows that correct ontology vocabulary alone does not ensure KGQA correctness: small errors in property choice, temporal filtering, aggregation, or administrative joins can preserve a plausible query form while changing the returned answers\.

The relatively higher SRS scores for GPT and Claude indicate that many generated queries preserve the broad spatial and temporal intent even when the answer rows differ from gold results\. This is typically because the query relaxes predicates, omits an aggregation, or returns less relevant evidence such as outage records rather than derived outage maps\. Complexity and template breakdowns show the same observation\. Two\-hop questions are much easier than multi\-hop questions for GPT \(0\.992 vs\. 0\.310 answer F1\) and Claude \(0\.833 vs\. 0\.359\), while Gemini drops most sharply on multi\-hop questions \(0\.151\)\. Template\-level results show that direct lookup and outage\-ranking queries are generally more reliable, while storm\-event/SVI threshold queries are model\-dependent, with strong performance for Claude but weaker results for Gemini\. In contrast, hurricane\-landfall ranking and event\-sequence questions remain difficult because they require coordinated event ranking, spatial aggregation, storm semantics, and a proper setting of spatiotemporal\-sensitive vulnerability thresholds\.

![Two-panel case-study figure comparing single-modal and multimodal KGQA. The single-modal panel asks for Florida counties in 2022 with high socioeconomic vulnerability and at least twelve wind-related storm events, returning SVI records, storm-event counts, and county rows. The multimodal panel requests nighttime-light images and derived outage maps around Hurricane Ian for those counties, returning county bindings along with image assets from September 29 and September 30, 2022.](https://arxiv.org/html/2609.36082v1/single-vs-multimodal-case-study.png)Figure 5\.Single\- versus multimodal querying for the outage\-and\-storm case study\. The multimodal variant supplements county\-level tabular answers with date\-aligned NTL images and outage maps, enabling evidence\-aware visual inspection and richer answer\-level assessment\.Two\-panel case\-study figure comparing single\-modal and multimodal KGQA\. The single\-modal panel asks for Florida counties in 2022 with high socioeconomic vulnerability and at least twelve wind\-related storm events, returning SVI records, storm\-event counts, and county rows\. The multimodal panel requests nighttime\-light images and derived outage maps around Hurricane Ian for those counties, returning county bindings along with image assets from September 29 and September 30, 2022\.Single Modal vs\. Multimodal Querying: A Case Study\. We use the ambiguous outage\-and\-storm question from Example[1\.1](https://arxiv.org/html/2609.36082#S1.Thmtheorem1)to illustrate what changes when KGQA moves from a single\-modal, tabular answer surface to a multimodal one\. A single\-modal variant asks:“Which Florida counties in 2022 had high socioeconomic vulnerability and at least twelve wind\-related storm events?”This query returns county, CDC SVI, and NOAA Storm Events bindings, but no remote\-sensing evidence of the outage footprint, leading to potentially biased or less interpretable results\.

The multimodal variant keeps the same county\-level analytic intent but adds an evidence requirement:“For those counties, which nighttime\-light images and derived outage maps around Hurricane Ian corroborate outage severity?”In this setting,𝗆𝗆𝖰𝟤𝖠\\mathsf\{mmQ2A\}or𝗆𝗆𝖮𝗇𝗍𝗈𝖰𝟤𝖠\\mathsf\{mmOntoQ2A\}returns not only the county, SVI, and storm\-event bindings, but also date\-alignedgoo:NTLImageandgoo:OutageMapassets linked through the same county and time constraints\. Figure[5](https://arxiv.org/html/2609.36082#S4.F5)displays how additional image evidence changes the assessment target: the benchmark can grade entity\-level correctness while also checking whether the retrieved multimodal bundle supports the intended spatiotemporal explanation\. This pattern is useful for downstream resilience applications in which analysts need both ranked county answers and inspectable evidence for outage extent, recovery timing, or vulnerability\-aware resource prioritization\.

## 5\.Conclusions

We have introduced𝖦𝖾𝗈𝖮𝗎𝗍𝖺𝗀𝖾𝖡𝖾𝗇𝖼𝗁\\mathsf\{GeoOutageBench\}, a domain\-specific KGQA benchmark for evaluating LLM\-based spatiotemporal KGQA for outage, social\-vulnerability, and storm\-resilience analysis\. The benchmark curates multimodal data into a geospatiotemporal knowledge graph \(GeoOutageKG\) and ontology \(GeoOutageOnto\)\. It supports automatic generation of diverse competency queries that cover seven geospatiotemporal templates and provides a built\-in library of primitive operators for efficient assessment\. It provides comprehensive performance metrics to assess LLMs’ spatiotemporal understanding, usability and utility of ontologies, and multimodal KGQA\. Our experimental evaluation verifies that Claude Opus 4\.7 achieves the best overall performance in NL spatiotemporal understanding, while GPT\-5\.5 remains competitive on executability and spatial grounding, and Gemini 3\.1 Pro is strongest on class recovery\. Meanwhile, our query\-level and finer\-grained assessment verifies the effectiveness of GeoOutageOnto and multimodal GeoOutageKG for improving the answer accuracy for power outage analysis\.

Future work will broaden the assessment to richer query semantics, including cross\-modal and cross\-model \(hybrid raster\-and\-tabular\) retrieval, counterfactual spatiotemporal reasoning, and other causes of energy disruption such as cyberattacks and grid overload\. We will also extend the Black Marble and EAGLE\-I layers beyond the sampled Florida counties\.

###### Acknowledgements\.

This material is based upon research in the Materials Data Science for Stockpile Stewardship Center of Excellence \(MDS3\-COE\), and supported by the Department of Energy’s National Nuclear Security Administration under Award Number\(s\) DE\-NA0004104\. The authors thank the CWRU University Technology center and the UCF Advanced Research Computing Center for their High Performance Computing \(HPC\) resources, which were utilized in this work\.

## Declaration of the use of Generative AI

The usage of Large Language Models and Agents, including ChatGPT, Google Gemini, Claude, and coding agents such as OpenAI Codex, was employed to automate code and experiment script generation, with human refinement and editing\. LLMs were also used to help in refining portions of text, tables, and figures\.

## References

- Aparcedoet al\.\(2024\)A\. Aparcedo, C\. Lopez, A\. Kotta, and M\. LiMultimodal power outage prediction for rapid disaster response and resource allocation\.In2024 IEEE 52nd Photovoltaic Specialist Conference \(PVSC\),Vol\.,pp\. 1093–1099\.External Links:[Document](https://dx.doi.org/10.1109/PVSC57443.2024.10749545)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p4.1)\.
- Aueret al\.\(2023\)S\. Auer, D\. A\. C\. Barone, C\. Bartz, E\. G\. Cortes, M\. Y\. Jaradeh, O\. Karras, M\. Koubarakis, D\. Mouromtsev, D\. Pliukhin, D\. Radyush, I\. Shilin, M\. Stocker, and E\. TsalapatiThe SciQA scientific question answering benchmark for scholarly knowledge\.Scientific Reports13\(1\),pp\. 7240\.External Links:[Document](https://dx.doi.org/10.1038/s41598-023-33607-z)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p13.1),[§1](https://arxiv.org/html/2609.36082#S1.p2.1)\.
- Battle and Kolas \(2012\)R\. Battle and D\. KolasEnabling the geospatial semantic web with parliament and GeoSPARQL\.Semantic Web3,pp\. 355–370\.External Links:[Document](https://dx.doi.org/10.3233/SW-2012-0065)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Böcklinget al\.\(2024\)M\. Böckling, H\. Paulheim, and S\. DetzlerComparing spatial\-temporal knowledge graph on spatial downstream tasks\.InProceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems,pp\. 581–584\.External Links:[Document](https://dx.doi.org/10.1145/3678717.3691321)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p1.1)\.
- Brelsfordet al\.\(2024\)C\. Brelsford, S\. Tennille, A\. Myers, S\. Chinthavali, V\. Tansakul, M\. Denman, M\. Coletti, J\. Grant, S\. Lee, K\. Allen,et al\.A dataset of recorded electricity outages by United States county 2014–2022\.Scientific Data11\(1\),pp\. 271\.External Links:[Document](https://dx.doi.org/10.1038/s41597-024-03095-5)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p6.1)\.
- Brickley \(2003\)D\. BrickleyBasic Geo \(WGS84 lat/long\) Vocabulary\.Note:W3C Semantic Web Interest GroupAccessed: 2026\-05\-25External Links:[Link](http://www.w3.org/2003/01/geo/wgs84_pos#)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry/Geospatial Research, Analysis, and Services Program \(2022\)Centers for Disease Control and Prevention/Agency for Toxic Substances and Disease Registry/Geospatial Research, Analysis, and Services ProgramCDC/ATSDR Social Vulnerability Index 2022 Database: U\.S\.\.Note:[https://www\.atsdr\.cdc\.gov/place\-health/php/svi/svi\-data\-documentation\-download\.html](https://www.atsdr.cdc.gov/place-health/php/svi/svi-data-documentation-download.html)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p9.1)\.
- Coleet al\.\(2017\)T\. A\. Cole, D\. W\. Wanik, A\. L\. Molthan, M\. O\. Román, and R\. E\. GriffinSynergistic use of nighttime satellite data, electric utility infrastructure, and ambient population to improve power outage detections in urban areas\.Remote Sensing9\(3\)\.External Links:[Link](https://www.mdpi.com/2072-4292/9/3/286),ISSN 2072\-4292,[Document](https://dx.doi.org/10.3390/rs9030286)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p4.1)\.
- Cox and Little \(2017\)S\. Cox and C\. LittleTime ontology in OWL\.Note:W3C RecommendationAvailable at[https://www\.w3\.org/TR/owl\-time/](https://www.w3.org/TR/owl-time/)External Links:[Link](https://www.w3.org/TR/owl-time/)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Cuiet al\.\(2023\)H\. Cui, S\. Qiu, Y\. Wang, Y\. Zhang, Z\. Liu, K\. Karila, J\. Jia, and Y\. ChenDisaster\-caused power outage detection at night using VIIRS DNB images\.Remote Sensing15\(3\)\.External Links:[Link](https://www.mdpi.com/2072-4292/15/3/640),ISSN 2072\-4292,[Document](https://dx.doi.org/10.3390/rs15030640)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p4.1)\.
- Dubeyet al\.\(2019\)M\. Dubey, D\. Banerjee, A\. Abdelkawi, and J\. LehmannLC\-quad 2\.0: a large dataset for complex question answering over wikidata and dbpedia\.InThe Semantic Web – ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand, October 26–30, 2019, Proceedings, Part II,Berlin, Heidelberg,pp\. 69–78\.External Links:ISBN 978\-3\-030\-30795\-0,[Link](https://doi.org/10.1007/978-3-030-30796-7_5),[Document](https://dx.doi.org/10.1007/978-3-030-30796-7%5F5)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p12.1),[§1](https://arxiv.org/html/2609.36082#S1.p2.1)\.
- Floratouet al\.\(2024\)A\. Floratou, F\. Psallidas, F\. Zhao, S\. Deep, G\. Hagleither, W\. Tan, J\. Cahoon, R\. Alotaibi, J\. Henkel, A\. Singla,et al\.Nl2sql is a solved problem… not\!\.InCIDR,Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p13.1)\.
- Frakeset al\.\(2025\)E\. Frakes, Y\. Wu, R\. H\. French, and M\. LiGeoOutageKG: a multimodal geospatiotemporal knowledge graph for multiresolution power outage analysis\.InThe Semantic Web – ISWC 2025,D\. Garijo, S\. Kirrane, A\. Salatino, C\. Shimizu, M\. Acosta, A\. G\. Nuzzolese, S\. Ferrada, T\. Soulard, K\. Kozaki, H\. Takeda, and A\. L\. Gentile \(Eds\.\),Cham,pp\. 221–239\.External Links:ISBN 978\-3\-032\-09530\-5,[Document](https://dx.doi.org/10.1007/978-3-032-09530-5%5F13),2507\.22878Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p1.1),[§1](https://arxiv.org/html/2609.36082#S1.p11.1),[§1](https://arxiv.org/html/2609.36082#S1.p6.1),[§1](https://arxiv.org/html/2609.36082#S1.p9.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p2.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p4.1)\.
- GOES\-R Series Program \(2019\)GOES\-R Series ProgramNOAA GOES\-R Series Advanced Baseline Imager \(ABI\) Level 0 Data\.Note:NOAA National Centers for Environmental InformationExternal Links:[Document](https://dx.doi.org/10.25921/tvws-w071)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p1.1)\.
- International Organization for Standardization and International Electrotechnical Commission \(2021\)International Organization for Standardization and International Electrotechnical CommissionISO/IEC 21838\-2:2021: Information technology – Top\-level ontologies \(TLO\) – Part 2: Basic Formal Ontology \(BFO\)\.International Organization for Standardization,Geneva, Switzerland\.Note:[https://www\.iso\.org/standard/74572\.html](https://www.iso.org/standard/74572.html)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1),[Example 2\.2](https://arxiv.org/html/2609.36082#S2.Thmtheorem2.p1.1)\.
- Janowiczet al\.\(2019\)K\. Janowicz, A\. Haller, S\. J\. D\. Cox, D\. Le Phuoc, and M\. LefrançoisSOSA: a lightweight ontology for sensors, observations, samples, and actuators\.Journal of Web Semantics56,pp\. 1–10\.External Links:[Document](https://dx.doi.org/10.1016/j.websem.2018.06.003)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Jensenet al\.\(2024\)M\. Jensen, G\. D\. Colle, S\. Kindya, C\. More, A\. P\. Cox, and J\. BeverleyThe Common Core Ontologies\.arXiv\.External Links:2404\.17758,[Document](https://dx.doi.org/10.48550/arXiv.2404.17758)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Kalbet al\.\(2018\)V\. Kalb, Z\. Wang, M\. Román, Q\. Sun, A\. Molthan, and L\. SchultzMonitoring disaster\-related power outages using NASA Black Marble nighttime light product\.ISPRS \- International Archives of the Photogrammetry, Remote Sensing and Spatial Information SciencesXLII\-3,pp\. 1853–1856\.External Links:[Document](https://dx.doi.org/10.5194/isprs-archives-XLII-3-1853-2018)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p4.1)\.
- Kostenet al\.\(2023\)C\. Kosten, P\. Cudré\-Mauroux, and K\. StockingerSpider4SPARQL: a complex benchmark for evaluating knowledge graph question answering systems\.In2023 IEEE International Conference on Big Data \(BigData\),Vol\.,pp\. 5272–5281\.External Links:[Document](https://dx.doi.org/10.1109/BigData59044.2023.10386182)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p12.1),[§1](https://arxiv.org/html/2609.36082#S1.p2.1)\.
- Krechet al\.\(2025\)RDFLibExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.6845245),[Link](https://github.com/RDFLib/rdflib)Cited by:[§4](https://arxiv.org/html/2609.36082#S4.p2.1)\.
- Kunduet al\.\(2025\)R\. Kundu, R\. Mehdi, V\. D\. Tran, E\. Frakes, A\. Daundkar, M\. Sumudumalie, V\. S\. Mandayam, J\. A\. Lample, M\. Li, L\. S\. Bruckman, E\. I\. Barcelos, A\. Sehirlioglu, R\. H\. French, and Y\. WuOntoCheck: query\-driven ontology assessments for scientific domain applications\.Note:[https://pypi\.org/project/OntoCheck/](https://pypi.org/project/OntoCheck/)Python software packageCited by:[§A\.2](https://arxiv.org/html/2609.36082#A1.SS2.p20.1),[Table 7](https://arxiv.org/html/2609.36082#A1.T7),[Figure 2](https://arxiv.org/html/2609.36082#S1.F2),[§3\.2](https://arxiv.org/html/2609.36082#S3.SS2.p3.1),[§4](https://arxiv.org/html/2609.36082#S4.p9.1)\.
- Liet al\.\(2026\)B\. Li, J\. Ma, O\. A\. Omitaomu, and A\. MostafaviRevealing growing and disparate vulnerability in the u\.s\. power system: a spatiotemporal analysis of nationwide outages from 2014 to 2023\.International Journal of Disaster Risk Reduction133,pp\. 105980–1856\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.ijdrr.2025.105980)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p4.1)\.
- Liebermanet al\.\(2007\)J\. Lieberman, R\. Singh, and C\. GoadW3C Geospatial Ontologies\.W3C Incubator Group ReportWorld Wide Web Consortium\.Note:Accessed: 2026\-05\-25External Links:[Link](https://www.w3.org/2005/Incubator/geo/XGR-geo-ont-20071023/)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Mulayimet al\.\(2025\)O\. B\. Mulayim, A\. Anwar, U\. M\. Saka, L\. Paul, A\. K\. Prakash, G\. Fierro, M\. Pritoni, and M\. BergésBuildingQA: a benchmark for natural language question answering over building knowledge graphs\.InProceedings of the 12th ACM International Conference on Systems for Energy\-Efficient Buildings, Cities, and Transportation,BuildSys ’25,New York, NY, USA,pp\. 65–75\.External Links:ISBN 9798400719455,[Link](https://doi.org/10.1145/3736425.3770097),[Document](https://dx.doi.org/10.1145/3736425.3770097)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p13.1)\.
- NASA Earthdata \(2026a\)NASA EarthdataAqua\.National Aeronautics and Space Administration\.Note:[https://www\.earthdata\.nasa\.gov/data/platforms/space\-based\-platforms/aqua](https://www.earthdata.nasa.gov/data/platforms/space-based-platforms/aqua)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p1.1)\.
- NASA Earthdata \(2026b\)NASA EarthdataTerra\.National Aeronautics and Space Administration\.Note:[https://www\.earthdata\.nasa\.gov/data/platforms/space\-based\-platforms/terra](https://www.earthdata.nasa.gov/data/platforms/space-based-platforms/terra)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p1.1)\.
- National Hurricane Center \(2026\)National Hurricane CenterAtlantic Hurricane Database \(HURDAT2\) 1851\-2025\.Note:National Oceanic and Atmospheric Administration \(NOAA\)External Links:[Link](https://www.nhc.noaa.gov/data/)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p8.1)\.
- NOAA National Centers for Environmental Information \(2026\)NOAA National Centers for Environmental InformationStorm Events Database\.National Oceanic and Atmospheric Administration\.Note:[https://www\.ncei\.noaa\.gov/stormevents/](https://www.ncei.noaa.gov/stormevents/)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p7.1)\.
- Ontotext \(2026\)OntotextGraphDB: GraphDB 11\.3 documentation\.Ontotext,New York, NY, USA\.External Links:[Link](https://graphdb.ontotext.com/documentation/11.3/)Cited by:[§A\.2](https://arxiv.org/html/2609.36082#A1.SS2.p23.1),[Figure 2](https://arxiv.org/html/2609.36082#S1.F2),[§1](https://arxiv.org/html/2609.36082#S1.p9.1)\.
- Rajamohanet al\.\(2025\)B\. P\. Rajamohan, A\. C\. H\. Bradley, V\. D\. Tran, J\. E\. Gordon, H\. W\. Caldwell, R\. Mehdi, G\. Ponon, Q\. D\. Tran, O\. Dernek, J\. Kaltenbaugh, B\. G\. Pierce, R\. Wieser, W\. Yue, K\. Lin, J\. Kambo, C\. Lopez, A\. Nihar, D\. J\. Savage, D\. W\. Brown, H\. Sharma, B\. Giera, P\. K\. Tripathi, Y\. Wu, M\. Li, K\. O\. Davis, L\. S\. Bruckman, E\. I\. Barcelos, and R\. H\. FrenchMaterials Data Science Ontology \(MDS\-Onto\): unifying domain knowledge in materials and applied data science\.Scientific Data12,pp\. 628\.External Links:[Document](https://dx.doi.org/10.1038/s41597-025-04938-5)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1),[Figure 3](https://arxiv.org/html/2609.36082#S2.F3),[Example 2\.2](https://arxiv.org/html/2609.36082#S2.Thmtheorem2.p1.1)\.
- Románet al\.\(2018\)M\. O\. Román, Z\. Wang, Q\. Sun, V\. Kalb, S\. D\. Miller, A\. Molthan, L\. Schultz, J\. Bell, E\. C\. Stokes, B\. Pandey, K\. C\. Seto, D\. Hall, T\. Oda, R\. E\. Wolfe, G\. Lin, N\. Golpayegani, S\. Devadiga, C\. Davidson, S\. Sarkar, C\. Praderas, J\. Schmaltz, R\. Boller, J\. Stevens, O\. M\. Ramos González, E\. Padilla, J\. Alonso, Y\. Detrés, R\. Armstrong, I\. Miranda, Y\. Conte, N\. Marrero, K\. MacManus, T\. Esch, and E\. J\. MasuokaNASA’s Black Marble nighttime lights product suite\.Remote Sensing of Environment210,pp\. 113–143\.External Links:ISSN 0034\-4257,[Document](https://dx.doi.org/10.1016/j.rse.2018.03.017)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p1.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p3.1)\.
- Rudnicki \(2019\)R\. RudnickiAn overview of the common core ontologies\.Technical ReportCUBRC, Inc\.,Buffalo, NY\.External Links:[Link](https://www.nist.gov/document/nist-ai-rfi-cubrcinc004pdf)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p11.1)\.
- Tansakulet al\.\(2024\)V\. Tansakul, A\. Myers, S\. Tennille, M\. Broughton, M\. Denman, A\. Hamaker, J\. Huihui, E\. Johnson, K\. Medlen, R\. Moncrief,et al\.EAGLE\-i power outage data 2023\.\.Note:The provided EAGLE\-I historic dataset includes 1 year of power outage information at the county level for 2023 at 15\-minute intervals collected by the EAGLE\-I program at ORNL\. The data has been collected from utility’s public outage maps using an ETL process\. The dataset details FIPS code, county name, state name, total number of customers without power, and a date/timestamp\. Also included is the EAGLE\-I coverage of each state for each year\. For detailed metadata, refer to the metadata DOI\.External Links:[Document](https://dx.doi.org/10.13139/OLCF/2297398),[Link](https://www.osti.gov/biblio/2297398)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p6.1)\.
- Tansakulet al\.\(2023\)V\. Tansakul, A\. Myers, S\. Tennille, M\. Denman, A\. Hamaker, J\. Huihui, K\. Medlen, K\. Allen, D\. Redmon, S\. Chinthavali,et al\.EAGLE\-I power outage data 2014 \- 2022\.Note:The provided EAGLE\-I historic dataset includes eight years of power outage information at the county level from 2014 to 2022 at 15\-minute intervals collected by the EAGLE\-I program at ORNL\. The data has been collected from utility’s public outage maps using an ETL process\. The dataset details FIPS code, county name, state name, total number of customers without power, and a date/timestamp\. Also included is the EAGLE\-I coverage of each state for each year\. For detailed metadata, refer to the metadata DOI\.External Links:[Document](https://dx.doi.org/10.13139/ORNLNCCS/1975202)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p6.1)\.
- Tansakulet al\.\(2026\)V\. Tansakul, A\. Myers, S\. Tennille, M\. Denman, J\. Huihui, K\. Medlen, M\. MacFarland, M\. Broughton, J\. Ibanez, R\. Ciego,et al\.EAGLE\-i power outage data 2025\.Note:The provided EAGLE\-I historic dataset includes power outage information at the county level for 2025 at 15\-minute intervals collected by the EAGLE\-I program at ORNL\. The data has been collected from utility’s public outage maps using an ETL process\. The dataset details FIPS code, county name, state name, total number of customers without power, and a date/timestamp\. For detailed metadata, refer to the linked metadata DOI\.External Links:[Document](https://dx.doi.org/10.13139/ORNLNCCS/3012826),[Link](https://www.osti.gov/biblio/3012826)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p6.1)\.
- Tansakulet al\.\(2025\)V\. Tansakul, A\. Myers, S\. Tennille, M\. Denman, J\. Huihui, K\. Medlen, M\. MacFarland, J\. Ibanez, R\. Ciego, and F\. QuinnEAGLE\-i power outage data 2024\.Note:The provided EAGLE\-I historic dataset includes power outage information at the county level for 2024 at 15\-minute intervals collected by the EAGLE\-I program at ORNL\. The data has been collected from utility’s public outage maps using an ETL process\. The dataset details FIPS code, county name, state name, total number of customers without power, total customers per county, and a date/timestamp\. For detailed metadata, refer to the linked metadata DOI\.External Links:[Document](https://dx.doi.org/10.13139/OLCF/2500278),[Link](https://www.osti.gov/biblio/2500278)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p6.1)\.
- U\.S\. Census Bureau \(2025\)U\.S\. Census BureauTIGER/Line Shapefiles\.Note:[https://www\.census\.gov/geographies/mapping\-files/time\-series/geo/tiger\-line\-file\.html](https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.html)Cited by:[§2\.1](https://arxiv.org/html/2609.36082#S2.SS1.p10.1)\.
- Usbecket al\.\(2018\)R\. Usbeck, R\. H\. Gusmita, A\. N\. Ngomo, and M\. Saleem9th challenge on question answering over linked data \(qald\-9\) \(invited paper\)\.InSemdeep/NLIWoD@ISWC,External Links:[Link](https://api.semanticscholar.org/CorpusID:53220210)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p12.1),[§1](https://arxiv.org/html/2609.36082#S1.p2.1)\.
- World Wide Web Consortium \(2004\)World Wide Web ConsortiumXML schema part 1: structures second edition\.Note:W3C RecommendationExternal Links:[Link](https://www.w3.org/TR/xmlschema-1/)Cited by:[Figure 3](https://arxiv.org/html/2609.36082#S2.F3)\.
- Yanget al\.\(2024\)R\. Yang, F\. D\. Salim, and H\. XueSSTKG: simple spatio\-temporal knowledge graph for intepretable and versatile dynamic information embedding\.InProceedings of the ACM Web Conference 2024,pp\. 551–559\.External Links:[Document](https://dx.doi.org/10.1145/3589334.3645441)Cited by:[§1](https://arxiv.org/html/2609.36082#S1.p1.1)\.

## Appendix AAppendix

### A\.1\.Query\-Interpretation: Prompts

Task 1 uses ontology\-grounded prompts to generate the query interpretations evaluated in Sections[3](https://arxiv.org/html/2609.36082#S3)and[4](https://arxiv.org/html/2609.36082#S4)\. This process implements the operator𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}with the following steps\. \(1\) The LLM receives a JSON prediction template with benchmark metadata, the GeoOutageOnto ontology in Turtle, and a compact vocabulary file containing core classes, properties, prefixes, and domain/range; gold SPARQL queries are not provided\. \(2\) The interpreter shares the template and consults the LLM to fillsparql\_querywith the best default interpretation\. To mitigate ambiguity, it includes the query set \(a memory spacesparql\_interpretations\) containing syntactically correct SPARQL queries\.

The prompt constrains the generated SPARQL interpretation to align with the ontology vocabulary when available, requires distinct interpretations for different spatial, temporal, semantic, or spatiotemporal readings, and discourages variants that merely rename variables or reorder triples\. It also standardizes SPARQL style through required prefixes, bound projected variables, ISO 8601xsd:dateTimefilters, half\-open day\-level ranges, aggregation and ranking rules, and namespace requirements\.

Figure[6](https://arxiv.org/html/2609.36082#A1.F6)summarizes the operative structure of the prompt; the full prompt is available in the GitHub repository\. Internally, the prompt asks the model to identify answer type, ontology class, connecting properties, spatial or temporal filters, aggregation, ranking, multimodal evidence, and ambiguity, but the returned output is only the completed JSON prediction file\.

### A\.2\.Evaluation Metrics

Role:expertGeoSPARQL,temporalSPARQL,andontology\-groundedquerygenerationsystem\.

Inputs:

1\.GeoOutageBenchpredictionJSONwithemptysparql\_queryfields\.

2\.FullontologyinTurtle\.

3\.Allowedclasses,properties,prefixes,anddomain/rangehints\.

Corerules:

\-Useonlytheprovidedontology/schemacontext\.

\-Donotusegold/referenceSPARQL\.

\-PreserveexistingJSONfields\.

\-Fillsparql\_querywiththedefaultinterpretation\.

\-Addsparql\_interpretationsonlyforanswer\-changingambiguity\.

\-ReturnonlyvalidJSON\.

\-GenerateexecutableSPARQLusingallowedprefixes\.

Ambiguityrules:

\-Generateseparatequeriesformateriallydifferentspatial,temporal,semantic,orspatiotemporalreadings\.

\-Donotcreatevariantsthatonlyrenamevariablesorreordertriples\.

\-Keepinterpretationssmall,usually1\-\-3\.

Outputforeachinterpretation:

label,ambiguity\_dimensions,rationale,sparql\_query\.

Figure 6\.Condensed structure of the𝖭𝖫𝟤𝖲𝖳𝖰\\mathsf\{NL2STQ\}prompt used to produce GeoOutageBench query interpretations\.Condensed structure of the NL2SPARQL prompt used to produce GeoOutageBench query interpretations\.GeoOutageBench evaluates formal query generation, ontology utility, and answer\-level spatiotemporal KGQA\. We useℛ​𝒬^im\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}for the realized generated interpretation set of modelmmon questionqiq\_\{i\}andℛ​𝒬i\{\\mathcal\{RQ\}\}\_\{i\}for the reference interpretation set\. Unless otherwise stated, corpus scores are macro\-averages over theNNevaluated questions:

\(1\)μ¯=1N​∑i=1Nμi\.\\overline\{\\mu\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mu\_\{i\}\.
Several metrics use the same set\-overlap precision, recall, and F1 form\. For a generated setX^i\\widehat\{X\}\_\{i\}, a gold setXiX\_\{i\}, and a matched\-correct countcic\_\{i\}, we compute

\(2\)Pi\\displaystyle P\_\{i\}=ci\|X^i\|,\\displaystyle=\\frac\{c\_\{i\}\}\{\|\\widehat\{X\}\_\{i\}\|\},Ri\\displaystyle R\_\{i\}=ci\|Xi\|,\\displaystyle=\\frac\{c\_\{i\}\}\{\|X\_\{i\}\|\},F​1i\\displaystyle F1\_\{i\}=\{2​Pi​RiPi\+Ri,if​Pi\+Ri≠0,0,if​Pi\+Ri=0\.\\displaystyle=\\begin\{cases\}\\dfrac\{2P\_\{i\}R\_\{i\}\}\{P\_\{i\}\+R\_\{i\}\},&\\text\{if \}P\_\{i\}\+R\_\{i\}\\neq 0,\\\\ 0,&\\text\{if \}P\_\{i\}\+R\_\{i\}=0\.\\end\{cases\}
When both generated and gold sets are empty, the corresponding feature\-family score is defined as11; when the prediction is empty and the gold set is non\-empty, it is00\.

Task 1 metrics\. The Task 1 evaluator first normalizes each prediction into a deduplicated list of query records, accepting either a singlesparql\_queryor richer ambiguity\-aware fields such assparql\_interpretations\. Each generated interpretation is scored against each reference interpretation, and the best aligned pair gives the question\-level structural score\. Class F1 is based on qnames used asrdf:typetargets, property F1 uses graph\-pattern predicates excludingrdf:type, spatial F1 uses GeoSPARQL/WGS84, geometry, coordinate, distance, and spatial\-predicate markers, and temporal F1 uses date/dateTime literals and temporal filter markers\. Axis\-level results group questions by non\-spatiotemporal, spatial, temporal, or spatiotemporal annotations\.

Letν⁡\(r\)\\nu\(r\)denote normalized SPARQL text after removing prefixes and comments, collapsing whitespace, and lowercasing\. LetN^im=\{ν⁡\(r^\)∣r^∈ℛ​𝒬^im\}\\widehat\{N\}\_\{i\}^\{m\}=\\\{\\nu\(\\hat\{r\}\)\\mid\\hat\{r\}\\in\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\\\}andNi=\{ν⁡\(r\)∣r∈ℛ​𝒬i\}N\_\{i\}=\\\{\\nu\(r\)\\mid r\\in\{\\mathcal\{RQ\}\}\_\{i\}\\\}\. Syntax validity is the fraction of generated interpretations that pass the coarse SELECT/ASK and WHERE/braces check:

\(3\)Synim=1\|ℛ​𝒬^im\|​∑r^∈ℛ​𝒬^im𝟏​\{r^​is syntactically valid\}\.\\mathrm\{Syn\}\_\{i\}^\{m\}=\\frac\{1\}\{\|\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\|\}\\sum\_\{\\hat\{r\}\\in\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\}\\mathbf\{1\}\\\{\\hat\{r\}\\text\{ is syntactically valid\}\\\}\.Ifℛ​𝒬^im=∅\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}=\\emptyset, syntax validity is defined as00\.

Compatibility \(also referred to as executability\) receives credit when at least one generated interpretation executes successfully over the configured graph or endpoint:

\(4\)Compatim=𝟏\{\\displaystyle\\mathrm\{Compat\}\_\{i\}^\{m\}=\\mathbf\{1\}\\\{∃r^∈ℛ​𝒬^im:\\displaystyle\\exists\\hat\{r\}\\in\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}:r^executes over𝒢\}\.\\displaystyle\\hat\{r\}\\text\{ executes over \}\{\\mathcal\{G\}\}\\\}\.
Exact original\-query match receives credit when the original source queryri​0r\_\{i0\}appears anywhere in the generated interpretation set, while ambiguity\-aware matching compares the full interpretation sets:

\(5\)Exactim\\displaystyle\\mathrm\{Exact\}\_\{i\}^\{m\}=𝟏\{ν\(ri​0\)∈N^im\},\\displaystyle=\\mathbf\{1\}\\\{\\nu\(r\_\{i0\}\)\\in\\widehat\{N\}\_\{i\}^\{m\}\\\},Anyim\\displaystyle\\mathrm\{Any\}\_\{i\}^\{m\}=𝟏\{N^im∩Ni≠∅\}\.\\displaystyle=\\mathbf\{1\}\\\{\\widehat\{N\}\_\{i\}^\{m\}\\cap N\_\{i\}\\neq\\emptyset\\\}\.
Interpretation recall is

\(6\)IRecim=\|N^im∩Ni\|\|Ni\|\.\\mathrm\{IRec\}\_\{i\}^\{m\}=\\frac\{\|\\widehat\{N\}\_\{i\}^\{m\}\\cap N\_\{i\}\|\}\{\|N\_\{i\}\|\}\.
For structural alignment, Eq\.[2](https://arxiv.org/html/2609.36082#A1.E2)is applied to extracted feature sets: classes, properties, spatial constraints, and temporal constraints\. The selected interpretation pair maximizes the average of these four F1 scores:

\(7\)Alignim=\\displaystyle\\mathrm\{Align\}\_\{i\}^\{m\}=maxr^∈ℛ​𝒬^imr∈ℛ​𝒬i⁡14​\(ClassF1r^,r\+PropF1r^,rCLOSE\\displaystyle\\max\_\{\\begin\{subarray\}\{c\}\\hat\{r\}\\in\\widehat\{\{\\mathcal\{RQ\}\}\}\_\{i\}^\{\\,m\}\\\\ r\\in\{\\mathcal\{RQ\}\}\_\{i\}\\end\{subarray\}\}\\frac\{1\}\{4\}\\bigl\(\\mathrm\{ClassF1\}\_\{\\hat\{r\},r\}\+\\mathrm\{PropF1\}\_\{\\hat\{r\},r\}OPEN\+SF1r^,r\+TF1r^,r\)\.\\displaystyle\+\\mathrm\{SF1\}\_\{\\hat\{r\},r\}\+\\mathrm\{TF1\}\_\{\\hat\{r\},r\}\\bigr\)\.
Schema\-linking F1 and spatiotemporal\-constraint F1 use harmonic means:

\(8\)SchemaF1im=\{2⋅ClassF1im⋅PropF1imClassF1im\+PropF1im,if​ClassF1im\+PropF1im≠0,0,otherwise,\\displaystyle\\mathrm\{SchemaF1\}\_\{i\}^\{m\}=\\begin\{cases\}\\dfrac\{2\\cdot\\mathrm\{ClassF1\}\_\{i\}^\{m\}\\cdot\\mathrm\{PropF1\}\_\{i\}^\{m\}\}\{\\mathrm\{ClassF1\}\_\{i\}^\{m\}\+\\mathrm\{PropF1\}\_\{i\}^\{m\}\},&\\begin\{gathered\}\\text\{if \}\\mathrm\{ClassF1\}\_\{i\}^\{m\}\\\\ \+\\mathrm\{PropF1\}\_\{i\}^\{m\}\\neq 0,\\end\{gathered\}\\\\ 0,&\\text\{otherwise,\}\\end\{cases\}
\(9\)STF1im=\{2⋅S​F​1im⋅T​F​1imS​F​1im\+T​F​1im,if​S​F​1im\+T​F​1im≠0,0,if​S​F​1im\+T​F​1im=0\.\\displaystyle\\mathrm\{STF1\}\_\{i\}^\{m\}=\\begin\{cases\}\\dfrac\{2\\cdot SF1\_\{i\}^\{m\}\\cdot TF1\_\{i\}^\{m\}\}\{SF1\_\{i\}^\{m\}\+TF1\_\{i\}^\{m\}\},&\\text\{if \}SF1\_\{i\}^\{m\}\+TF1\_\{i\}^\{m\}\\neq 0,\\\\ 0,&\\text\{if \}SF1\_\{i\}^\{m\}\+TF1\_\{i\}^\{m\}=0\.\\end\{cases\}
Table 7\.Task\-Based Assessment of GeoOutageOnto through OntoCheck\([Kundu et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib36)\)\.QClose\.QClose\.QClose\.QClose\.QClose\.QClose\.10\.0294130\.0686250\.0392370\.0686490\.0784610\.088220\.0392140\.0196260\.0392380\.0686500\.0784620\.137330\.0294150\.0196270\.0392390\.0686510\.0784630\.127540\.0294160\.0196280\.0294400\.0392520\.0490640\.068650\.0392170\.0392290\.0686410\.0686530\.0490650\.068660\.0392180\.0392300\.0294420\.0882540\.1176660\.098070\.0196190\.0588310\.0490430\.0882550\.0686670\.098080\.0196200\.0588320\.0588440\.0882560\.1176680\.098090\.0196210\.0588330\.0490450\.0882570\.0686690\.0980100\.0196220\.0490340\.1078460\.1078580\.1078700\.1078110\.0196230\.0490350\.1078470\.1078590\.0980710\.1078120\.0490240\.0490360\.1078480\.1078600\.1275720\.1078Average Closeness:0\.0673

Task 2 metrics\. OntoCheck\([Kundu et al\., 2025](https://arxiv.org/html/2609.36082#bib.bib36)\)constructs a task term setTa​\(ri​j\)T\_\{a\}\(r\_\{ij\}\)from extractedgoo:terms in each SPARQL interpretationri​j∈ℛ​𝒬ir\_\{ij\}\\in\{\\mathcal\{RQ\}\}\_\{i\}and compares it with the scoped ontology term setToT\_\{o\}\. Query Coverage \(or Relevance\) measures how much workload vocabulary is declared by the ontology; Query Closeness \(or Accuracy\) measures how densely the scoped ontology vocabulary is activated by the query:

\(10\)Covi​j\\displaystyle\\mathrm\{Cov\}\_\{ij\}=\|Ta​\(ri​j\)∩To\|\|Ta​\(ri​j\)\|,\\displaystyle=\\frac\{\|T\_\{a\}\(r\_\{ij\}\)\\cap T\_\{o\}\|\}\{\|T\_\{a\}\(r\_\{ij\}\)\|\},Cloi​jo​n​t​o\\displaystyle\\mathrm\{Clo\}^\{onto\}\_\{ij\}=\|Ta​\(ri​j\)∩To\|\|To\|\.\\displaystyle=\\frac\{\|T\_\{a\}\(r\_\{ij\}\)\\cap T\_\{o\}\|\}\{\|T\_\{o\}\|\}\.
The full list of Closeness scores for all 72 queries in the ground\-truth set is listed in Table[7](https://arxiv.org/html/2609.36082#A1.T7)\(all queries returned a Coverage of 1\.00, hence are not shown\)\.

Task 3 metrics\. Task 3 treats Task 1 outputs as query workload to be processed\. Each SPARQL interpretation is executed against GeoOutageKG, using GraphDB\([Ontotext, 2026](https://arxiv.org/html/2609.36082#bib.bib27)\)for the reported experiments, and gold answers are obtained by executing ground\-truth interpretations, or by using stored human\-validated benchmark answers if all interpretations return empty results\. Returned bindings are normalized by compacting IRIs, simplifying datatype and language\-tag variants, normalizing numeric and UTC date/time values, and aligning common aliases such asntl/image,rec/record, andnum/numOutages\.

Answer scoring uses Eq\.[2](https://arxiv.org/html/2609.36082#A1.E2)withX^i=𝒜^im\\widehat\{X\}\_\{i\}=\\widehat\{\{\\mathcal\{A\}\}\}\_\{i\}^\{\\,m\},Xi=𝒜iX\_\{i\}=\{\\mathcal\{A\}\}\_\{i\}, andcic\_\{i\}equal to the number of generated rows matched to distinct gold rows after normalization\. For ranked outputs, letyi​ℓ∈\{0,1\}y\_\{i\\ell\}\\in\\\{0,1\\\}indicate whether the row at rankℓ\\ellis relevant\. We compute

\(11\)Hits​@​ki\\displaystyle\\mathrm\{Hits@\}k\_\{i\}=𝟏\{∑ℓ=1kyi​ℓ\>0\},\\displaystyle=\\mathbf\{1\}\\Bigl\\\{\\sum\_\{\\ell=1\}^\{k\}y\_\{i\\ell\}\>0\\Bigr\\\},RRi=\{1min⁡\{ℓ:yi​ℓ=1\},if any​yi​ℓ=1,0,otherwise,\\displaystyle\\mathrm\{RR\}\_\{i\}=\\begin\{cases\}\\dfrac\{1\}\{\\min\\\{\\ell:y\_\{i\\ell\}=1\\\}\},&\\text\{if any \}y\_\{i\\ell\}=1,\\\\ 0,&\\text\{otherwise,\}\\end\{cases\}
and report MRR as the macro\-average ofRRi\\mathrm\{RR\}\_\{i\}\. Ranking quality at depth 10 is

\(12\)DCG​@​10i\\displaystyle\\mathrm\{DCG@10\}\_\{i\}=∑ℓ=110yi​ℓlog2⁡\(ℓ\+1\),\\displaystyle=\\sum\_\{\\ell=1\}^\{10\}\\frac\{y\_\{i\\ell\}\}\{\\log\_\{2\}\(\\ell\+1\)\},nDCG​@​10i\\displaystyle\\mathrm\{nDCG@10\}\_\{i\}=DCG​@​10iIDCG​@​10iifIDCG@10i\>0,\\displaystyle=\\frac\{\\mathrm\{DCG@10\}\_\{i\}\}\{\\mathrm\{IDCG@10\}\_\{i\}\}\\quad\\text\{if \}\\mathrm\{IDCG@10\}\_\{i\}\>0,=0otherwise\.\\displaystyle=0\\quad\\text\{otherwise\.\}
Spatial correctnessSiS\_\{i\}and temporal correctnessTiT\_\{i\}are defined from explicit annotations when available; otherwise, they are derived from overlap between the selected generated and gold SPARQL constraints\. Dimensions not required by the question receive full credit, while required dimensions fall back to answer F1 only when no query\-level signal is available\. The Spatiotemporal Relevance Score is

\(13\)SRSi=\{2​Si​TiSi\+Ti,if​Si\+Ti≠0,0,if​Si\+Ti=0\.\\displaystyle\\mathrm\{SRS\}\_\{i\}=\\begin\{cases\}\\dfrac\{2S\_\{i\}T\_\{i\}\}\{S\_\{i\}\+T\_\{i\}\},&\\text\{if \}S\_\{i\}\+T\_\{i\}\\neq 0,\\\\ 0,&\\text\{if \}S\_\{i\}\+T\_\{i\}=0\.\\end\{cases\}
With multiple generated and reference interpretations, answer\-level alignment selects the generated–gold pair maximizing the interpretation\-selection score

\(14\)Selim=14​\(AnswerF1im\+Si\+TiCLOSE\\displaystyle\\mathrm\{Sel\}\_\{i\}^\{m\}=\\frac\{1\}\{4\}\\bigl\(\\mathrm\{AnswerF1\}\_\{i\}^\{m\}\+S\_\{i\}\+T\_\{i\}OPEN\+SRSi\)\.\\displaystyle\\hskip 52\.00005pt\+\\mathrm\{SRS\}\_\{i\}\\bigr\)\.