GeoFitBench-LLM：评估大型语言模型在地理空间分析任务中获取地理空间数据需求能力的基准

arXiv cs.CL 论文
geospatial benchmark llm geoai evaluation natural-language

摘要

GeoFitBench-LLM 是一个评估大语言模型在地理分析任务中推断地理空间数据需求能力的新基准，围绕时空关系与语义落地两个推理维度，实验显示 GPT-5.4 Thinking 在资源识别上表现最佳（66.67%），但属性级适用性判定仍具挑战。

arXiv:2610.12669v1 Announce Type: new Abstract: Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements. This omission is critical as retrieval oriented geo-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints. GeoFitBench-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo-analytic requests in natural language. The benchmark is structured around two reasoning dimensions, Spatio-Temporal Relational and Semantic Grounding, reflecting long-standing challenges in geospatial information retrieval and related natural language tasks. Ground truth requirements are encoded in a fixed, provider-independent schema for reproducible evaluation of resource identification and attribute-level specification. Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable, with GPT-5.4 Thinking achieving the best result by recovering the complete set of resources in 66.67% of benchmark requests. Results further indicate that accurate fitness-for-use specification remains more difficult as inferential demands increase.
查看原文
查看缓存全文

缓存时间: 2026/10/12 06:48 

# GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks
Source: [https://arxiv.org/abs/2610.12669](https://arxiv.org/abs/2610.12669)
[View PDF](https://arxiv.org/pdf/2610.12669)

> Abstract:Large language models are becoming central to GeoAI systems, yet their evaluation rarely considers the elicitation of geospatial data requirements\. This omission is critical as retrieval oriented geo\-analytic tasks cannot be supported reliably without first identifying the necessary datasets and constraints\. GeoFitBench\-LLM is introduced as a benchmark for assessing whether LLMs can infer the datasets, the supporting resources and the constraints required by geo\-analytic requests in natural language\. The benchmark is structured around two reasoning dimensions, Spatio\-Temporal Relational and Semantic Grounding, reflecting long\-standing challenges in geospatial information retrieval and related natural language tasks\. Ground truth requirements are encoded in a fixed, provider\-independent schema for reproducible evaluation of resource identification and attribute\-level specification\. Experiments on seven models under minimal prompting indicate that identifying the necessary resources is often achievable, with GPT\-5\.4 Thinking achieving the best result by recovering the complete set of resources in 66\.67% of benchmark requests\. Results further indicate that accurate fitness\-for\-use specification remains more difficult as inferential demands increase\.

## Submission history

From: Thierry Badard \[[view email](https://arxiv.org/show-email/3cd1a1f5/2610.12669)\] **\[v1\]**Thu, 8 Oct 2026 19:45:40 UTC \(912 KB\)

相似文章

MA-Bench：衡量语言模型的操作规划能力

arXiv cs.LG

Scale AI 与 Data Machines 推出 MA-Bench 基准，通过军事任务分析场景评估语言模型在不完整、矛盾信息下的操作规划判断能力；20 个前沿模型中最佳仅答对 62% 的评分项，最弱环节是需要推断的判断。

评估大语言模型在社交媒体分析中的能力：多任务探索

arXiv cs.CL

犹他州立大学和范德堡大学的研究人员对GPT-4、Gemini 1.5 Pro、DeepSeek-V3、Llama 3.2与BERT在三大社交媒体任务——作者身份验证、帖子生成与用户属性推断——进行了基准测试，引入新的采样方案与分类体系以减少偏差，打造可复现的评测基准。

GlobeBench：评估用于可扩展环境模拟的文本世界模型能力

arXiv cs.AI

GlobeBench 提出一个包含 1,632 个任务的标准化基准，用于评估文本世界模型模拟 RL 环境状态转移的能力；评测 20 个 LLM 后发现即使是表现最好的 GPT-6 Astra 任务成功率也仅 49.57%，且 631 个任务无任何模型解决，凸显模型作为通用世界模型仍存在显著差距。