UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Tasks

# UrbanDS: A Graph-Guided LLM Multi-Agent System for Data-Intensive Urban Tasks
Source: [https://arxiv.org/html/2607.26724](https://arxiv.org/html/2607.26724)
\\correspondingauthor

Zhilun ZhouDepartment of Electronic Engineering, BNRist, Tsinghua UniversityBeijingChina[zzl22@mails\.tsinghua\.edu\.cn](https://arxiv.org/html/2607.26724v1/mailto:[email protected])Jianghao YuDepartment of Electronic Engineering, BNRist, Tsinghua UniversityBeijingChina[yujh2703@gmail\.com](https://arxiv.org/html/2607.26724v1/mailto:[email protected]),Yuming LinDepartment of Urban Planning, Tsinghua UniversityBeijingChina[yujh2703@gmail\.com](https://arxiv.org/html/2607.26724v1/mailto:[email protected]),Yongjun YangJiangmen Municipal Smart Social Governance Technology Innovation CenterJiangmenChina[muyigmail@gmail\.com](https://arxiv.org/html/2607.26724v1/mailto:[email protected]),Yongquan SunTsinghua UniversityBeijingChina[syq23@mails\.tsinghua\.edu\.cn](https://arxiv.org/html/2607.26724v1/mailto:[email protected]),Depeng JinDepartment of Electronic Engineering, BNRist, Tsinghua UniversityBeijingChina[jindp@tsinghua\.edu\.cn](https://arxiv.org/html/2607.26724v1/mailto:[email protected])andYong LiDepartment of Electronic Engineering, BNRist, Tsinghua UniversityBeijingChina[liyong07@tsinghua\.edu\.cn](https://arxiv.org/html/2607.26724v1/mailto:[email protected])

\(2018\)

###### Abstract\.

Large language model \(LLM\) agents have been widely applied in various domains, with numerous studies exploring their applications in automating data science tasks\. However, existing methods typically rely on a limited set of provided datasets, and they face challenges in data\-intensive scenarios that require discovering and leveraging relevant information from large\-scale and heterogeneous data repositories\. Urban tasks are representative examples of such scenarios, as urban data are not only large\-scale and multi\-sourced, but also exhibit complex spatial, temporal, and semantic relationships across datasets\. To address these challenges, we propose UrbanDS, a graph\-guided LLM multi\-agent system for data\-intensive urban tasks\. We first construct a unified dataset graph to organize reusable dataset skills and the relationships among datasets\. Specifically, we develop a Data Profiling Agent that constructs a skill for each dataset, describing the content, schema, statistics, and usage\. Moreover, a Relation Agent identifies relationships among datasets and integrates these relationships into the dataset graph\. At runtime, a Planner Agent retrieves task\-relevant datasets and relationships from the graph based on user queries and generates execution plans\. Multiple Execution Agents then collaboratively perform code\-based data processing and analysis, while their execution progress and intermediate results are shared through a common memory\. Finally, a Report Agent synthesizes all the experimental logs into a report, which can be further refined based on user feedback\. To systematically evaluate the capability of LLM agents in handling data\-intensive urban scenarios, we further construct UrbanDS\-Bench, an urban data science benchmark covering representative data analysis and modeling tasks\. Experiments on both general and urban benchmarks demonstrate that UrbanDS consistently outperforms existing data science agents on data\-intensive tasks\. Furthermore, UrbanDS has been deployed on the urban operations platform of Dongxihu District, Wuhan, demonstrating its effectiveness in real\-world urban applications\. Our code and benchmark are released at https://github\.com/zhou\-zl18/UrbanDS\.

††copyright:acmlicensed††journalyear:2018††doi:XXXXXXX\.XXXXXXX††conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY††isbn:978\-1\-4503\-XXXX\-X/2018/06## 1\.Introduction

Large language models \(LLMs\) are rapidly evolving from passive assistants into autonomous agents that can reason, use tools, write and execute code, and coordinate with other agents\. This development creates a promising way to automate data science workflows, which traditionally require substantial human effort in data understanding, preprocessing, analysis, modeling, and interpretation\. Consequently, many existing studies have developed LLM agent systems for data science tasks ranging from data analysis to data modeling and machine learning\(Hollmannet al\.,[2023](https://arxiv.org/html/2607.26724#bib.bib7); Guoet al\.,[2024](https://arxiv.org/html/2607.26724#bib.bib5); Honget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib4); Triratet al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib3); Zhanget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib2)\)\.

Existing data science agents usually rely on one or a few datasets provided directly with each task\. However, in real\-world applications, a data repository may contain a large number of heterogeneous datasets, most of which are irrelevant to a given task\. Therefore, an agent must not only generate analysis code, but also discover relevant datasets from large\-scale repositories before performing data processing and modeling\. This challenge is particularly pronounced in urban scenarios\. Urban data are typically multi\-source and heterogeneous, covering domains such as mobility, transportation, population, land use, points of interest, economy, and public services\. Moreover, these datasets often exhibit complex spatial, temporal, and semantic relationships, making it difficult for agents to identify relevant data sources and correctly integrate them for downstream analysis\(Zhanget al\.,[2026b](https://arxiv.org/html/2607.26724#bib.bib12)\)\.

To address these limitations, in this work, we proposeUrbanDS, a graph\-guided LLM multi\-agent system for data\-intensive urban tasks\. UrbanDS consists of two stages: dataset graph construction and task execution\. In the first stage, we organize the datasets into a graph to model their attributes and relationships\. Specifically, a Data Profiling Agent explores each dataset via Python code and creates a reusable*dataset skill*that describes the dataset’s content, schema, statistics, spatiotemporal coverage, and usage\. Based on these, a Relation agent discovers relations between fields from different datasets via a semantic codebook\. Moreover, we identify spatial and temporal relations from geographic coverage and time ranges between datasets\. Finally, UrbanDS organizes the dataset skills and relations into a dataset graph, which helps agents quickly understand what each dataset contains, how to use it, and which other datasets may be relevant\. In the task execution stage, a Planner Agent retrieves relevant datasets from the dataset graph and creates a step\-by\-step analysis plan\. Then, Execution Agents carry out the steps by writing and running code with a shared progress memory\. After all steps are completed, a Reporter Agent generates the final report, which can be further revised based on user feedback\.

To evaluate UrbanDS’s ability in data\-intensive urban tasks, we construct a benchmark for urban data science namedUrbanDS\-Bench\. Specifically, we collect 94 datasets from ten major cities in China, covering geospatial, mobility, and socioeconomic data\. Based on these, we construct 90 data analysis tasks and eight data modeling tasks, which comprehensively evaluate agents’ capability of discovering and utilizing data from large\-scale repositories\.

We evaluate our method on UrbanDS\-Bench and CoDA\-Bench\(Zhanget al\.,[2026b](https://arxiv.org/html/2607.26724#bib.bib12)\)\. Results show that UrbanDS surpasses existing data science agents as well as general coding agents by over 10%, demonstrating its capability to handle data\-intensive urban tasks\. Moreover, UrbanDS has been deployed on the urban operations platform of Dongxihu District, Wuhan, where it supports practical analysis over real municipal data repositories\. User studies further show that our system can significantly accelerate human data analysis processes\.

Our main contributions are summarized as follows:

- •We propose UrbanDS, a LLM multi\-agent system for data\-intensive urban tasks\. We organize large\-scale datasets into an easy\-to\-use graph, and further design a workflow that automatically discovers datasets, creates plans, executes code and generates reports\.
- •We construct UrbanDS\-Bench to evaluate agents’ ability to handle data\-intensive urban tasks\. It consists of 94 datasets from ten Chinese cities, 450 data analysis instances, and eight data modeling tasks\.
- •We conduct extensive experiments on two benchmarks, demonstrating the effectiveness of our method\. Moreover, we deploy UrbanDS in the real\-world to support urban data analysis\.

## 2\.Related Work

### 2\.1\.LLM Agents for Data Science

In recent years, there has been a growing interest in using LLM agents for data science tasks\. DS\-Agent uses case\-based reasoning to adapt expert solutions and improve machine learning experiments through feedback\(Guoet al\.,[2024](https://arxiv.org/html/2607.26724#bib.bib5)\)\. Data Interpreter represents a complex workflow as a hierarchical graph and progressively verifies and refines its execution\(Honget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib4)\)\. AutoML\-Agent coordinates specialized agents to cover the full AutoML pipeline from data retrieval to model deployment\(Triratet al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib3)\), while LAMBDA employs programmer and inspector agents to make data analysis accessible through natural\-language interaction\(Sunet al\.,[2026](https://arxiv.org/html/2607.26724#bib.bib6)\)\. More recently, DeepAnalyze trains an 8B agentic model to complete the process from raw data sources to analytical reports without relying on a fixed workflow\(Zhanget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib2)\)\. These systems show the potential of LLMs to support data understanding, code generation, analysis, modeling, and reporting\.

Progress in data science agents has also motivated benchmarks at different levels of complexity\. DS\-1000 and LLM4DS focus primarily on data science code generation\(Laiet al\.,[2023](https://arxiv.org/html/2607.26724#bib.bib11); Nascimentoet al\.,[2024](https://arxiv.org/html/2607.26724#bib.bib15)\)\. DSBench, DABstep, and DataSciBench further evaluate realistic analysis, modeling, multistep reasoning, and executable workflow completion\(Jinget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib8); Egget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib9); Zhanget al\.,[2026a](https://arxiv.org/html/2607.26724#bib.bib10)\)\. However, these benchmarks generally provide agents with the datasets needed for each task, so they do not directly evaluate dataset discovery from a large repository\. CoDA\-Bench takes an important step toward this setting by placing code agents in noisy file systems that contain hundreds of datasets\(Zhanget al\.,[2026b](https://arxiv.org/html/2607.26724#bib.bib12)\)\. Its results reveal that strong agents still struggle to combine data discovery with correct code execution\. UrbanDS addresses this capability through reusable dataset skills, explicit dataset relations, and graph\-guided task execution\.

![Refer to caption](https://arxiv.org/html/2607.26724v1/x1.png)Figure 1\.The overall framework of UrbanDS, including \(a\) dataset graph construction, and \(b\) task execution\.
### 2\.2\.Evaluation of LLM Agents on Urban Tasks

Several benchmarks have evaluated LLMs as urban agents\. CityBench integrates urban data with an interactive simulator and evaluates perception, understanding, and decision making across representative urban tasks and cities\(Fenget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib17)\)\. USTBench focuses on spatiotemporal reasoning and provides fine\-grained evaluation of understanding, forecasting, planning, and reflection in an interactive city environment\(Laiet al\.,[2026](https://arxiv.org/html/2607.26724#bib.bib16)\)\. These benchmarks demonstrate both the promise of LLMs for urban applications and their limitations in professional numerical reasoning, long\-horizon planning, and adaptation\.

Our benchmark draws inspiration from their design of representative urban tasks, but evaluates a different capability\. In CityBench and USTBench, the task environment and required observations are prepared for the agent, and the main goal is to assess reasoning or decision\-making within that environment\. In UrbanDS\-Bench, the agent receives a question and access to a large data pool, without being told which datasets are required\. It must discover the relevant datasets, understand their content, determine how to use them together, and execute the analysis\. UrbanDS\-Bench therefore evaluates urban data science in a data\-intensive repository rather than the inherent urban knowledge or reasoning ability of an LLM alone\.

## 3\.Preliminaries

In this section, we introduce the basic concepts and formally define the task studied in this work\.

##### Dataset Repository\.

Let𝒟=\{D1,D2,…,Dn\}\\mathcal\{D\}=\\\{D\_\{1\},D\_\{2\},\\ldots,D\_\{n\}\\\}denote a dataset repository containingnndatasets\. Each datasetDiD\_\{i\}can be a single file or a directory containing a collection of related files\. Urban datasets commonly use formats such as CSV, GeoJSON, XLSX, and Shapefile, and may contain tabular, spatial, temporal, or spatiotemporal information\. The datasets in𝒟\\mathcal\{D\}can differ in their formats, schemas, spatial coverage, temporal coverage, and levels of detail\.

###### Definition 3\.1 \(Data\-intensive Question Answering Task\)\.

Based on the dataset repository𝒟\\mathcal\{D\}, a data\-intensive question answering task is defined as a tupleTk=\(Qk,𝒟k,Ak\)T\_\{k\}=\(Q\_\{k\},\\mathcal\{D\}\_\{k\},A\_\{k\}\), whereQkQ\_\{k\}is a question expressed in natural language,𝒟k⊆𝒟\\mathcal\{D\}\_\{k\}\\subseteq\\mathcal\{D\}is the set of datasets required to answer the question, andAkA\_\{k\}is the expected answer\. The goal of the task is to identify the relevant datasets𝒟k\\mathcal\{D\}\_\{k\}from the complete repository𝒟\\mathcal\{D\}and perform the necessary data analysis to produce the answerAkA\_\{k\}, given only the questionQkQ\_\{k\}\.

This task is challenging for the following reasons\. First, the number of datasetsnnin the repository may be very large in real\-world scenarios, making it difficult to correctly identify the relevant datasets\. Second, there are complex relationships between datasets, and failing to consider these relationships can lead to selecting incorrect datasets or missing necessary ones\.

## 4\.Methods

### 4\.1\.Framework Overview

To address the aforementioned challenges, we design UrbanDS with two stages: dataset graph construction and task execution\. For large\-scale dataset repositories, inspecting every dataset when doing each task is costly and may exceed the context window of LLMs\. Therefore, UrbanDS instead explores each dataset once and stores the resulting knowledge for later use\. Specifically, A Data Profiling Agent explores the dataset through code and summarizes its content and usage in a dataset skill\. UrbanDS then identifies how datasets may be connected and organizes the skills and relations into a dataset graph\. In the task execution stage, UrbanDS leverages this graph to answer a user query\. Specifically, a Planner Agent progressively inspects and selects promising datasets and decomposes the query into executable steps\. Then an Execution Agent completes each step by writing and executing code\. All steps communicate through a shared progress memory that records observations, findings, and generated files\. Finally, a Report Agent summarizes the progress into a data analysis report, and a Revision Agent updates the report or its artifacts according to user feedback\. The overall framework of UrbanDS is presented in Figure[1](https://arxiv.org/html/2607.26724#S2.F1)\.

### 4\.2\.Dataset Graph Construction

To model the attributes of datasets and the complex relationships between them, we construct a dataset graph, which can be formulated as𝒢=\(𝒱,ℰ\)\\mathcal\{G\}=\(\\mathcal\{V\},\\mathcal\{E\}\)\. Each nodevi∈𝒱v\_\{i\}\\in\\mathcal\{V\}corresponds to a datasetDiD\_\{i\}and stores its dataset skill\. The edge setℰ=ℰspa∪ℰtem∪ℰsem\\mathcal\{E\}=\\mathcal\{E\}\_\{\\mathrm\{spa\}\}\\cup\\mathcal\{E\}\_\{\\mathrm\{tem\}\}\\cup\\mathcal\{E\}\_\{\\mathrm\{sem\}\}contains spatial, temporal, and semantic field relations\. The graph is built through the following two steps\.

#### 4\.2\.1\.Skill Extraction

To avoid exploring every dataset for each question, we propose to explore each dataset only once and store its information into a reusable skill\. Specifically, we construct a Data Profiling Agent to explore each dataset by writing Python code\. The agent examines schemas, samples, value distributions, and spatial and temporal coverage\. After obtaining a basic understanding of the dataset, it produces a dataset skill that summarizes the dataset content, field meanings, data scale, spatial and temporal ranges, loading method, and important usage notes\. The skills give later agents practical knowledge of how to use the data without repeatedly opening the raw files\.

#### 4\.2\.2\.Relation Identification

Urban datasets often contain spatial and temporal information\. Therefore, we model three types of relations between datasets: spatial, temporal, and semantic relations\. For spatial and temporal relations, we compare the corresponding coverage recorded in the dataset skills\. We create a spatial relation when two datasets have overlapping spatial coverage and a temporal relation when their time ranges overlap\.

Apart from spatial and temporal overlap, datasets may be connected through fields that refer to the same entities\. We call these connections semantic relations\. For example,region\_idin a population dataset may use the same region identifiers asidin a region boundary dataset\. A semantic relation allows agents to recognize that the two datasets can be joined even though their field names are different\. It is promising to identify semantic relations using LLMs\. However, it would be very costly to compare every pair of datasets\. To address these, we propose to first store potential relations in a semantic codebook, and then verify each relation via an LLM agent\.

As shown in Algorithm[1](https://arxiv.org/html/2607.26724#alg1), UrbanDS first constructs an incremental semantic codebook while profiling the datasets\. Each code represents one type of entity identifier or reference, such as a region identifier\. The Data Profiling Agent processes the datasets one by one\. For each field that may connect to another dataset, it compares the field meaning and sample values with the existing codes\. If the field matches an existing code, it is added to that code\. Otherwise, the agent creates a new code to describe its meaning\. As more datasets are profiled, the codebook grows to cover their fields without requiring a predefined ontology\.

After the codebook has been constructed, UrbanDS processes each code with a Relation Agent\. Fields assigned to the same code are possible connections, but their identifier systems may still be incompatible\. The Relation Agent therefore checks their meanings and values, removes incorrect assignments, and determines how the validated fields can be connected\. It then creates a semantic relation that records the involved datasets, their related fields, and a short connection description\.

Algorithm 1Codebook and Semantic Relation Construction1:Datasets

\{Di\}\\\{D\_\{i\}\\\}and dataset skills

\{Si\}\\\{S\_\{i\}\\\}
2:Codebook

𝒞\\mathcal\{C\}and semantic relation set

ℰsem\\mathcal\{E\}\_\{\\mathrm\{sem\}\}
3:

𝒞←∅\\mathcal\{C\}\\leftarrow\\emptyset
4:

ℰsem←∅\\mathcal\{E\}\_\{\\mathrm\{sem\}\}\\leftarrow\\emptyset
5:foreach dataset

DiD\_\{i\}do

6:foreach field

ffin

DiD\_\{i\}do

7:

c←ProfilingAgent​\(Di,f,Si,𝒞\)c\\leftarrow\\textsc\{ProfilingAgent\}\(D\_\{i\},f,S\_\{i\},\\mathcal\{C\}\)
8:⊳\\trianglerightMatch an existing code, create a new code, or returnNone

9:if

c≠Nonec\\neq\\textsc\{None\}then

10:

𝒞​\[c\]←𝒞​\[c\]∪\{\(Di,f\)\}\\mathcal\{C\}\[c\]\\leftarrow\\mathcal\{C\}\[c\]\\cup\\\{\(D\_\{i\},f\)\\\}
11:foreach code

ccin

𝒞\\mathcal\{C\}do

12:

ℰc←RelationAgent​\(c,𝒞​\[c\],\{Si\}\)\\mathcal\{E\}\_\{c\}\\leftarrow\\textsc\{RelationAgent\}\(c,\\mathcal\{C\}\[c\],\\\{S\_\{i\}\\\}\)
13:

ℰsem←ℰsem∪ℰc\\mathcal\{E\}\_\{\\mathrm\{sem\}\}\\leftarrow\\mathcal\{E\}\_\{\\mathrm\{sem\}\}\\cup\\mathcal\{E\}\_\{c\}
14:return

𝒞,ℰsem\\mathcal\{C\},\\mathcal\{E\}\_\{\\mathrm\{sem\}\}

### 4\.3\.Task Execution

Given a user query, UrbanDS first retrieves the required datasets and builds an analysis plan, then executes the plan one step at a time\. The two phases use the same dataset graph but require different levels of information\. Planning needs a broad view of possible data sources, whereas execution needs detailed data access and actual runtime feedback\.

#### 4\.3\.1\.Dataset Retrieval and Planning

Providing every dataset skill to the Planner Agent would fill its context with mostly irrelevant information, while providing only dataset names would not support reliable selection\. UrbanDS therefore retrieves data progressively, moving from broad and compact information to detailed and focused information\. The planner first sees the names and short descriptions of all datasets and selects promising candidates based on the user query\. It then reads the complete skills of these candidates to understand their contents and usage\.

When the planner inspects a dataset, it also sees compact summaries of its neighboring datasets in the graph\. These relations may reveal useful data whose descriptions alone do not appear directly relevant\. The planner can inspect the full skill of a useful neighbor and continue following relations until it has enough information\. A graph edge is treated as a possible connection rather than proof that a dataset must be used, so the planner still verifies each choice from its skill\. It finally produces an analysis plan containing the selected datasets and an ordered set of executable steps\.

#### 4\.3\.2\.Sub\-task Execution

Based on the execution plan, we assign one Execution Agent to each planned step\. Focusing an agent on one concrete step reduces the complexity of code generation, while shared results preserve dependencies across the full analysis\. Specifically, each Execution Agent writes and runs code, observes the actual output, and revises its approach when errors or unexpected values occur\. The resulting observations and findings are added to a shared progress memory, while generated files are also recorded for later steps\. Subsequent agents can therefore reuse earlier results instead of repeating the same computation\. Moreover, if the datasets selected by the planner are insufficient during execution, the agent can also retrieve additional datasets from the dataset graph\.

#### 4\.3\.3\.Report Generation and Revision

After all steps are completed, the Report Agent uses the plan, progress memory, and generated artifacts to produce a report grounded in the executed analysis\. A Revision Agent can then update the report or its artifacts according to user feedback\. This allows users to refine the results without restarting the complete task\.

![Refer to caption](https://arxiv.org/html/2607.26724v1/x2.png)Figure 2\.The construction and statistics of UrbanDS\-Bench\.

## 5\.Benchmark for Urban Data Science

Existing data science benchmarks evaluate the ability of LLM agents to perform data analysis, visualization, and modeling\(Jinget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib8); Egget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib9); Zhanget al\.,[2026a](https://arxiv.org/html/2607.26724#bib.bib10); Laiet al\.,[2023](https://arxiv.org/html/2607.26724#bib.bib11)\)\. However, in these benchmarks, the datasets required by a task are directly provided, so an agent can begin by writing analysis code\. As a result, these benchmarks do not evaluate whether an agent can discover a small set of relevant datasets from a large, heterogeneous repository and correctly connect them before analysis\. This capability is essential in urban data science, where data are distributed across many sources and linked through nontrivial spatial, temporal, and semantic relations\.

To address this limitation and systematically evaluate the capability of UrbanDS on data\-intensive urban data science tasks, we construct a benchmark named UrbanDS\-Bench\. Specifically, we collect urban datasets from various sources for ten major Chinese cities and construct both data analysis and data modeling tasks\. The overview of UrbanDS\-Bench is shown in Figure[2](https://arxiv.org/html/2607.26724#S4.F2)\.

### 5\.1\.Data Collection

UrbanDS\-Bench contains 94 datasets covering the 10 largest cities in China, including Beijing, Chongqing, Shanghai, Chengdu, Guangzhou, Shenzhen, Wuhan, Tianjin, Xi’an, and Zhengzhou\. For every city, we collect eight core geospatial datasets from OpenStreetMap: administrative divisions, point\-based points of interest \(POIs\), area\-based points of interest \(AOIs\), buildings, roads, railways, waterways, and water bodies\. These datasets provide a common basis for within\-city and cross\-city spatial analysis\. We further collect human activity data and socioeconomic data to enrich the mobility and socioeconomic dimensions of the benchmark\. They cover crowd inflow and outflow, OD flow, user trajectories, regional population, firm, restaurant, takeaway\-order and price indicators, as well as POI category, brand, and commercial\-area metadata\. The resulting collection spans tabular, geospatial, and spatiotemporal data stored in CSV, GeoJSON, and JSON formats\.

To create a genuinely data\-intensive environment, we place all datasets in a single flat data pool, remove city names and most topic names from filenames, randomly shuffle the files, and assign anonymous identifiers\. As a result, an agent generally cannot select a dataset from its filename alone; it must inspect file contents and schemas to infer the dataset semantics\. The same pool also contains many plausible but irrelevant datasets for each task, requiring both dataset retrieval and valid cross\-dataset integration\.

### 5\.2\.Data Analysis Tasks

Following existing studies\(Laiet al\.,[2026](https://arxiv.org/html/2607.26724#bib.bib16); Fenget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib17)\), we construct question–answer pairs that require executable analysis over the collected data\. We organize them along two complementary dimensions: the spatiotemporal nature of the required reasoning and the number of datasets required to answer the question\. The first dimension contains three categories\.*Spatial*tasks involve operations such as containment, intersection, proximity, buffering, and aggregation over geographic entities\.*Temporal*tasks require ordering, filtering, aggregation, or extrema detection over timestamps and time series\.*Spatiotemporal*tasks jointly reason about space and time, for example by mapping trajectories to regions or comparing time\-varying flows among regions crossed by a transport feature\. The second dimension ranges from one to six datasets and measures the breadth of dataset discovery and integration required by a task\.

For each task type, we construct 5 questions by sampling from the data\. For example, for the task of identifying the region of a POI \( Figure[2](https://arxiv.org/html/2607.26724#S4.F2)\), we randomly sample five POIs and compute the corresponding regions, resulting in five questions\. Overall, the benchmark contains 90 task types and 450 questions\. Among them, 56 task types \(280 questions\) are spatial, 11 types \(55 questions\) are temporal, and 23 types \(115 questions\) are spatiotemporal\. Questions range from direct single\-dataset aggregation to multi\-stage analyses that combine as many as six independent files\. Each question explicitly specifies the expected answer format, while its reference answer and required dataset set are generated from deterministic data operations\. During evaluation, however, the agent is given the question and the full anonymized data pool rather than the identities of the required datasets\.

### 5\.3\.Data Modeling Tasks

Beyond analytical QA, we construct eight Kaggle\-style data modeling tasks, covering representative problems in urban computing\. For each task, the agent receives a supervised training table, an unlabeled test table, and access to the remaining anonymized data pool\. The agent needs to identify useful datasets, construct features, choose a validation strategy and model, and produce a submission in the specified CSV schema\. The tasks are as follows\.

##### Land use classification\(Xuet al\.,[2023](https://arxiv.org/html/2607.26724#bib.bib23)\)

Given polygon boundaries and labels for training AOIs, the agent classifies held\-out AOIs into nine land\-use categories by exploiting their geometry and surrounding urban context\. We report accuracy and macro\-F1\.

##### Crowd flow generation\(Zhouet al\.,[2023a](https://arxiv.org/html/2607.26724#bib.bib20)\)

The agent generates a representative 24\-hour weekday inflow and outflow profile for 362 spatially held\-out regions after observing weekday flows in 648 other regions\. We use MAE, RMSE, and SMAPE\.

##### OD flow prediction\(Ronget al\.,[2024](https://arxiv.org/html/2607.26724#bib.bib22)\)

We hide a stratified 20% of zero and nonzero pairs in the1,010×1,0101\{,\}010\\times 1\{,\}010regional OD flow matrix and ask the agent to complete their flow values\. We evaluate the predictions using the Common Part of Commuters \(CPC\) and MAE\.

##### POI check\-in prediction\(Zhanget al\.,[2026c](https://arxiv.org/html/2607.26724#bib.bib24)\)

The agent estimates check\-in counts for held\-out POIs using labels stratified by check\-in magnitude and any non\-target POI or urban\-context data it discovers\. We report MAE, RMSE, andR2R^\{2\}\.

##### Economic prediction\(Zhouet al\.,[2023b](https://arxiv.org/html/2607.26724#bib.bib18)\)

The agent predicts firm counts for regions held out as complete spatial blocks, encouraging generalization beyond immediately neighboring labeled regions\. We report MAE, RMSE, andR2R^\{2\}\.

##### Population prediction\(Zhouet al\.,[2023b](https://arxiv.org/html/2607.26724#bib.bib18)\)

The agent estimates population for the same spatially held\-out regions by constructing explanatory features from the available urban datasets\. We report MAE, RMSE, andR2R^\{2\}\.

##### Restaurant number prediction\(Zhouet al\.,[2023b](https://arxiv.org/html/2607.26724#bib.bib18)\)

The agent predicts the number of restaurants in spatially held\-out regions from non\-target urban context, evaluated by MAE, RMSE, andR2R^\{2\}\.

##### Takeaway orders prediction\(Zhouet al\.,[2023b](https://arxiv.org/html/2607.26724#bib.bib18)\)

The agent predicts takeaway\-order volume for spatially held\-out regions; total order price is removed because it is a direct target\-derived proxy\. We report MAE, RMSE, andR2R^\{2\}\.

Details about evaluation of data modeling tasks are presented in[A](https://arxiv.org/html/2607.26724#A1)\. For each modeling task, we split the corresponding target data into public training and test tables and treat the remaining files as candidate auxiliary datasets\. We remove the original target file and any strongly target\-derived source that could leak test labels\. Unlike the data analysis tasks, a modeling task has no single ground\-truth set of required auxiliary datasets: an agent may select and combine any leakage\-safe sources that improve its model\. This setting therefore evaluates whether an agent can make effective use of a large urban data repository under a more open\-ended objective, in addition to training a predictive model\.

## 6\.Experiments

### 6\.1\.Experiment Settings

#### 6\.1\.1\.Benchmarks

We evaluate UrbanDS on two benchmarks\.UrbanDS\-Bench, introduced in the previous section, includes 450 data analysis tasks and eight data modeling tasks\.CoDA\-Bench\(Zhanget al\.,[2026b](https://arxiv.org/html/2607.26724#bib.bib12)\)evaluates whether code agents can jointly perform data discovery and executable analysis in large file systems\. We use its hard subset, which contains 119 challenging tasks from 15 Kaggle communities\. Each task is placed in its complete community repository, requiring an agent to locate relevant resources among approximately 1,422 files on average\.

For evaluation metrics, we report answer accuracy for the data analysis tasks in UrbanDS\-Bench and CoDA\-Bench\. The data modeling tasks use their corresponding predictive metrics introduced previously\.

#### 6\.1\.2\.Baselines

We compare UrbanDS with the following baselines:

- •DS\-Agent\(Guoet al\.,[2024](https://arxiv.org/html/2607.26724#bib.bib5)\): A data science agent that uses case\-based reasoning to adapt expert solutions from Kaggle and improves generated experiments through iterative feedback\.
- •Data Interpreter\(Honget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib4)\): An LLM agent that represents complex data science workflows with a hierarchical graph and progressively verifies and refines each step during execution\.
- •DeepAnalyze\(Zhanget al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib2)\): An 8B agentic model trained with a curriculum of data science tasks to complete the full process from raw data sources to analytical reports\.
- •AutoGen\(Wuet al\.,[2023](https://arxiv.org/html/2607.26724#bib.bib1)\): A general multi\-agent framework in which customizable agents communicate through natural language and code and use tools to complete a task\.
- •Claude Code111[https://claude\.com/product/claude\-code](https://claude.com/product/claude-code): A general\-purpose coding agent from Anthropic that can inspect files, write code, and execute commands in a working environment\.

These baselines cover state\-of\-the\-art data science agents, a general multi\-agent framework, and a commercial coding agent\. This range allows us to compare UrbanDS with both systems designed for data science and agents designed for general coding tasks\.

#### 6\.1\.3\.Implementation Details

DeepAnalyze is an open\-source agentic model with 8B parameters, so we use its official model and implementation\. For all other baselines and our method, we use DeepSeek\-V4\-Pro\(Xuet al\.,[2026](https://arxiv.org/html/2607.26724#bib.bib25)\)with a temperature of 0 to ensure fair comparison\. All methods receive the same task instruction, dataset repository, execution environment, and required answer format\.

### 6\.2\.Overall Performance

Table 1\.Performance comparison with baselines on UrbanDS\-Bench and CoDA\-Bench\. The best results are in bold, and the second\-best results are underlined\.MethodUrbanDS\-BenchCoDA\-BenchLand Use ClassificationCrowd Flow GenerationCheck\-in PredictionSpatial/%Temporal/%ST/%Overall/%Overall/%AccF1MAERMSESMAPEMAERMSER2DeepAnalyze0\.70\.00\.00\.40\.80\.3710\.3196\.70510\.9111\.1162\.8243\.458\-1\.063DS\-Agent3\.918\.20\.04\.72\.50\.3990\.3322\.0005\.7620\.6513\.8414\.319\-2\.219Data Interpreter4\.63\.60\.03\.35\.00\.3810\.3057\.48611\.0651\.2493\.7244\.199\-2\.043AutoGen40\.4\\ul76\.443\.545\.631\.1\\ul0\.502\\ul0\.4362\.8325\.7620\.8412\.404\\ul2\.792\\ul\-0\.345Claude Code\\ul62\.556\.4\\ul67\.0\\ul62\.9\\ul42\.00\.4440\.386\\ul1\.821\\ul5\.681\\ul0\.656\\ul2\.1113\.209\-0\.777UrbanDS65\.783\.673\.970\.046\.20\.5500\.4831\.7955\.6220\.6170\.9421\.7630\.464

Table 2\.Performance comparison with baselines on the remaining data modeling tasks of UrbanDS\-Bench\. The best results are in bold, and the second\-best results are underlined\.MethodOD Flow PredictionEconomic PredictionPopulation PredictionRestaurant PredictionOrders PredictionCPCMAEMAERMSER2MAERMSER2MAERMSER2MAERMSER2DeepAnalyze\\ul0\.645\\ul3\.0291\.2731\.6770\.4131\.1131\.5110\.1240\.7931\.0320\.7163\.6085\.171\-0\.725DS\-Agent0\.3156\.9332\.0152\.488\-0\.2920\.9841\.2620\.3900\.9531\.2600\.5761\.9623\.0680\.393Data Interpreter0\.1478\.7352\.0152\.488\-0\.2920\.9841\.2620\.3902\.1092\.654\-0\.8792\.1623\.2280\.327AutoGen0\.2975\.1921\.3161\.7020\.395\\ul0\.773\\ul1\.075\\ul0\.5570\.8411\.1520\.6462\.1153\.2560\.316Claude Code0\.4724\.116\\ul1\.065\\ul1\.314\\ul0\.6400\.7921\.0940\.541\\ul0\.656\\ul0\.871\\ul0\.797\\ul1\.710\\ul2\.484\\ul0\.602UrbanDS0\.7302\.5190\.8311\.1030\.7460\.6810\.9300\.6690\.5840\.8050\.8271\.6292\.3610\.640

![Refer to caption](https://arxiv.org/html/2607.26724v1/x3.png)Figure 3\.Accuracy under different dataset numbers in UrbanDS\-Bench\.Tables[1](https://arxiv.org/html/2607.26724#S6.T1)and[2](https://arxiv.org/html/2607.26724#S6.T2)report the overall results, from which we have the following findings\.

First, UrbanDS achieves the best accuracy in every reasoning category of UrbanDS\-Bench, reaching 65\.7% on spatial tasks, 83\.6% on temporal tasks, and 73\.9% on spatiotemporal tasks\. Its overall accuracy is 70\.0%, compared with 62\.9% for the strongest baseline, Claude Code\. This corresponds to a relative improvement of 11\.2%\. UrbanDS also achieves 46\.2% accuracy on CoDA\-Bench, improving upon Claude Code by 10\.0%\. The consistent gains on both the urban benchmark and the general data discovery benchmark show that UrbanDS is effective beyond a single data domain\. UrbanDS also performs strongly on the eight data modeling tasks\. It obtains the best result on all metrics\. The improvement is especially clear on tasks that benefit from auxiliary data\. For example, UrbanDS achieves anR2R^\{2\}of 0\.464 for POI check\-in prediction, while all baselines produce negativeR2R^\{2\}values\. These results indicate that UrbanDS can discover and use relevant datasets to build more accurate models\.

Among the baselines, the data science agents DS\-Agent and Data Interpreter perform poorly on UrbanDS\-Bench and CoDA\-Bench\. This is probably because these agents are mainly designed to plan an analysis after the input data have already been specified\. In our setting, however, an agent must first explore a large repository, identify the required datasets, and adjust its plan as new dataset information becomes available\. General coding agents, especially Claude Code and AutoGen, handle this exploration more effectively, but still lack an explicit representation of dataset content and connections\. UrbanDS addresses this difficulty with dataset skills, the dataset graph, and progressive retrieval by the Planner Agent\. Additionally, DeepAnalyze performs worst on the two analysis benchmarks, suggesting that agentic training alone does not fully compensate for the limited capability of its 8B backbone on tasks that require extensive data discovery and reasoning\.

Figure[3](https://arxiv.org/html/2607.26724#S6.F3)further compares accuracy by the number of datasets required for each UrbanDS\-Bench task\. UrbanDS performs best on tasks involving one, two, or three datasets, with accuracies of 78\.5%, 79\.4%, and 69\.3%, respectively\. When tasks require four or more datasets, the accuracy of every method is below 53%\. Performance generally becomes lower as more datasets must be jointly identified and used, particularly from three datasets onward\. This result confirms that finding the correct data and combining information across datasets are central challenges in data\-intensive analysis\. Although UrbanDS substantially reduces this difficulty, tasks involving many datasets remain challenging\.

### 6\.3\.Ablation Study

![Refer to caption](https://arxiv.org/html/2607.26724v1/x4.png)Figure 4\.Ablation study results\.We evaluate two variants of UrbanDS to understand the contribution of dataset relations and dataset skills\. The first variant removes the relations between datasets, so the Planner Agent cannot use the dataset graph to locate connected data\. The second removes dataset skills and exposes the agent to the original files without the structured descriptions produced during data preparation\.

As shown in Figure[4](https://arxiv.org/html/2607.26724#S6.F4), both variants perform worse on UrbanDS\-Bench and CoDA\-Bench\. Removing dataset relations reduces the overall accuracy from 70\.0% to 66\.7% on UrbanDS\-Bench and from 46\.2% to 36\.1% on CoDA\-Bench\. The relative decrease is 13\.3% on average across the two benchmarks\. This result shows that explicit relations help the agent move from an initially relevant dataset to other datasets needed by the task\. The effect is particularly clear on CoDA\-Bench, where useful resources are distributed across a large file repository\.

Removing dataset skills causes a larger decrease\. Accuracy falls to 58\.9% on UrbanDS\-Bench and 34\.5% on CoDA\-Bench, giving an average relative decrease of 20\.6%\. The largest change within UrbanDS\-Bench occurs on spatiotemporal tasks, where accuracy drops from 73\.9% to 39\.1%\. These tasks require the agent to understand both the content and usage of heterogeneous datasets before combining them\. The results therefore show that dataset relations mainly support data discovery, while dataset skills provide the detailed knowledge required to inspect, select, and correctly use the discovered data\. Both components are important for reliable analysis over large dataset repositories\.

### 6\.4\.Deployment and User Study

#### 6\.4\.1\.Real\-world Deployment

![Refer to caption](https://arxiv.org/html/2607.26724v1/x5.png)Figure 5\.Interface of the deployed system\.To evaluate UrbanDS beyond controlled benchmarks, we deployed it on the urban operations platform of Dongxihu District, Wuhan\. The deployed repository currently contains 103 dataset skills constructed directly from real urban and operational data\. These datasets cover urban governance events, transportation and mobility, emergency response, population and communities, industrial economy, public facilities, municipal networks, environment, and land\-use planning\. They include heterogeneous tables, geographic layers, time series, trajectories, and text stored in formats such as CSV, Excel, GeoJSON, JSON, and SQL\. This setting reflects a practical difficulty of urban operations: relevant data are distributed across many domains and formats, making manual discovery and integration costly\.

The profiling and relation construction stages convert this data into reusable skills and a dataset graph\. The current graph contains more than 2,200 spatial, temporal, and semantic relations, allowing the Planner Agent to discover useful data connections when answering natural\-language requests\. Figure[5](https://arxiv.org/html/2607.26724#S6.F5)shows an interface of the deployed platform that visualizes the dataset graph, where different colors represent different relation types\. Users can conveniently search for a dataset, view its skill to understand its content and usage, and quickly locate related datasets through the graph\. The deployment demonstrates that UrbanDS can organize a real urban data repository and make its data knowledge more accessible for practical analysis\.

#### 6\.4\.2\.User Study

In addition to supporting urban operations, the deployed platform allows users to register their own datasets and use UrbanDS for their analysis tasks\. We invited 12 users to use the platform for approximately one month and collected questionnaires and usage logs\. The participants were mainly graduate researchers and urban professionals from fields including urban planning, transportation, economy, public services, environment, and population\. They were familiar with urban analysis but had different levels of programming experience\. By analyzing the questionnaires and usage logs, we have the following findings\.

First, UrbanDS substantially reduced analysis time\. According to participants’ estimates, manually completing a data analysis task typically takes around 4\.08 hours on average, whereas using UrbanDS reduces the required time to only 0\.73 hours, corresponding to an estimated speedup of 5\.6 times\. Second, the tasks performed by users typically involved two to six datasets\. Nine users usually worked with two or three datasets, while three users used four to six, which is consistent with the setting of up to six datasets in our UrbanDS\-Bench\. Third, users generally found UrbanDS useful, with eight participants rating its task completion and helpfulness at 4 or 5 out of 5\. The logs also show that users inspected and revised the generated results, indicating that UrbanDS is most useful as an analysis assistant whose outputs remain open to human verification\.

## 7\.Conclusion

In this paper, we presented UrbanDS, a multi\-agent system for data\-intensive urban data science tasks\. UrbanDS first creates a reusable skill for each dataset and identifies spatial, temporal, and semantic relations between datasets, resulting in a dataset graph\. Based on this graph, the Planner Agent progressively finds relevant datasets and constructs an analysis plan\. Execution Agents then complete the plan through code execution and runtime feedback, while shared progress records connect the results of different steps\. The system finally generates a report and supports further revision based on user feedback\. We also construct UrbanDS\-Bench to evaluate agents’ ability to handle data\-intensive urban tasks\. Experiments show that UrbanDS outperforms existing data science agents and strong general coding agents\. The real\-world deployment and user study further demonstrate its practical value for organizing urban data and supporting users in complex analysis tasks\.

Like existing data science agents, UrbanDS currently focuses on completing analysis and modeling tasks specified by users\. It remains limited in more open\-ended settings, such as exploring a repository without a predefined question, identifying unexpected patterns, and proposing new research questions from data\(Xiaet al\.,[2025](https://arxiv.org/html/2607.26724#bib.bib14)\)\. Future work will extend UrbanDS from executing user requests toward actively discovering and validating useful insights, with the broader goal of building a more capable urban data scientist\.

## References

- A\. Egg, M\. I\. Goyanes, F\. Kingma, A\. Mora, L\. von Werra, and T\. Wolf \(2025\)Dabstep: data agent benchmark for multi\-step reasoning\.arXiv preprint arXiv:2506\.23719\.Cited by:[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p2.1),[§5](https://arxiv.org/html/2607.26724#S5.p1.1)\.
- J\. Feng, J\. Zhang, T\. Liu, X\. Zhang, T\. Ouyang, J\. Yan, Y\. Du, S\. Guo, and Y\. Li \(2025\)Citybench: evaluating the capabilities of large language models for urban tasks\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5413–5424\.Cited by:[§2\.2](https://arxiv.org/html/2607.26724#S2.SS2.p1.1),[§5\.2](https://arxiv.org/html/2607.26724#S5.SS2.p1.1)\.
- S\. Guo, C\. Deng, Y\. Wen, H\. Chen, Y\. Chang, and J\. Wang \(2024\)DS\-agent: automated data science by empowering large language models with case\-based reasoning\.InInternational Conference on Machine Learning,pp\. 16813–16848\.Cited by:[§1](https://arxiv.org/html/2607.26724#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p1.1),[1st item](https://arxiv.org/html/2607.26724#S6.I1.i1.p1.1)\.
- N\. Hollmann, S\. Müller, and F\. Hutter \(2023\)Large language models for automated data science: introducing caafe for context\-aware automated feature engineering\.Advances in Neural Information Processing Systems36,pp\. 44753–44775\.Cited by:[§1](https://arxiv.org/html/2607.26724#S1.p1.1)\.
- S\. Hong, Y\. Lin, B\. Liu, B\. Liu, B\. Wu, C\. Zhang, D\. Li, J\. Chen, J\. Zhang, J\. Wang,et al\.\(2025\)Data interpreter: an llm agent for data science\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 19796–19821\.Cited by:[§1](https://arxiv.org/html/2607.26724#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p1.1),[2nd item](https://arxiv.org/html/2607.26724#S6.I1.i2.p1.1)\.
- L\. Jing, Z\. Huang, X\. Wang, W\. Yao, W\. Yu, K\. Ma, H\. Zhang, X\. Du, and D\. Yu \(2025\)DSBench: how far are data science agents from becoming data science experts?\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 32597–32649\.Cited by:[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p2.1),[§5](https://arxiv.org/html/2607.26724#S5.p1.1)\.
- S\. Lai, Y\. Ning, Z\. Yuan, Z\. Chen, and H\. Liu \(2026\)USTBench: benchmarking and dissecting spatiotemporal reasoning capabilities of LLMs as urban agents\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ETzBStUFJy)Cited by:[§2\.2](https://arxiv.org/html/2607.26724#S2.SS2.p1.1),[§5\.2](https://arxiv.org/html/2607.26724#S5.SS2.p1.1)\.
- Y\. Lai, C\. Li, Y\. Wang, T\. Zhang, R\. Zhong, L\. Zettlemoyer, W\. Yih, D\. Fried, S\. Wang, and T\. Yu \(2023\)DS\-1000: a natural and reliable benchmark for data science code generation\.InInternational Conference on Machine Learning,pp\. 18319–18345\.Cited by:[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p2.1),[§5](https://arxiv.org/html/2607.26724#S5.p1.1)\.
- N\. Nascimento, E\. Guimaraes, S\. S\. Chintakunta, and S\. A\. Boominathan \(2024\)Llm4ds: evaluating large language models for data science code generation\.arXiv preprint arXiv:2411\.11908\.Cited by:[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p2.1)\.
- C\. Rong, J\. Ding, and Y\. Li \(2024\)An interdisciplinary survey on origin\-destination flows modeling: theory and techniques\.ACM Computing Surveys57\(1\),pp\. 1–49\.Cited by:[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px3)\.
- M\. Sun, R\. Han, B\. Jiang, H\. Qi, D\. Sun, Y\. Yuan, and J\. Huang \(2026\)Lambda: a large model based data agent\.Journal of the American Statistical Association121\(553\),pp\. 1–13\.Cited by:[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p1.1)\.
- P\. Trirat, W\. Jeong, and S\. J\. Hwang \(2025\)AutoML\-agent: a multi\-agent llm framework for full\-pipeline automl\.InInternational Conference on Machine Learning,pp\. 60099–60146\.Cited by:[§1](https://arxiv.org/html/2607.26724#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p1.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.\(2023\)Autogen: enabling next\-gen llm applications via multi\-agent conversation\.arXiv preprint arXiv:2308\.08155\.Cited by:[4th item](https://arxiv.org/html/2607.26724#S6.I1.i4.p1.1)\.
- T\. Xia, J\. Zhang, R\. You, A\. Xu, L\. Zhang, T\. Tu, J\. Wang, J\. Piao, Y\. Zhang, F\. Xu,et al\.\(2025\)AI urban scientist: multi\-agent collaborative automation for urban research\.arXiv preprint arXiv:2512\.07849\.Cited by:[§7](https://arxiv.org/html/2607.26724#S7.p2.1)\.
- A\. Xu, B\. Lin, B\. Xue, B\. Wang, B\. Xu, B\. Wu, B\. Zhang, C\. Lin, C\. Dong, C\. Ling,et al\.\(2026\)Deepseek\-v4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[§6\.1\.3](https://arxiv.org/html/2607.26724#S6.SS1.SSS3.p1.1)\.
- R\. Xu, W\. Huang, J\. Zhao, M\. Chen, and L\. Nie \(2023\)A spatial and adversarial representation learning approach for land use classification with pois\.ACM Transactions on Intelligent Systems and Technology14\(6\),pp\. 1–25\.Cited by:[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px1)\.
- D\. Zhang, S\. Zhoubian, M\. Cai, F\. Li, L\. Yang, W\. Wang, T\. Dong, Z\. Hu, J\. Tang, and Y\. Yue \(2026a\)Datascibench: an llm agent benchmark for data science\.InFindings of the Association for Computational Linguistics: ACL 2026,pp\. 3685–3728\.Cited by:[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p2.1),[§5](https://arxiv.org/html/2607.26724#S5.p1.1)\.
- S\. Zhang, J\. Fan, M\. Fan, G\. Li, and X\. Du \(2025\)Deepanalyze: agentic large language models for autonomous data science\.arXiv preprint arXiv:2510\.16872\.Cited by:[§1](https://arxiv.org/html/2607.26724#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p1.1),[3rd item](https://arxiv.org/html/2607.26724#S6.I1.i3.p1.1)\.
- Y\. Zhang, J\. Fan, M\. Fan, S\. Zhang, and X\. Du \(2026b\)CoDA\-bench: can code agents handle data\-intensive tasks?\.InProceedings of the 43rd International Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2607.26724#S1.p2.1),[§1](https://arxiv.org/html/2607.26724#S1.p5.1),[§2\.1](https://arxiv.org/html/2607.26724#S2.SS1.p2.1),[§6\.1\.1](https://arxiv.org/html/2607.26724#S6.SS1.SSS1.p1.1)\.
- Z\. Zhang, M\. Xie, Y\. Li, L\. Cai, S\. Luo, and G\. Cong \(2026c\)CausalPOI: spatio\-temporal graph\-based causal modeling for cold\-start poi check\-in forecasting\.arXiv preprint arXiv:2606\.05413\.Cited by:[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px4)\.
- Z\. Zhou, J\. Ding, Y\. Liu, D\. Jin, and Y\. Li \(2023a\)Towards generative modeling of urban flow through knowledge\-enhanced denoising diffusion\.InProceedings of the 31st ACM international conference on advances in geographic information systems,pp\. 1–12\.Cited by:[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px2)\.
- Z\. Zhou, Y\. Liu, J\. Ding, D\. Jin, and Y\. Li \(2023b\)Hierarchical knowledge graph learning enabled socioeconomic indicator prediction in location\-based social network\.InProceedings of the ACM web conference 2023,pp\. 122–132\.Cited by:[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px5),[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px6),[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px7),[§5\.3](https://arxiv.org/html/2607.26724#S5.SS3.SSS0.Px8)\.

## Appendix AEvaluation Metrics for Data Modeling Tasks

Table 3\.Metrics used for the data modeling tasks in UrbanDS\-Bench\. An upward arrow indicates that a larger value is better, and a downward arrow indicates that a smaller value is better\.This section describes the metrics used for the eight data modeling tasks in UrbanDS\-Bench\. For every task, a valid submission must contain exactly the same sample keys as the ground truth\. All submitted numeric values must be finite and nonnegative\. LetNNdenote the number of evaluated values, and letyiy\_\{i\}andy^i\\hat\{y\}\_\{i\}denote the ground truth and prediction for theiith value, respectively\. Higher values are better for Accuracy, Macro F1, CPC, andR2R^\{2\}, while lower values are better for MAE, RMSE, and SMAPE\.

### A\.1\.Classification Metrics

##### Accuracy\.

Accuracy measures the proportion of samples assigned the correct class:

\(1\)Accuracy=1N​∑i=1N𝟏​\(yi=y^i\),\\mathrm\{Accuracy\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbf\{1\}\(y\_\{i\}=\\hat\{y\}\_\{i\}\),where𝟏​\(⋅\)\\mathbf\{1\}\(\\cdot\)equals 1 when its argument is true and 0 otherwise\.

##### Macro F1\.

For each classccin the set of ground truth classes𝒞\\mathcal\{C\}, its F1 score is

\(2\)F1c=2​T​Pc2​T​Pc\+FPc\+FNc,\\mathrm\{F1\}\_\{c\}=\\frac\{2\\mathrm\{TP\}\_\{c\}\}\{2\\mathrm\{TP\}\_\{c\}\+\\mathrm\{FP\}\_\{c\}\+\\mathrm\{FN\}\_\{c\}\},whereTPc\\mathrm\{TP\}\_\{c\},FPc\\mathrm\{FP\}\_\{c\}, andFNc\\mathrm\{FN\}\_\{c\}are the numbers of true positives, false positives, and false negatives for classcc\. The score is set to 0 if the denominator is 0\. Macro F1 assigns equal weight to every class:

\(3\)MacroF1=1\|𝒞\|​∑c∈𝒞F1c\.\\mathrm\{MacroF1\}=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{c\\in\\mathcal\{C\}\}\\mathrm\{F1\}\_\{c\}\.

### A\.2\.Regression Metrics

##### Mean Absolute Error\.

MAE is the average absolute difference between predictions and ground truth values:

\(4\)MAE=1N​∑i=1N\|yi−y^i\|\.\\mathrm\{MAE\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\|y\_\{i\}\-\\hat\{y\}\_\{i\}\|\.

##### Root Mean Squared Error\.

RMSE gives more weight to large errors by squaring each difference before averaging:

\(5\)RMSE=1N​∑i=1N\(yi−y^i\)2\.\\mathrm\{RMSE\}=\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(y\_\{i\}\-\\hat\{y\}\_\{i\}\)^\{2\}\}\.

##### Symmetric Mean Absolute Percentage Error\.

SMAPE measures error relative to the magnitudes of both the ground truth and prediction:

\(6\)SMAPE=1N​∑i=1N\|yi−y^i\|\(\|yi\|\+\|y^i\|\)/2\+ϵ,\\mathrm\{SMAPE\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{\|y\_\{i\}\-\\hat\{y\}\_\{i\}\|\}\{\(\|y\_\{i\}\|\+\|\\hat\{y\}\_\{i\}\|\)/2\+\\epsilon\},whereϵ=10−8\\epsilon=10^\{\-8\}prevents division by zero\. The implementation reports the ratio directly rather than multiplying it by 100\.

##### Common Part of Commuters\.

CPC measures the overlap between two nonnegative flow distributions:

\(7\)CPC=2​∑i=1Nmin⁡\(yi,y^i\)∑i=1Nyi\+∑i=1Ny^i\.\\mathrm\{CPC\}=\\frac\{2\\sum\_\{i=1\}^\{N\}\\min\(y\_\{i\},\\hat\{y\}\_\{i\}\)\}\{\\sum\_\{i=1\}^\{N\}y\_\{i\}\+\\sum\_\{i=1\}^\{N\}\\hat\{y\}\_\{i\}\}\.If both the total ground truth flow and total predicted flow are zero, CPC is defined as 1\. A larger CPC indicates greater agreement between the two flow distributions\.

##### Coefficient of Determination\.

The coefficient of determination compares prediction error with the variation in the ground truth:

\(8\)R2=1−∑i=1N\(yi−y^i\)2∑i=1N\(yi−y¯\)2,y¯=1N​∑i=1Nyi\.R^\{2\}=1\-\\frac\{\\sum\_\{i=1\}^\{N\}\(y\_\{i\}\-\\hat\{y\}\_\{i\}\)^\{2\}\}\{\\sum\_\{i=1\}^\{N\}\(y\_\{i\}\-\\bar\{y\}\)^\{2\}\},\\qquad\\bar\{y\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}y\_\{i\}\.A value closer to 1 indicates a better fit\. The score can be negative when a model performs worse than predicting the ground truth mean\.

### A\.3\.Task Specific Evaluation

Table[3](https://arxiv.org/html/2607.26724#A1.T3)lists the metrics used by each task\. For Crowd Flow Generation, the outflow and inflow values of all region and hour pairs are flattened into one vector before MAE, RMSE, and SMAPE are calculated\. For OD Flow Prediction, CPC and MAE are calculated over the flow values of all origin and destination pairs\.

Five count prediction tasks use a natural logarithm before calculating MAE, RMSE, andR2R^\{2\}\. Specifically, each ground truth or predicted valuevvis transformed independently as

\(9\)g​\(v\)=\{log⁡\(v\),v\>1,0,0≤v≤1\.g\(v\)=\\begin\{cases\}\\log\(v\),&v\>1,\\\\ 0,&0\\leq v\\leq 1\.\\end\{cases\}The formulas above are then applied tog​\(yi\)g\(y\_\{i\}\)andg​\(y^i\)g\(\\hat\{y\}\_\{i\}\)rather than the original values\. This transformation is used for POI Check\-in Prediction, Economic Prediction, Population Prediction, Restaurant Number Prediction, and Takeaway Orders Prediction\.