UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks

# UrbanAgent: A Tool-Augmented Agent for Cross-System Urban Tasks
Source: [https://arxiv.org/html/2608.03018](https://arxiv.org/html/2608.03018)
Jiayu Cao, Xingyuan Zeng, Feiyu Li, Zhijin Huang, Xujie Yuan, Rongxiang Chen, Shimin Di, Libin Zheng, and Jian Yin

###### Abstract

Modern cities rely on an increasing number of digital services to operate, but residents’ daily needs are still difficult to meet\. Services are fragmented and have little interoperability, placing a heavy operational burden on users\. Existing digital platforms, urban foundation models, and intelligent assistants each address only isolated aspects of an urban task\. But they struggle to reliably convert complex natural\-language requests into executable cross\-system workflows\. We propose UrbanAgent, a tool\-augmented agent framework for cross\-system urban tasks\. It couples the cognitive and reasoning capabilities of a large language model with a toolset supporting code execution, API calls, and Model Context Protocol\. Through one adaptive closed loop, it clarifies missing information before acting, grounds tool use in live observations, and aligns the final response with observed evidence and task constraints\. To address the evaluation gap, we introduce UrbanEval, a benchmark specifically designed for cross\-system urban request\. Unlike prior benchmarks that assess either general tool use or urban knowledge and reasoning, UrbanEval evaluates both task results and execution quality, including required tool coverage, dependency validity, and evidence traceability\. Experimental results indicate that UrbanAgent reaches a 71% task success rate, 10points above the strongest baseline\. This lead holds across GPT\-5\-mini, Gemini\-2\.5\-flash, DeepSeek\-V4\-flash, and Qwen3\-235B\-A22B\.

## Introduction

Modern cities have deployed extensive digital systems for transportation, urban planning, environment, energy, and public services\(Zhenget al\.[2014](https://arxiv.org/html/2608.03018#bib.bib1)\)\.Yet this digitalization has not translated into convenience\. These functions are scattered across separate systems with independent data, interfaces, and procedures\. Even a simple daily request involves multiple aspects\. Figure[1](https://arxiv.org/html/2608.03018#Sx1.F1)illustrates a representative request that combines weather\-related route planning with restrictions on restaurants along the way\. Existing services can satisfy individual subtasks, but the user must manually transfer intermediate results, compare alternatives, and verify the final combination\. Such requests are typically multi hop and compositional, spanning multiple systems, locations, and constraints\(Heet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib51)\)\. The manual coordination required is especially burdensome for older residents, who need more time to find the right function in feature\-rich interfaces and make more errors\(Yu and Chattopadhyay[2024](https://arxiv.org/html/2608.03018#bib.bib52)\)\.

![Refer to caption](https://arxiv.org/html/2608.03018v1/x1.png)Figure 1:Fragmented urban services vs UrbanAgent\. \(a\)Existing approaches each cover one fragment of a compositional request\. \(b\) UrbanAgent executes the same request as one coordinated, evidence\-aligned cross\-system workflow\.Existing research addresses separate components of the coordination problem\. Integrated urban platforms such as Hangzhou City Brain aggregate municipal data and services into a unified operating layer for traffic, public safety, and environmental management\(Zhanget al\.[2019](https://arxiv.org/html/2608.03018#bib.bib61)\)\. However each deployment remains bounded by a single domain or operating organization and does not act on an individual resident’s cross\-system request\. Urban foundation models such as UrbanGPT and CityGPT support spatiotemporal prediction and spatial reasoning, but do not interact with real\-time service systems\(Liet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib55); Fenget al\.[2024](https://arxiv.org/html/2608.03018#bib.bib56)\)\. Agent systems go a step further and take actions on real applications and services, yet each agent is confined either to a single application’s interface or to a single predefined task\. GUI agents such as AppAgent and Mobile\-Agent focus on interactions within a single application\(Zhanget al\.[2025](https://arxiv.org/html/2608.03018#bib.bib53); Wanget al\.[2024](https://arxiv.org/html/2608.03018#bib.bib54)\), while urban vertical agents such as UrbanKGent and LLMLight execute specific domain tasks, including knowledge graph construction and traffic signal control\(Ning and Liu[2024](https://arxiv.org/html/2608.03018#bib.bib57); Laiet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib58)\)\. Existing approaches integrate urban data, reason about cities, or act within a single application or domain, but do not coordinate heterogeneous services around a complete user request\.

Tool\-augmented LLM agents can coordinate independent services through standardized tool interfaces\. They select and invoke tools, then pass intermediate results between them\. Prior work has applied such agents to web navigation and software engineering\(Yaoet al\.[2023](https://arxiv.org/html/2608.03018#bib.bib5); Qinet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib42); Zhouet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib10); Xieet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib11); Jimenezet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib44)\)\. These advances make general tool agents a promising basis for executing cross\-system urban tasks\. However, direct transfer is not feasible\. Urban requests are usually expressed as high\-level goals rather than complete execution paradigms\. Fulfilling them may require real\-time environmental conditions, spatial relations, transportation options, and local\-service information from providers with different geographic coverage\. The workflow also emerges during execution because later calls may depend on places or conditions returned earlier\. Although none of these challenges is unique in isolation, their combination makes urban workflows particularly fail\. When a user asks for the “nearest” service without providing a location, an agent may silently assume one\. Then it can retrieve a valid place, obtain current weather, and compute an accurate route for the wrong location\. Each tool call may appear successful, but their combination produces a plausible but unusable recommendation\. Therefore, reliable execution requires clarification before action, consistency checks, ordered tool use, and constraint verification\.

To bridge this gap, we propose UrbanAgent, a tool\-augmented agent framework for open urban tasks\. It owns four components including cognition, a reasoning–execution core, tool invocation, and synthesis\. They clarify missing critical task inputs before taking action, select each operation based on previous observations, rely on call consumption of upstream values, recover from empty or geo\-invalid returns, and align the final response with collected evidence and task constraints\. Every action and statement is grounded in tool observations rather than the model’s prior knowledge\.

Furthermore, we find no benchmark evaluates agents on coordinating heterogeneous urban services across domains\. General tool\-use benchmarks test function calls in non\-urban settings\(Patilet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib12); Qinet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib42); Liuet al\.[2023](https://arxiv.org/html/2608.03018#bib.bib43)\)\. Urban benchmarks test city knowledge, reasoning, or simulation, not live service execution\(Fenget al\.[2025](https://arxiv.org/html/2608.03018#bib.bib16); Zhouet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib17)\)\. Agentic local life search stays within a single domain category\(Heet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib51)\)\. To bridge this gap, we construct UrbanEval\. The requests range from ambiguous tasks that require clarification to constrained tasks with ordered dependencies\. Because a plausible answer can conceal an incomplete or invalid execution trace, UrbanEval assesses both task results and execution quality, including tool coverage, dependency order, and evidence grounding\.

![Refer to caption](https://arxiv.org/html/2608.03018v1/x2.png)Figure 2:Framework of UrbanAgent\. \(1\) Cognition maps a query to a structured task𝒰\\mathcal\{U\}and requests clarification when a key task parameter is missing\. \(2\) The reasoning–execution core selects the next action from prior observations, and \(3\) tool invocation executes that action through the unified registry𝒯\\mathcal\{T\}; the two alternate until the agent commits a preliminary answerℛ^\\hat\{\\mathcal\{R\}\}\. \(4\) Synthesis aligns the draft with multi\-source observations, verifies task constraints, and commits the final answerℛ\\mathcal\{R\}\.To address these gaps, we make the following contributions:

- •We propose UrbanAgent to bridge the gap between high\-level user requests and fragmented urban services\. It clarifies missing parameters, preserves cross\-service dependencies, and grounds the final response in validated tool observations\.
- •We introduce UrbanEval to make cross\-system urban execution evaluable\. It pairs 250 requests with trace\-aware criteria that verify missing handling, required capability coverage, dependency preservation, and evidence support\. Therefore, it distinguishes between truly completed workflows and plausible answers produced through incomplete execution\.
- •We show that UrbanAgent improves cross\-system task completion over general orchestration baselines under matched models and tools\. With GPT\-5\-mini, UrbanAgent completes 64\.0% of executable tasks vs 53\.5% for the strongest baseline, and it achieves the highest TSR across all tested base models\.

## UrbanAgent

### Problem Formulation and Overview

Let𝒬\\mathcal\{Q\}denote a natural\-language query and𝒯=\{τ1,…,τm\}\\mathcal\{T\}=\\\{\\tau\_\{1\},\\ldots,\\tau\_\{m\}\\\}a registry of urban tools\. Each tool records its name, description, source server, and input JSON Schema\. UrbanAgent accesses the external urban state𝒲t\\mathcal\{W\}\_\{t\}through tool calls: invokingτk\\tau\_\{k\}with argumentsptp\_\{t\}yields an observationoto\_\{t\}\. The agent combines𝒬\\mathcal\{Q\}with these observations to produceℛ=Φ​\(𝒬,𝒲t;𝒯\)\\mathcal\{R\}=\\Phi\(\\mathcal\{Q\},\\mathcal\{W\}\_\{t\};\\mathcal\{T\}\)\(Yaoet al\.[2023](https://arxiv.org/html/2608.03018#bib.bib5); Qinet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib42); Model Context Protocol[2025](https://arxiv.org/html/2608.03018#bib.bib15)\)\.

UrbanAgent addresses four recurring failures of cross\-system urban tasks through the components in Figure[2](https://arxiv.org/html/2608.03018#Sx1.F2)\. Cognition detects missing inputs \(e\.g\., location or time\) and clarifies before acting\. The reasoning–execution core selects actions incrementally so downstream calls consume upstream values\. Tool invocation validates service returns and recovers from failed or geographically inconsistent calls\. And synthesis checks the draft against task constraints and collected evidence\. We evaluate this integrated design rather than attributing gains to any single component\.

### Cognition and Clarification

Cognition maps the query to a structured task,fcogπ:𝒬↦𝒰f\_\{\\text\{cog\}\}^\{\\pi\}:\\mathcal\{Q\}\\mapsto\\mathcal\{U\}, where𝒰=⟨ℐ,ℰ,𝒞hard,𝒞soft,ℳ⟩\\mathcal\{U\}=\\langle\\mathcal\{I\},\\mathcal\{E\},\\mathcal\{C\}\_\{\\text\{hard\}\},\\mathcal\{C\}\_\{\\text\{soft\}\},\\mathcal\{M\}\\rangle\. The intentℐ\\mathcal\{I\}specifies the task objective, andℰ\\mathcal\{E\}records the entities and parameters required for subsequent tool calls\.𝒞hard\\mathcal\{C\}\_\{\\text\{hard\}\}encodes mandatory conditions and𝒞soft\\mathcal\{C\}\_\{\\text\{soft\}\}ranking preferences\.

The setℳ\\mathcal\{M\}collects missing user\-specific inputs that cannot be inferred without changing the task\. Ifℳ≠∅\\mathcal\{M\}\\neq\\varnothing, UrbanAgent requests these inputs and terminates before any tool call, rather than inferring a location, destination, or other task\-defining value\(Liet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib2); Qinet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib42)\)\.

### Reasoning–Execution Core

The reasoning–execution core implements a reason–act–observe policy over the structured task𝒰\\mathcal\{U\}and observation history𝒪<t\\mathcal\{O\}\_\{<t\}\. It selects actions incrementally rather than constructing a complete tool sequence in advance\. At turntt,π\\piselects either a toolτk\\tau\_\{k\}with argumentsptp\_\{t\}or the terminal actionCommit⁡\(ℛ^\)\\operatorname\{Commit\}\(\\hat\{\\mathcal\{R\}\}\):

at\\displaystyle a\_\{t\}=π​\(𝒰,𝒪<t∣Schema⁡\(𝒯\)\),\\displaystyle=\\pi\\\!\\left\(\\mathcal\{U\},\\mathcal\{O\}\_\{<t\}\\mid\\operatorname\{Schema\}\(\\mathcal\{T\}\)\\right\),\(1\)at\\displaystyle a\_\{t\}∈\{\(τk,pt\),Commit⁡\(ℛ^\)\}\.\\displaystyle\\in\\\{\(\\tau\_\{k\},p\_\{t\}\),\\operatorname\{Commit\}\(\\hat\{\\mathcal\{R\}\}\)\\\}\.Forat=\(τk,pt\)a\_\{t\}=\(\\tau\_\{k\},p\_\{t\}\), the runtime executes the call and appends the result,𝒪≤t=𝒪<t∪\{ot\}\\mathcal\{O\}\_\{\\leq t\}=\\mathcal\{O\}\_\{<t\}\\cup\\\{o\_\{t\}\\\}; selecting the next action from this history lets a downstream call consume upstream values\. Execution ends whenat=Commit⁡\(ℛ^\)a\_\{t\}=\\operatorname\{Commit\}\(\\hat\{\\mathcal\{R\}\}\)\.

![Refer to caption](https://arxiv.org/html/2608.03018v1/x3.png)Figure 3:Overview of UrbanEval\. \(a\) The 250 queries span five task categories and three difficulty levels, and every category is scored by result\-layer metrics, and the Level\-2 category additionally by process\-layer metrics\. \(b\) Representative query examples and tool workflows for the five categories\.
### Tool Invocation and Grounding

At each turn, the function\-calling interface exposes the registered tool names, descriptions, and input schemas toπ\\pi\. These schemas constrain the action space and the shape ofptp\_\{t\}but cannot ensure semantic validity\. The runtime converts unknown tool names, malformed arguments, or backend exceptions into error observations, from whichπ\\pirevises the arguments or switches tools on the next turn\(Model Context Protocol[2025](https://arxiv.org/html/2608.03018#bib.bib15); Patilet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib12)\)\. Each observationoto\_\{t\}records the tool name, source server, parameters, raw return, status, and latency\. The runtime caches a call signature only after it returns usable data and rejects an exact repeat, while still allowing failed calls to be retried\. After eight consecutive failures on the same tool, it advises switching tools or reporting the subtask as unavailable\.

Urban services may also return syntactically valid but unusable results\. A geographic\-consistency check rejects a geocode whose city or province conflicts with the requested region, and an empty\-result check rejects a return without usable content\. Both set the observation status to failed and report the failure toπ\\pi\. Policy instructions further encode provider coverage and call prerequisites, soπ\\piavoids mainland\-only geocoders for overseas locations and resolves POI coordinates before route planning\. All observations remain in𝒪\\mathcal\{O\}, but only valid returns support factual claims\.

### Evidence\-Aligned Synthesis

The reasoning–execution core returns a draftℛ^\\hat\{\\mathcal\{R\}\}, which synthesis refines conditioned on the structured task and observations:

ℛ=fsynπ​\(𝒰,𝒪,ℛ^\)\.\\mathcal\{R\}=f\_\{\\text\{syn\}\}^\{\\pi\}\(\\mathcal\{U\},\\mathcal\{O\},\\hat\{\\mathcal\{R\}\}\)\.\(2\)A draft may narrate the execution process, defer requested actions to a follow\-up turn, or overlook the most specific observation\. Synthesis corrects these through the three operations in Figure[2](https://arxiv.org/html/2608.03018#Sx1.F2):*Clean*removes process descriptions and redundant follow\-up offers;*Align*selects the most relevant evidence in𝒪\\mathcal\{O\}and matches it to the requested format;*Verify*checks the conclusion against the constraints in𝒰\\mathcal\{U\}and admits no fact absent from𝒪\\mathcal\{O\}andℛ^\\hat\{\\mathcal\{R\}\}\. When observations lack the needed detail,ℛ\\mathcal\{R\}states the limitation rather than inferring a value\. If synthesis fails or returns empty, the runtime falls back toℛ^\\hat\{\\mathcal\{R\}\}\.

### Implementation Details

UrbanAgent requires no task\-specific training\. The evaluated configuration uses the ReAct core with evidence\-aligned rewrite synthesis, and remote MCP services and local computational tools share one registry interface\(Anthropic[2024](https://arxiv.org/html/2608.03018#bib.bib14); Model Context Protocol[2025](https://arxiv.org/html/2608.03018#bib.bib15); Ouyanget al\.[2025](https://arxiv.org/html/2608.03018#bib.bib18)\)\. The controlled experiments use GPT\-5\-mini at temperatureT=1\.0T=1\.0with a per\-query budget of 25 LLM calls or 600 seconds\. When at most two calls or 90 seconds remain, the runtime disables tool calls and requests a draft from the available observations\.

## UrbanEval Benchmark

LayerMetricDefinitionResultTSRTask success rate — a correct, complete, constraint\-satisfying answerMIRMissing\-information identification rate — guess\-free flagging of a missing task\-critical inputProcessTICTool invocation completeness — coverage of the required tool capabilitiesIOAInvocation order accuracy — dependency pairs called in the required orderVPARValid proactive augmentation rate — useful augmentation or verification beyond the minimumEfficiencyTokens per queryMean over all queriesTokens per successMean over successful tasks onlyTable 1:UrbanEval’s three\-layer metric mechanism\. TIC, IOA, and the efficiency measures are computed deterministically from recorded traces; TSR, MIR, and VPAR are scored by an LLM judge \(Qwen3\-Max\) under a shared rubric\. Process metrics are reported on Level 2, and VPAR only on runs having at least one tool call\.Existing tool\-use benchmarks are not designed around cross system urban workflows\(Patilet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib12); Qinet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib42); Liuet al\.[2023](https://arxiv.org/html/2608.03018#bib.bib43)\), while urban benchmarks focus on scene understanding or simulation rather than live service execution\(Fenget al\.[2025](https://arxiv.org/html/2608.03018#bib.bib16); Zhouet al\.[2025](https://arxiv.org/html/2608.03018#bib.bib17)\)\. UrbanEval instead tests high level requests that combine missing inputs, service dependencies, time\-varying observations, and evidence\-grounded constraints\.

Table 2:Main results on UrbanEval for tool\-augmented systems, grouped by base model\. GPT\-5\-mini defines the controlled main setting; the other base models evaluate generalization under the same tool set, budget, and protocol\.UrbanEval addresses this gap with 250 natural\-language requests executed against external services exposed through MCP servers across seven urban\-computing domains\(Zhenget al\.[2014](https://arxiv.org/html/2608.03018#bib.bib1)\)\. Its challenge structure varies the form of execution rather than merely increasing the number of tool calls\. As shown in Figure[3](https://arxiv.org/html/2608.03018#Sx2.F3), five query types cover factual retrieval, complex retrieval across independent services, sequential dependency, conditional or combinatorial decision\-making, and ambiguous or incomplete intent\. These types are organized into three difficulty levels\. Level 1 tests information acquisition and composition\. Level 2 tests dependency\-aware execution and constrained decision\-making\. Level 3 tests whether an agent identifies task\-critical omissions before acting\. Each type contains 50 queries, providing balanced coverage of retrieval, coordination, branching, and clarification failures\.

A fluent answer does not prove the task was carried out: an agent can skip a required tool, call tools in the wrong order, or state a fact, such as a travel time, that no tool ever returned\. UrbanEval therefore scores each request against a human\-written checklist of small, independently verifiable criteria instead of one gold answer\. Each criterion states what must hold \(e\.g\., the returned hospital is currently open\), which tool capability and upstream result it depends on \(a place search must run before, and feed, the route query\), and any user input the agent must clarify first\. Evidence is matched by type\. Specifically, fixed facts such as a district’s official name are checked against fixed references\. Facts that change over time such as current weather are compared with the raw tool outputs from\. And computed values such as an arrival estimate are compared with the specific observations\. Therefore, the value is only recognized when the trajectory actually generates the input behind it\.

The schema drives a three\-layer metric mechanism \(Table[1](https://arxiv.org/html/2608.03018#Sx3.T1)\) that separates three questions answer\-only evaluation conflates: was the request solved, was the execution valid, and what did success cost? Process metrics are reported on Level 2, where explicit tool dependencies are defined, and VPAR only on runs that issue at least one tool call\. Together, these layers distinguish plausible answers from grounded task completion, valid orchestration, and execution cost\.

## Experiments

Our experiments test one mechanism: clarifying missing inputs before acting, then executing under explicit tool dependencies and task constraints\. We ask whether this structure completes the complex urban tasks that baseline agents leave unfinished, under matched model and tool access\. Three questions organize the study\.Q1 \(Effectiveness and cost\):Does UrbanAgent improve end\-to\-end task success, and at what token cost?Q2 \(Process\):Which execution behaviors, tool coverage, dependency order, augmentation, and clarification, accompany its gains?Q3 \(Robustness\):Does the advantage persist across closed and open base models?

### Experimental Setup

Baselines\.We evaluate four tool\-augmented baselines that represent established orchestration strategies: Native tool\-calling, which delegates action selection to the model’s function\-calling interface; ReAct, which interleaves reasoning and tool use\(Yaoet al\.[2023](https://arxiv.org/html/2608.03018#bib.bib5)\); Plan\-and\-Execute, which plans before execution\(Wanget al\.[2023a](https://arxiv.org/html/2608.03018#bib.bib62)\); and AutoGen, which pairs an assistant with a tool executor\(Wuet al\.[2024](https://arxiv.org/html/2608.03018#bib.bib40)\)\. To compare orchestration rather than access, every tool\-augmented system within a base\-model block receives the same queries, MCP tools, tool descriptions, JSON schemas, temperature, and budget \(25 LLM calls or 600 seconds per query\)\. Because clarification is one of the behaviors we measure, every system may ask the user for a missing parameter instead of assuming one\. When a tool call returns an error, the agent receives it and may retry\. So the error alone does not count as a task failure\. We also evaluate ten Single LLMs without tool access or live observations\.

Evaluation\.We follow the UrbanEval protocol\. TIC, IOA, and the cost derive directly from the execution traces; the remaining metrics require judging the content of an answer or trajectory rather than applying a rule\. Qwen3\-Max at temperature0scores them with system identities masked to limit bias\(Wanget al\.[2023b](https://arxiv.org/html/2608.03018#bib.bib60); Zhenget al\.[2023](https://arxiv.org/html/2608.03018#bib.bib59)\), grading TSR and MIR against fixed per\-query criteria and VPAR against a fixed trajectory rubric\. To keep the benchmark neutral, we froze the UrbanEval queries and schema before finalizing UrbanAgent, and annotated the schema independently of its development\. All systems query the same MCP services, and every raw return is logged\. Because these responses vary over time, the schema grades time\-varying facts against each run’s own returns rather than a fixed key\. So no system is credited or penalized for the particular values a service returned during its run\.

### Main Results

Controlled comparison \(Q1\)\.Over all 250 queries UrbanAgent attains an overall TSR of71%, against 61% for the strongest baselines, Native tool\-calling and AutoGen \(Table[2](https://arxiv.org/html/2608.03018#Sx3.T2)\)\. The 50 ambiguous or incomplete intent queries test clarification rather than execution, so we isolate the 200 executable queries separately: UrbanAgent completes 128 \(64\.0%\) against 107 \(53\.5%\) for Native tool\-calling, a 10\.5\-point gain at matched model and tool access\. The gain incurs a higher token cost\. 118\.3k per successful task is about three times the 37\.3–42\.8k of Native tool\-calling, ReAct, and AutoGen\.\(Figure[4](https://arxiv.org/html/2608.03018#Sx4.F4)b\)\. The higher token cost buys reliability on the hard requests\. On the 200 executable queries UrbanAgent solves 21 more than Native tool\-calling\.

![Refer to caption](https://arxiv.org/html/2608.03018v1/x4.png)Figure 4:Main results and analysis on GPT\-5\-mini\. \(a\) TSR by query type: factual retrieval \(FR\), complex retrieval across independent services \(CR\), sequential dependency \(SD\), conditional or combinatorial decision\-making \(CC\), and ambiguous or incomplete intent \(AI\)\. \(b\) TSR vs tokens per successful task\. \(c\) Failure counts by class including factual error, requirement mismatch, incompleteness, and other over the 200 executable queries\.The margin concentrates in two categories \(Figure[4](https://arxiv.org/html/2608.03018#Sx4.F4)a\): UrbanAgent exceeds the strongest baseline by 16 points on factual retrieval \(0\.68 vs\. 0\.52\) and 12 on conditional or combinatorial decision\-making \(0\.68 vs\. 0\.56\)\. It narrows on sequential dependency, where the leading systems fall between 0\.68 and 0\.72, and reverses on complex retrieval across independent services, where AutoGen edges UrbanAgent 0\.50 to 0\.48\.

A stronger base model does not bridge the gap\.The ten Single LLMs \(Table[3](https://arxiv.org/html/2608.03018#Sx4.T3)\) include the strongest available models\. But all fall within a narrow 0\.22–0\.26 overall TSR\. Their successes are almost entirely clarification: 70–91% arise on the ambiguous or incomplete intent category, while no executable category exceeds 0\.22 and the best scores on the conditional or combinatorial decision\-making and sequential dependency categories reach only 0\.12\. Without tools, even a frontier model flags a missing input but cannot satisfy a request that depends on live urban observations\. Tool use closes this gap: on GPT\-5\-mini, TSR rises from 0\.22 without tools to 0\.71 with UrbanAgent, a 49\-point difference\. Scale alone does not close this gap\. The newest models stay within the same narrow band as the older ones\. The bottleneck is access to live data, not model capacity\.

Table 3:Results for ten single LLMs prompted directly, without tools or live observations\. The process metrics therefore do not apply\.
### Analysis

Locating the advantage \(Q2\)\.By separating results from process, UrbanEval also localizes the 10\.5\-point gap\. On GPT\-5\-mini the tool\-augmented systems already coincide on whether the right tools are called in the right order: TIC is 0\.89 for UrbanAgent against 0\.86 for the strongest baseline, and IOA 1\.00 against 0\.99\. Clarification is equally saturated: MIR ranges from 0\.90 to 0\.98, and every system identifies the missing input and requests it\. UrbanAgent departs only in never acting on an assumed value, returning no assumption across the 50 ambiguous or incomplete intent queries against 5–8 for the baselines\. Coverage, ordering, and clarification thus do not explain the gap\. There are two behaviors can explain including augmentation beyond the minimal toolchain and completion of constrained tasks\.

Proactive augmentation\.Among the process metrics, only augmentation separates the systems: VPAR reaches 0\.96 for UrbanAgent against 0\.84 for the strongest baseline\. Valid augmentation takes four forms: precision refinement \(narrowing a coarse result to an exact one\), expanded retrieval \(evidence gathered beyond the minimal toolchain\), error recovery \(a failed call retried with new arguments or another tool\), and cross\-validation \(a fact confirmed from two independent calls\)\. The judge credits at least one on 95 of the 99 Level\-2 runs in which UrbanAgent issued a tool call; the remaining run issued none, leaving nothing to augment\. Refinement appears in 85 trajectories and expanded retrieval in 54, error recovery in 8 and cross\-validation in 10; the categories overlap, so a trajectory may bear several\. UrbanAgent thus supplements and cross\-checks steps that a minimal, correctly ordered toolchain omits\. Refinement is the most common form\. A coarse match becomes the exact value that the request needs\.

Task completeness\.The second locus is completion, which the failure classes expose\. Each failed run receives one dominant class: a factual error \(a claim its observations do not support\), a requirement mismatch \(a result that violates a stated constraint\), task incompleteness \(a required step left undone\), or other\. UrbanAgent fails 72 of the 200 executable queries, against 93 for Native tool\-calling, 104 for ReAct, 127 for Plan\-and\-Execute, and 96 for AutoGen, and the classes diverge most on incompleteness \(Figure[4](https://arxiv.org/html/2608.03018#Sx4.F4)c\): UrbanAgent leaves 20 tasks incomplete against 31–45 for the baselines\. One factual retrieval query illustrates the pattern: it requests hotels near Kunming South Railway Station with availability from 12–15 July 2026\. All four baselines retrieve hotels but none verifies availability for those dates\. Native tool\-calling and AutoGen defer the check, ReAct returns no verified hotel, and Plan\-and\-Execute spends 25 LLM calls on candidates and routes yet never queries rooms\. UrbanAgent checks several nearby hotels for the dates and reports that none has rooms, completing the query the baselines abandon\. This case is typical of constrained retrieval\. UrbanAgent tests each candidate before it answers\.

Robustness across base models \(Q3\)\.UrbanAgent ranks first in TSR across all four base models and first in MIR within each tool\-augmented block \(Table[2](https://arxiv.org/html/2608.03018#Sx3.T2); a few tool\-free Single LLMs reach a higher MIR of 1\.00, Table[3](https://arxiv.org/html/2608.03018#Sx4.T3)\)\. The rank is stable but the margin is not: \+10 points on GPT\-5\-mini, \+14 on Gemini\-2\.5\-flash, and \+9 on Qwen3\-235B\-A22B, yet \+2 on DeepSeek\-V4\-flash\. The margin tracks how strong the baselines already are\. On DeepSeek\-V4\-flash the best baseline already reaches 0\.50 TSR\.

Limitations\.UrbanAgent is more token\-intensive, and its gains vary across tasks and backbones; evaluation with time\-varying services and an LLM judge does not establish generalization to unseen cities or domains\.

## Conclusion

We propose UrbanAgent, a tool\-augmented agent that turns a natural\-language urban request into an executable cross\-system workflow\. Meanwhile, we built UrbanEval, a 250\-query benchmark that scores not only whether a request succeeds but how it is executed\. Under GPT\-5\-mini, UrbanAgent reaches 71% TSR, 10 points above the strongest baseline, and completes 64\.0% of executable requests against 53\.5%\. Future work will isolate each component’s contribution and add user studies and in\-the\-wild deployment\.

## References

- Introducing the model context protocol\.Note:Anthropic NewsExternal Links:[Link](https://www.anthropic.com/news/model-context-protocol)Cited by:[Implementation Details](https://arxiv.org/html/2608.03018#Sx2.SSx6.p1.1)\.
- J\. Feng, Y\. Du, T\. Liu, S\. Guo, Y\. Lin, and Y\. Li \(2024\)CityGPT: empowering urban spatial cognition of large language models\.External Links:2406\.13948Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- J\. Feng, J\. Zhang, T\. Liu, X\. Zhang, T\. Ouyang, J\. Yan, Y\. Du, S\. Guo, and Y\. Li \(2025\)Citybench: evaluating the capabilities of large language models for urban tasks\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5413–5424\.Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p5.1),[UrbanEval Benchmark](https://arxiv.org/html/2608.03018#Sx3.p1.1)\.
- H\. He, C\. Yue, C\. Dong, M\. Tian, Z\. Liu, J\. Chai, X\. Wang, Y\. Zhang, Q\. Liao, G\. Yin,et al\.\(2025\)LocalSearchBench: benchmarking agentic search in real\-world local life services\.External Links:2512\.07436Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p1.1),[Introduction](https://arxiv.org/html/2608.03018#Sx1.p5.1)\.
- C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world github issues?\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p3.1)\.
- S\. Lai, Z\. Xu, W\. Zhang, H\. Liu, and H\. Xiong \(2024\)LLMLight: large language models as traffic signal control agents\.External Links:2312\.16044Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- Z\. Li, L\. Xia, X\. Ren, J\. Tang, T\. Chen, Y\. Xu, and C\. Huang \(2025\)Urban computing in the era of large language models\.ACM Transactions on Intelligent Systems and Technology16\(6\),pp\. 1–43\.Cited by:[Cognition and Clarification](https://arxiv.org/html/2608.03018#Sx2.SSx2.p2.2)\.
- Z\. Li, L\. Xia, J\. Tang, Y\. Xu, L\. Shi, L\. Xia, D\. Yin, and C\. Huang \(2024\)UrbanGPT: spatio\-temporal large language models\.External Links:2403\.00813Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- X\. Liu, H\. Yu, H\. Zhang, Y\. Xu, X\. Lei, H\. Lai, Y\. Gu, H\. Ding, K\. Men, K\. Yang,et al\.\(2023\)Agentbench: evaluating llms as agents\.External Links:2308\.03688Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p5.1),[UrbanEval Benchmark](https://arxiv.org/html/2608.03018#Sx3.p1.1)\.
- Model Context Protocol \(2025\)Model context protocol specification\.Note:Official specificationVersion dated 2025\-03\-26External Links:[Link](https://modelcontextprotocol.io/specification/2025-03-26)Cited by:[Problem Formulation and Overview](https://arxiv.org/html/2608.03018#Sx2.SSx1.p1.8),[Tool Invocation and Grounding](https://arxiv.org/html/2608.03018#Sx2.SSx4.p1.4),[Implementation Details](https://arxiv.org/html/2608.03018#Sx2.SSx6.p1.1)\.
- Y\. Ning and H\. Liu \(2024\)UrbanKGent: a unified large language model agent framework for urban knowledge graph construction\.External Links:2402\.06861Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- C\. Ouyang, L\. Yue, S\. Di, L\. Zheng, L\. Yue, S\. Pan, J\. Yin, and M\. Zhang \(2025\)Code2MCP: transforming code repositories into mcp services\.External Links:2509\.05941Cited by:[Implementation Details](https://arxiv.org/html/2608.03018#Sx2.SSx6.p1.1)\.
- S\. G\. Patil, H\. Mao, F\. Yan, C\. C\. Ji, V\. Suresh, I\. Stoica, and J\. E\. Gonzalez \(2025\)The berkeley function calling leaderboard \(bfcl\): from tool use to agentic evaluation of large language models\.InForty\-second International Conference on Machine Learning,Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p5.1),[Tool Invocation and Grounding](https://arxiv.org/html/2608.03018#Sx2.SSx4.p1.4),[UrbanEval Benchmark](https://arxiv.org/html/2608.03018#Sx3.p1.1)\.
- Y\. Qin, S\. Liang, Y\. Ye, K\. Zhu, L\. Yan, Y\. Lu, Y\. Lin, X\. Cong, X\. Tang, B\. Qian,et al\.\(2024\)ToolLLM: facilitating large language models to master 16000\+ real\-world apis\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p3.1),[Introduction](https://arxiv.org/html/2608.03018#Sx1.p5.1),[Problem Formulation and Overview](https://arxiv.org/html/2608.03018#Sx2.SSx1.p1.8),[Cognition and Clarification](https://arxiv.org/html/2608.03018#Sx2.SSx2.p2.2),[UrbanEval Benchmark](https://arxiv.org/html/2608.03018#Sx3.p1.1)\.
- J\. Wang, H\. Xu, J\. Ye, M\. Yan, W\. Shen, J\. Zhang, F\. Huang, and J\. Sang \(2024\)Mobile\-agent: autonomous multi\-modal mobile device agent with visual perception\.External Links:2401\.16158Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- L\. Wang, W\. Xu, Y\. Lan, Z\. Hu, Y\. Lan, R\. K\. Lee, and E\. Lim \(2023a\)Plan\-and\-solve prompting: improving zero\-shot chain\-of\-thought reasoning by large language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(ACL\),pp\. 2609–2634\.Cited by:[Experimental Setup](https://arxiv.org/html/2608.03018#Sx4.SSx1.p1.1)\.
- P\. Wang, L\. Li, L\. Chen, Z\. Cai, D\. Zhu, B\. Lin, Y\. Cao, Q\. Liu, T\. Liu, and Z\. Sui \(2023b\)Large language models are not fair evaluators\.External Links:2305\.17926Cited by:[Experimental Setup](https://arxiv.org/html/2608.03018#Sx4.SSx1.p2.1)\.
- Q\. Wu, G\. Bansal, J\. Zhang, Y\. Wu, B\. Li, E\. Zhu, L\. Jiang, X\. Zhang, S\. Zhang, J\. Liu,et al\.\(2024\)Autogen: enabling next\-gen llm applications via multi\-agent conversations\.InFirst conference on language modeling,Cited by:[Experimental Setup](https://arxiv.org/html/2608.03018#Sx4.SSx1.p1.1)\.
- T\. Xie, D\. Zhang, J\. Chen, X\. Li, S\. Zhao, R\. Cao, T\. J\. Hua, Z\. Cheng, D\. Shin, F\. Lei,et al\.\(2024\)OSWorld: benchmarking multimodal agents for open\-ended tasks in real computer environments\.Advances in Neural Information Processing Systems37,pp\. 52040–52094\.Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p3.1)\.
- S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. R\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p3.1),[Problem Formulation and Overview](https://arxiv.org/html/2608.03018#Sx2.SSx1.p1.8),[Experimental Setup](https://arxiv.org/html/2608.03018#Sx4.SSx1.p1.1)\.
- J\. E\. Yu and D\. Chattopadhyay \(2024\)Reducing the search space on demand helps older adults find mobile ui features quickly, on par with younger adults\.InProceedings of the CHI Conference on Human Factors in Computing Systems,pp\. 1–22\.External Links:[Document](https://dx.doi.org/10.1145/3613904.3642796)Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p1.1)\.
- C\. Zhang, Z\. Yang, J\. Liu, Y\. Li, Y\. Han, X\. Chen, Z\. Huang, B\. Fu, and G\. Yu \(2025\)AppAgent: multimodal agents as smartphone users\.InProceedings of the CHI Conference on Human Factors in Computing Systems,pp\. 1–20\.External Links:[Document](https://dx.doi.org/10.1145/3706598.3713600)Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- J\. Zhang, X\. Hua, J\. Huang, X\. Shen, J\. Chen, Q\. Zhou, Z\. Fu, and Y\. Zhao \(2019\)City brain: practice of large\-scale artificial intelligence in the real world\.IET Smart Cities1\(1\),pp\. 28–37\.Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. P\. Xing, H\. Zhang, J\. E\. Gonzalez, and I\. Stoica \(2023\)Judging LLM\-as\-a\-judge with MT\-bench and chatbot arena\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Cited by:[Experimental Setup](https://arxiv.org/html/2608.03018#Sx4.SSx1.p2.1)\.
- Y\. Zheng, L\. Capra, O\. Wolfson, and H\. Yang \(2014\)Urban computing: concepts, methodologies, and applications\.ACM Transactions on Intelligent Systems and Technology \(TIST\)5\(3\),pp\. 1–55\.Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p1.1),[UrbanEval Benchmark](https://arxiv.org/html/2608.03018#Sx3.p2.1)\.
- B\. Zhou, H\. Yang, D\. Chen, J\. Ye, T\. Bai, J\. Yu, S\. Zhang, D\. Lin, C\. He, and W\. Li \(2025\)URBench: a comprehensive benchmark for evaluating large multimodal models in multi\-view urban scenarios\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 10707–10715\.Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p5.1),[UrbanEval Benchmark](https://arxiv.org/html/2608.03018#Sx3.p1.1)\.
- S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried,et al\.\(2024\)WebArena: a realistic web environment for building autonomous agents\.InInternational Conference on Learning Representations,Cited by:[Introduction](https://arxiv.org/html/2608.03018#Sx1.p3.1)\.