ChatPlanner: A Large Language Model Framework for Personalized Public Transit Routing

# ChatPlanner: A Large Language Model Framework for Personalized Public Transit Routing
Source: [https://arxiv.org/html/2606.15315](https://arxiv.org/html/2606.15315)
Tingting Yang 1School of Engineering and Materials Science, Queen Mary University of London Mile End Road, London, E1 4NS, United Kingdom t\.yang@qmul\.ac\.uk Chenhao Xue11footnotemark:1 2Department of Engineering Science, University of Oxford Parks Road, Oxford, OX1 3PJ, United Kingdom chenhao\.xue@eng\.ox\.ac\.uk &Jun Chen 1School of Engineering and Materials Science, Queen Mary University of London Mile End Road, London, E1 4NS, United Kingdom jun\.chen@qmul\.ac\.uk Corresponding author

###### Abstract

Personalized public transit routing in public transit systems remains challenging due to the difficulty of capturing and integrating diverse user preferences into routing algorithms\. This paper presents ChatPlanner, a novel framework that leverages Large Language Models \(LLMs\) to enable preference aware public transit routing\. Our approach employs fine\-tuned LLMs with Retrieval\-Augmented Generation \(RAG\) to extract routing parameters and interpret nuanced user preferences from natural language queries, subsequently integrating these preferences into the objective function of a public transit routing algorithm\. This study designs preference aware datasets incorporating eight personas and five contexts to establish scoring standards for both fine\-tuning and RAG\. This work conducted three experiments to validate the solutions’ feasibility, extraction of routing information and preferences, and solution set quality and completeness\. Results demonstrate that ChatPlanner generates feasible solutions reliably\. Fine\-tuning enforces the required output structure and learns general preference patterns, while RAG provides query\-specific context to resolve imprecise or conversational expressions and calibrate continuous scores\. The combination of both achieves the highest accuracy in routing information extraction and user preference interpretation\. Results based on selected case studies show that by capturing user preferences, ChatPlanner identifies valuable solutions across different dimensions that existing route planners overlook, generating more valuable route alternatives\. This research establishes a new paradigm for integrating natural language understanding into transportation optimization\.

*K*eywordsPublic transit routing⋅\\cdotUser preferences⋅\\cdotRAPTOR algorithm⋅\\cdotPersonalized transportation⋅\\cdotMulti\-criteria optimization⋅\\cdotUrban mobility⋅\\cdotLarge language models⋅\\cdotPublic transit routing

## 1Introduction

Public transit, as defined by the Federal Transit Administration, is transportation by bus, rail, or other conveyance, either publicly or privately owned, that provides general or special service to the public on a regular and continuing basis\(Daganzo and Ouyang,[2019](https://arxiv.org/html/2606.15315#bib.bib13)\)\. These systems are characterized by large\-capacity vehicles operating on predetermined routes and timetables\. They serve as the foundation of urban mobility in major cities worldwide, particularly in areas with high population density\(Hörcher and Tirachini,[2021](https://arxiv.org/html/2606.15315#bib.bib108)\)\. Public transit routing identifies feasible passenger paths across existing transit networks\. Public transit routing algorithms provide decision support for route planner apps, for example, Google Maps, Apple Maps, and Citymapper\. However, most commercial journey planners still primarily optimize for travel time and the number of transfers as fixed objectives\. These planners do not interpret users’ preferences directly and translate those into extra objectives dynamically\.\(Apple Inc\.,[2025](https://arxiv.org/html/2606.15315#bib.bib98); Google,[2025](https://arxiv.org/html/2606.15315#bib.bib100); Citymapper,[2025](https://arxiv.org/html/2606.15315#bib.bib99)\)\.

Traveler preferences are heterogeneous across time sensitivity, trip purpose, accessibility, affordability, and perceived safety\. Commuters typically prioritize time efficiency\(Mackieet al\.,[2001](https://arxiv.org/html/2606.15315#bib.bib45)\), leisure travelers often trade time for itinerary quality\(Zhouet al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib46)\), and riders with limited mobility require step\-free access\(Ceccatoet al\.,[2020](https://arxiv.org/html/2606.15315#bib.bib32)\)\. Low\-income travelers, non\-drivers, children, and elderly adults frequently rely on public transit for affordability, safety, and convenience\(Fan,[2004](https://arxiv.org/html/2606.15315#bib.bib103)\), while safety concerns, particularly among women, children, and students, also shape route choice\(The Mayor of London,[2023](https://arxiv.org/html/2606.15315#bib.bib101)\)\. Because public transit is essential to daily life for many groups, system performance has tangible consequences for well\-being\(Heapset al\.,[2021](https://arxiv.org/html/2606.15315#bib.bib48); Molneret al\.,[2023](https://arxiv.org/html/2606.15315#bib.bib47)\)\. Incorporating user preferences into routing algorithms is therefore critical to serving diverse needs and improving the overall transit experience\.

When specific preferences are captured through the user interfaces of those commercial journey planners, they are often used as post\-hoc filters\. That is, the solutions set and the process of finding solutions remain unchanged\. The routes are simply re\-ranked by preferences\. Public transit routing problems with multiple conflicting objectives seek to find Pareto optimal journeys for travelers\(Dellinget al\.,[2015](https://arxiv.org/html/2606.15315#bib.bib107); Fan and Mumford,[2010](https://arxiv.org/html/2606.15315#bib.bib106)\)\. However, considering many conflicting preferences as various objective functions in the public transit routing algorithm may cause high computational cost, and lead to information clusters as more than necessary solutions are presented to the end users\.

Apart from the limitations of fixed objectives, a more fundamental issue lies in how preferences are captured in the first place\. A structured preference interface with sliders or ranking widgets could let travelers rate criteria such as accessibility or safety directly\. However, research in public transit routing shows that it is difficult for users to accurately express their preferences for each criterion through seemingly more guaranteed inputs\(Wanget al\.,[2021](https://arxiv.org/html/2606.15315#bib.bib35); Huanget al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib37)\)\. Such an interface faces an inherent trade\-off: simplified but more guaranteed methods require less cognitive effort but are less expressive, while more expressive methods demand greater cognitive effort and reduce usability\(Pommeranzet al\.,[2012](https://arxiv.org/html/2606.15315#bib.bib44)\)\. However, even when the preference dimensions are low, asking users to assign numerical importance scores creates a cognitive mismatch\. Travelers naturally think in terms of situational context \(e\.g\., “I have a stroller and it is rush hour”\) rather than abstract ratings of preferences on a scale\(Kostricet al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib38)\)\.

Given the above circumstance, this study contemplates three research questions: \(i\) What constitutes an effective, systematic methodology for formulating a useful framework to identify travelers’ preferences? \(ii\) How does the proposed framework better understand traveler requests across different preferences? \(iii\) How does the proposed work perform in both theoretical and practical perspectives?

To address the above challenges, ChatPlanner, a Large Language Model \(LLM\) Framework for Personalized Public Transit Routing, is proposed\. Within this context,Chatcorresponds to the conversational interaction and natural language handling functionalities provided by LLM\.Plannerapplies a multi\-criterion round\-based public transit routing \(MC\-RAPTOR\) algorithm\(Dellinget al\.,[2015](https://arxiv.org/html/2606.15315#bib.bib107)\)\. Practically, ChatPlanner provides a flexible conversational interface that interprets user preferences and returns diverse, feasible route sets\. Methodologically, a fine\-tuned transit\-domain LLM translates natural\-language queries into journey requests and user preferences based on four extra criteria in addition to travel time and number of transfers: \(i\) accessibility level, \(ii\) crowding level, \(iii\) safety level, and \(iv\) sightseeing level\. The LLM estimates the importance of these criteria, allowing the framework to select the most relevant ones for the traveler\. The route planner incorporates these preferences from the traveler into the multi\-objective search and selects potentially different objective functions during routing\. These preference\-aware objectives thereby yield distinct Pareto fronts over candidate routes, combining personalized user preferences with the transit routing algorithm\.

The contribution of this framework is that, first, the state\-of\-the\-art LLM models’ performances in generating public transit routing solutions are investigated, providing insights into the feasibility, completeness, and correctness\. Second, this paper introduces ChatPlanner, a framework that integrates the LLM with a public transit routing algorithm with a preference\-aware dataset\. This work designs the datasets for both fine\-tuning and RAG with an input\-output structure and incorporates preference information through eight distinct personas and five contexts to establish preference scoring standards\. Third, this study evaluates the proposed RAG and fine\-tuned LLM under the proposed framework for extracting routing information and interpreting user preferences, compared to general purpose base LLM models\. Evaluation demonstrates that fine\-tuning is essential for accurate parsing of origin, destination, and requested arrival time, while the combination of fine\-tuning and RAG best enhances preference interpretation accuracy\. Especially, the RAG’s contextual examples ensure consistent and accurate preference scoring, aligning with the designed scoring standards\. Finally, this study conducted case studies to confirm that by capturing user preferences systematically, the framework generates more valuable route alternatives overlooked by existing route planners\.

The rest of this article is organized as follows\. Section 2 describes the related work on classic public transit routing algorithms, multi\-objective routing algorithms, personalized route recommendation systems, LLM in transportation, and LLM in public transit routing\. Section 3 provides a detailed presentation of the proposed ChatPlanner framework\. Section 4 presents the experiments conducted to test the parsing performance, solution feasibility and completeness, and the analysis of solution diversity\. It also discusses the major findings of this study and the academic and industrial implications\. Finally, the conclusion is drawn in Section 5\.

## 2Literature Review

A public transit system is generally referred to as public transport and mass transit\([ScienceDirect Topics,](https://arxiv.org/html/2606.15315#bib.bib9)\)\. When it comes to public transit, bus, railway, and airline transit network design play a key role in optimization\(Kepaptsoglou and Karlaftis,[2009](https://arxiv.org/html/2606.15315#bib.bib8); Ciprianiet al\.,[2012](https://arxiv.org/html/2606.15315#bib.bib7); Lianget al\.,[2025](https://arxiv.org/html/2606.15315#bib.bib6)\)\.Kepaptsoglou and Karlaftis \([2009](https://arxiv.org/html/2606.15315#bib.bib8)\)systematically reviews research on the Transit Route Network Design Problem, addressing the challenges of efficiently designing public transportation networks that enhance mobility while reducing congestion and pollution\.Lianget al\.\([2025](https://arxiv.org/html/2606.15315#bib.bib6)\)propose a novel multi\-objective evolutionary algorithm based on objective space decomposition to solve the transit network design and frequency\-setting problem\. The network design problem determines the stations that the bus, subway, or railway needs to go through at a specific frequency\. By contrast, the public transit routing aims to select the stations that travelers need to go through from the origin to the destination, given the scheduled timetable\.

Early approaches to public transit routing adapt classical shortest path algorithms, particularly Dijkstra’s algorithm, by modeling timetables as either time\-expanded or time\-dependent graphs\(Pyrgaet al\.,[2008](https://arxiv.org/html/2606.15315#bib.bib15); Disseret al\.,[2008](https://arxiv.org/html/2606.15315#bib.bib14)\)\.Dellinget al\.\([2015](https://arxiv.org/html/2606.15315#bib.bib107)\)introduce Round\-based Public Transit Optimized Router \(RAPTOR\), which is achieved by examining routes that have been marked and incrementally exploring connected paths in a round\-by\-round manner, identifying the earliest trip on that line that can be boarded\. Unlike previous Dijkstra\-based graph approaches, RAPTOR operates in rounds, processing each route at most once per round, making it simpler and faster\. MC\-RAPTOR is a multiple\-criteria Round\-based Public Transit Optimized Router, a multiple\-criteria extension of RAPTOR for computing Pareto\-optimal journeys in public transit networks that minimizes arrival time, number of transfers, and any extra criteria in a round\-based manner\. In each iteration, it determines the earliest arrival times for journeys withn−1n\-1transfers, wherenncorresponds to the current iteration\. The multiple objective values are saved in different labels\. Not all public transit routing problems are NP\-Hard, but they become NP\-Hard when multiple objectives need to be considered\.

Regarding the traveler’s preference,Jakobet al\.\([2014](https://arxiv.org/html/2606.15315#bib.bib5)\)presents an advanced journey planner that combines all types of urban mobility services while personalizing recommendations based on each user’s past travel choices and preferences\.Ceder and Jiang \([2019](https://arxiv.org/html/2606.15315#bib.bib4)\)considers multiple traveler preferences, including time, cost, and convenience, and performs multiple searches across different criteria to generate alternative routes, with each route optimized for different attribute combinations to provide personalized options\.Zografoset al\.\([2010](https://arxiv.org/html/2606.15315#bib.bib3)\)identify traveler preferences by conducting surveys that capture travelers’ needs, which are then analyzed to determine the requirements for personalized journey planning services\.Ludwiget al\.\([2009](https://arxiv.org/html/2606.15315#bib.bib2)\)propose ROSE, a mobile application that combines event recommendation with public transportation navigation, using the A\* algorithm to incorporate multi\-criteria user preferences for optimal public transit routing\.Bajajet al\.\([2016](https://arxiv.org/html/2606.15315#bib.bib1)\)develop a personalized route recommendation system that identifies commuter preferences through a participatory sensing Android application, which collects convenience feedback from users during their public transit journeys, and stores this feedback in a knowledge base to provide personalized route recommendations\.

In recent years, a new paradigm has been proposed to create greener, safer, and more inclusive future transportation\(Linet al\.,[2023](https://arxiv.org/html/2606.15315#bib.bib12); Shamsuddohaet al\.,[2025](https://arxiv.org/html/2606.15315#bib.bib11)\)\. This new paradigm is reflected as transportation 5\.0, which employs emerging technologies such as Internet of Things, artificial intelligence \(AI\), and data analytics\(Linet al\.,[2023](https://arxiv.org/html/2606.15315#bib.bib12); webdemia,[2024](https://arxiv.org/html/2606.15315#bib.bib10)\)\. An LLM is an AI model that learns language patterns by training on large datasets\. These models are built to understand and generate human language\. Although language models have been developed for decades, transfer learning is what makes foundation models possible, and scale is what makes them powerful in recent years\(Wallace and Dunlop,[2016](https://arxiv.org/html/2606.15315#bib.bib113)\)\(Bommasani,[2021](https://arxiv.org/html/2606.15315#bib.bib112)\)\. The introduction of the self\-attention mechanism enabled the transformer architecture to capture long\-range dependencies more effectively\(Vaswaniet al\.,[2017](https://arxiv.org/html/2606.15315#bib.bib111)\)\. LLMs possess remarkable capabilities, producing text that can match human writing in both quality and natural flow\. These powerful AI systems have rapidly become integrated into people’s everyday lives\(Rappet al\.,[2025](https://arxiv.org/html/2606.15315#bib.bib110)\)\.

The emergence of LLMs in transportation can be seen in recent years\.Chenget al\.\([2025](https://arxiv.org/html/2606.15315#bib.bib75)\)propose LLM\-TFP, which integrates LLMs with spatio\-temporal features through a tokenizer that captures timestep, time\-of\-day, and spatial embeddings for urban traffic flow prediction\.Ronget al\.\([2024](https://arxiv.org/html/2606.15315#bib.bib74)\)propose an architecture that integrates LLMs with spatio\-temporal features for large\-scale traffic flow prediction in intelligent autonomous transport systems\.Wanget al\.\([2025](https://arxiv.org/html/2606.15315#bib.bib97)\)introduces LLMTraveler, an LLM\-empowered agent with a memory system that learns from past experiences to simulate human route choice behavior in transportation networks\.Zhaoet al\.\([2025](https://arxiv.org/html/2606.15315#bib.bib95)\)integrates multi\-modal crash data, including driver, vehicle, and environment information, into a textualized dataset of prompts to fine\-tune an LLM for crash prediction, and provides event\-level and conditional risk analysis for identifying critical features and guiding targeted interventions\.

For the LLM in public transit routing problem,Jonnalaet al\.\([2024](https://arxiv.org/html/2606.15315#bib.bib91)\)use LLMs to handle questions about the General Transit Feed Specification \(GTFS\) data, which is a data format that contains a series of \.txt files for public transit\. Questions such as “What is the meaning of the accessibility file?” target the understanding of the GTFS data format\. Furthermore, there is a gap between analyzing the GTFS data and providing journey solutions\.Wang and Shalaby \([2024](https://arxiv.org/html/2606.15315#bib.bib90)\)design an LLM\-based trip advisor application to provide trip suggestions\. The key inputs are the Origin\-Destination pair \(OD\), travel time, and user preference\. This study uses LLMs to generate paths directly based on information queried from their database, without using any public transit routing algorithms\.Fanget al\.\([2024](https://arxiv.org/html/2606.15315#bib.bib83)\)apply LLMs in public transit routing for network disruption\. The LLM works as both planner and summarizer\. The LLM planner analyzes the subway map for connections between stops and routes through stops\. It can identify nearby stations and possible routes as alternatives during disruption\. The LLM summarizer organizes the content about possible routes and stops\. However, these suggestions do not include timetable information for suggested routes and stops\.Fanget al\.\([2024](https://arxiv.org/html/2606.15315#bib.bib83)\)show that it is difficult to conduct the experiments since there is no benchmark model for LLM in public transit routing\. Also, the challenges of LLM in public transit routing lie in its lack of transportation\-specific domain knowledge, public transit data, and the accuracy of its answers\.

Despite recent advances in applying LLMs to transportation problems, significant gaps remain in personalized public transit routing\. Existing personalized routing approaches rely heavily on historical data collection through surveys, past travel records, or participatory sensing applications to learn user preferences\. These methods assume static preference patterns and require substantial user data before providing personalized recommendations\. They fail to address the reality that travelers’ requests are dynamic and change daily based on varying contexts, needs, and circumstances\. Recent LLM\-based transit applications have limitations\. Some focus only on interpreting GTFS data structures without providing actual journey solutions\. Others employ routes recorded in the past directly without using a routing algorithm to find the most suitable routes according to the current dynamic real\-world environment\. Most lack the ability to dynamically interpret and adapt to user preferences expressed in natural language during the planning process\.

Therefore, this work addresses the gap by proposing an LLM\-based framework that enables intelligent, personalized preferences by integrating LLM with MC\-RAPTOR to find solutions based on real timetable data\. The proposed framework supports conversational preference refinement throughout the journey planning process and allows public transit routing to be personalized, intelligent, and adaptive to the dynamic nature of daily travel requests\. Within the large language models for transportation taxonomy\(Nieet al\.,[2025](https://arxiv.org/html/2606.15315#bib.bib43)\), ChatPlanner operates simultaneously as an information processor, interpreting natural language preference expressions, and a decision facilitator, integrating LLM reasoning with MC\-RAPTOR to produce timetable\-grounded journey plans\.

## 3Methodology

This section explains the ChatPlanner framework and its core components\. Section[3\.1](https://arxiv.org/html/2606.15315#S3.SS1)presents the overall framework architecture, which focuses on the information transition across each node\. Furthermore, two core components are explained in detail in Sections[3\.2](https://arxiv.org/html/2606.15315#S3.SS2)and[3\.3](https://arxiv.org/html/2606.15315#S3.SS3), including the fine\-tuned LLM parser with RAG for understanding traveler requests in Node 1, and the process of integrating the multi\-criteria public transit algorithm in Node 4\.

### 3\.1ChatPlanner Agent Framework

ChatPlanner is structured as a directed graph of seven nodes, implemented via the LangGraph software library\.

The LangGraph architecture comprises three core elements: nodes, edges, and a shared data structure State, as illustrated in Figure[1](https://arxiv.org/html/2606.15315#S3.F1)\. State stores structured information throughout execution, including Preference Scores, Journey Request, Criteria List, and Solution Table\. Nodes represent discrete actions\. Each node operates on State through a read\-modify\-write cycle: it reads the structured information, performs its designated action, and updates State with results\. Edges define conditional transitions between nodes, determining control flows\.

![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/Langgraph-Copy_12.png)Figure 1:Overall framework of the LangGraph for ChatPlanner#### 3\.1\.1Initial Planning Phase

Node 1contains three actions: RAG, Prompt, and Fine\-tuned LLM parse\. It processes the user’s free\-text input and records the requested origin, destination, and arrival time\. The input may also include the user’s preferences and intentions\. First, the semantic similarity is computed between the user’s input query and the query instances stored in the RAG dataset\. Second, the most similar five RAG instances are retrieved and injected into the Prompt together with the user’s input query\. The Prompt instructs the fine\-tuned LLM to parse the correct request of the user’s information, including the origin, destination, requested arrival time, and preferences\. The parsed information from Node 1 updates the Preference Scores and Journey Request in the State\.

Node 2\.The origin and destination from the Journey Request, parsed by the fine\-tuned LLM, may not match the standard format required by the Route Planner in Node 4\. The parsed origin and destination are mapped to standard names that appear in the available timetables for the Route Planner to carry out the search\. The mapping is supported by both the string matching check and a further geographic matching check using latitude and longitude to avoid misinterpretation\. The geographic matching check using the geopy package\(Esmukov and others,[2023](https://arxiv.org/html/2606.15315#bib.bib39)\)improves the mapping accuracy, even when the user uses conventions or abbreviations for stations\. The accurate origin and destination are written to the State\.

Node 3\.The Preference Scores from State contain four criteria: accessibility, crowding, safety, and sightseeing\. Each criterion is parsed with a score, from 0 to 1, in Node 1 by the fine\-tuned LLM\. These preference scores require further processing in Node 3 for active preferences selection\. To accurately identify true user needs and maintain computational tractability during route search, Node 3 implements a two\-step preference selection process\. First, criteria with scores below0\.60\.6are filtered out\. This threshold was determined empirically through preliminary experiments\. By evaluating 30 semantically neutral journey requests \(i\.e\., requests containing only origin, destination, and time information\), it is observed that the average predicted scores naturally hover around0\.50\.5rather than absolute zero \(detailed results of this empirical test are provided in Appendix A\)\. Therefore,0\.60\.6is established as the minimum activation threshold\. Second, from the retained set, a maximum ofK=2K=2criteria with the highest scores are selected as active user preferences for the initial planning phase\. Because the MC\-RAPTOR algorithm inherently optimizes travel time and the number of transfers, addingK=2K=2user preferences creates a four objectives Pareto optimization problem\. To guarantee that moderately important preferences will not be discarded, and to ensure this selection policy does not systematically suppress meaningful solutions or cause permanent information loss, ChatPlanner leverages its conversational architecture\. The parameterKKcan be adjusted, and any specific preferences filtered out in Node 3 can be reselected during the follow\-up conversation phase \(Node 6\) if the user is unsatisfied with the initial solutions\. The selection is saved to the Criteria List in State\. Actions in Nodes 1, 2, and 3 work together to support the understanding of users’ preferences\.

Node 4reads the Journey Request and the Criteria List from State, including the canonical origin and destination, the requested arrival time, and the active user preferences as inputs\. Then these inputs are passed to MC\-RAPTOR, with the active user preferences determining the objectives that drive search\. The solutions provided by MC\-RAPTOR are saved in the Solution Table in State\. More information about MC\-RAPTOR is explained in Section[3\.3](https://arxiv.org/html/2606.15315#S3.SS3)\.

Node 5is designed to present the solutions, which simplifies the Solution Table essential information\. The original table contains case metadata, algorithm parameters, intermediate metrics, and final criteria values\. The simplified table retains only the solution ID and six solution criteria: travel time, number of transfers, accessibility, crowding, safety, and sightseeing\. This simplification reduces users’ effort in comprehending solutions\. A base LLM then presents this simplified table and generates a brief text summary highlighting the tradeoffs between solutions, such as which solution balances the user’s preferences best and how solutions differ in terms of the criteria\.

#### 3\.1\.2Follow\-up Phase

Nodes 6 and 7 work together to handle the follow\-up phase\. Node 6 interprets the follow\-up request from users, leveraging a base LLM for conversational interaction\. The main responsibility of Node 6 is to decide which actions to take in Node 7\. The actions include: \(i\) Show Specific Solutions, \(ii\) Criteria Adjustment, and \(iii\) Solution Regeneration\. Show Specific Solutions is executed when the user requests to provide more details for the existing solutions from the initial planning phase\. Criteria Adjustment happens when Node 6 detects that the user is particularly emphasizing one of the four criteria\. It triggers the solutions re\-ranking process based on the emphasized criterion\. Solution Regeneration initiates re\-planning based on the follow\-up request by calling the route planner in Node 4\. When the users provide an extra criterion and are not satisfied with the current solutions, the regeneration starts\.

### 3\.2Fine\-tuning LLM Parse and RAG

In the LangGraph architecture, Nodes 1, 5, and 6 all invoke an LLM, but each node assigns the LLM a distinct role\. Nodes 5 and 6 rely on the base LLM to present solutions and to classify user intended follow\-up actions\. These do not require fine\-tuning\. By contrast, Node 1 employs a fine\-tuned LLM for parsing the origin, destination, and requested arrival time from the user’s query and to interpret the user’s travel preferences\.

#### 3\.2\.1Fine\-tuned Data Preparation

Two established datasets serve as the foundation for fine\-tuning: the Multi\-Domain Wizard\-of\-Oz \(MultiWOZ\) and Schema\-Guided Dialogue \(SGD\) datasets Buses domain\. MultiWOZ was chosen because it contains over 10,000 human\-to\-human dialogues, including many in train and taxi transportation contexts\(Budzianowskiet al\.,[2018](https://arxiv.org/html/2606.15315#bib.bib55)\)\. SGD was selected because it contains 3135 annotated dialogues for bus services\(Rastogiet al\.,[2020](https://arxiv.org/html/2606.15315#bib.bib54)\)\. In MultiWOZ, stop names are real UK city names and railway stations, while in SGD, they are primarily US city names\. This geographic diversity in the training data provides varied location naming patterns\.

To prepare the fine\-tuning dataset, the MultiWOZ dataset is filtered to retain only transportation\-related dialogues\. Because the focus is on urban public transit in this work rather than intercity trains or taxis, this study replaces train and taxi terms with urban transit terms in both datasets\. For example, “find me a train” becomes “find me a route,” and “book a taxi” becomes “plan a route,” to ensure the terminology is consistent with urban transit\. Since MC\-RAPTOR searches timetables backward from a requested arrival time, this study uses an “arrive by” convention rather than “depart at” throughout the dataset\. While the current system does not implement departure time logic, the same data processing approach could be applied by converting “arrive by” to “depart at” convention and integrating forward search functionality in MC\-RAPTOR\.

##### Preference Data Argumentation

Table 1:Persona Types, Preference Weights, and Examples\.Note: Personas are not included in the LLM’s inputs and predicted outputs during both training and generation\.PersonaAccess\-ibilityCrowd\-ednessSafetySight\-seeingExampleSightseeing Tourist0\.40\.30\.70\.9I’d love to see landmarks on the wayMobility Impaired0\.950\.80\.80\.2I don’t want stairs or gapsSafety Conscious0\.40\.60\.950\.2I want to feel secure in my journeyCrowd Averse0\.40\.90\.60\.3I prefer quieter, less busy routesDaily Commuter0\.60\.70\.70\.2I go to officeFamily with Children0\.80\.80\.90\.6I’m traveling with childrenSenior Citizen0\.90\.850\.90\.5I am elderly; I want less crowded routesFlexible Travel0\.50\.30\.60\.7I’m flexible with conditionsTable 2:Context Types, Preference Weights, and Examples\.Note: Contexts are not included in the LLM’s inputs and predicted outputs during both training and generation\.ContextAccess\-ibilityCrowd\-ednessSafetySight\-seeingExampleRush Hour0\.60\.80\.50\.5during rush hour; in the morning rush; when it’s busyLate Night0\.50\.30\.90\.2late at night; after dark; in the eveningWeekend0\.50\.60\.00\.8on the weekend; for a weekend trip; on Saturday/SundayTourist Area0\.50\.60\.70\.9in the tourist district; near the main attractionsBad Weather0\.80\.70\.50\.3when it’s raining; during bad weatherOur fine\-tuning approach follows a supervised learning setting, where the model learns to map traveler queries in the form of natural language to structured JSON outputs containing origin, destination, requested arrival time, and preference scores\. In supervised fine\-tuning, the model requires labeled training examples with input\-output pairs that demonstrate the desired behavior\. Since existing traveler query dialogue datasets lack preference annotations, this study conducts systematic preference data augmentation guided by established travel behavior research to inject preference information into journey requests and create the necessary ground truth labels for our training dataset\.

An augmentation system is designed based on eight user personas and five context types \(Tables[1](https://arxiv.org/html/2606.15315#S3.T1),[2](https://arxiv.org/html/2606.15315#S3.T2)\)\. The eight personas defined include mobility impaired users\(Ceccatoet al\.,[2020](https://arxiv.org/html/2606.15315#bib.bib32); Bezyaket al\.,[2017](https://arxiv.org/html/2606.15315#bib.bib31)\), safety\-conscious travelers\(Loukaitou\-Sideris,[2014](https://arxiv.org/html/2606.15315#bib.bib30); Delbosc and Currie,[2012](https://arxiv.org/html/2606.15315#bib.bib29)\), crowd\-averse passengers\(Roncoliet al\.,[2023](https://arxiv.org/html/2606.15315#bib.bib33); Tirachiniet al\.,[2013](https://arxiv.org/html/2606.15315#bib.bib28)\), sightseeing tourists\(Zhouet al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib46); De Voset al\.,[2016](https://arxiv.org/html/2606.15315#bib.bib27)\), daily commuters\(Cascetta and Cartenì,[2014](https://arxiv.org/html/2606.15315#bib.bib26)\), families with children\(Delbosc and Currie,[2012](https://arxiv.org/html/2606.15315#bib.bib29); Ceccatoet al\.,[2020](https://arxiv.org/html/2606.15315#bib.bib32)\), senior citizens\(National Centre for Accessible Transport \(ncat\),[2024](https://arxiv.org/html/2606.15315#bib.bib25)\), and flexible travelers\(Cascetta and Cartenì,[2014](https://arxiv.org/html/2606.15315#bib.bib26)\)\.

Preference weights are assigned to each persona across four dimensions: accessibility, safety, crowdedness, and sightseeing, with the score of preferences referencing the above literature\. For example, high accessibility preferences are assigned to mobility\-impaired users with accessibility scores set to 0\.95 in the dataset, consistent with findings on accessibility measurement\(Ceccatoet al\.,[2020](https://arxiv.org/html/2606.15315#bib.bib32)\)\. Similarly, high sightseeing scores \(0\.9\) are assigned to tourist personas, followingZhouet al\.\([2024](https://arxiv.org/html/2606.15315#bib.bib46)\)\. Contextual modifiers are designed to assign preference scores for situational factors such as rush hour, late night, weekend, tourist area, and bad weather conditions, where score magnitudes are guided by\(Loukaitou\-Sideris,[2014](https://arxiv.org/html/2606.15315#bib.bib30); Cascetta and Cartenì,[2014](https://arxiv.org/html/2606.15315#bib.bib26)\)\. These assignments serve as the ground truth labels for the training dataset, though alternative scoring schemes could be developed for different preference frameworks\.

The augmentation process randomly selects personas or contexts for each training example, injects appropriate preference language into the instruction, and generates the corresponding preference scores in structured JSON output, creating aligned input\-output pairs for supervised learning\. The JSON structure ensures the fine\-tuned model produces consistently parseable output\.

#### 3\.2\.2Fine\-tune

Fine\-tuning adapts pre\-trained language models to domain\-specific tasks by optimizing model parametersθ\\thetato minimize the loss function\.

ℒ​\(θ\)=1N​∑i=1Nℓ​\(fθ​\(xi\),yi\)\\mathcal\{L\}\(\\theta\)=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\ell\(f\_\{\\theta\}\(x\_\{i\}\),y\_\{i\}\)\(1\)wherefθf\_\{\\theta\}represents the language model,\(xi,yi\)\(x\_\{i\},y\_\{i\}\)are input\-output pairs from the fine\-tuned dataset, andℓ\\ellis the loss function\.

Parameter Efficient Fine\-Tuning \(PEFT\)\(Hanet al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib49)\)is employed to reduce computational costs\. Specifically, Low\-Rank Adaptation \(LoRA\)\(Huet al\.,[2022](https://arxiv.org/html/2606.15315#bib.bib50)\)is used, which avoids updating all model parameters by learning low\-rank small adjustment matrices\. For each linear layer of weight matrixW0∈ℝd×kW\_\{0\}\\in\\mathbb\{R\}^\{d\\times k\}in the original model, LoRA introduces two smaller matricesB∈ℝd×rB\\in\\mathbb\{R\}^\{d\\times r\}andA∈ℝr×kA\\in\\mathbb\{R\}^\{r\\times k\}, whererris the rank andr≪min⁡\(d,k\)r\\ll\\min\(d,k\)\. The adapted transformation becomes:

h=W0​x\+B​A​xh=W\_\{0\}x\+BAx\(2\)
The original weightsW0W\_\{0\}remain frozen, and onlyBBandAAare trained\. This reduces trainable parameters fromd×kd\\times ktor×\(d\+k\)r\\times\(d\+k\), significantly lowering computational requirements\.

LoRA is applied to the LLaMA\-3\-8B\-Instruct and Qwen2\.5\-7B\-Instruct models with rankr=16r=16and scaling factorα=32\\alpha=32\. The method targets the attention and feed\-forward layers of these models\. Training uses 4\-bit quantization for memory efficiency and runs for 2 epochs with learning rate1\.5×10−41\.5\\times 10^\{\-4\}\. The objective is to train the model to parse natural language transportation queries into structured JSON outputs containing origin, destination, arrival time, and preference scores\. Please note that the persona and context labels are not used to form the training dataset; the LLM is trained to perform direct regression from the raw natural language request to predict the four continuous preference scores\.

#### 3\.2\.3Retrieval\-Augmented Generation \(RAG\)

RAG\(Lewiset al\.,[2020](https://arxiv.org/html/2606.15315#bib.bib51)\)enhances language model performance by incorporating relevant external knowledge during inference\. While fine\-tuning updates the model’s global parameters to learn the required JSON output structure and general preference scoring patterns, RAG provides instance\-specific context; by retrieving similar examples as local reference points to guide the fine\-tuned model’s output generation, it helps the model resolve users’ imprecise and scenario\-specific expressions and addresses scenario\-specific preference understanding\.

The retrieval process identifies semantically similar examples using text embeddings\. An embedding is a mathematical representation that converts text into a numerical vector, where texts with similar meanings produce vectors that are close together in vector space\. The BAAI/bge\-small\-en\-v1\.5 embedding model\(Xiaoet al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib52)\)is used to compute 384\-dimensional vectors for query instanceqiq\_\{i\}in our RAG dataset, denoted as𝐞i\\mathbf\{e\}\_\{i\}\. Similarly, the user queryqqis converted to a query embedding𝐞q\\mathbf\{e\}\_\{q\}\.

The semantic similarity between a user query and a query instance stored in the RAG dataset is measured using cosine similarity:

sim​\(q,qi\)=𝐞q⋅𝐞i‖𝐞q‖2⋅‖𝐞i‖2\\text\{sim\}\(q,q\_\{i\}\)=\\frac\{\\mathbf\{e\}\_\{q\}\\cdot\\mathbf\{e\}\_\{i\}\}\{\|\|\\mathbf\{e\}\_\{q\}\|\|\_\{2\}\\cdot\|\|\\mathbf\{e\}\_\{i\}\|\|\_\{2\}\}\(3\)where𝐞q⋅𝐞i\\mathbf\{e\}\_\{q\}\\cdot\\mathbf\{e\}\_\{i\}is the dot product and\|\|⋅\|\|2\|\|\\cdot\|\|\_\{2\}denotes the L2 norm\. This metric ranges from \-1 to 1, with higher values indicating greater semantic similarity\.

For each user query, the topk=5k=5most similar queries are retrieved from the RAG dataset\. These retrieved examples are then formatted as demonstration examples in the prompt together with the current user query\. This approach, known as in\-context learning, provides the fine\-tuned model with concrete examples showing how similar queries were previously mapped to structured preference scores\. The retrieved examples help the model maintain consistency with our designed preference score assignments based on personas and contexts\.

The retrieval library is constructed using the same training dataset generated through the preference data augmentation process\. Consequently, highly similar patterns exist between the training set and the RAG corpus\. It is important to note that the retrieved examples contain only the raw natural language request and the corresponding origin, destination, arrival time, and preference scores; they do not explicitly contain the persona or context labels used during preference data augmentation\. Furthermore, the retrieval library is strictly disjoint from the human\-written evaluation test set used in Section 4\.2, ensuring that the model is tested on its ability to generalize to out\-of\-distribution human language rather than simply memorizing training distributions\.

### 3\.3Multi\-Criteria Route Planning Algorithm

The Multi\-criteria Round\-Based Public Transit Routing \(MC\-RAPTOR\) algorithm is used as the default solver in Node 4\. The original MC\-RAPTOR algorithm uses timetable data to optimize travel time and the number of transfers\. MC\-RAPTOR runs backward in the timetable from the destination, marking routes and stops that can reach the destination\. The search proceeds in roundsNN; stops marked in the previous round seed the expansion in the next round\. The number of transfers is equal toN−1N\-1\. Per\-stop labels store objective values, including travel time labelsL1L\_\{1\}, accessibilityL2L\_\{2\}, crowdingL3L\_\{3\}, safetyL4L\_\{4\}, and sightseeingL5L\_\{5\}\. A Pareto\-dominance check removes labels dominated by others at each stop\.

![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/MCRaptor_Data_Preparation.png)Figure 2:Data Preparation for MC\-RAPTORThe MC\-RAPTOR algorithm is a timetable search algorithm that minimizes travel time and the number of transfers\. To support the algorithm in considering additional and user\-centric criteria, such as accessibility, crowding, safety, and sightseeing, this study integrates additional datasets as shown in Figure[2](https://arxiv.org/html/2606.15315#S3.F2)\. In particular, the accessibility level, crowding level, safety level, and sightseeing level are the extra criteria considered in this study\. The selected active preferences determined in Node 3 directly weight the third and fourth criteria\.

Crowding and safety levels at each station are simulated using a normal distribution and then normalized to the range\[0,1\]\[0,1\], where 0 represents the lowest level and 1 represents the highest level\. Tourist\-friendly level is based on whether the distance of each station to the London tourist attractions is within 100 meters\. The London tourist locations are downloaded from the Open Trip Map by enabling the tourism option\([2](https://arxiv.org/html/2606.15315#bib.bib94)\)\. The step\-free data are collected from Transport for London \(TfL\) to assess accessibility of stops; all buses and bus stops are step\-free, featuring the low\-floor vehicles with ramps that deploy to make boarding easier\. The step\-free data are mainly collected for the underground and tram\([75](https://arxiv.org/html/2606.15315#bib.bib93)\)\. The timetable data is based on the London city TfL GTFS data from June 2024 to December 2024\.

## 4Experiments and Discussions

To validate the effectiveness of our proposed ChatPlanner, three complementary experiments are designed to evaluate different components and capabilities\. Experiment 1 investigates and compares the feasibility of end\-to\-end route planning between LLM\-based approaches and our ChatPlanner framework\. Experiment 2 evaluates the LLM parsing component’s ability to extract travel information and predict user preferences from natural language requests\. Experiment 3 demonstrates how ChatPlanner enlarges the solution space by finding diverse routes that serve different user preferences beyond traditional optimization criteria\.

### 4\.1Experiment 1: Route Planning Feasibility by LLM with Tool

This experiment aims to evaluate the end\-to\-end public transit routing performance of the available LLM equipped with public transit routing tools\. Specifically, whether it can generate realistic, feasible, and complete transit routes when given the same journey planning queries as ChatPlanner\.

GPT\-5\.0 \(released in August 2025\) is utilized and accessed via the OpenAI API in September 2025\(Singhet al\.,[2025](https://arxiv.org/html/2606.15315#bib.bib40)\)\), a state\-of\-the\-art closed\-source model\. For each test case, GPT\-5\.0 first parses the origin, destination, and requested arrival time from the natural language query, then calls a Google Maps API tool, which is implemented as a Python script that receives the parsed variables from LLM, queries the API, and returns the structured routing response to the LLM\. If the Google Maps API does not return a valid result, GPT\-5\.0 is instructed to use its web search tool to query Moovit\(Moovit Inc\.,[2025](https://arxiv.org/html/2606.15315#bib.bib41)\)and Rome2Rio\(Rome2Rio Pty Ltd\.,[2025](https://arxiv.org/html/2606.15315#bib.bib42)\)as fallback routing services\. All three services contain the whole London GTFS data used by ChatPlanner\.

##### Test Dataset

45 basic journey planning query instances are constructed using the template "I need to get from \{start\} to \{end\} by \{time\}" with actual London locations and realistic arrival times\. Each query contains information: start and end stops, and the requested arrival time\.

For each case, the origin and destination routes were manually verified using Citymapper and Google Maps, with the requested arrival time specified accordingly\. For routes that did not appear in either map solver but were returned by the tool\-augmented LLM, further verification was conducted by searching the route number of each segment against the London TfL dataset to confirm its existence\.

#### 4\.1\.1Evaluation Metrics

To assess practical feasibility, the generated route planning solutions are evaluated across four key performance indicators \(KPIs\)\. These KPIs determine whether a generated route is useful for real transit users\. Table[3](https://arxiv.org/html/2606.15315#S4.T3)provides the scope for each KPI\.

\(i\) Station Identification \(SI\) captures inaccuracies in location specification\. This includes wrong station names for origins, destinations, or transfers; vague descriptions where no explicit station name is provided; and ambiguous stop references that prevent a valid API call from being issued\. Such issues prevent the system from retrieving route information and prevent users from identifying the correct physical locations to begin or continue their journeys\. \(ii\) Timetable Accuracy \(TA\) captures inaccuracies in timing information for valid services\. This includes wrong departure or arrival times and incorrect duration estimates\. TA excludes cases where the route itself does not exist\. Timetable inaccuracies mean users may miss the timed service, though the route itself may remain feasible via an alternative departure on the same route\. \(iii\) Route Validity \(RV\) captures wrong or unsupported services that do not serve the specified journey, and missing transit legs that leave a route incomplete\. RV excludes timing issues for otherwise valid services\. \(iv\) Walking Completeness \(WC\) concerns the pedestrian portions of a route, including incorrect estimates of walking time in a segment\. Solutions missing the necessary walking segment are classified under route validity\.

KPI Rate=Number of solutions with specific KPI typeTotal number of solutions\\text\{KPI Rate\}=\\frac\{\\text\{Number of solutions with specific KPI type\}\}\{\\text\{Total number of solutions\}\}
The Feasibility Rate serves as our overall success metric, representing the proportion of solutions that can actually be executed by real users:

Feasibility Rate=Number of feasible solutionsTotal number of solutions\\text\{Feasibility Rate\}=\\frac\{\\text\{Number of feasible solutions\}\}\{\\text\{Total number of solutions\}\}
A solution is considered feasible if a user can complete the trip from the origin to the destination\.

Table 3:Scope and exclusion criteria for each error dimension\.DimensionScopeExcludeStation Identification \(SI\)Wrong name, vague description, ambiguous stop for OD pair that blocks calling for external tools\-Timetable Accuracy \(TA\)Wrong departure/arrival times, wrong durationRoute does not existRoute Validity \(RV\)Any public transit portion being invalid or incomplete, e\.g\., wrong service, unsupported line, and missing transit legTimings of a valid serviceWalking Completeness \(WC\)Covers walking portion: wrong walking time estimates and missing walking segmentMissing public transport segment that makes a transfer impossible
#### 4\.1\.2Performance Discussion and Comparison

Table[4](https://arxiv.org/html/2606.15315#S4.T4)presents a case\-by\-case summary of the KPIs observed for the tool\-augmented GPT\-5\.0\. Despite having access to external routing APIs and web search, the overall feasibility rate is only 35%\.RV records the highest value among all KPIs, achieving 35%\.

Table 4:Analysis of solutions from the Tool\-Augmented LLM across 45 test cases \(Cases 0–29\)\. S indicates a feasible route\.CaseOptionSITARVWCFeasibleEvidence01TAWCWrong time \(17:21→\\rightarrow17:33\); walking omitted on first leg11TARVOnly first segment shown; wrong time \(9:40→\\rightarrow10:32\)21WCWalk 2 mins stated as 10 mins31WCSFinal walk 9 mins stated as 4 mins41TASTold to leave 20 mins before earliest possible2S51TARV81–90 mins shown as 23 mins; bus fails destination61STransfer very tight71RVTransfer should be Waterloo, not London Bridge81WCFirst walking leg is \(15 mins\) not mentioned91SIRVAPI failed; transfer station wrong101RVBus 206 does not support this journey111SIRVAPI failed121WCSWalk 6 mins stated as 1 mins131SIAPI failed; station name ambiguous141SIAPI failed151S2RVBarbican wrong transfer; Barbican and Tower Hill station do not exist in the Circle Line161RVSpecific route is missing171SStart time should be 30 mins earlier181SIBus stop name not specified191SIAPI failed; station name ambiguous201SIAPI failed; station name ambiguous211SIAPI failed; station name ambiguous221RVWCFirst segment missing; how to reach Marble Arch omitted231RVSpecific route not provided2TARVBus 115 does not serve this journey241TASSpecific departure time not mentioned251SIBoth start and end stations are ambiguous261TASNeed 50 mins but shown as 35–40 mins2RVBus 13 does not serve this journey3WCWalking missing on first leg; train and tube not specified271S2RVDistrict line does not support this journey281S291S2RVBus 73 does not serve the final legTable 5:Analysis of solutions from the Tool\-Augmented LLM \(continued, Cases 30–44\)\. S indicates a feasible route\.CaseOptionSITARVWCFeasibleEvidence301RVBus 428 does not serve this OD pair311S321RVBus 410 does not serve this OD pair331TASTotal time is 41 mins not 25 mins341WCAmbiguous origin; failed to call Google API351Second segment requires 20 mins walk; walking omitted361SI“Dixon Way” is ambiguous371S381S391SIFailed to identify start point401WCSMissing second walking segment \(16 mins walk\)411S421RVFinal bus segment is missing431SIGoogle API failed; station name ambiguous441RVInvalid solutionTotal \(45 cases, 52 options\)12818918Error / feasibility rate23%15%35%17%35%Table 6:Summary comparison of GPT\-5\.0 with Tools and ChatPlanner across 30 test cases\.MethodSolutioncountSIrateTArateRVrateWCrateFeasibilityrateGPT\-5\.0 with Tools373724%19%38%19%32%ChatPlanner48482%0%0%0%98%A primary cause of this low feasibility is the inherent instability of autonomous tool use\. When querying the Google Maps API, the LLM can generate malformed requests to the API even when query codes and formats are provided\. When forced to fall back on web search tools \(e\.g\., Moovit or Rome2Rio\), the model must parse complex, unstructured web page layouts\. This extraction process is prone to information loss\. If the LLM misses an intermediate step or misinterprets the retrieved text, it leads to an invalid route\. These manifest as incorrect transfer stations \(e\.g\., London Bridge instead of Waterloo in Case 7\), bus route misassignments where real service numbers are applied to segments they do not operate \(e\.g\., Bus 13 and 73 in Cases 26 and 29\), and completely omitted transit legs \(Cases 16 and 22\)\.

SI further illustrates the challenge of using an LLM without a tightly integrated location database\. The location names the LLM extracts from the user’s prompt often do not exactly match the location strings expected by the external routing tools\. For example, Cases 19\-21 and 25 fail because the LLM submits ambiguous station names that the external APIs cannot resolve, returning zero results or routing the user to a location\.

Furthermore, TA and WC occur when the model incorrectly transcribes and summarizes the retrieved data\. For instance, in Case 5, the LLM extracts an incorrect travel duration \(stating 23 minutes for an 80\-minute journey\), rendering the suggested departure time unusable\. LLM could overlook walking details during summarization, resulting in omitted walking segments \(Case 0\) or incorrect walking time estimates \(Case 12\)\.

Table[6](https://arxiv.org/html/2606.15315#S4.T6)compares these baseline results against the proposed ChatPlanner framework\. ChatPlanner achieves a 98% feasibility rate across 52 generated solutions, significantly outperforming the tool\-augmented LLM\. This performance gap is a direct result of ChatPlanner’s modular architecture\. Rather than relying on the LLM to execute tool calls or summarize path planning results, ChatPlanner strictly limits the LLM’s role to natural language parsing and conversation handling\. The extracted location names are handled and mapped to the same stops locations database used by the routing algorithm, which minimizes API mismatches and reduces station identification errors to 2%\. Once the locations are accurately resolved, the deterministic MC\-RAPTOR algorithm computes the routes\. Because the actual path planning is handled by an exact search algorithm rather than language generation, ChatPlanner is not affected by route validity, timetable accuracy, and walking completeness errors \(0% across all three dimensions\)\.

### 4\.2Experiment 2: User Request Parsing

This experiment evaluates the performance of the LLM component in Node 1 of our system\. The LLM processes natural language requests and produces structured JSON output for three tasks: \(1\) extraction of stop names and request arrival time, \(2\) regression of user preference scores for four criteria, and \(3\) active preferences selection, involving multi\-label classification and ranking of criteria based on the regression scores\. Tasks \(1\) and \(2\) directly reflect the LLM component effectiveness for understanding and parsing the user request in Node 1, while task \(3\) reflects the effectiveness for guiding subsequent criteria selection in Node 3\. Finally, to explicitly validate the value of the LLM’s semantic understanding, LLM’s preference interpretation performance is compared against a traditional keyword matching system as the baseline\.

#### 4\.2\.1Experimental Setup

The test set comprises 120 instances collected through a user study\. Participants were asked to provide transit journey requests, organized across five travel scenarios: an everyday journey, a special occasion, a journey planned to help another person, a journey under an unusual constraint, and a free\-choice journey\. Participants completed all ten requests before receiving any information about the four preferences \(accessibility, crowdedness, safety, and sightseeing\)\. After completing all requests, participants were introduced to the four preferences with a scoring rubric and scored example transit requests\. Each preference is scored on a continuous scale from 0 to 1 for each of their ten requests\. Participants also provided the origin, destination, and time request for each instance\. From the 17 returned questionnaires, five were excluded \(one blank, two missing preference labels, and two incomplete\), yielding 120 labeled requests from 12 fully valid participants\. These labeled requests were then processed into the User Study Test Set, where each instance consists of a natural language travel request as the instruction, and a structured label containing the origin, destination, time specification, and four preference scores as the output\. In evaluation, the models are given each travel request and are expected to generate the corresponding output\.

Three properties distinguish this user study test set from our training and RAG data\. First, the requests capture the essences of public transit queries, including ambiguous expressions and mixed priorities, as they were composed freely without template constraints or domain\-specific priming\. Second, because participants wrote their requests without prior knowledge of the four preferences, the request text carries no systematic bias toward those preferences\. Third, participants had no access to the eight personas and five travel contexts used for preference injection in constructing the fine\-tuning and RAG datasets; the test set is therefore out\-of\-distribution \(OOD\) with respect to both, providing a more realistic assessment of generalization than evaluation based on in\-distribution synthetic data, where inputs map cleanly to predefined personas or contexts\.

To assess label reliability, three independent annotators scored all 120 instances across the four preferences using the same rubric and the same examples provided to the original participants\. The intraclass correlation coefficient \(ICC\(A,1\), two\-way random, absolute agreement, single measures\(McGraw and Wong,[1996](https://arxiv.org/html/2606.15315#bib.bib20)\)\) was computed across three annotators and the original participant, followingKoo and Li \([2016](https://arxiv.org/html/2606.15315#bib.bib21)\)\. The ICC values were 0\.79 \(95% CI: \[0\.76, 0\.82\]\) for accessibility, 0\.75 \(\[0\.71, 0\.80\]\) for crowdedness, 0\.77 \(\[0\.74, 0\.79\]\) for safety, and 0\.81 \(\[0\.78, 0\.83\]\) for sightseeing, indicating moderate to good inter\-annotator agreement\(Artstein and Poesio,[2008](https://arxiv.org/html/2606.15315#bib.bib22); Koo and Li,[2016](https://arxiv.org/html/2606.15315#bib.bib21)\)and supporting the use of the original participant scores as ground truth in Experiment 2\.

Two open\-source LLMs are selected, including Llama3\-8B\-Instruct\(Dubeyet al\.,[2024](https://arxiv.org/html/2606.15315#bib.bib57)\)and Qwen2\.5\-7B\-Instruct\(Qwenet al\.,[2025](https://arxiv.org/html/2606.15315#bib.bib56)\)\. Open\-source models differ from closed\-source models in that they allow access to model weights and architecture, enabling fine\-tuning for domain\-specific tasks\.

#### 4\.2\.2Experimental Design

Four different model configurations are compared: \(i\) ChatPlanner \(LoRA fine\-tuned LLM with RAG\), \(ii\) raw base LLM, \(iii\) base LLM with RAG, and \(iv\) LoRA fine\-tuned LLM without RAG\.

For each data instance in the test set, the journey request is input into the parsing component of each configuration, and the output JSON is compared with the ground truth label across three evaluation tasks: \(1\) exact matching, \(2\) preference score regression, and \(3\) active preferences selection\.

#### 4\.2\.3Task 1: Exact Matching

Task 1 examines the model’s ability to extract the exact origin, destination names, and the requested arrival time from the user’s natural language input query\. Exact string matching is evaluated for stop names and requested arrival time values\.

For discrete slots \(start stop, end stop, requested arrival time\), exact match accuracy is used as the evaluation metric, following the information extraction practice established in\(Rajpurkaret al\.,[2016](https://arxiv.org/html/2606.15315#bib.bib71)\)\.

Exact match accuraciesfor stop names \(EM startandEM end\) and requested arrival time \(EM time\) extraction directly measure the model’s ability to extract origin and destination locations and requested arrival time without any tolerance for variation\. They are calculated as

EMstop=1N​∑i=1N𝟙​\[s^i=si\],EMtime=1N​∑i=1N𝟙​\[t^i=ti\]\\text\{EM\}\_\{\\text\{stop\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\[\\hat\{s\}\_\{i\}=s\_\{i\}\],\\qquad\\text\{EM\}\_\{\\text\{time\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\[\\hat\{t\}\_\{i\}=t\_\{i\}\]
whereNNis the total number of test instances,s^i\\hat\{s\}\_\{i\}andsis\_\{i\}are the predicted and true stop names,t^i\\hat\{t\}\_\{i\}andtit\_\{i\}are the predicted and true time values for instanceii, and𝟙​\[⋅\]\\mathbbm\{1\}\[\\cdot\]is the indicator function that returns 1 if the condition is true and 0 otherwise\.

EM allis additionally defined as the joint exact match, where all three fields must be simultaneously correct:

EMall=1N​∑i=1N𝟙​\[s^istart=sistart∧s^iend=siend∧t^i=ti\]\\text\{EM\}\_\{\\text\{all\}\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\[\\hat\{s\}\_\{i\}^\{\\text\{start\}\}=s\_\{i\}^\{\\text\{start\}\}\\wedge\\hat\{s\}\_\{i\}^\{\\text\{end\}\}=s\_\{i\}^\{\\text\{end\}\}\\wedge\\hat\{t\}\_\{i\}=t\_\{i\}\]where∧\\wedgedenotes logical AND\.

A time tolerance accuracy is also included as an operational relaxation for practical transportation applications\.Time tolerance accuracyallowing 5\-minute deviation \(TimeTol@5mins\) provides a measure with tolerance, as small time deviations are sometimes acceptable in real\-world scenarios\. It is calculated as

TimeTol@5mins=1N​∑i=1N𝟙​\[\|t^i−ti\|≤5\]\\text\{TimeTol@5mins\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\[\|\\hat\{t\}\_\{i\}\-t\_\{i\}\|\\leq 5\]
Table[7](https://arxiv.org/html/2606.15315#S4.T7)presents the exact matching metrics for stop name and requested arrival time extraction\. These metrics measure the model’s ability to parse discrete fields from natural language input\.

Table 7:Exact matching performance for stop names and requested arrival time\.LabelEMstartEMendEMtimeTimeTol@5minsEMallLlama Base0\.8670\.8670\.7330\.7330\.6420\.6420\.6750\.6750\.4750\.475Qwen Base0\.8750\.8750\.7500\.7500\.8000\.8000\.8500\.8500\.6080\.608Llama Base \+ RAG0\.8580\.8580\.8080\.8080\.8500\.8500\.8920\.8920\.6920\.692Qwen Base \+ RAG0\.8830\.8830\.8330\.8330\.9170\.9170\.9580\.9580\.7750\.775Llama Fine\-tuned0\.8920\.8920\.8750\.8750\.9080\.9080\.9540\.9540\.8080\.808Qwen Fine\-tuned0\.9000\.9000\.8920\.8920\.9000\.9000\.9500\.9500\.8250\.825Llama Fine\-tuned \+ RAG1\.0001\.0000\.9920\.9920\.9920\.9921\.0001\.0000\.983Qwen Fine\-tuned \+ RAG1\.0001\.0001\.0001\.0000\.9750\.9751\.0001\.0000\.9750\.975The results show that the base LLMs struggle with exact matching on the User Study Test Set, achieving only 47\.5\-60\.8% overall accuracy\. Adding RAG components improves performance to 69\.2\-77\.5%, but still leaves significant parsing errors\. The exact parsing of stops and time is important, and any error at this stage will result in route planner failure\. After fine\-tuning with LoRA, accuracy improves to 80\.8\-82\.5%\. Combining fine\-tuning with RAG leads to the best performance at 97\.5\-98\.3%\. These results demonstrate their complementary roles\. Fine\-tuning enforces the strict output structure, while RAG provides query\-specific examples to resolve location\-specific, imprecise, conversational, and implicit human expressions, making their combination necessary for more reliable exact matching\. The fine\-tuning corpus draws from both US\-context \(SGD\) and UK\-context \(MultiWOZ\) data, which introduces variation in location\-specific terminology\. However, the RAG component helps ground the stop name resolution directly in the target TfL corpus\. As shown in Table[7](https://arxiv.org/html/2606.15315#S4.T7), while the fine\-tuned models alone perform well, adding the RAG component achieves near\-perfect stop name extraction \(up to 100\.0% EM start and end\)\. This demonstrates that the retrieval mechanism can compensate for distributional mismatches in transport terminology \(e\.g\., US vs\. UK vocabulary\) between the training and test domains\.

#### 4\.2\.4Task 2: Preference Score Regression

Given the user query, the model predicts one numerical score between 0 and 1 for each of the four criteria: accessibility, crowding, safety, and sightseeing, which reflect user preferences in public transit route planning\. Standard regression metrics are used for evaluation, along with agreement measures\(Hyndman and Koehler,[2006](https://arxiv.org/html/2606.15315#bib.bib70)\)\.

Mean Absolute Error \(MAE\)measures the average magnitude of prediction errors for preference scores, providing an intuitive measure of how far predictions deviate from true user preferences\.NNis the total number of test instances,y^i\\hat\{y\}\_\{i\}andyiy\_\{i\}are the predicted and true preference scores for instanceii\. MAE is calculated as

MAE=1N​∑i=1N\|y^i−yi\|\\text\{MAE\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|
Root Mean Square Error \(RMSE\)penalizes larger errors more heavily than MAE, indicating how well the model avoids substantial errors in estimating user preferences\. It is calculated as

RMSE=1N​∑i=1N\(y^i−yi\)2\\text\{RMSE\}=\\sqrt\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(\\hat\{y\}\_\{i\}\-y\_\{i\}\)^\{2\}\}
Symmetric Mean Absolute Percentage Error \(sMAPE\)normalizes errors across different score scales, providing a scale\-independent measure of relative prediction accuracy\. It is calculated as

sMAPE=100N​∑i=1N\|y^i−yi\|\(\|y^i\|\+\|yi\|\)/2\\text\{sMAPE\}=\\frac\{100\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{\|\\hat\{y\}\_\{i\}\-y\_\{i\}\|\}\{\(\|\\hat\{y\}\_\{i\}\|\+\|y\_\{i\}\|\)/2\}
While error metrics \(MAE, RMSE, sMAPE\) measure prediction deviations, theConcordance Correlation Coefficient \(CCC\)evaluates whether predictions agree with true values in both their correlation and absolute scale\(Lawrence and Lin,[1989](https://arxiv.org/html/2606.15315#bib.bib69)\)\. CCC is calculated as

CCC=2​ρ​σy^​σyσy^2\+σy2\+\(μy^−μy\)2\\text\{CCC\}=\\frac\{2\\rho\\sigma\_\{\\hat\{y\}\}\\sigma\_\{y\}\}\{\\sigma\_\{\\hat\{y\}\}^\{2\}\+\\sigma\_\{y\}^\{2\}\+\(\\mu\_\{\\hat\{y\}\}\-\\mu\_\{y\}\)^\{2\}\}
whereρ\\rhois the Pearson correlation between predicted and true scores,σy^\\sigma\_\{\\hat\{y\}\}andσy\\sigma\_\{y\}denote their standard deviations, andμy^\\mu\_\{\\hat\{y\}\}andμy\\mu\_\{y\}denote their means\. CCC ranges from \-1 to 1, with 1 indicating perfect agreement, 0 no concordance, and negative values indicating systematic disagreement\. CCC captures both relative ordering and magnitude matching, rewarding models that both rank correctly and match magnitudes\. This is useful because a model might correctly rank preferences but systematically over/underestimate scores, which error metrics alone cannot fully reveal\.

Table[8](https://arxiv.org/html/2606.15315#S4.T8)presents the regression metrics for predicting user preference scores averaged across the four criteria\.

Table 8:Regression performance for preference score prediction\.LabelMAERMSEsMAPECCCLlama Base0\.4060\.4060\.4800\.4800\.5070\.5070\.0760\.076Qwen Base0\.4110\.4110\.5010\.5010\.4750\.4750\.0600\.060Llama Base \+ RAG0\.1380\.1380\.1850\.1850\.1900\.1900\.6020\.602Qwen Base \+ RAG0\.1910\.1910\.2750\.2750\.2940\.2940\.5560\.556Llama Fine\-tuned0\.1090\.1090\.1770\.1770\.1930\.1930\.7440\.744Qwen Fine\-tuned0\.1800\.1800\.2520\.2520\.2320\.2320\.6530\.653Llama Fine\-tuned \+ RAG0\.0980\.1420\.1580\.779Qwen Fine\-tuned \+ RAG0\.1290\.1290\.1850\.1850\.2020\.2020\.6800\.680The regression results demonstrate clear improvements from both RAG and fine\-tuning \([8](https://arxiv.org/html/2606.15315#S4.T8)\. Base LLMs show MAE values around 0\.4, which represents substantial error given the\[0,1\]\[0,1\]score range\. Adding either RAG or fine\-tuning individually reduces MAE considerably, while the combination of our designed fine\-tuning and RAG achieves the best performance across all four metrics\. The best\-performing configuration, Llama Fine\-tuned \+ RAG, achieves an MAE of below 0\.1 and a CCC near 0\.8, indicating both accurate preference magnitude prediction and substantial correlation with human\-annotated preference scores\.

##### Statistical Robustness of Tasks 1 and 2

To quantify the variability in LLM performance for the extraction \(Task 1\) and regression \(Task 2\) across different test samples, paired bootstrap resampling is applied, and confidence intervals are computed following standard practices\(Tibshirani and Efron,[1993](https://arxiv.org/html/2606.15315#bib.bib61); Droret al\.,[2018](https://arxiv.org/html/2606.15315#bib.bib60)\)\. Since LLM performance can vary depending on which specific test examples are evaluated, a single test set may not fully represent the model’s true capability\. Bootstrap resampling addresses this limitation by drawing 1,000 resampled test sets of equal size \(120 instances each\) with replacement from the User Study Test Set, where some instances may appear multiple times, and others may be omitted in each resample\. EM and MAE are recomputed on each resampled set, and the resulting 95% confidence intervals \(CIs\), reported in Table[9](https://arxiv.org/html/2606.15315#S4.T9), indicate the range within which true performance likely falls\.

Table 9:Bootstrapped EM and MAE \(means and 95% CIs\)\.LabelEM allmeanEM allCI lowEM allCI highScore MAEmean \(items\)MAECI lowMAECI highLlama Base0\.4920\.4920\.4080\.4080\.5580\.5580\.4050\.4050\.3830\.3830\.4280\.428Qwen Base0\.6830\.6830\.6330\.6330\.7170\.7170\.4290\.4290\.4040\.4040\.4570\.457Llama Base \+ RAG0\.6170\.6170\.5250\.5250\.6490\.6490\.1310\.1310\.1180\.1180\.1450\.145Qwen Base \+ RAG0\.7670\.7670\.7420\.7420\.7920\.7920\.1910\.1910\.1760\.1760\.2080\.208Llama Fine\-tuned0\.8170\.8170\.7750\.7750\.8580\.8580\.1120\.1120\.1050\.1050\.1200\.120Qwen Fine\-tuned0\.8170\.8170\.7750\.7750\.8670\.8670\.1780\.1780\.1640\.1640\.1950\.195Llama Fine\-tuned \+ RAG0\.9580\.9251\.0000\.0990\.0900\.109Qwen Fine\-tuned \+ RAG0\.9500\.9500\.9250\.9251\.0001\.0000\.1280\.1280\.1190\.1190\.1380\.138The confidence intervals confirm two key findings\. First, the performance gains from combining fine\-tuning with RAG are statistically robust: both ChatPlanner configurations \(LLM Fine\-tuned \+ RAG\) achieve EM CIs whose lower bounds remain above 0\.92, with no overlap against any other configurations\. This rules out the possibility that the observed improvements are accidentally achieved due to a few simple examples\. Second, configurations with better performance also produce more stable results\. The best configuration has a CI width of only 0\.019 for MAE, whereas base LLMs without RAG exhibit CI widths roughly three times larger, meaning their performance fluctuates more depending on which particular test instances are included\. Together, these results provide statistical evidence that the combination of fine\-tuning and RAG is not only more accurate but also more reliable across different test samples\.

#### 4\.2\.5Task 3: Active Preferences Selection

In Node 3, the four predicted preference scores are used to select a subset of criteria to guide route planning\. The selection process consists of \(i\) filtering out criteria with scores below 0\.6, and \(ii\) selecting the top two highest\-scoring criteria among the remaining criteria \(or all remaining criteria if fewer than two remain after filtering\)\. Task 3 is evaluated from two complementary perspectives: \(i\) as a multi\-label classification problem and \(ii\) as a ranking preservation problem\.

##### \(I\) As a multi\-label classification problem

Each query can activate up to two criteria from accessibility, crowdedness, safety, and sightseeing, making this a multi\-label classification problem of predicting which subset of criteria should be selected\. Unlike standard classification, where each instance gets exactly one label, here each query can receive multiple labels simultaneously\. For instance, a user might care about both safety and accessibility\. This experiment assesses whether the model selects the correct subset using \(1\) multi\-label classification metrics\(Madjarovet al\.,[2012](https://arxiv.org/html/2606.15315#bib.bib68); Zhang and Zhou,[2013](https://arxiv.org/html/2606.15315#bib.bib67); Chicco and Jurman,[2020](https://arxiv.org/html/2606.15315#bib.bib66)\); and \(2\) calibration metrics\(Guoet al\.,[2017](https://arxiv.org/html/2606.15315#bib.bib65); Glenn and others,[1950](https://arxiv.org/html/2606.15315#bib.bib64)\), to evaluate whether the model’s confidence in its selections is well\-calibrated\.

The multi\-label classification metrics are defined as follows\.

Micro\-averaged F1 score \(Micro\-F1\)measures overall classification performance across all criteria by treating each criterion selection as an independent binary decision\. It is calculated as

Micro\-F1=2⋅Micro\-P⋅Micro\-RMicro\-P\+Micro\-R\\text\{Micro\-F1\}=\\frac\{2\\cdot\\text\{Micro\-P\}\\cdot\\text\{Micro\-R\}\}\{\\text\{Micro\-P\}\+\\text\{Micro\-R\}\}
where Micro\-P =∑j=1LT​Pj∑j=1L\(T​Pj\+F​Pj\)\\frac\{\\sum\_\{j=1\}^\{L\}TP\_\{j\}\}\{\\sum\_\{j=1\}^\{L\}\(TP\_\{j\}\+FP\_\{j\}\)\}and Micro\-R =∑j=1LT​Pj∑j=1L\(T​Pj\+F​Nj\)\\frac\{\\sum\_\{j=1\}^\{L\}TP\_\{j\}\}\{\\sum\_\{j=1\}^\{L\}\(TP\_\{j\}\+FN\_\{j\}\)\}, are micro\-averaged precision and recall\.LLis the number of criteria labels, andT​PjTP\_\{j\},F​PjFP\_\{j\},F​NjFN\_\{j\}are true positives, false positives, and false negatives for criterionjj\.

Intersection over Union \(Jaccard\)measures the similarity between predicted and true criteria sets, directly evaluating how well the model selects the correct combination of transportation criteria\.Y^i\\hat\{Y\}\_\{i\}andYiY\_\{i\}are predicted and true label set\. Jaccard is calculated as

Jaccard=1N​∑i=1N\|Y^i∩Yi\|\|Y^i∪Yi\|\\text\{Jaccard\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\frac\{\|\\hat\{Y\}\_\{i\}\\cap Y\_\{i\}\|\}\{\|\\hat\{Y\}\_\{i\}\\cup Y\_\{i\}\|\}
Exact match accuracy for label sets \(Subset Accuracy\)measures the proportion of instances where the model selects exactly the correct set of criteria, providing the strictest evaluation of criteria selection performance\.Y^i\\hat\{Y\}\_\{i\}andYiY\_\{i\}are the set of predicted and true criteria for instanceii, and𝟙​\[⋅\]\\mathbbm\{1\}\[\\cdot\]is the indicator function\. Subset Accuracy is calculated as

Subset Accuracy=1N​∑i=1N𝟙​\[Y^i=Yi\]\\text\{Subset Accuracy\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\mathbbm\{1\}\[\\hat\{Y\}\_\{i\}=Y\_\{i\}\]
Area Under the ROC Curve \(AUC\)evaluates the model’s ability to distinguish between criteria that should and should not be selected across different decision thresholds\. It is calculated using the trapezoidal rule for each label and then averaged\.

Matthews Correlation Coefficient \(MCC\)provides a robust measure under class imbalance, which is important given that not all criteria are equally likely to be selected\(Chicco and Jurman,[2020](https://arxiv.org/html/2606.15315#bib.bib66)\)\. It is calculated for each criterion and then averaged, as

MCC=T​P×T​N−F​P×F​N\(T​P\+F​P\)​\(T​P\+F​N\)​\(T​N\+F​P\)​\(T​N\+F​N\)\\text\{MCC\}=\\frac\{TP\\times TN\-FP\\times FN\}\{\\sqrt\{\(TP\+FP\)\(TP\+FN\)\(TN\+FP\)\(TN\+FN\)\}\}
Calibration metrics are defined below\.

Expected Calibration Error \(ECE\)measures the difference between predicted confidence and actual accuracy, with lower values indicating better calibration\(Guoet al\.,[2017](https://arxiv.org/html/2606.15315#bib.bib65)\)\. It is calculated as

ECE=∑m=1M\|Bm\|N​\|acc​\(Bm\)−conf​\(Bm\)\|\\text\{ECE\}=\\sum\_\{m=1\}^\{M\}\\frac\{\|B\_\{m\}\|\}\{N\}\|\\text\{acc\}\(B\_\{m\}\)\-\\text\{conf\}\(B\_\{m\}\)\|
whereMMis the number of confidence bins,BmB\_\{m\}is the set of samples in binmm,\|Bm\|\|B\_\{m\}\|is the number of samples in binmm,NNis the total number of samples,acc​\(Bm\)\\text\{acc\}\(B\_\{m\}\)andconf​\(Bm\)\\text\{conf\}\(B\_\{m\}\)are the accuracy and average confidence of samples in binmm\.

Brier Scoremeasures both calibration and sharpness of probability predictions, with lower values indicating better probabilistic predictions\(Glenn and others,[1950](https://arxiv.org/html/2606.15315#bib.bib64)\)\.pip\_\{i\}andyiy\_\{i\}are the predicted probability and actual binary outcome \(0 or 1\) for sampleii\. The Brier score is calculated as

Brier=1N​∑i=1N\(pi−yi\)2\\text\{Brier\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(p\_\{i\}\-y\_\{i\}\)^\{2\}
Table[10](https://arxiv.org/html/2606.15315#S4.T10)presents performance measured by the multi\-label classification metrics and calibration metrics for Task 3\.

Table 10:Multi\-label classification performance for criteria selection\.Labelmicro F1JaccardSubsetAccuracyAUCECEBrierLlama Base0\.5160\.5160\.4030\.4030\.1420\.1420\.5630\.5630\.5350\.5350\.5190\.519Qwen Base0\.4830\.4830\.3720\.3720\.1250\.1250\.5100\.5100\.5470\.5470\.5580\.558Llama Base \+ RAG0\.7260\.7260\.6690\.6690\.5500\.5500\.7690\.7690\.2720\.2720\.2780\.278Qwen Base \+ RAG0\.6500\.6500\.5690\.5690\.4500\.4500\.7430\.7430\.3280\.3280\.3030\.303Llama Fine\-tuned0\.7700\.7700\.7290\.7290\.6170\.6170\.8130\.8130\.2310\.2310\.2200\.220Qwen Fine\-tuned0\.6680\.6680\.6010\.6010\.5580\.5580\.7670\.7670\.2750\.2750\.2750\.275Llama Fine\-tuned \+ RAG0\.7810\.7460\.6750\.8480\.2170\.170Qwen Fine\-tuned \+ RAG0\.7320\.7320\.6820\.6820\.6080\.6080\.7950\.7950\.2310\.2310\.2150\.215The multi\-label classification metrics reveal a direct link between regression accuracy in Task 2 and criterion selection quality\. Configurations that better predict preference scores also more accurately select which criteria to activate\. Base LLMs perform at a near\-chance level, with subset accuracy below 0\.15, meaning nearly every request would trigger route planning with incorrect criteria\. RAG and fine\-tuning each improve performance substantially, but even with either component alone, roughly half of requests still select the wrong criteria combination\. The combined approach achieves the strongest results, with a subset accuracy of 0\.675 from the best configuration\. While not perfect, this is acceptable for the initial planning stage, as only about one\-third of requests would require adjustment through follow\-up interaction, either re\-ranking solutions or rerunning the route planning algorithm\.

Calibration metrics assess whether high\-confidence predictions are indeed more likely to be correct\. Base LLMs produce confidence estimates that are essentially uninformative, with ECE values above 0\.5\. Both RAG and fine\-tuning substantially improve calibration, with fine\-tuning being more effective of the two\. The combined approach again achieves the best results, producing the lowest ECE and Brier scores across all configurations\. This indicates that fine\-tuning and retrieval are complementary\. Fine\-tuning aligns the model’s internal confidence with actual correctness, while RAG provides grounding examples that further sharpen prediction reliability\.

##### \(II\) As a ranking preservation problem

Beyond selecting the correct criteria, the model should also preserve the relative ordering of user preferences\. For example, if a user cares more about safety \(scoring 0\.8\) than crowdedness \(scoring 0\.5\), the model should maintain this ranking even if both criteria are eventually filtered out or selected\. Ranking quality is measured using three rank correlation coefficients, as shown below\. All rank correlation metrics range from−1\-1to11, with higher values indicating better preservation of preference ordering\.

Spearman’s rank correlation coefficient \(Spearmanρ\\rho\)is a nonparametric metric that measures monotonic relationships between the predicted and true score rankings, robust to score rescaling\(Spearman,[1961](https://arxiv.org/html/2606.15315#bib.bib63)\)\. It is calculated as

ρ=1−6​∑di2n​\(n2−1\)\\rho=1\-\\frac\{6\\sum d\_\{i\}^\{2\}\}\{n\(n^\{2\}\-1\)\}
wherennis the number of criteria, anddid\_\{i\}is the difference between the ranks of criterioniiin the predicted and true scores\.

Kendall’s rank correlation coefficient \(Kendallτ\\tau\)provides an alternative measure of rank correlation that emphasizes pairwise rank agreements\(Kendall,[1938](https://arxiv.org/html/2606.15315#bib.bib62)\)\. It is calculated as

τ=nc−nd12​n​\(n−1\)\\tau=\\frac\{n\_\{c\}\-n\_\{d\}\}\{\\frac\{1\}\{2\}n\(n\-1\)\}
wherencn\_\{c\}andndn\_\{d\}are the number of concordant pairs \(where both the predicted and true rankings agree on relative order\) and discordant pairs\.

Pearson correlation coefficient \(Pearsonrr\)serves as a reference for linear relationships between predicted and true scores, though it is less robust to monotonic transformations than the nonparametric alternatives\(Lee Rodgers and Nicewander,[1988](https://arxiv.org/html/2606.15315#bib.bib53)\)\. It is calculated as

r=∑\(xi−x¯\)​\(yi−y¯\)∑\(xi−x¯\)2​∑\(yi−y¯\)2r=\\frac\{\\sum\(x\_\{i\}\-\\bar\{x\}\)\(y\_\{i\}\-\\bar\{y\}\)\}\{\\sqrt\{\\sum\(x\_\{i\}\-\\bar\{x\}\)^\{2\}\\sum\(y\_\{i\}\-\\bar\{y\}\)^\{2\}\}\}
To evaluate whether models preserve the relative ordering of user preference scores across criteria, rank correlation is examined throughρ\\rho,τ\\tau, andrr\. These metrics are presented in Table[11](https://arxiv.org/html/2606.15315#S4.T11)\.

Table 11:Ranking performance for criteria selection\.LabelSpearmanKendallPearsonLlama Base0\.1120\.1120\.1000\.1000\.0760\.076Qwen Base0\.1610\.1610\.1380\.1380\.1300\.130Llama Base \+ RAG0\.5630\.5630\.5290\.5290\.5660\.566Qwen Base \+ RAG0\.5560\.5560\.5180\.5180\.5460\.546Llama Fine\-tuned0\.7030\.7030\.6870\.6870\.7010\.701Qwen Fine\-tuned0\.6250\.6250\.6020\.6020\.6460\.646Llama Fine\-tuned \+ RAG0\.7170\.6910\.738Qwen Fine\-tuned \+ RAG0\.6300\.6300\.6040\.6040\.6560\.656Base LLMs show near\-zero correlations, meaning they largely fail to distinguish which criteria a user considers more important\. This is a critical limitation, since Task 3 depends on correctly prioritizing preferences\. Both RAG and fine\-tuning substantially improve ranking quality\. The combined approach again achieves the strongest performance, with all three correlation measures around 0\.7\. In practical terms, this means that when a user emphasizes safety over crowdedness, for example, the model consistently reflects this ordering in its predicted scores\. The results suggest that retrieval provides relevant preference context from similar past queries, while fine\-tuning teaches the model to assign and order criteria weights appropriately\.

In summary, the combination of fine\-tuning and RAG consistently achieves the best performance across all three tasks, with bootstrap analysis confirming these gains are statistically robust\. The two components serve complementary roles\. Fine\-tuning enforces structured output formatting and teaches the model to map natural language to preference scores, while RAG grounds predictions in query\-specific examples that resolve ambiguous or location\-specific expressions\.

#### 4\.2\.6Comparison Against a Keyword Matching Baseline

Rule\-based frameworks, such as bag\-of\-words and keyword matching, have long served as standard baselines in transportation research for extracting user preferences and network features from unstructured natural language data\(Aliet al\.,[2019](https://arxiv.org/html/2606.15315#bib.bib84)\)\. To directly assess whether the semantic understanding capacity of our ChatPlanner provides measurable gains over such approaches, a keyword matching baseline is developed, and both systems are evaluated on the same binary preference detection task\.

##### Keyword Matching System Design

The keyword matching system is designed as a binary relevance classifier reliant on a hand\-crafted, domain\-specific dictionary\.

\(i\)Accessibility\(25 keywords, e\.g\., “wheelchair”, “step\-free”, “elevator”\);

\(ii\)Safety\(31 keywords, e\.g\., “safe”, “well\-lit”, “crime”\);

\(iii\)Crowdedness\(31 keywords, e\.g\., “crowded”, “rush hour”, “personal space”\); and

\(iv\)Sightseeing\(39 keywords, e\.g\., “scenic”, “landmarks”, “architecture”\)\.

The complete lists of curated keywords for each preference are provided in[B](https://arxiv.org/html/2606.15315#A2)\.

The classifier employs a token\-level matching approach to prevent sub\-word matching errors inherent to basic regular expressions\(Zhanget al\.,[2018](https://arxiv.org/html/2606.15315#bib.bib86)\)\. Utilizing the Natural Language Toolkit \(NLTK\)\(Birdet al\.,[2009](https://arxiv.org/html/2606.15315#bib.bib85)\), the user’s journey request is first lowercased and segmented into discrete word tokens\. If the intersection between the user’s tokenized request and a predefined keyword list is non\-empty, that corresponding preference is activated \(assigned a value of11\); otherwise, it remains inactive \(assigned a value of0\)\.

##### Experimental Setup and Evaluation Metrics

To evaluate ChatPlanner against the keyword matching system in the binary preference detection task, the User Study Test Set described in Section[4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1)is adapted by binarizing the human\-labeled continuous ground\-truth scores\. A preference is considered active if its score is≥0\.6\\geq 0\.6, and inactive otherwise\. To ensure a fair comparison,0\.60\.6is chosen as the same threshold to convert the continuous preference scores generated by the LLM in ChatPlanner into binary active and inactive labels\.

Performance is evaluated using three standard metrics:

- •Exact Match Accuracy: The proportion of user requests where the model correctly predicts the binary activation states for all four criteria simultaneously\.
- •F1\-Score: The harmonic mean of precision and recall, providing a robust evaluation metric when positive and negative samples are highly imbalanced \(e\.g\., when a specific travel constraint is rarely requested\)\. It is mathematically formulated as: F1=2​T​P2​T​P\+F​P\+F​NF\_\{1\}=\\frac\{2TP\}\{2TP\+FP\+FN\}whereT​PTP,F​PFP, andF​NFNdenote the number of true positives, false positives, and false negatives, respectively\. We report both the overall F1\-score to capture global performance, and the per\-dimension F1\-scores to evaluate robustness across different preference types\.

##### Results and Discussion

Table[12](https://arxiv.org/html/2606.15315#S4.T12)presents the comparative results of the keyword matching baseline and our fine\-tuned LLMs equipped with RAG \(Llama Fine\-tuned \+ RAG and Qwen Fine\-tuned \+ RAG\)\. The table also reports Support, which is the number of active instances per criterion out of the 120 instances in the User Study Test Set\.

Table 12:Performance comparison between the Keyword Matching Baseline and ChatPlanner on binary preference detection\.Model / MetricOverall PerformancePer\-Criterion F1\-ScoresExact MatchAccuracyOverallF1\-ScoreAccessibilityCrowdednessSafetySightseeingKeyword Matching System Baseline0\.2420\.5600\.5900\.6190\.4600\.571Qwen Fine\-tuned \+ RAG0\.6500\.8330\.8000\.8330\.8480\.889Llama Fine\-tuned \+ RAG0\.7250\.8840\.8350\.9020\.8990\.957Support \(Active Instances\)\-\-41546324The results show that keyword matching fails to identify the complete set of user preferences in the majority of cases, while ChatPlanner achieves substantially higher performance across both overall and per\-criterion metrics\. The performance gap is most significant for criteria that users tend to express implicitly rather than through explicit keywords\. For example, a request such as "I am traveling with my grandmother" implies safety needs, and "I want a beautiful walk" implies sightseeing interest, but neither contains vocabulary that a keyword system would match, while the fine\-tuned LLM with RAG can correctly identify these preferences\. The per\-criterion F1\-scores confirm this pattern\. Criteria where implicit expression could be more common \(i\.e\., Safety and Sightseeing\) show the largest gains over the keyword baseline\. These findings demonstrate that the core value of integrating an LLM into the planning pipeline lies in its ability to interpret varied and indirect human expressions of travel preferences, ensuring the downstream routing solver optimizes for the correct objectives\.

### 4\.3Experiment 3: ChatPlanner Enlarge Solution Space for User Preference

ChatPlanner is compared against the baseline MC\-RAPTOR algorithm, which only considers minimizing travel time and the number of transfers in journey planning\. Case A focuses on mobility\-impaired users requiring accessibility considerations\. Case B examines commuters seeking less crowded routes\. Case C explores tourist preferences for routes with sightseeing opportunities\.

Table 13:Example user inquiries used for evaluation\.User CaseUser queryCase A: Mobility\-impaired peopleI’m traveling with a stroller and need elevator access\. I want to plan my journey from Farringdon to Bow Road by 12:00\.Case B: Crowding\-adverse commuterAs a daily commuter, I need spacious routes\. I need to get from Charing Cross Underground Station to Euston Station by 15:45\.Case C: TouristGetting around London: Sloane Square to Tate Britain by 10:00\. I prefer routes that pass interesting architecture\.##### Case A: Accessibility for Mobility\-Impaired Users

For Case A, the journey is from Farringdon to Bow Road with a requested arrival time of 12:00\. Figure[3](https://arxiv.org/html/2606.15315#S4.F3)shows the routes\. In the lower half figure, the baseline MC\-RAPTOR algorithm finds a direct path that goes through fewer stations, start\-A\-B\-C\-D\-E\-F\-G\-H\-End, but requires the user to exit at station H, which lacks step\-free access\. ChatPlanner \(upper half\) identifies an alternative solution with higher accessibility scores that suggests transferring at station I to reach another station J that both have step\-free access\. This route also ensures direct exiting at the end stop\. Traditional optimization methods would reject this option because it requires more time and transfers\. However, this alternative route provides essential value for users with limited mobility\.

![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/Case1.jpg)Figure 3:Case A: Route Planning with Accessibility Need![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/pareto_fronts/corrected_pareto_Case_1_Accessibility.png)\(a\)Case A: Pareto front comparison between ChatPlanner and MC\-RAPTOR baseline in travel time vs number of transfers space\.
![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/pareto_fronts/corrected_pareto_third_Case_1_Accessibility.png)\(b\)Case A: Pareto front in travel time vs accessibility, showing how ChatPlanner enlarges the solution space along accessibility\.

Figure 4:Case A: Pareto front comparisons for ChatPlanner\.
##### Case B: Crowding Avoidance for Commuters

In Case B, the user needs to travel from Charing Cross Underground Station to Euston Station by 15:45 while avoiding crowded areas\. Figures[5\(a\)](https://arxiv.org/html/2606.15315#S4.F5.sf1)and[5\(b\)](https://arxiv.org/html/2606.15315#S4.F5.sf2)compare two route options\. Route 1 shows higher crowding levels at most stops, while Route 2 maintains lower crowding throughout the journey\. Although Route 2 requires more stops and takes longer travel time, it successfully avoids crowded areas, providing a more comfortable travel experience for daily commuters who prioritize space over speed\.

![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/Case2_sol1.jpg)\(a\)Case B: Route Planning with Crowding Need,
![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/Case2_sol2.jpg)\(b\)Case B: Route Planning with Crowding Need

Figure 5:Case B: Route Planning with Crowding Need \(two solutions\)\.![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/pareto_fronts/corrected_pareto_Case_2_Crowding.png)\(a\)Case B: Pareto front comparison between ChatPlanner and MC\-RAPTOR baseline in travel time vs number of transfers space\.
![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/pareto_fronts/corrected_pareto_third_Case_2_Crowding.png)\(b\)Case B: Pareto front in travel time vs crowding space, showing how ChatPlanner enlarges the solution space along the crowding dimension\.

Figure 6:Case B: Pareto front comparisons for ChatPlanner\.
##### Case C: Sightseeing Opportunities for Tourists

In Case C, the journey goes from Sloane Square to Tate Britain by 10:00 for a tourist seeking interesting architecture\. Figures[7\(a\)](https://arxiv.org/html/2606.15315#S4.F7.sf1)and[7\(b\)](https://arxiv.org/html/2606.15315#S4.F7.sf2)compare different routing approaches\. The first figure shows a direct route that passes through segments \(A’–F’\) with few sightseeing opportunities within 100 meters, offering limited value for tourists\. The second figure demonstrates ChatPlanner’s tourist\-friendly alternative that prioritizes points of interest and areas with high sightseeing potential, even though it requires one transfer\. This route creates a more engaging journey \(A–E\) while still efficiently connecting the same start and end locations\.

![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/Case3_sol1.jpg)\(a\)Case C: Route Planning with Sightseeing Need
![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/Case3_sol2.jpg)\(b\)Case C: Route Planning with Sightseeing Need

Figure 7:Case C: Route Planning with Sightseeing Need \(two solutions\)\.![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/pareto_fronts/corrected_pareto_Case_3_Sightseeing.png)\(a\)Case C: Pareto front comparison between ChatPlanner and MC\-RAPTOR baseline in travel time vs number of transfers space\.
![Refer to caption](https://arxiv.org/html/2606.15315v1/figures/pareto_fronts/corrected_pareto_third_Case_3_Sightseeing.png)\(b\)Case C: Pareto front in travel time vs sightseeing space, showing how ChatPlanner enlarges the solution space along the sightseeing dimension\.

Figure 8:Case C: Pareto front comparisons for ChatPlanner\.Across all three cases, ChatPlanner successfully discovers and preserves valuable personalized solutions that address specific user needs\. Figures[4](https://arxiv.org/html/2606.15315#S4.F4),[6](https://arxiv.org/html/2606.15315#S4.F6), and[8](https://arxiv.org/html/2606.15315#S4.F8)present the Pareto front comparisons, showing that ChatPlanner expands the solution set along preference\-specific dimensions \(accessibility, crowding, sightseeing\) compared to the baseline MC\-RAPTOR algorithm\. These results confirm that the framework effectively helps travelers customize their journeys\.

#### 4\.3\.1Quantitative Analysis of Multi\-objective performance

While visual inspections demonstrate ChatPlanner’s ability to accommodate specific user preferences, this study provides a quantitative evaluation of the performance of the solutions to statistically demonstrate its advantage over the baseline\.

To evaluate across different travel scales, this study randomly sampled a subset of 30 journey requests to form our Routing Test Set\. To reflect the spatial distribution of the transit network, these cases are divided into three distance categories based on the direct distance between the origin and destination, leading to 10 short journeys \(<5<5km\), 10 medium journeys \(55\-1010km\), and 10 long journeys \(\>10\>10km\)\.

The baseline MC\-RAPTOR algorithm optimizes only two objectives: travel time and the number of transfers\. In contrast, ChatPlanner dynamically incorporates additional user\-specific preferences as objectives\. Post\-hoc evaluation can be applied to MC\-RAPTOR, allowing their solution sets to be analyzed and compared within a unified 6\-dimensional objective space\.

This study compares the Pareto fronts using three standard multi\-objective performance indicators\(Audetet al\.,[2021](https://arxiv.org/html/2606.15315#bib.bib124)\): the Coverage metric \(C\-metric\)\(Zitzler and Thiele,[2002](https://arxiv.org/html/2606.15315#bib.bib125)\), Inverted Generational Distance \(IGD\)\(Zhang and Li,[2007](https://arxiv.org/html/2606.15315#bib.bib126)\), and Hypervolume \(HV\)\(Zitzler and Thiele,[2002](https://arxiv.org/html/2606.15315#bib.bib125)\)\. The results across 30 journey cases are summarized in Table[14](https://arxiv.org/html/2606.15315#S4.T14)\.

C\-metric:This metric measures the proportion of solutions in one set that are dominated by at least one solution in another set\.C​\(A,B\)=1C\(A,B\)=1meansAAdominates all solutions inBB, whereasC​\(A,B\)=0C\(A,B\)=0meansAAdominates none\. As shown in Table[14](https://arxiv.org/html/2606.15315#S4.T14),C​\(RAPTOR,ChatPlanner\)=0\.00C\(\\text\{RAPTOR\},\\text\{ChatPlanner\}\)=0\.00across all 30 cases\. This indicates that the baseline algorithm never dominates any solution generated by ChatPlanner in the 6D space\. Conversely,C​\(ChatPlanner,RAPTOR\)C\(\\text\{ChatPlanner\},\\text\{RAPTOR\}\)is greater than zero in several cases \(mean = 0\.25\), demonstrating that ChatPlanner solutions frequently and strictly dominate the baseline solutions\.

IGDmeasures the distance from a reference Pareto front to the algorithm’s approximation set, evaluating both convergence and coverage\. Since the true Pareto front is unknown, we construct the reference set as the union of non\-dominated solutions from both MC\-RAPTOR and ChatPlanner\. ChatPlanner achieves a near\-perfect mean IGD of 0\.02, indicating that its solution set essentially constitutes the entire reference front\. In contrast, the baseline algorithm yields a mean IGD of 0\.88, remaining far from the true multi\-objective front due to its exclusive focus on time and transfers\.

HVmeasures the volume of the objective space dominated by the solution set; higher values indicate better diversity and optimality\. In the full 6\-dimensional objective space, ChatPlanner achieves substantially higher hypervolume than the baseline in 26 out of 30 cases, with a mean of 0\.44 compared to 0\.11 for the baseline\. In some cases, this improvement is considerable, with ChatPlanner achieving substantially higher hypervolume where the baseline performs poorly\. This result directly reflects ChatPlanner’s ability to explore multiple user\-specific criteria that MC\-RAPTOR does not, producing solutions that collectively dominate a much larger region of the objective space\.

Table 14:Per\-case comparison of MC\-RAPTOR \(R\) and ChatPlanner \(CP\)\.Pareto size\|𝒫\|\|\\mathcal\{P\}\|C\-metricIGD↓\\downarrowHV↑\\uparrowCaseRCPC​\(CP,R\)↑C\(\\text\{CP,R\}\)\{\\uparrow\}C​\(R,CP\)↓C\(\\text\{R,CP\}\)\{\\downarrow\}RCPRCPLong journeyslong\_11220\.000\.001\.490\.030\.010\.69long\_22160\.000\.000\.710\.000\.040\.13long\_3130\.000\.001\.000\.000\.010\.02long\_42231\.000\.000\.870\.000\.050\.23long\_5150\.000\.001\.050\.000\.010\.05long\_61170\.000\.001\.170\.000\.040\.44long\_72681\.000\.000\.730\.000\.070\.74long\_8181\.000\.001\.340\.000\.010\.85long\_92130\.000\.000\.580\.000\.110\.42long\_102471\.000\.000\.890\.000\.060\.42Medium journeysmedium\_1230\.500\.000\.530\.000\.010\.14medium\_22111\.000\.000\.880\.000\.160\.29medium\_32190\.000\.000\.820\.000\.050\.72medium\_41290\.000\.001\.160\.000\.040\.50medium\_5260\.000\.000\.650\.100\.020\.26medium\_61280\.000\.001\.220\.000\.010\.57medium\_7130\.000\.000\.980\.000\.000\.02medium\_82230\.000\.000\.790\.020\.080\.60medium\_9250\.000\.000\.460\.000\.020\.03medium\_10260\.000\.000\.700\.080\.050\.05Short journeysshort\_12330\.000\.000\.980\.020\.260\.56short\_21530\.000\.001\.080\.000\.721\.33short\_3180\.000\.000\.960\.000\.010\.46short\_4161\.000\.001\.300\.000\.010\.16short\_5240\.000\.000\.640\.180\.140\.10short\_6250\.000\.000\.740\.130\.130\.12short\_72120\.000\.000\.780\.090\.040\.68short\_8250\.500\.000\.540\.000\.170\.54short\_92180\.500\.000\.650\.000\.010\.36short\_10110\.000\.000\.000\.001\.771\.77Summary \(30 cases\)Mean1\.5716\.770\.250\.000\.880\.020\.110\.44Std0\.5016\.130\.390\.000\.340\.050\.330\.37Median2\.0010\.000\.000\.000\.820\.000\.040\.38Table 15:Average language processing time per user request across 30 test cases\.Model ConfigurationMean \(s\)Std DevMedian90th PercentileQwen Base \+ RAG4\.840\.124\.824\.98Qwen Base5\.140\.205\.145\.31Llama Base5\.300\.315\.225\.44Llama Base \+ RAG5\.800\.165\.806\.01Qwen Fine\-tuned8\.550\.368\.568\.85Llama Fine\-tuned9\.040\.328\.949\.54Qwen Fine\-tuned \+ RAG9\.140\.229\.129\.39Llama Fine\-tuned \+ RAG9\.760\.249\.7410\.14

### 4\.4Experiment 4: Latency and Computational Tractability

To assess the viability of ChatPlanner for real\-world deployment, the computation time of both blocks was evaluated for latency and computational tractability\. The ChatPlanner pipeline consists of two main computational blocks in the initial planning phase: the natural language processing block, where the LLM interprets the user’s request, and the public transit routing block, where the MC\-RAPTOR algorithm calculates paths\.

#### 4\.4\.1Language Processing Latency

First, this experiment measured the time required for the LLM to process a user’s natural language travel request\. This includes retrieving similar examples from the database \(RAG\), processing the text, and generating the structured output with the interpreted preference scores\. 30 user requests were randomly sampled from the User Study Test Set \(described in Section[4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1)\) and measured the processing time on our server with an NVIDIA A40 GPU for different model configurations\.

As shown in Table[15](https://arxiv.org/html/2606.15315#S4.T15), our best\-performing configuration from Experiment 2 \(Llama Fine\-tuned \+ RAG\) takes an average of 9\.76 seconds to process requests\. This processing time is considered reasonable and acceptable, as it falls within the established 10\-second usability threshold for keeping a user’s attention focused on a continuous dialogue task\(Nielsen,[1994](https://arxiv.org/html/2606.15315#bib.bib78)\)\. Furthermore, this running time can still be optimized significantly in real\-world product deployment settings through hardware scaling and server\-side memory management optimizations, such as continuous batching and PagedAttention\(Kwonet al\.,[2023](https://arxiv.org/html/2606.15315#bib.bib79)\)\.

#### 4\.4\.2Route Search Latency and Computational Tractability

While the language processing takes about 10 seconds, the majority of the system’s total response time is determined by the public transit routing algorithm\. MC\-RAPTOR is highly sensitive to the number of objectives considered\. The algorithm tracks and compares non\-dominated solutions at every transit stop\. Therefore, increasing the number of objectives causes an exponential increase in calculation time\.

To manage this computational burden, a core design choice in the ChatPlanner framework is the preference selection mechanism in Node 3, which restricts the number of active user\-defined criteria toK=2K=2\. To provide a principled computational justification for this choice, the sensitivity of the MC\-RAPTOR algorithm to the number of simultaneous objectives is analyzed\. Specifically, experiments are conducted by incrementally adding user preferences \(crowdedness, accessibility, safety, and sightseeing\) to the base routing objectives \(travel time and number of transfers\)\. The average execution time is measured across the same 30 journey cases evaluated in Section[4\.3\.1](https://arxiv.org/html/2606.15315#S4.SS3.SSS1)\(10 short, 10 medium, and 10 long journeys\)\. As shown in Table[16](https://arxiv.org/html/2606.15315#S4.T16), a standard routing search \(2 objectives\) requires an average of 12\.6 seconds\. Adding two additional user preferences increases the average search time to around 50 seconds\. However, expanding the number of objectives to 5 causes an immediate combinatorial explosion, pushing average computation times past 8 minutes\. Attempting to optimize all 6 criteria simultaneously results in complete algorithm failure, exhausting the 86GB RAM of an NVIDIA A40 server on long and medium journeys\.

#### 4\.4\.3Overall Deployment Feasibility

When combining the LLM processing time and the MC\-RAPTOR search time for aK=2K=2configuration, ChatPlanner requires an average end\-to\-end processing time of approximately 1 minute to generate its initial response\.

Compared to standard routing engines such as Google Maps, which primarily optimize only for travel time and compute paths in a few seconds, ChatPlanner’s latency is notably higher\. However, ChatPlanner acts as a personalized travel assistant, solving a highly complex 4\-objectives optimization problem tailored specifically to users’ preferences\. Research on human\-chatbot interaction suggests that users tolerate higher response times and even report increased trust when the system is perceived to be performing complex, personalized computation\(Gnewuchet al\.,[2018](https://arxiv.org/html/2606.15315#bib.bib82); Buell and Norton,[2011](https://arxiv.org/html/2606.15315#bib.bib80)\)\. Also, the current implementation is not optimized for running speed, and further reductions are achievable through code optimization, hardware scaling, and parallelizing independent route traversals across multiple cores\(Dellinget al\.,[2015](https://arxiv.org/html/2606.15315#bib.bib107)\)\. Importantly, the modular design of the ChatPlanner framework allows the routing solver to be replaced with any faster multi\-objective algorithm without modifying the rest of the pipeline\.

Table 16:Sensitivity analysis of MC\-RAPTOR computation time relative to the number of simultaneous optimization objectives\.Algorithm ConfigurationTotalObjectivesLong Journeys\(\>10\>10km\) \[s\]Medium Journeys\(5–10km\) \[s\]Short Journeys\(<5<5km\) \[s\]AverageTime \[s\]Base \(Time, Transfers\) \(K=0K=0\)217\.3216\.264\.1212\.57Base \+ 1 Preference \(K=1K=1\)343\.6942\.1141\.8942\.56ChatPlanner Initial Max \(K=2K=2\)464\.8343\.2543\.1650\.41Base \+ 3 Preferences \(K=3K=3\)5907\.27457\.91197\.73520\.97Base \+ 4 Preferences \(K=4K=4\)6−\-−\-857\.79−\-

## 5Conclusions

This research presents ChatPlanner, a novel framework for applying LLMs to public transit routing, addressing the critical gap in personalized route planning\. This approach integrates LLM capabilities into the public transit routing algorithm to capture users’ implicit and explicit preferences alongside traditional routing inputs extracted from natural language queries, such as origin, destination, and requested arrival time\. These extracted preferences guide the selection of criteria, which are then incorporated into the objective function of the public transit routing algorithm, enabling the generation of personalized, preference\-aware solution sets\.

Three experiments are conducted to validate the framework: solutions’ feasibility compared with tool\-augmented LLM, extraction of routing information and preferences, and solution set quality and completeness\.

Comparative analysis reveals that even tool\-augmented LLMs, such as GPT\-5\.0 equipped with external routing APIs including Google Map API, Moovit, and Rome2rio, cannot stably perform end\-to\-end public transit routing, achieving only one\-third feasibility rate, while ChatPlanner generates feasible solutions reliably\. These limitations of tool\-augmented LLMs stem from the inherent instability, as the model cannot always guarantee successful API calls\. The model cannot reliably resolve ambiguous location names across different external databases and could misinterpret transit information when parsing unstructured web layouts\. The model can also misreport travel durations and walking segments when summarizing route details from external sources\. ChatPlanner addresses these constraints by strictly limiting the LLM to natural language parsing and delegating the path planning to a routing algorithm, with both components sharing the same database of stop names and transit network data\.

Testing on an out\-of\-distribution, human\-written dataset, our evaluation across three user query extraction tasks validates the necessity of combining fine\-tuning with RAG\. Fine\-tuning with our designed input\-output pairs is essential for extracting origin, destination, and requested arrival time from natural language queries, while RAG is needed to provide query\-specific context that resolves imprecise, conversational human expressions for locations and times\. For preference interpretation, datasets incorporating eight personas and five contexts establish scoring standards for both fine\-tuning and RAG components\. The combination achieves better preference interpretation compared to either component alone\. Fine\-tuning learns preference patterns, and RAG provides query\-specific examples that guide the model toward consistent and accurate scoring aligned with the designed standards\.

Our case studies demonstrate that by capturing user preferences, ChatPlanner identifies valuable solutions across different criteria dimensions that existing route planners overlook\. From a user\-centric perspective, ChatPlanner provides a richer set of meaningful route alternatives\. From an algorithmic standpoint, this demonstrates how distinct objective functions operating in different search regions of the solution space provide differentiated Pareto frontiers, thereby expanding the solution set for personalized transit routing\.

## 6Future Directions

Looking ahead, the integration of LLMs in public transit systems warrants further investigation in several key areas\. First, enhanced solution algorithms could provide users with a broader range of valuable routing alternatives\. If alternative route planners employing different search mechanisms, such as evolutionary algorithms, can generate more diverse solution sets, the current ChatPlanner framework could similarly benefit from incorporating these approaches to expand its solution space\. Second, improving preference interpretation accuracy remains a critical research direction\. LLMs trained on general text may associate words in ways that do not match their meaning in a transportation context\. Developing techniques to strengthen domain\-specific understanding of transit terminology represents a promising avenue for more reliable LLM\-based transit planning systems\.

## Acknowledgments

The authors would like to thank \[Chinese Scholarship Council\] for their support and contributions to this research\.

## References

- F\. Ali, S\. El\-Sappagh, and D\. Kwak \(2019\)Fuzzy ontology and lstm\-based text mining: a transportation network monitoring system for assisting travel\.Sensors19\(2\),pp\. 234\.Cited by:[§4\.2\.6](https://arxiv.org/html/2606.15315#S4.SS2.SSS6.p1.1)\.
- \[2\]\(2025\)APIs\.OpenTripPlanner\.External Links:[Link](https://docs.opentripplanner.org/en/latest/apis/Apis/)Cited by:[§3\.3](https://arxiv.org/html/2606.15315#S3.SS3.p3.1)\.
- Apple Inc\. \(2025\)Apple maps\.Note:[https://maps\.apple\.com/](https://maps.apple.com/)Accessed: October 22, 2025Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p1.1)\.
- R\. Artstein and M\. Poesio \(2008\)Survey article: inter\-coder agreement for computational linguistics\.Computational linguistics34\(4\),pp\. 555–596\.Cited by:[§4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1.p3.1)\.
- C\. Audet, J\. Bigeon, D\. Cartier, S\. Le Digabel, and L\. Salomon \(2021\)Performance indicators in multiobjective optimization\.European journal of operational research292\(2\),pp\. 397–422\.Cited by:[§4\.3\.1](https://arxiv.org/html/2606.15315#S4.SS3.SSS1.p4.1)\.
- G\. Bajaj, R\. Agarwal, G\. Bouloukakis, P\. Singh, N\. Georgantas, and V\. Issarny \(2016\)Towards building real\-time, convenient route recommendation system for public transit\.In2016 IEEE International Smart Cities Conference \(ISC2\),Vol\.,pp\. 1–5\.External Links:[Document](https://dx.doi.org/10.1109/ISC2.2016.7580779)Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p3.1)\.
- J\. L\. Bezyak, S\. A\. Sabella, and R\. H\. Gattis \(2017\)Public transportation: an investigation of barriers for people with disabilities\.Journal of Disability Policy Studies28\(1\),pp\. 52–60\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1)\.
- S\. Bird, E\. Klein, and E\. Loper \(2009\)Natural language processing with python: analyzing text with the natural language toolkit\." O’Reilly Media, Inc\."\.Cited by:[§4\.2\.6](https://arxiv.org/html/2606.15315#S4.SS2.SSS6.Px1.p7.2)\.
- R\. Bommasani \(2021\)On the opportunities and risks of foundation models\.arXiv preprint arXiv:2108\.07258\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. Gasic \(2018\)MultiWOZ\-a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,pp\. 5016–5026\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.p1.1)\.
- R\. W\. Buell and M\. I\. Norton \(2011\)The labor illusion: how operational transparency increases perceived value\.Management Science57\(9\),pp\. 1564–1579\.Cited by:[§4\.4\.3](https://arxiv.org/html/2606.15315#S4.SS4.SSS3.p2.1)\.
- E\. Cascetta and A\. Cartenì \(2014\)The hedonic value of railways terminals\. a quantitative analysis of the impact of stations quality on travellers behaviour\.Transportation Research Part A: Policy and Practice61,pp\. 41–52\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1),[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p3.1)\.
- R\. Ceccato, F\. Deflorio, M\. Diana, and M\. Pirra \(2020\)Measure of urban accessibility provided by transport services in turin: a traveller perspective through a mobility survey\.Transportation Research Procedia45,pp\. 301–308\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1),[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1),[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p3.1)\.
- A\. A\. Ceder and Y\. Jiang \(2019\)Personalized public transport mobility service: a journey ranking approach for route guidance\.Transportation Research Procedia38,pp\. 935–955\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p3.1)\.
- H\. Cheng, Z\. Gong, and C\. Wang \(2025\)LLM\-tfp: integrating large language models with spatio\-temporal features for urban traffic flow prediction\.Applied Soft Computing,pp\. 113174\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p5.1)\.
- D\. Chicco and G\. Jurman \(2020\)The advantages of the matthews correlation coefficient \(mcc\) over f1 score and accuracy in binary classification evaluation\.BMC genomics21\(1\),pp\. 6\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p1.1),[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p11.1)\.
- E\. Cipriani, S\. Gori, and M\. Petrelli \(2012\)Transit network design: a procedure and an application to a large urban area\.Transportation Research Part C: Emerging Technologies20\(1\),pp\. 3–14\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p1.1)\.
- Citymapper \(2025\)Citymapper\.Note:[https://citymapper\.com/?lang=en](https://citymapper.com/?lang=en)Accessed: October 22, 2025Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p1.1)\.
- C\. F\. Daganzo and Y\. Ouyang \(2019\)Public transportation systems: principles of system design, operations planning and real\-time control\.World Scientific Publishing Co\.\.External Links:[Document](https://dx.doi.org/10.1142/10553),ISBN 978\-981\-3224\-10\-0,[Link](https://doi.org/10.1142/10553)Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p1.1)\.
- J\. De Vos, P\. L\. Mokhtarian, T\. Schwanen, V\. Van Acker, and F\. Witlox \(2016\)Travel mode choice and travel satisfaction: bridging the gap between decision utility and experienced utility\.Transportation43\(5\),pp\. 771–796\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1)\.
- A\. Delbosc and G\. Currie \(2012\)Modelling the causes and impacts of personal safety perceptions on public transport ridership\.Transport Policy24,pp\. 302–309\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1)\.
- D\. Delling, T\. Pajor, and R\. F\. Werneck \(2015\)Round\-based public transit routing\.Transportation Science49\(3\),pp\. 591–604\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p3.1),[§1](https://arxiv.org/html/2606.15315#S1.p6.1),[§2](https://arxiv.org/html/2606.15315#S2.p2.2),[§4\.4\.3](https://arxiv.org/html/2606.15315#S4.SS4.SSS3.p2.1)\.
- Y\. Disser, M\. Müller–Hannemann, and M\. Schnee \(2008\)Multi\-criteria shortest paths in time\-dependent train networks\.InInternational Workshop on Experimental and Efficient Algorithms,pp\. 347–361\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p2.2)\.
- R\. Dror, G\. Baumer, S\. Shlomov, and R\. Reichart \(2018\)The hitchhiker’s guide to testing statistical significance in natural language processing\.InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 1383–1392\.Cited by:[§4\.2\.4](https://arxiv.org/html/2606.15315#S4.SS2.SSS4.Px1.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv e\-prints,pp\. arXiv–2407\.Cited by:[§4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1.p4.1)\.
- K\. Esmukovet al\.\(2023\)Geopy: geocoding library for pythonExternal Links:[Link](https://github.com/geopy/geopy)Cited by:[§3\.1\.1](https://arxiv.org/html/2606.15315#S3.SS1.SSS1.p2.1)\.
- L\. Fan and C\. L\. Mumford \(2010\)A metaheuristic approach to the urban transit routing problem\.Journal of Heuristics16\(3\),pp\. 353–372\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p3.1)\.
- W\. Fan \(2004\)Optimal transit route network design problem: algorithms, implementations, and numerical results\.The University of Texas at Austin\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1)\.
- B\. Fang, Z\. Yang, S\. Wang, and X\. Di \(2024\)TraveLLM: could you plan my new public transit route in face of a network disruption?\.External Links:2407\.14926,[Link](https://arxiv.org/abs/2407.14926)Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p6.1)\.
- W\. B\. Glennet al\.\(1950\)Verification of forecasts expressed in terms of probability\.Monthly weather review78\(1\),pp\. 1–3\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p1.1),[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p17.3)\.
- U\. Gnewuch, S\. Morana, M\. Adam, and A\. Maedche \(2018\)Faster is not always better: understanding the effect of dynamic response delays in human\-chatbot interaction\.Cited by:[§4\.4\.3](https://arxiv.org/html/2606.15315#S4.SS4.SSS3.p2.1)\.
- Google \(2025\)Google maps\.Note:[https://www\.google\.com/maps/@51\.7716261,\-1\.2590835,14z](https://www.google.com/maps/@51.7716261,-1.2590835,14z)Accessed: October 22, 2025Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p1.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InInternational conference on machine learning,pp\. 1321–1330\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p1.1),[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p14.1)\.
- Z\. Han, C\. Gao, J\. Liu, J\. Zhang, and S\. Q\. Zhang \(2024\)Parameter\-efficient fine\-tuning for large models: a comprehensive survey\.arXiv preprint arXiv:2403\.14608\.Cited by:[§3\.2\.2](https://arxiv.org/html/2606.15315#S3.SS2.SSS2.p3.5)\.
- W\. Heaps, E\. Abramsohn, and E\. Skillen \(2021\)Public transportation in the us: a driver of health and equity\.Health Affairs Health Policy Brief10\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1)\.
- D\. Hörcher and A\. Tirachini \(2021\)A review of public transport economics\.Economics of transportation25,pp\. 100196\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§3\.2\.2](https://arxiv.org/html/2606.15315#S3.SS2.SSS2.p3.5)\.
- Y\. Huang, X\. Jin, M\. Fan, X\. Yang, and F\. Jiang \(2024\)Personalized route recommendation based on user habits for vehicle navigation\.InProceedings of the 2024 International Conference on Intelligent Driving and Smart Transportation,pp\. 28–32\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p4.1)\.
- R\. J\. Hyndman and A\. B\. Koehler \(2006\)Another look at measures of forecast accuracy\.International journal of forecasting22\(4\),pp\. 679–688\.Cited by:[§4\.2\.4](https://arxiv.org/html/2606.15315#S4.SS2.SSS4.p1.1)\.
- M\. Jakob, J\. Hrncir, L\. Oliva, F\. Ronzano, P\. Zilecky, and J\. Finnegan \(2014\)Personalized fully multimodal journey planner\.InECAI 2014,Frontiers in Artificial Intelligence and Applications, Vol\.263,pp\. 1225–1226\.External Links:[Document](https://dx.doi.org/10.3233/978-1-61499-419-0-1225)Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p3.1)\.
- R\. Jonnala, G\. Liang, J\. Yang, and I\. Alsmadi \(2024\)Using large language models in public transit systems, san antonio as a case study\.External Links:2407\.11003,[Link](https://arxiv.org/abs/2407.11003)Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p6.1)\.
- M\. G\. Kendall \(1938\)A new measure of rank correlation\.Biometrika30\(1\-2\),pp\. 81–93\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px2.p5.1)\.
- K\. Kepaptsoglou and M\. Karlaftis \(2009\)Transit route network design problem\.Journal of transportation engineering135\(8\),pp\. 491–505\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p1.1)\.
- T\. K\. Koo and M\. Y\. Li \(2016\)A guideline of selecting and reporting intraclass correlation coefficients for reliability research\.Journal of chiropractic medicine15\(2\),pp\. 155–163\.Cited by:[§4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1.p3.1)\.
- I\. Kostric, K\. Balog, and F\. Radlinski \(2024\)Generating usage\-related questions for preference elicitation in conversational recommender systems\.ACM Transactions on Recommender Systems2\(2\),pp\. 1–24\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p4.1)\.
- W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with pagedattention\.InProceedings of the 29th symposium on operating systems principles,pp\. 611–626\.Cited by:[§4\.4\.1](https://arxiv.org/html/2606.15315#S4.SS4.SSS1.p2.1)\.
- I\. Lawrence and K\. Lin \(1989\)A concordance correlation coefficient to evaluate reproducibility\.Biometrics,pp\. 255–268\.Cited by:[§4\.2\.4](https://arxiv.org/html/2606.15315#S4.SS2.SSS4.p8.1)\.
- J\. Lee Rodgers and W\. A\. Nicewander \(1988\)Thirteen ways to look at the correlation coefficient\.The American Statistician42\(1\),pp\. 59–66\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px2.p8.1)\.
- P\. Lewis, E\. Perez, A\. Piktus, F\. Petroni, V\. Karpukhin, N\. Goyal, H\. Küttler, M\. Lewis, W\. Yih, T\. Rocktäschel,et al\.\(2020\)Retrieval\-augmented generation for knowledge\-intensive nlp tasks\.Advances in neural information processing systems33,pp\. 9459–9474\.Cited by:[§3\.2\.3](https://arxiv.org/html/2606.15315#S3.SS2.SSS3.p1.1)\.
- M\. Liang, M\. Xu, and S\. Wang \(2025\)A novel multi\-objective evolutionary algorithm for transit network design and frequency\-setting problem considering passengers’ choice behaviors under station congestion\.Transportation Research Part B: Methodological197,pp\. 103238\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p1.1)\.
- Y\. Lin, X\. Na, D\. Wang, X\. Dai, and F\. Wang \(2023\)Mobility 5\.0: smart logistics and transportation services in cyber\-physical\-social systems\.IEEE Transactions on Intelligent Vehicles8\(6\),pp\. 3527–3532\.External Links:[Document](https://dx.doi.org/10.1109/TIV.2023.3286995)Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- A\. Loukaitou\-Sideris \(2014\)Fear and safety in transit environments from the women’s perspective\.Security journal27\(2\),pp\. 242–256\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1),[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p3.1)\.
- B\. Ludwig, B\. Zenker, and J\. Schrader \(2009\)Recommendation of personalized routes with public transport connections\.InInternational Conference on Intelligent Interactive Assistance and Mobile Multimedia Computing,pp\. 97–107\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p3.1)\.
- P\. J\. Mackie, S\. Jara\-Dıaz, and A\. S\. Fowkes \(2001\)The value of travel time savings in evaluation\.Transportation Research Part E: Logistics and Transportation Review37\(2\-3\),pp\. 91–106\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1)\.
- G\. Madjarov, D\. Kocev, D\. Gjorgjevikj, and S\. Džeroski \(2012\)An extensive experimental comparison of methods for multi\-label learning\.Pattern recognition45\(9\),pp\. 3084–3104\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p1.1)\.
- K\. O\. McGraw and S\. P\. Wong \(1996\)Forming inferences about some intraclass correlation coefficients\.\.Psychological methods1\(1\),pp\. 30\.Cited by:[§4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1.p3.1)\.
- B\. Molner, K\. D\. Curtin, K\. Kongats, J\. A\. Brown, M\. Thomson, K\. Atkey, and C\. I\. Nykiforuk \(2023\)Policy influencer support for active transportation policy in two canadian provinces: implications for advocacy\.Journal of Transport & Health28,pp\. 101537\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1)\.
- Moovit Inc\. \(2025\)Moovit: multimodal mobility and transit routing\.Note:2025\-9\-15External Links:[Link](https://moovitapp.com/)Cited by:[§4\.1](https://arxiv.org/html/2606.15315#S4.SS1.p2.1)\.
- National Centre for Accessible Transport \(ncat\) \(2024\)Understanding and identifying barriers to accessing transport: the experiences of disabled people in the uk\.Technical reportNational Centre for Accessible Transport\.Note:Accessed: 2025\-10\-30External Links:[Link](https://www.ncat.uk/wp-content/uploads/2024/12/ncat-Understanding-and-identifying-barriers-to-accessing-transport-Full-Report-Accessible-PDF-FINAL-1.pdf)Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1)\.
- T\. Nie, J\. Sun, and W\. Ma \(2025\)Exploring the roles of large language models in reshaping transportation systems: a survey, framework, and roadmap\.Artificial Intelligence for Transportation1,pp\. 100003\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p8.1)\.
- J\. Nielsen \(1994\)Usability engineering\.Morgan Kaufmann\.Cited by:[§4\.4\.1](https://arxiv.org/html/2606.15315#S4.SS4.SSS1.p2.1)\.
- A\. Pommeranz, J\. Broekens, P\. Wiggers, W\. Brinkman, and C\. M\. Jonker \(2012\)Designing interfaces for explicit preference elicitation: a user\-centered investigation of preference representation and elicitation process\.User Modeling and User\-Adapted Interaction22\(4\),pp\. 357–397\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p4.1)\.
- E\. Pyrga, F\. Schulz, D\. Wagner, and C\. Zaroliagis \(2008\)Efficient models for timetable information in public transportation systems\.Journal of Experimental Algorithmics \(JEA\)12,pp\. 1–39\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p2.2)\.
- Qwen, :, A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Tang, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, and Z\. Qiu \(2025\)Qwen2\.5 technical report\.External Links:2412\.15115,[Link](https://arxiv.org/abs/2412.15115)Cited by:[§4\.2\.1](https://arxiv.org/html/2606.15315#S4.SS2.SSS1.p4.1)\.
- P\. Rajpurkar, J\. Zhang, K\. Lopyrev, and P\. Liang \(2016\)SQuAD: 100,000\+ questions for machine comprehension of text\.InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pp\. 2383–2392\.Cited by:[§4\.2\.3](https://arxiv.org/html/2606.15315#S4.SS2.SSS3.p2.1)\.
- A\. Rapp, C\. Di Lodovico, and L\. Di Caro \(2025\)How do people react to chatgpt’s unpredictable behavior? anthropomorphism, uncanniness, and fear of ai: a qualitative study on individuals’ perceptions and understandings of llms’ nonsensical hallucinations\.International Journal of Human\-Computer Studies198,pp\. 103471\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- A\. Rastogi, X\. Zang, S\. Sunkara, R\. Gupta, and P\. Khaitan \(2020\)Towards scalable multi\-domain conversational agents: the schema\-guided dialogue dataset\.InProceedings of the AAAI conference on artificial intelligence,Vol\.34,pp\. 8689–8696\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.p1.1)\.
- Rome2Rio Pty Ltd\. \(2025\)Rome2Rio: global multi\-modal transport search engine\.Note:2025\-09\-15External Links:[Link](https://www.rome2rio.com/)Cited by:[§4\.1](https://arxiv.org/html/2606.15315#S4.SS1.p2.1)\.
- C\. Roncoli, E\. Chandakas, and I\. Kaparias \(2023\)Estimating on\-board passenger comfort in public transport vehicles using incomplete automatic passenger counting data\.Transportation Research Part C: Emerging Technologies146,pp\. 103963\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1)\.
- Y\. Rong, Y\. Mao, H\. Cui, X\. He, and M\. Chen \(2024\)Edge computing enabled large\-scale traffic flow prediction with gpt in intelligent autonomous transport system for 6g network\.IEEE Transactions on Intelligent Transportation Systems\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p5.1)\.
- \[71\]ScienceDirect TopicsPublic transport\.Note:[https://www\.sciencedirect\.com/topics/social\-sciences/public\-transport](https://www.sciencedirect.com/topics/social-sciences/public-transport)Accessed: 2025\-11\-04Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p1.1)\.
- M\. Shamsuddoha, M\. A\. Kashem, and T\. Nasir \(2025\)A review of transportation 5\.0: advancing sustainable mobility through intelligent technology and renewable energy\.Future Transportation5\(1\),pp\. 8\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- A\. Singh, A\. Fry, A\. Perelman, A\. Tart, A\. Ganesh, A\. El\-Kishky, A\. McLaughlin, A\. Low, A\. Ostrow, A\. Ananthram,et al\.\(2025\)Openai gpt\-5 system card\.arXiv preprint arXiv:2601\.03267\.Cited by:[§4\.1](https://arxiv.org/html/2606.15315#S4.SS1.p2.1)\.
- C\. Spearman \(1961\)The proof and measurement of association between two things\.\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px2.p2.1)\.
- \[75\]\(2025\)Step free map\.Transport for London\.External Links:[Link](https://tfl.gov.uk/info-for/open-data-users/our-open-data)Cited by:[§3\.3](https://arxiv.org/html/2606.15315#S3.SS3.p3.1)\.
- The Mayor of London \(2023\)Violent incidents on the London Underground\.London Assembly\.External Links:[Link](https://www.london.gov.uk/who-we-are/what-london-assembly-does/questions-mayor/find-an-answer/violent-incidents-london-underground-1)Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1)\.
- R\. J\. Tibshirani and B\. Efron \(1993\)An introduction to the bootstrap\.Monographs on statistics and applied probability57\(1\),pp\. 1–436\.Cited by:[§4\.2\.4](https://arxiv.org/html/2606.15315#S4.SS2.SSS4.Px1.p1.1)\.
- A\. Tirachini, D\. A\. Hensher, and J\. M\. Rose \(2013\)Crowding in public transport systems: effects on users, operation and implications for the estimation of demand\.Transportation research part A: policy and practice53,pp\. 36–52\.Cited by:[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- M\. Wallace and G\. Dunlop \(2016\)ELIZA: a very basic rogerian psychotherapist chatbot\.Retrieved July10,pp\. 2019\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- J\. Wang and A\. Shalaby \(2024\)Leveraging large language models for enhancing public transit services\.arXiv preprint arXiv:2410\.14147\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p6.1)\.
- L\. Wang, P\. Duan, Z\. He, C\. Lyu, X\. Chen, N\. Zheng, L\. Yao, and Z\. Ma \(2025\)Agentic large language models for day\-to\-day route choices\.Transportation Research Part C: Emerging Technologies180,pp\. 105307\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p5.1)\.
- R\. Wang, M\. Zhou, K\. Gao, A\. Alabdulwahab, and M\. J\. Rawa \(2021\)Personalized route planning system based on driver preference\.Sensors22\(1\),pp\. 11\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p4.1)\.
- webdemia \(2024\)Mobipar\.External Links:[Link](https://mobipar.it/en/https-mobipar-it-en-mobility-5-0-future-transport/)Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p4.1)\.
- S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. Nie \(2024\)C\-pack: packed resources for general chinese embeddings\.InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval,pp\. 641–649\.Cited by:[§3\.2\.3](https://arxiv.org/html/2606.15315#S3.SS2.SSS3.p2.4)\.
- M\. Zhang and Z\. Zhou \(2013\)A review on multi\-label learning algorithms\.IEEE transactions on knowledge and data engineering26\(8\),pp\. 1819–1837\.Cited by:[§4\.2\.5](https://arxiv.org/html/2606.15315#S4.SS2.SSS5.Px1.p1.1)\.
- Q\. Zhang and H\. Li \(2007\)MOEA/d: a multiobjective evolutionary algorithm based on decomposition\.IEEE Transactions on evolutionary computation11\(6\),pp\. 712–731\.Cited by:[§4\.3\.1](https://arxiv.org/html/2606.15315#S4.SS3.SSS1.p4.1)\.
- Z\. Zhang, Q\. He, J\. Gao, and M\. Ni \(2018\)A deep learning approach for detecting traffic accidents from social media data\.Transportation research part C: emerging technologies86,pp\. 580–596\.Cited by:[§4\.2\.6](https://arxiv.org/html/2606.15315#S4.SS2.SSS6.Px1.p7.2)\.
- Y\. Zhao, P\. Wang, Y\. Zhao, H\. Du, and H\. F\. Yang \(2025\)SafeTraffic copilot: adapting large language models for trustworthy traffic safety assessments and decision interventions\.Nature Communications16\(1\),pp\. 8846\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p5.1)\.
- H\. Zhou, X\. Chi, R\. Norman, Y\. Zhang, and C\. Song \(2024\)Tourists’ urban travel modes: choices for enhanced transport and environmental sustainability\.Transportation Research Part D: Transport and Environment129,pp\. 104144\.Cited by:[§1](https://arxiv.org/html/2606.15315#S1.p2.1),[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p2.1),[§3\.2\.1](https://arxiv.org/html/2606.15315#S3.SS2.SSS1.Px1.p3.1)\.
- E\. Zitzler and L\. Thiele \(2002\)Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach\.IEEE transactions on Evolutionary Computation3\(4\),pp\. 257–271\.Cited by:[§4\.3\.1](https://arxiv.org/html/2606.15315#S4.SS3.SSS1.p4.1)\.
- K\. G\. Zografos, K\. N\. Androutsopoulos, and J\. D\. Nelson \(2010\)Identifying travelers’ information needs and services for an integrated international real time journey planning system\.In13th International IEEE Conference on Intelligent Transportation Systems,pp\. 998–1004\.Cited by:[§2](https://arxiv.org/html/2606.15315#S2.p3.1)\.

## Appendix AEmpirical Justification of the Activation Threshold

To determine the appropriate threshold for converting continuous LLM preference predictions into active routing criteria \(Node 3\), we analyzed the default output behavior of the models\. We evaluated 30 semantically neutral journey requests\. These requests contained only basic travel information \(origin, destination, and arrival time\) without any explicit or implicit preference keywords\. An example of such a request is:“I need to get from Temple Fortune Lane to Suffolk Road by 10:54\.”

Table[17](https://arxiv.org/html/2606.15315#A1.T17)presents the average preference scores predicted by different model configurations across these 30 neutral sentences\. The results show that LLMs do not output scores of zero for neutral inputs\. Base models often generate inaccurately high scores, such as Llama Base predicting an average safety score of 0\.961\.

While the combination of fine\-tuning and RAG successfully reduces these extreme values, the average predicted scores stabilize around 0\.5 rather than 0\.0\. Because the default output for a neutral request naturally centers around 0\.5, setting an activation threshold is a necessary structural requirement to separate the LLM’s default output values from actively expressed travel preferences\. This threshold is set to 0\.6 to ensure the route planning algorithm only optimizes for criteria the user actually cares about\.

Table 17:Average LLM preference score predictions across 30 semantically neutral test sentences\.Model SetupAccessibilityCrowdednessSafetySightseeingLlama Base0\.8350\.4270\.9610\.615Qwen Base0\.9000\.6760\.9480\.250Llama Base \+ RAG0\.6760\.4160\.5100\.046Qwen Base \+ RAG0\.6100\.6230\.6300\.020Llama FT0\.5260\.5590\.5780\.477Qwen FT0\.5440\.5440\.5760\.441Llama FT \+ RAG0\.5460\.5330\.4860\.516Qwen FT \+ RAG0\.5440\.5260\.4830\.510
## Appendix BCurated Keyword Dictionary for Baseline Classifier

This section details the complete sets of curated keywords utilized by the rule\-based baseline system \(described in Section[4\.2\.6](https://arxiv.org/html/2606.15315#S4.SS2.SSS6)\) to detect user travel preferences from natural language requests\. The dictionary consists of 126 unique natural language triggers and semantic proxies, categorized into four core transportation preference dimensions\. The keywords for each dimension are presented below in alphabetical order:

- •Accessibility \(25 keywords\):access needs, accessible, accessible platform, assistance, barrier\-free, carrying, disabled, easy access, elevator, heavy bags, lift, luggage, mobility, mobility equipment, physical, platform, ramp, stairs, step free, step\-free, stroller, suitcase, walking, walking distance, wheelchair\.
- •Safety \(31 keywords\):alone, area safety, avoid, bright, busy area, cctv, crime, dangerous, dark, emergency, feel secure, gangster, late night, lighting, main road, monitored, neighborhood, night time, personal security, police, populated, risk, safe, safety, secure, security, threat, unsafe, visible, well lit, well\-lit\.
- •Crowdedness \(31 keywords\):anxiety, avoid crowds, breathing room, busy, claustrophobic, comfortable, congested, crowd, crowded, empty, full, jam, less busy, less crowded, not busy, overwhelming, packed, peak hour, personal space, quiet, room, rush, rush hour, sardines, seat, seating, space, spacious, standing, stress, uncrowded\.
- •Sightseeing \(39 keywords\):above ground, architecture, art, attraction, attractions, beautiful, bridge, cathedral, church, cultural, culture, enjoyable, experience, explore, exploring, gallery, garden, historic, holiday, interesting, journey, landmarks, leisure, monument, museum, overground, palace, park, river, scenic, sights, sightseeing, street view, tourism, tourist, vacation, view, views, window\.

## Appendix CMulti\-turn User Request Parsing

The evaluations in Section[4\.2](https://arxiv.org/html/2606.15315#S4.SS2)assessed the parsing component on single\-turn requests from the User Study Test Set\. In practice, users may also interact with the system across multiple turns, for example, omitting key trip details initially or revising their preferences during the conversation\. To evaluate parsing performance under these conditions, an additional multi\-turn test set was constructed\. Each request in the User Study Test Set was randomly assigned with a fixed seed to four dialogue patterns, with each base request used only once\.

### C\.1Multi\-turn Test Set Construction

The multi\-turn test set was created by rule\-based transformations of the original single\-turn requests and their structured labels\. Four dialogue patterns were considered: \(i\) requests with the origin omitted, \(ii\) requests with the requested arrival time omitted, \(iii\) requests with both origin and requested arrival time omitted, and \(iv\) complete requests followed by later preference revisions\. For incomplete\-information cases \(i\-iii\), the missing fields were removed from the initial request and then supplied in subsequent user turns\. For the preference\-revision cases \(iv\), the original request was preserved and followed by short, templated updates \(e\.g\., ’I care more about safety’\), and the corresponding preference score in the ground\-truth output was set to 1\.0\. An example dialogue is shown in Figure[9](https://arxiv.org/html/2606.15315#A3.F9)\. The Multi\-turn Test Set should be interpreted as a controlled test of conversational parsing under common interaction patterns, rather than a substitute for naturally collected multi\-turn dialogues, which is left for future work\.

System instruction \(abridged\):“You are a strict information extractor for public transit requests\. If start\_stop, end\_stop, requested\_arrival\_time, or preference is missing, ask one short clarification question\. If all are provided, output only one JSON object …”Turn 1User:“I want to get to Stratford from the Sainsbury’s in Whitechapel by 2 pm\. I am carrying groceries, so please make sure the route doesn’t have crowded spaces\.”Assistant \(ChatPlanner response\):\{\.\.\.\}Turn 2User:“I care more about safety\.”Assistant \(ChatPlanner response\):\{\.\.\.\}Turn 3User:“Please avoid crowds more\.”Assistant \(ChatPlanner response\):\{\.\.\.\}Ground\-truth \(for evaluation\):\{"start\_stop": "Whitechapel", "end\_stop": "Stratford",
"requested\_arrival\_time": "14:00:00",
"scores":\{"accessibility": 0\.7, "crowdedness": 1,
"safety": 1, "sightseeing": 0\.2\}\}Figure 9:Each dialogue instance consists of the user’s messages across turns and a ground\-truth structured output used for evaluation\. The system instruction shown at the top is shared across all dialogues and is not part of any individual dialogue\. During evaluation, the model generates its own response at each assistant turn, and the final output is compared against the ground\-truth\.
### C\.2Evaluation Setup

Each configuration was evaluated by replaying the Multi\-turn Test Set dialogue one turn at a time\. The system prompt and user messages were provided sequentially, and whenever an assistant turn was reached, the LLM was asked to generate its own response based on the preceding dialogue history\. The LLM reply was then added to the dialogue history before the next user turn was provided\. In this way, the evaluation tests whether the LLM can ask for missing information when needed and produce the final structured output once sufficient information has been supplied\. The same LLM configurations as in the single\-turn experiments in Section[4\.2](https://arxiv.org/html/2606.15315#S4.SS2)were tested here\. To maintain comparability, the RAG\-based configurations used the same retrieval library as in the single\-turn evaluation\.

### C\.3Multi\-turn Evaluation Metrics

The same metrics as above are reported: EM\_start, EM\_end, EM\_time, EM\_all, MAE, Subset Accuracy, and Pearson correlation coefficient\. In addition, ARF\_turn0 \(Ask\-Right\-Field Accuracy at the first assistant turn\) is introduced, which measures whether the LLM’s first clarification question asks for the field or fields that are missing from the request\.

### C\.4Results and Discussions

Table 18:Multi\-turn performance on the Multi\-turn Test Set\.LabelEMstartEMendEMtimeEMallMAESubsetAccuracyPearsonARFturn0Llama Base0\.4580\.4580\.4170\.4170\.4420\.4420\.3420\.3420\.3900\.3900\.0580\.0580\.1030\.1030\.3330\.333Qwen Base0\.3580\.3580\.3420\.3420\.4170\.4170\.2580\.2580\.3980\.3980\.0170\.0170\.1330\.1330\.4330\.433Llama Base \+ RAG0\.6170\.6170\.5080\.5080\.6420\.6420\.4170\.4170\.2700\.2700\.3420\.3420\.5060\.5060\.5000\.500Qwen Base \+ RAG0\.6420\.6420\.5830\.5830\.6830\.6830\.5420\.5420\.2780\.2780\.3170\.3170\.4390\.4390\.5000\.500Llama Fine\-tuned0\.6420\.6420\.5920\.5920\.7000\.7000\.5000\.5000\.2650\.2650\.4000\.4000\.5110\.5110\.6170\.617Qwen Fine\-tuned0\.6580\.6580\.6170\.6170\.7250\.7250\.5670\.5670\.2740\.2740\.3920\.3920\.4510\.4510\.6500\.650Llama Fine\-tuned \+ RAG0\.7420\.7420\.7500\.8330\.8330\.6580\.6580\.2260\.2260\.5830\.5830\.5390\.7330\.733Qwen Fine\-tuned \+ RAG0\.7580\.7330\.7330\.8580\.6750\.2190\.5920\.5130\.5130\.767The results in Table[18](https://arxiv.org/html/2606.15315#A3.T18)show that the same performance hierarchy observed in the single\-turn experiments holds in the multi\-turn setting\. Fine\-tuning combined with RAG consistently outperforms all other configurations across all metrics\. For ARF\_turn0, the best configuration correctly identifies the missing field\(s\) in more than 75% of cases, indicating the ChatPlanner’s configuration can often recognize what information is lacking, while there is clear room for improvement\. Overall, performance is lower than in the single\-turn setting, which is expected given the added complexity of handling clarification and preference revision across multiple turns, but the results still demonstrate that the parsing component can function in a conversational setting\. This performance could be further improved through several potential approaches, such as incorporating multi\-turn dialogue examples into the fine\-tuning data, collecting and training on real user conversation dialogues from a deployed system, and so on\.