车载对话助手多轮对话的自动化评测

# Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants
Source: [https://arxiv.org/html/2609.35812](https://arxiv.org/html/2609.35812)
Vaishnav NegiAffiliation:Friedrich\-Alexander\-Universität Erlangen\-Nürnberg, Erlangen, Germanyvaishnav\.negi@fau\.deAffiliation:BMW Group, GermanyLev SorokinAffiliation:BMW Group, GermanyAffiliation:Technical University of Munich, Munich, Germanylev\.sorokin@tum\.de \| andrea\.stocco@tum\.deSoroosh Tayebi ArastehAffiliation:Friedrich\-Alexander\-Universität Erlangen\-Nürnberg, Erlangen, Germanyvaishnav\.negi@fau\.deAffiliation:RWTH Aachen University, Aachen, Germanysoroosh\.arasteh@rwth\-aachen\.deAndrea StoccoAffiliation:Technical University of Munich, Munich, Germanylev\.sorokin@tum\.de \| andrea\.stocco@tum\.deAffiliation:fortiss GmbH, Munich, Germanystocco@fortiss\.org

###### Abstract

In\-car conversational assistants \(ICAs\) are increasingly integrated into vehicles to support route planning, vehicle control, and information access\. Ensuring their reliability is challenging due to multi\-turn interactions, the absence of explicit ground truth, and strict safety constraints\. Existing evaluation techniques fall short, as they target single\-turn settings and fail to capture constraint handling, context retention, and safety\-critical behavior across turns\. We propose an automated framework for testing the multi\-turn conversational capabilities of ICAs\. The system is treated as a black box and evaluated via closed\-loop simulation with a strategy\-guided user simulator, an adversarial strategy manager, and a two\-tier LLM judge assessing turn\-level failures and conversation\-level quality\. We evaluate the approach on an industrial ICA with six LLM backends and twelve human annotators\. The automated judge shows substantial agreement with humans, and strategy guidance uncovers 2\.96×\\timesmore unique failure types per conversation and more than doubles the number of unique failing conversations compared to unguided simulation\.

## IIntroduction

In\-car conversational assistants \(ICAs\) based on Large Language Models \(LLMs\) are becoming an integral part of modern vehicles, facilitating voice interaction for navigation, climate control, media, and vehicle information retrieval\[[1](https://arxiv.org/html/2609.35812#bib.bib6),[2](https://arxiv.org/html/2609.35812#bib.bib7),[3](https://arxiv.org/html/2609.35812#bib.bib16),[4](https://arxiv.org/html/2609.35812#bib.bib17),[5](https://arxiv.org/html/2609.35812#bib.bib20)\]\. However, these systems require higher reliability due to their critical application domain\[[6](https://arxiv.org/html/2609.35812#bib.bib18),[7](https://arxiv.org/html/2609.35812#bib.bib19)\]\. A violated constraint or an overlooked contextual feature during interaction may reduce driver trust and compromise vehicle safety\[[1](https://arxiv.org/html/2609.35812#bib.bib6)\]\.

Assessing the reliability of such systems automatically, however, poses three interrelated challenges\. Firstly, such dialogues aremulti\-turn, and users may refine their requests and constraints incrementally, such that errors may not be apparent immediately, especially due to long\-range dependencies\[[8](https://arxiv.org/html/2609.35812#bib.bib11),[9](https://arxiv.org/html/2609.35812#bib.bib14)\]\. Secondly, there isno unique ground truthas navigation requests may have multiple correct answers, and thus, quality is determined based on task completion, satisfaction of constraints, and safety, rather than similarity to a benchmark\[[10](https://arxiv.org/html/2609.35812#bib.bib10),[11](https://arxiv.org/html/2609.35812#bib.bib9)\]\. Thirdly, such dialogues require special attention todomain\-specific constraints, where failures during route planning or adhering to an unsafe driver request may compromise safety\[[12](https://arxiv.org/html/2609.35812#bib.bib8)\]\.

Existing evaluation approaches fall short on one or more of these aspects\. Reference\-based metrics such as BLEU\[[13](https://arxiv.org/html/2609.35812#bib.bib3)\]correlate poorly with human judgment for open\-domain dialogue\[[10](https://arxiv.org/html/2609.35812#bib.bib10)\]\. Embedding\-based metrics such as BERTScore\[[14](https://arxiv.org/html/2609.35812#bib.bib4)\]capture semantic similarity more effectively but still require reference responses\. Human evaluation remains the most reliable but is expensive to scale\[[11](https://arxiv.org/html/2609.35812#bib.bib9)\]\. Recent multi\-turn evaluation benchmarks such as MT\-Eval\[[8](https://arxiv.org/html/2609.35812#bib.bib11)\]and MultiChallenge\[[15](https://arxiv.org/html/2609.35812#bib.bib12)\]expose limitations of LLMs in multi\-turn settings, but none target the automotive domain with its safety requirements and black\-box deployment constraints\. Automated testing approaches like MORTAR\[[16](https://arxiv.org/html/2609.35812#bib.bib13)\], STELLAR\[[17](https://arxiv.org/html/2609.35812#bib.bib5)\], and ASTRAL\[[18](https://arxiv.org/html/2609.35812#bib.bib2)\]reveal degradation across turns or simplified interaction settings, which do not reflect the multi\-turn nature of real\-world conversational systems\.

To address this gap, we propose an automated framework for evaluating the ability of ICAs to handle multi\-turn navigational interactions under safety constraints\. The framework treats the ICA as a black box and performs closed\-loop simulation using a strategy\-guided LLM\-based user simulator, an adversarial strategy manager targeting specific failure modes, and a two\-tier LLM\-based evaluator that assesses both turn\-level failures and conversation\-level quality across four evaluation dimensions, such as constraint adherence and safety compliance\.

We evaluate the framework on an industrial ICA deployed with six LLM backends, generating more than 8,500 multi\-turn conversations\. The automated judge achieves substantial agreement with twelve human annotators \(Cohen’sκ=0\.68\\kappa=0\.68,F1=0\.94F\_\{1\}=0\.94\)\. Compared to unguided simulation, strategy\-guided probing uncovers2\.96×2\.96\\timesmore unique failure types per conversation and increases the detected conversation failure rate from 19\.2% to 82\.7%\.

## IIBackground and Motivation

### II\-AMulti\-Turn Testing Problem

A multi\-turn test for an ICA is defined as a tupleP=\(SUT,s,𝐜,Λ,E\)P=\(\\textit\{SUT\},s,\\mathbf\{c\},\\Lambda,E\), whereSUTis the assistant under test, treated as a black box;ssis a seed phrase describing the driver’s goals \(e\.g\., “Navigate to Marienplatz and avoid tolls”\);𝐜=\(location,date,time\)\\mathbf\{c\}=\(\\textit\{location\},\\textit\{date\},\\textit\{time\}\)is a context vector;Λ\\Lambdais a set of adversarial strategies that target specific failure modes; andEEis an evaluation function that maps the resulting conversation transcript to failure labels and quality scores\.

The testing problem is to generate a sequence of user utterancesu0,u1,…,uNu\_\{0\},u\_\{1\},\\dots,u\_\{N\}such that the resulting conversation𝒞=\{\(ut,at\)\}t=0N\\mathcal\{C\}=\\\{\(u\_\{t\},a\_\{t\}\)\\\}\_\{t=0\}^\{N\}, whereata\_\{t\}is the assistant’s response at turntt, maximises the number of detected failuresℱ=⋃tE⁡\(𝒞,t\)\\mathcal\{F\}=\\bigcup\_\{t\}E\(\\mathcal\{C\},t\)\.

### II\-BMotivating Example

Let’s consider an example of interaction with an ICA\. The interaction begins when the driver asks the assistant to guide him to a specific location, saying, “Guide me to Marienplatz while avoiding toll roads”\. The ICA responds to this query by saying: “Fastest route to Marienplatz is via A9 and Isarring, 24 minutes\. Two other routes available”\. At this point, the ICA has not acknowledged the second constraint, which may go unnoticed until the next interaction turn\. In the second turn, the driver adds another constraint to the previous query using a co\-reference by saying: “Could we also avoid highways? I’d like a quieter drive”\. Again, the ICA responds to this query by saying the same route, i\.e\., the A9 highway route\. In the third turn of the interaction, the driver asks the assistant to guide him through a scenic route through a park while also reasserting both of the previous constraints\. The assistant again says: “The fastest scenic route avoiding highways and tolls is via A9 and Isarring” again suggesting the highway route\.

Subsequent turns by the driver also introduce more requests, such as a stop at a park, a visit to a pharmacy, and a time constraint, to which the assistant responds by repeating its previous deferral statements \(“Let me set the route first”\) or continuing to repeat its initial route on the highway\. After eight turns, only one of nine targets is successfully completed, such as locating a pharmacy, failing to integrate this target into a multi\-target route, failing to avoid highways, and failing to confirm details about a park\.

This example illustrates the complexity of user interactions that automated multi\-turn testing systems must handle\. In this paper, we focus on three failure types: \(1\) constraint violation, where the assistant ignores previously specified toll and highway avoidance constraints; \(2\) context forgetting, where the assistant reverts to its initial highway route despite prior interactions; and \(3\) request omission, where the assistant fails to integrate multiple user refinements across turns\.

A single\-turn test \(e\.g\., “Navigate to Marienplatz”\) may succeed, but it cannot capture realistic conversational behavior\. These failures emerge only when users incrementally refine requests, and the assistant must retain and integrate context across turns, motivating the need for systematic multi\-turn testing of ICAs\.

## IIIApproach

We propose a framework that aims to support the automation of multi\-turn testing for in\-car conversational interfaces\. Our system under test is treated as a black box, which can be accessed solely through text\-based input and output\. Although ICAs used in production settings rely on voice\-based interactions, our framework focuses on text\-based input to isolate dialog quality from speech recognition errors, as performed in prior studies\[[17](https://arxiv.org/html/2609.35812#bib.bib5),[19](https://arxiv.org/html/2609.35812#bib.bib1)\]\. In the following, we introduce the components of our framework\.

### III\-AFramework Components

User Simulator\.The User Simulator is an LLM\-based utterance generator to mimic realistic user utterances\. The interaction is initiated with a*seed phrase*ss, which describes the driver’s goals, e\.g\., “Plan a route with a charging stop and dinner before 8pm”\. The seed phrase is accompanied by a*context vector*𝐜=\(location,date,time\)\\mathbf\{c\}=\(\\textit\{location\},\\textit\{date\},\\textit\{time\}\)\. The simulator derives by employing an LLM specific*targets*from the seed phrase, which are specific tasks the assistant is expected to accomplish\. Each targetgkg\_\{k\}has a statusσk∈\{−1,0,1\}\\sigma\_\{k\}\\in\\\{\-1,0,1\\\}\(dropped, incomplete, completed\)\. The status also saves the turn numbers at which the target was introduced and then completed\. Possible*personas*of the driver are also considered \(e\.g\., polite, impatient, tech\-savvy\) to influence the utterance style\.

System Under Test\.The SUT is the ICA under test\. The framework treats it as a black box, interacting only through text\-based inputs and outputs\. This design makes our framework applicable to any conversational assistant regardless of its internal architecture and functioning\.

Strategy Manager\.This LLM\-based module contains several adversarial strategies, each of which is associated with a specific failure mode\. The strategies are embedded in the user utterances\. Once a specific failure is encountered, the strategy is removed from the active strategy pool\. This ensures that the framework covers new failure modes\.

LLM Judge\.This component consists of a two\-tier evaluator\. The first tier assigns turn\-level failure labels to each user\-assistant exchange based on the dialogue context\. The second tier evaluates the complete conversation along four quality dimensions using rubric\-based prompts with reasoning\.

### III\-BConversation Loop

[Algorithm 1](https://arxiv.org/html/2609.35812#alg1)describes the simulation loop\. A conversation consists of user–assistant exchange pairs\(ut,at\)\(u\_\{t\},a\_\{t\}\), whereutu\_\{t\}is the user utterance andata\_\{t\}is the assistant response at turntt\. The conversation historyHtH\_\{t\}denotes all exchanges prior to turntt, withH0=∅H\_\{0\}=\\emptyset\. The algorithm takes as input a seed phrasess, a context vector𝐜\\mathbf\{c\}, a maximum number of turnsMM, and the set of adversarial strategiesΛ\\Lambda\. It produces a conversation historyHH, dimension scores𝐝\\mathbf\{d\}, and a success scoreCC\.

The algorithm operates in three phases:

Initialisation \(lines 1–3\):The framework extracts a set of targets𝒢\\mathcal\{G\}from the seed phrasess, generates the first user utteranceu0u\_\{0\}from the seed and context, and copies all strategies into an active poolΠ\\Pi\.

Conversation loop \(lines 4–12\):At each turntt, the utteranceutu\_\{t\}is sent to the SUT, which returns a responseata\_\{t\}\(line 5\)\. The LLM\-based turn evaluator analyses the exchange against the historyHtH\_\{t\}and the targets𝒢\\mathcal\{G\}to produce failuresℱt\\mathcal\{F\}\_\{t\}\(line 6\)\. Target statuses in𝒢\\mathcal\{G\}are updated accordingly \(line 7\)\. Any strategyλ∈Π\\lambda\\in\\Piwhose associated failure modeξ⁡\(λ\)\\xi\(\\lambda\)has been triggered is removed from the active pool \(line 8\), whereξ\\xiis the strategy\-to\-failure mapping \([Section III\-C](https://arxiv.org/html/2609.35812#S3.SS3)\)\. The exchange is appended to the history \(line 9\)\. The loop terminates early if all targets are resolved and at least half the turn budget is spent \(line 10\); otherwise, the framework selects a strategy fromΠ\\Piand generates the next utteranceut\+1u\_\{t\+1\}\(line 11\)\.

Final evaluation \(lines 13–15\):The conversation\-level evaluator scores the full dialogue on each quality dimension, producing a score vector𝐝\\mathbf\{d\}\. The success scoreCCis then computed from𝐝\\mathbf\{d\}and the final target statuses in𝒢\\mathcal\{G\}\(see[Section III\-D](https://arxiv.org/html/2609.35812#S3.SS4)\)\.

Algorithm 1Multi\-turn conversation simulation1:Seed phrase

ss, context

𝐜\\mathbf\{c\}, max turns

MM, strategies

Λ\\Lambda
2:History

HH, dimension scores

𝐝\\mathbf\{d\}, success score

CC
3:Initialisation

4:

𝒢←ExtractTargets⁡\(s\)\\mathcal\{G\}\\leftarrow\\mathrm\{ExtractTargets\}\(s\)
5:

u0←GenerateUtterance⁡\(s,𝐜\)u\_\{0\}\\leftarrow\\mathrm\{GenerateUtterance\}\(s,\\mathbf\{c\}\)
6:

Π←Λ\\Pi\\leftarrow\\Lambda
7:Conversation loop

8:for

t←0t\\leftarrow 0to

M−1M\{\-\}1do

9:

at←SUT\.Send⁡\(ut,𝐜\)a\_\{t\}\\leftarrow\\mathrm\{SUT\.Send\}\(u\_\{t\},\\mathbf\{c\}\)
10:

ℱt←EvaluateTurn⁡\(Ht,ut,at,𝒢\)\\mathcal\{F\}\_\{t\}\\leftarrow\\mathrm\{EvaluateTurn\}\(H\_\{t\},u\_\{t\},a\_\{t\},\\mathcal\{G\}\)
11:

𝒢←UpdateTargets⁡\(𝒢,ℱt\)\\mathcal\{G\}\\leftarrow\\mathrm\{UpdateTargets\}\(\\mathcal\{G\},\\mathcal\{F\}\_\{t\}\)
12:

Π←Π∖\{λ∈Π:ξ⁡\(λ\)∈ℱt\}\\Pi\\leftarrow\\Pi\\setminus\\\{\\lambda\\in\\Pi:\\xi\(\\lambda\)\\in\\mathcal\{F\}\_\{t\}\\\}
13:

Ht\+1←Ht∪\{\(ut,at\)\}H\_\{t\+1\}\\leftarrow H\_\{t\}\\cup\\\{\(u\_\{t\},a\_\{t\}\)\\\}
14:if

AllResolved⁡\(𝒢\)\\mathrm\{AllResolved\}\(\\mathcal\{G\}\)and

t≥⌈M/2⌉t\\geq\\lceil M/2\\rceilthen break

15:

ut\+1←SelectAndGenerate⁡\(Ht\+1,𝒢,Π,𝐜\)u\_\{t\+1\}\\leftarrow\\mathrm\{SelectAndGenerate\}\(H\_\{t\+1\},\\mathcal\{G\},\\Pi,\\mathbf\{c\}\)
16:endfor

17:Final evaluation

18:

𝐝←EvaluateConversation⁡\(H\)\\mathbf\{d\}\\leftarrow\\mathrm\{EvaluateConversation\}\(H\)
19:

C←ComputeScore⁡\(𝐝,𝒢\)C\\leftarrow\\mathrm\{ComputeScore\}\(\\mathbf\{d\},\\mathcal\{G\}\)
20:return

H,𝐝,CH,\\mathbf\{d\},C

### III\-CAdversarial Strategies

A strategy corresponds to a target failure mode through a mapping function:ξ:Λ→ℱall\\xi:\\Lambda\\rightarrow\\mathcal\{F\}\_\{\\text\{all\}\}\.[Table I](https://arxiv.org/html/2609.35812#S3.T1)lists the five strategies used\. If the target failure mode of a strategy is detected in the failuresℱt\\mathcal\{F\}\_\{t\}of a turn, the strategy is removed from the active pool:Π←Π∖\{λ∈Π:ξ⁡\(λ\)∈ℱt\}\\Pi\\leftarrow\\Pi\\setminus\\\{\\lambda\\in\\Pi:\\xi\(\\lambda\)\\in\\mathcal\{F\}\_\{t\}\\\}, redirecting subsequent turns to unexplored failure modes\.

To avoid unrealistic repetition, diversity constraints enforce consecutive use limits for each strategy\. For example, safety probes rarely succeed on immediate retry, and are used at most once consecutively \(Lsafety=1L\_\{\\mathrm\{safety\}\}=1\), while stacking constraints may benefit from accumulation over turns \(Lconstraint=2L\_\{\\mathrm\{constraint\}\}=2\)\.

TABLE I:Adversarial strategies and their target failure modes
### III\-DEvaluation Rubric

Turn\-Level Failures\.The turn evaluator identifies two types of failure\. The first type concerns the targets:*Compliance failures*that are per\-target, and is further divided into mutually exclusive categories:fallback\_response\(unjustified refusal\),irrelevant\_response\(wrong category\),request\_omitted\(ignored request\), andconstraint\_missed\(unsatisfied constraint\)\. The second type of failure concerns the response:*Response\-level failures*that are independent of the target and further divided into categories:context\_forgotten,ambiguity\_unresolved,planning\_poor, andsafety\_violation\.

The materiality criteria filter out trivial issues\. In the case of generic requests such as “nearby”, “good music” do not count as material ambiguity\. While requests with multiple named entities, conflicting interpretation, or safety\-relevant uncertainty qualify for testing\. The same applies to the other categories\. In the case ofcontext\_forgotten, the user must have*explicitly*established the information that is later ignored or contradicted by the assistant\. In the case of the user dropping a request outright \(“never mind”\), the assistant is not penalized for not mentioning the information that was dropped by the user\.

Conversation\-Level Scoring\.At the end of the conversation, the four dimension evaluators independently assign a three\-point score \(dj∈\{0,1,2\}d\_\{j\}\\in\\\{0,1,2\\\}\) to each quality dimensionjj:Instruction & Constraint Adherence,Context & Ambiguity Handling,Plan Coherence, andSafety Compliance\. The final success score is computed as the weighted sum of the normalized dimension scores and the target completion ratioT=\|\{g∈𝒢:σg=1\}\|/\|𝒢\|T=\|\\\{g\\in\\mathcal\{G\}:\\sigma\_\{g\}=1\\\}\|/\|\\mathcal\{G\}\|:

C=∑j=14wj⋅\(dj/2\)\+wT⋅T∑j=14wj\+wT,C=\\frac\{\\sum\_\{j=1\}^\{4\}w\_\{j\}\\cdot\(d\_\{j\}/2\)\+w\_\{T\}\\cdot T\}\{\\sum\_\{j=1\}^\{4\}w\_\{j\}\+w\_\{T\}\},\(1\)wherewjw\_\{j\}andwTw\_\{T\}are configurable non\-negative weights for the dimensions and target completion, respectively\. A conversation passes ifC≥θC\\geq\\thetafor a thresholdθ\\theta\.

## IVEvaluation

### IV\-AResearch Questions

RQ0\(Judge Accuracy\)\.How accurately do LLM judges evaluate multi\-turn in\-car conversations compared to humans?

RQ1\(Effectiveness\)\.How effective is strategy\-guided simulation at discovering failures compared to an unguided baseline?

RQ2\(Failure Diversity\)\.How diverse are the failures uncovered by strategy\-guided simulation?

### IV\-BExperimental Setup

System Under Test\.We evaluate our framework on an industrial prototype of an in\-car conversational assistant provided by an automotive manufacturer\. The system supports route planning, POI recommendations, vehicle settings, and general information queries via a RAG architecture\. The system under test is treated as a black box, accessed only through text\-based sessions\.

SUT Models\.We deploy the assistant with six interchangeable LLM backends to assess its generalisability: four cloud\-based models \(GPT\-4o,GPT\-4o\-Mini,GPT\-5\-chat,DeepSeek\-V3\) accessed via Azure, and two locally hosted models \(Ministral\-3\-14b,Qwen3\-14b\-8k\) served via Ollama\. Each configuration is tested over six independent 4\-hour runs\.

Scenarios\.We maintain seven distinct seed\-phrase pools of 230–254 phrases each, covering navigation, POI search, trip planning, vehicle control, media playback, and combined multi\-domain requests\. Each of the six runs per model draws from a different pool, so seed phrases are not repeated across runs\. Within a run, a representative sampling strategy pairs seeds with two personas, 16 task combinations \(spanning one to three domains at varying complexity\), and two turn budgets \(M∈\{8,10\}M\\in\\\{8,10\\\}\)\. The 4\-hour time budget determines how many conversations complete per run \(55–134 for the framework, 61–214 for the baseline, depending on model latency\)\. Two configurable personas \(*polite*,*impatient*\) influence utterance style, while four context vectors provide diverse spatio\-temporal settings\.

Baseline\.We compare the strategy\-guided framework against a*simulation baseline*that uses the same user simulator, LLM judge, seed phrases, and evaluation rubric, but without adversarial strategies or evaluator feedback during the conversation loop\. The baseline employs a single neutral persona and a fixed turn budget \(M=8M=8\), isolating the effect of strategy\-guided probing while controlling the underlying LLM components\.

Metrics\.ForRQ0, we assess judge quality at two levels\. At the*dimension level*, each of the four quality dimensions is rated on a three\-point ordinal scale \(0/1/2\); we measure agreement using Cohen’s weightedκ\\kappaand Spearman’sρ\\rhofor each LLM–human pair per dimension, and report the mean across all 12 pairs and four dimensions\. The exact agreement percentage is the proportion of ratings where the LLM and a human assign the same ordinal score\. At the*conversation level*, each conversation is classified as pass or fail; we computeF1F\_\{1\}of the LLM’s binary verdict against the human majority vote\. These metrics follow the evaluation methodology of STELLAR\[[17](https://arxiv.org/html/2609.35812#bib.bib5)\]and G\-Eval\[[20](https://arxiv.org/html/2609.35812#bib.bib15)\]\. ForRQ1, we assess testing effectiveness through the conversation\-level success scoreCC\(Equation[1](https://arxiv.org/html/2609.35812#S3.E1)\) and the resulting pass/fail decision \(C≥θC\\geq\\theta\)\. A conversation is classified as a*failure*\(i\.e\., a detected defect of the SUT\) whenC<θC<\\theta\. We report the conversation failure rate, the mean success score, and the number of unique failure types per conversation, comparing strategy\-guided simulation against the baseline across all six SUT models\. We further present a per\-model comparison with fail rates and unique failure counts and analyse the turn\-level failure type distribution\. Statistical significance is assessed using the Mann–Whitney U test \(α=0\.05\\alpha=0\.05\) with Vargha–DelaneyA^12\\hat\{A\}\_\{12\}effect size\[[17](https://arxiv.org/html/2609.35812#bib.bib5)\]\. ForRQ2, we evaluate failure diversity following the STELLAR methodology\[[17](https://arxiv.org/html/2609.35812#bib.bib5)\]: \(i\) embedding\-based deduplication usingall\-MiniLM\-L6\-v2sentence embeddings and the STELLAR distance formulad=\(1−cos​\_​sim\)/2d\{=\}\(1\{\-\}\\mathrm\{cos\\\_sim\}\)/2, with a cosine similarity threshold\>0\.8\{\>\}\\,0\.8, reporting the count of unique failing conversations; and \(ii\) cluster coverage via k\-medoids clustering with Silhouette\-basedkkselection \(repeated10×10\{\\times\}\), measuring the fraction of failure clusters reached by each approach\. Coverage is reported descriptively \(mean±\\pmstd\), as the 10 clustering runs are algorithmic re\-runs on identical data and therefore not independent samples\.

TABLE II:Experimental configurationHuman Annotation\.We generated 37 multi\-turn conversations using our framework across four scenario configurations varying in domain complexity \(navigation, POI search, trip planning, and combined multi\-domain requests\)\. Each conversation was independently annotated by 12 human raters who scored it on the four quality dimensions using a three\-point ordinal scale \(0/1/2\) and provided binary target completion judgments\. A detailed annotation guideline with worked examples was distributed to all raters\. We assessed inter\-rater reliability using Light’s extension of Cohen’sκ\\kappa, yielding a human baseline of mean pairwiseκHH=0\.65\\kappa\_\{\\text\{HH\}\}=0\.65andρHH=0\.70\\rho\_\{\\text\{HH\}\}=0\.70, indicating substantial agreement\.

Judge Candidates\.We evaluated six LLMs as candidate judges:GPT\-5,GPT\-5\-Mini,GPT\-4\.1,GPT\-4o,GPT\-4o\-Mini, andDeepSeek\-V3\. Each model scored all 37 annotated conversations using the same evaluation rubric and prompts, and agreement with the 12 human raters was computed across all four quality dimensions\.

## VResults

Fig\. 1:RQ1: Effectiveness results\. Top: failures per run \(6 runs\), showing that strategy\-guided simulation consistently outperforms the baseline\. Bottom: cumulative failures over 4 hours \(mean±\\pmSD across 6 runs\)\.### V\-ARQ0: Judge Accuracy

[Table III](https://arxiv.org/html/2609.35812#S5.T3)presents the agreement between six candidate LLM judges and 12 human annotators on 37 multi\-turn conversations\. On the ordinal dimension scores,GPT\-5andGPT\-5\-Minimeet or exceed the human–human baseline, both achievingκ=0\.68\\kappa=0\.68\(vs\.κHH=0\.65\\kappa\_\{\\text\{HH\}\}=0\.65\) andρ=0\.70\\rho=0\.70\(vs\.ρHH=0\.70\\rho\_\{\\text\{HH\}\}=0\.70\), constituting substantial agreement on the Landis–Koch scale\. On the conversation\-level binary pass/fail,GPT\-5achievesF1F\_\{1\}=0\.97=0\.97andGPT\-5\-MiniF1F\_\{1\}=0\.94=0\.94, both at or above the human baseline of0\.940\.94\. The remaining four models attain moderate ordinal agreement \(κ∈\[0\.57,0\.60\]\\kappa\\in\[0\.57,0\.60\]\), below the human baseline but above theF1F\_\{1\}≥0\.71\\geq 0\.71threshold of STELLAR\[[17](https://arxiv.org/html/2609.35812#bib.bib5)\]and theρ≥0\.50\\rho\\geq 0\.50threshold of G\-Eval\[[20](https://arxiv.org/html/2609.35812#bib.bib15)\]\.

At the dimension level,GPT\-5\-Miniachieves the highest or tied\-highestκ\\kappaon three of four dimensions, including Safety Compliance \(κ=0\.70\\kappa=0\.70\) and Plan Coherence \(κ=0\.73\\kappa=0\.73\)\.GPT\-5leads on Instruction Adherence \(κ=0\.66\\kappa=0\.66\)\. Both models exhibit the lowest variability across dimensions \(coefficient of variation≤0\.08\\leq 0\.08forκ\\kappa\), compared to0\.110\.11–0\.310\.31for the other models, indicating consistent evaluation\.

We selectedGPT\-5\-Minias the automated judge for the remainder of this study\. It matchesGPT\-5on ordinal agreement \(κ=0\.68\\kappa=0\.68,ρ=0\.70\\rho=0\.70\), meets the humanF1F\_\{1\}baseline \(0\.940\.94\), and achieves within\-one agreement of93\.8%93\.8\\%, matching the human–human adjacent baseline\. Compared toGPT\-5, it offers substantially lower inference cost and latency suited to large\-scale automated evaluation\.

TABLE III:RQ0: Judge accuracy results\.*Finding 1:GPT\-5andGPT\-5\-Miniare the only models to meet or exceed the human baseline \(κHH=0\.65\\kappa\_\{\\text\{HH\}\}=0\.65,ρHH=0\.70\\rho\_\{\\text\{HH\}\}=0\.70\)\. We selectGPT\-5\-Minias the automated judge, as it matchesGPT\-5in ordinal agreement at lower cost\.*

### V\-BRQ1: Effectiveness

[Figure 1](https://arxiv.org/html/2609.35812#S5.F1)\(top\) reports the total conversation failures \(i\.e\., conversations withC<0\.75C<0\.75\) per 4\-hour run for each SUT model\. Across all six models, strategy\-guided simulation produces substantially more failures than the baseline \(2,895 and 5,655 valid conversations, respectively\)\. The overall failure rate rises from 19\.2% \(baseline\) to 82\.7% \(strategy\-guided\), with mean success scores ofC=0\.49±0\.23C=0\.49\\pm 0\.23andC=0\.87±0\.16C=0\.87\\pm 0\.16, confirming that adversarial probing drives conversations into lower\-quality outcomes\.

[Figure 1](https://arxiv.org/html/2609.35812#S5.F1)\(bottom\) shows that the cumulative failure count diverges early and widens steadily over the 4\-hour budget for every model, indicating that strategy\-guided simulation sustains a higher discovery rate throughout the run rather than merely front\-loading easy failures\.

*Finding 2: Strategy\-guided simulation identifies2\.96×2\.96\{\\times\}more unique failure types per conversation than the unguided baseline across six SUT models\.*

### V\-CRQ2: Failure Diversity

[Table IV](https://arxiv.org/html/2609.35812#S5.T4)reports the diversity analysis\. After embedding\-based deduplication, strategy\-guided simulation produces no duplicate conversations across all six models, while the baseline shows 0\.5–8\.3% redundancy\. Overall, strategy\-guided simulation yields 2,394 unique failing conversations versus 1,084 for the baseline \(2\.21×2\.21\{\\times\}\), with the largest gains forMinistral\-3\-14b\(2\.75×2\.75\{\\times\}\) andQwen3\-14b\-8k\(2\.49×2\.49\{\\times\}\)\.

Both approaches achieve near\-complete cluster coverage: 96\.9% for strategy\-guided simulation and 99\.0% for the baseline\. The low optimal cluster counts \(k=2\.2k=2\.2–4\.44\.4\) and silhouette scores \(0\.080\.08–0\.250\.25\) suggest a relatively homogeneous failure landscape, making high coverage attainable\.

*Finding 3: Strategy\-guided simulation produces exclusively unique conversations \(zero duplicates\) and yields2\.21×2\.21\{\\times\}more unique failing conversations than the baseline\.*

TABLE IV:RQ2: Failure diversity results\.Unique FailCov\. \(%\)SUT ModelSBS/BSBDeepSeek\-V33521582\.23100\.0100\.0GPT\-4o4792521\.9098\.6100\.0GPT\-4o\-Mini4372391\.83100\.0100\.0GPT\-5\-chat3731522\.4584\.894\.0Ministral\-3\-14b5191892\.75100\.0100\.0Qwen3\-14b\-8k234942\.4998\.0100\.0Overall2,3941,0842\.2196\.999\.0

## VIQualitative Evaluation

[Table V](https://arxiv.org/html/2609.35812#S7.T5)compares normalized failure distributions\.planning\_poorshows the largest gap, appearing in 62\.3% of strategy\-guided conversations versus 0\.6% in the baseline \(24\.1×24\.1\\times\), confirming the effectiveness of the*Planning Challenge*strategy\.safety\_violationalso increases substantially \(3\.0×3\.0\\times\)\. In contrast, baseline failures are dominated byconstraint\_missed,fallback\_response, andcontext\_forgotten\.*Planning Challenge*\(44\.9%\) and*Constraint Stacking*\(43\.0%\) most reliably trigger their target failures, while*Safety Probe*mainly inducesfallback\_response\(48\.1%\)\. All strategies also generate substantial collateral failures \(\>\>61%\)\. Finally, 81\.1% of framework\-generated conversations contain≥\\geq3 distinct failure types, compared to 20\.5% for the baseline\. Baseline failures mainly occur in early turns \(75\.2%\), whereas our framework distributes failures more evenly across conversations\.

## VIIConclusions and Future Work

Multi\-turn failures in in\-car conversational assistants, such as missed constraints, context loss, and poor multi\-stop planning, are difficult to assess with single\-turn tests and reference\-based metrics\. Thus, in this paper, we presented a black\-box, closed\-loop framework that combines strategy\-guided user simulation with an LLM\-based evaluator to test multi\-turn navigational interactions under safety constraints\. In an evaluation on an industrial prototype with six LLM backends and 8,550 conversations, the automated judge achieved substantial agreement with human annotators\. Strategy guidance significantly increased the detected and unique failure rates with comparable failure diversity\. Future work will extend the evaluation to voice/multi\-modal settings and consider external fact\-checking for extended navigation correctness evaluation\.

TABLE V:Failure\-type profile across 21,260 \(S\) and 12,035 \(B\) failures\. Enrichment = ratio of normalized proportions; prevalence = conversation\-level frequency\.
## References

- \[1\]Emotion\-awareness for intelligent vehicle assistants: a research agenda\.InSEFAIAS ’18,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[2\]M\. Schmidt, D\. Stier, S\. Werner, and W\. MinkerExploration and assessment of proactive use cases for an in\-car voice assistant\.InStudientexte zur Sprachkommunikation: Elektronische Sprachsignalverarbeitung 2019 \(ESSV\),Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[3\]P\. Habicht, L\. Sorokin, A\. Saydemir, K\. E\. Friedl, and A\. Stocco\(2026\)Benchmarking contextual understanding for in\-car conversational systems\.Journal of Systems and Software\.Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[4\]M\. R\. A\. H\. e\. al\. RonyCarExpert: leveraging large language models for in\-car conversational question answering\.InACL ’23: Industry Track,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[5\]Y\. Gao, M\. Piccinini, Y\. Zhang, D\. Wang, K\. Moller, R\. Brusnicki, B\. Zarrouki, A\. Gambi, J\. F\. Totz, K\. Storms, S\. Peters, A\. Stocco, B\. Alrifaee, M\. Pavone, and J\. Betz\(2026\)Foundation models in autonomous driving: a survey on scenario generation and scenario analysis\.IEEE Open Journal of Intelligent Transportation Systems\.External Links:[Link](https://arxiv.org/abs/2506.11526)Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[6\]V\. Riccio, G\. Jahangirova, A\. Stocco, N\. Humbatova, M\. Weiss, and P\. Tonella\(2020\)Testing Machine Learning based Systems: A Systematic Mapping\.Empirical Software Engineering25\(6\),pp\. 5193–5254\.External Links:[Document](https://dx.doi.org/10.1007/s10664-020-09881-0)Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[7\]N\. Humbatova, G\. Jahangirova, G\. Bavota, V\. Riccio, A\. Stocco, and P\. Tonella\(2020\)Taxonomy of real faults in deep learning systems\.InProceedings of the 42nd International Conference on Software Engineering,ICSE ’20,pp\. 12 pages\.External Links:[Document](https://dx.doi.org/10.1145/3377811.3380395)Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p1.1)\.
- \[8\]W\. Kwan, X\. Zeng, Y\. Jiang, Y\. Wang, L\. Li, L\. Shang, X\. Jiang, Q\. Liu, and K\. WongMT\-eval: a multi\-turn capabilities evaluation benchmark for large language models\.InACL 2024,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p2.1),[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[9\]S\. Guan, J\. Wang, J\. Bian, B\. Zhu, J\. Lou, and H\. XiongEvaluating llm\-based agents for multi\-turn conversations: a survey\.arXiv preprint arXiv:2503\.22458\.Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p2.1)\.
- \[10\]C\. Liuet al\.How NOT to evaluate your dialogue system: an empirical study of unsupervised evaluation metrics for dialogue response generation\.InACL ’16,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p2.1),[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[11\]J\. M\. Deriu, A\. Rodrigo, A\. Otegi,et al\.Survey on evaluation methods for dialogue systems\.Artificial Intelligence Review\.Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p2.1),[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[12\]A\. Chu and G\. HuangThe intersection of voice assistants and autonomous vehicles: a scoping review\.InProceedings of the Human Factors and Ergonomics Society Annual Meeting,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p2.1)\.
- \[13\]K\. Papineni, S\. Roukos, T\. Ward, and W\. ZhuBLEU: a method for automatic evaluation of machine translation\.InProceedings of the 40th annual meeting on ACL,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[14\]T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with bert\.External Links:1904\.09675Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[15\]V\. Sirdeshmukhet al\.MultiChallenge: a realistic multi\-turn conversation evaluation benchmark challenging to frontier llms\.InACL 2025,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[16\]G\. Guo, A\. Aleti, N\. Neelofar, C\. Tantithamthavorn, Y\. Qi, and T\. Y\. ChenMORTAR: multi\-turn metamorphic testing for llm\-based dialogue systems\.arXiv preprint arXiv:2412\.15557\.Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[17\]L\. Sorokin, I\. Vasilev, K\. E\. Friedl, and A\. StoccoSTELLAR: a search\-based testing framework for large language model applications\.InSANER ’26,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p3.1),[§III](https://arxiv.org/html/2609.35812#S3.p1.1),[§IV\-B](https://arxiv.org/html/2609.35812#S4.SS2.p5.1),[§V\-A](https://arxiv.org/html/2609.35812#S5.SS1.p1.1)\.
- \[18\]M\. Ugarte, P\. Valle, J\. A\. Parejo, S\. Segura, and A\. ArrietaASTRAL: a tool for the automated safety testing of large language models\.InISSTA Companion ’25,Cited by:[§I](https://arxiv.org/html/2609.35812#S1.p3.1)\.
- \[19\]K\. E\. Friedl, A\. G\. Khan, S\. R\. Sahoo, M\. R\. A\. H\. Rony, J\. Germies, and C\. SüßInCA: rethinking in\-car conversational system assessment leveraging large language models\.External Links:2311\.07469Cited by:[§III](https://arxiv.org/html/2609.35812#S3.p1.1)\.
- \[20\]Y\. Liu, D\. Iter, Y\. Xu, S\. Wang, R\. Xu, and C\. ZhuG\-eval: NLG evaluation using GPT\-4 with better human alignment\.InACL 2023,Cited by:[§IV\-B](https://arxiv.org/html/2609.35812#S4.SS2.p5.1),[§V\-A](https://arxiv.org/html/2609.35812#S5.SS1.p1.1)\.