Towards Detecting Inconsistencies in End-to-end Generated TODs

# Towards Detecting Inconsistencies in End-to-end Generated TODs
Source: [https://arxiv.org/html/2607.09338](https://arxiv.org/html/2607.09338)
###### Abstract\.

Generative AI is profoundly transforming the core technologies behind conversational systems, shifting from component\-based to end\-to\-end approaches\. However, Large Language Models \(LLMs\) may still generate inconsistencies, a critical issue particularly in Task\-Oriented Dialogues \(TODs\), where system responses must strictly adhere to information from a domain knowledge base \(e\.g\., restaurants in a city\)\. A single hallucination \(e\.g\., suggesting a non\-existent restaurant\) can lead to severe task failures\. We investigate a method for automatically detecting inconsistencies by conceptualizing TODs as a Constraint Satisfaction Problem \(CSP\), where variables represent dialogue segments referencing the conversational domain, and constraints among variables capture dialogue properties such as turn coherence and adherence to domain knowledge\. We propose a pipeline that first identifies variables in a target dialogue and then applies a CSP solver to identify valid solutions\. By comparing the target dialogue with valid variable assignments, we can detect inconsistencies and suggest minimal changes to ensure dialogue consistency\. We demonstrate the high accuracy of the CSP\-based approach in detecting inconsistencies, and provide a detailed analysis of our findings\.

Task\-Oriented Dialogue Systems, Dialogue Consistency, Constraint Satisfaction Problem, Large Language Models\.

††conference:Proceedings of the SIGIR Workshop on Search\-Oriented Conversational AI; 2026; Melbourne, Australia††ccs:Computing methodologies Discourse, dialogue and pragmatics††ccs:Computing methodologies Natural language generation††ccs:Computing methodologies Language resources## 1\.Introduction

Knowledge Base

IDNameAreaFoodPriceR1TabernacentrespanishcheapR2EspanacentrespanishmoderateR3Beirutcentrelebanesecheap
Dialogue User:I am looking for a restaurant servingSpanishfood\.System:There arethreerestaurants servingSpanishfood, one ischeapand the other ismoderateprice range\. Which price range would you prefer?User:I am looking for acheaprestaurant inanyarea that servesSpanishfood\.System:Beirutischeapand servesLebanesefood\. Would you like the location information?

Figure 1\.An inconsistent task\-oriented dialogue \(TOD\) and a restaurant KB with slot\-value pairs \(N = Name, A = Area, F = Food, P = Price\)\. Bold text highlights slot values in the dialogue\.Redvalues indicate dialogic inconsistencies, whilepurplevalues indicate inconsistencies with the KB\.Task\-oriented dialogue \(TOD\) systems\(Balaramanet al\.,[2021](https://arxiv.org/html/2607.09338#bib.bib89); Budzianowskiet al\.,[2018](https://arxiv.org/html/2607.09338#bib.bib45); McTear,[2020](https://arxiv.org/html/2607.09338#bib.bib53); Qinet al\.,[2023](https://arxiv.org/html/2607.09338#bib.bib236)\)play a crucial role in human\-computer interaction, facilitating seamless communication between users and machines to perform specific tasks\. In recent years, transformer\-based neural models have become the core technology behind TODs\. In particular, pre\-trained large language models \(LLMs\) allow end\-to\-end approaches\(Banget al\.,[2023](https://arxiv.org/html/2607.09338#bib.bib104); Laiet al\.,[2023](https://arxiv.org/html/2607.09338#bib.bib136); Qinet al\.,[2023](https://arxiv.org/html/2607.09338#bib.bib236)\)that greatly simplify the development of conversational systems, with respect to more complex component\-based pipelines\(Younget al\.,[2013](https://arxiv.org/html/2607.09338#bib.bib65)\)\. However, despite their impressive generative capabilities, it is well known that LLMs exhibit significant limitations in producing outputs that adhere to the requirements of task\-specific domains\(Choet al\.,[2022](https://arxiv.org/html/2607.09338#bib.bib237); Ji and others,[2022](https://arxiv.org/html/2607.09338#bib.bib15)\)\. In a recent study\(Labrunaet al\.,[2024](https://arxiv.org/html/2607.09338#bib.bib232)\)it has been shown that, when asked to generate a dialogue according to a given knowledge base \(K​BKB\), as required by TODs, state\-of\-the\-art open source LLMs produce up to 59% of per dialogue disalignments with respect to the underlying KB\. Failing to align their outputs with a domainK​BKB, leads to inconsistencies that undermine LLMs reliability in real\-world applications\.

Figure[1](https://arxiv.org/html/2607.09338#S1.F1)shows an example of a fragment of a Knowledge Base \(three restaurants in a city\) and a short TOD dialogue generated by a LLM\. There are two hallucinations in this dialogue: first, at turn S1, the system mentions three restaurants serving Spanish food, which is not consistent with the knowledge base, where there are two such restaurants \(this is adomain inconsistency\)\. Second, at turn S2, the system introduces a Lebanese restaurant, which, although existing in theK​BKB, it is not coherent with the previous dialogue turns, as a Spanish restaurant would have been expected \(this is adialogic inconsistency\)\. Intuitively, both domain and dialogic inconsistencies need the whole dialogue context in order to be detected: for instance,Lebaneseappears inconsistent because the user is looking for a Spanish restaurant since the beginning of the conversation, while considering turn S2 alone would result in a well formed dialogue\. In addition, notice that three changes would make the whole dialogue consistent: \(i\) changingthreewithtwoat turn S1; \(ii\) changingLebanesewithSpanishat turn S2; and \(iii\) changingBeirutwithTabernaat turn S2\. Detecting TOD inconsistencies and, if possible, suggesting how to solve them, is the goal of this paper\. The novel intuition of the paper is to consider dialogue consistency as a kind ofConstraint Satisfaction Problem\(CSP\(Brailsfordet al\.,[1999](https://arxiv.org/html/2607.09338#bib.bib238)\)\), under the following working hypothesis: \(i\) first, dialogue consistency can be modeled with a limited number of domain independent constraints that need to be respected by appropriate linguistic realizations; \(ii\) such constraints can be well represented to define a CSP, whose allowed solutions can be identified by a CSP solver; \(iii\) a TOD is consistent if its linguistic realizations belong to the set of solutions allowed by a CSP solver for that dialogue\. In the paper, we discuss how dialogue constraints are defined, how they can be extracted and modeled as a CSP, and how to set up an experimental setting where we can empirically prove that a CSP solver can detect inconsistencies in a dialogue and suggest possible changes that make the dialogue consistent\.

The contributions of the paper are the following:

- •We model TOD consistency as a Constraint Satisfaction Problem \(CSP\): to the best of our knowledge, this is a fully original approach\.
- •We set up a reusable experimental setting where TOD consistency can be automatically evaluated against a CSP solver\.
- •We show that the proposed CSP approach allows for effective detection of inconsistent TODs, achieving an accuracy of 75\.9%\.

## 2\.Dialogue Consistency as a Constraint Satisfaction Problem

In this section, we explore the conceptualization of dialogue consistency in the CSP framework\. We first describe the fundamental component of a conversational domain \(Section[2\.1](https://arxiv.org/html/2607.09338#S2.SS1)\), then we elucidate the various constraints that contribute to dialogue coherence \(Section[2\.2](https://arxiv.org/html/2607.09338#S2.SS2)\), encompassing linguistic, dialogic, and domain\-based considerations\. We finally expound upon the formalization of dialogue constraints as CSPs \(Section[2\.3](https://arxiv.org/html/2607.09338#S2.SS3)\), delineating the process of modeling dialogue coherence as a constraint satisfaction task\.

### 2\.1\.TOD Conversational Domain

TODs typically need specific knowledge about the conversational domain \(e\.g\., a database of restaurants, a playlist of songs, etc\.\)\. As in literature\(Henderson and others,[2014](https://arxiv.org/html/2607.09338#bib.bib17)\), we assume a domain ontology providing a schema of the concepts \(e\.g\.,Restaurant,Hotel,Movie\), a set of slotsSS\(e\.g\.,Food,Area,Price\) for the concepts, and the set of values that each slot can assume \(e\.g\.,Expensive,Moderate, andCheapfor thePriceslot\)\. Then, a domain knowledge base \(K​BKB\) comprises a collection of instances for the ontology concepts, each consisting of\[s​l​o​t−v​a​l​u​e\]\[slot\-value\]pairs, adhering to the domain ontology schema\.

![Refer to caption](https://arxiv.org/html/2607.09338v1/x1.png)Figure 2\.Overview of the CSP\-based methodology\. In step 1 GPT\-4o is used to annotate the given dialogue for variable identification\. Step 2 allocates the constraints that need to be true for the dialogue to be consistent\. Step 3 uses CSP to find the possible solutions, and in step 4 the original dialogue is matched with the CSP solutions to assess its correctness\.
### 2\.2\.Dialogue Consistency

A TOD can be considered as a sequence of conversational turns between a user and a system aimed at achieving a specific goal\. Within this framework, ensuring the consistency of the dialogue is crucial for effective communication between the user and the system\. We consider three types of constraints, which need to be respected for a dialogue to be consistent: linguistic, dialogic and domain constraints\. Figure[3](https://arxiv.org/html/2607.09338#S2.F3)provides a concrete example of how violations of these constraints can lead to inconsistencies, showing a case where the system response contradicts the underlying knowledge base despite a seemingly coherent dialogue flow\.

IDNameAreaFoodPriceR1Marioeastitalianexpens\.R2Napolicentreitaliancheap
U:I’m looking for a restaurant serving<MASK\>food in any area\.S:There are no<MASK\>restaurants in the area\.U:Well, can I get the phone number to a<MASK\>restaurant?S:Restaurant R2 serves<MASK\>food\. Their phone number is 01223 355166\. Can I help you with anything else?U:That’s it, goodbye\.S:You’re welcome, goodbye\.

U:I’m looking for a restaurant servingitalianfood in any area\.S:There are noitalianrestaurants in the area\.U:Well, can I get the phone number to acheaprestaurant?S:Restaurant R2 servesitalianfood\. Their phone number is 01223 355166\. Can I help you with anything else?U:That’s it, goodbye\.S:You’re welcome, goodbye\.

Figure 3\.Example of an inconsistent re\-lexicalization\. The system is claiming there are no Italian restaurant, even though there are 2 in the Knowledge Base\.##### Linguistic Constraints\.

They are necessary to respect general rules of language, including morpho\-syntactic rules \(e\.g\., genre and number agreement\) and syntax\-based rules \(e\.g\., the correct use of a preposition\)\. For instance, consider the following masked utterance:

U:I look for a restaurant in<MASK\>\.

The choice ofcentreas a substitute for the masked token is valid, whereasexpensivewould not be suitable because the prepositioninis rarely used to introduce a price in English\.

##### Dialogic Constraints\.

They maintain the semantic coherence across successive turns of the dialogue, ensuring that each utterance logically aligns with the preceding context, thereby facilitating a seamless flow of information\. As an example, suppose the following masked dialogue turns:

U:I would like an Italian restaurant\.S:There is no<MASK\>restaurant in the centre\.

Here bothItalianandcheapwould be eligible choices from a linguistic point of view, but onlyItalianwould maintain the coherence with the previous turn in the dialogue\.

##### Domain Constraints\.

They ensure alignment between the dialogue content and the knowledge base of the system, thereby maintaining the dialogue’s alignment with relevant factual information\. Consider, for instance, aK​BKBwith the following restaurants:

IDNameAreaFoodPriceR1Marioeastitalianexpens\.R2Napolicentreitaliancheap
And the following piece of masked dialogue:

U:I am looking for anItalianrestaurant in thecentre\.S:We have<MASK\>restaurants available for your preferences\.

Then, the only admissible choice for the masked token would beone, as selecting any other number would introduce an inconsistency with the information provided in theK​BKB\.

### 2\.3\.TOD Consistency as CSP

A Constraint Satisfaction Problem \(CSP\)\(Brailsfordet al\.,[1999](https://arxiv.org/html/2607.09338#bib.bib238); Kumar,[1992](https://arxiv.org/html/2607.09338#bib.bib170)\)defines a set of variables, each associated with a finite domain of values, and a set of constraints specifying the allowed combinations of values\. A solution is an assignment of values to all variables that satisfies all constraints; if no such assignment exists, the CSP is unsolvable\.

We model TOD consistency as a CSP, where variables correspond to dialogue elements to be instantiated \(e\.g\.,M​A​S​KMASKtokens\), and their domains are derived from the dialogue\-specificK​BKB\. Constraints capture linguistic, dialogic, and domain requirements \(Section[2\.2](https://arxiv.org/html/2607.09338#S2.SS2)\)\.

Formally, given a dialoguedid\_\{i\}with variablesx1,…,xnx\_\{1\},\\ldots,x\_\{n\}, domainsD1,…,DnD\_\{1\},\\ldots,D\_\{n\}, and a set of constraints𝒞\\mathcal\{C\}, the task is to determine whether there exists an assignmentA=\{\(x1,a1\),…,\(xn,an\)\}A=\\\{\(x\_\{1\},a\_\{1\}\),\\ldots,\(x\_\{n\},a\_\{n\}\)\\\}withai∈Dia\_\{i\}\\in D\_\{i\}such that:

Satisfies​\(A,Cj\)∀Cj∈𝒞\\text\{\{Satisfies\}\}\(A,C\_\{j\}\)\\quad\\forall C\_\{j\}\\in\\mathcal\{C\}whereSatisfies​\(A,Cj\)\\text\{\{Satisfies\}\}\(A,C\_\{j\}\)indicates whether assignmentAAsatisfies constraintCjC\_\{j\}\.

## 3\.Methodology

This section outlines the process of modeling a TOD as a CSP, and then to assess the dialogue consistency using a CSP solver\. The assessment involves three key steps for a\[d,K​B\]\[d,KB\]pair, whereddis a dialogue andK​BKBis a Knowledge Base: \(1\) identification of the variables within the dialoguedd\(Section[3\.1](https://arxiv.org/html/2607.09338#S3.SS1)\); \(2\) definition of dialogue constraints and construction of a CSP solver for the\[d,K​B\]\[d,KB\]pair \(Section[3\.2](https://arxiv.org/html/2607.09338#S3.SS2)\); and \(3\) application of the CSP solver to determine if the dialogueddrepresents a feasible solution with respect to the defined constraints \(Section[3\.3](https://arxiv.org/html/2607.09338#S3.SS3)\)\. These phases of the methodology are illustrated in Figure[2](https://arxiv.org/html/2607.09338#S2.F2)\.

### 3\.1\.Identifying TOD Variables

At step 1 \(see Figure[2](https://arxiv.org/html/2607.09338#S2.F2)\), we consider a TODddand aK​BKB\(i\.e\., a set of entities described by slot\-value pairs\) related to the conversational domain of the dialogue\. We do not assume any particular dependency betweenddandK​BKB:ddcould be either fully covered byK​BKB\(i\.e\., all mentions of slot values inddare present inK​BKB\), only partially covered, or not covered at all\. We consider CSP variables all text portions inddeither referring to a slot value inK​BKBor mentioning amounts of instances inK​BKB\. The rationale is that both slot values and instance amounts are elements that better characterize a TOD and are responsible for its consistency\. In our example in Figure[1](https://arxiv.org/html/2607.09338#S1.F1), we will obtain the following variables with their assignments, corresponding to highlighted tokens:

\[x1=S​p​a​n​i​s​h\]\[x\_\{1\}=Spanish\],\[x2=t​h​r​e​e\]\[x\_\{2\}=three\],\[x3=S​p​a​n​i​s​h\]\[x\_\{3\}=Spanish\],\[x4=c​h​e​a​p\]\[x\_\{4\}=cheap\]…\[x11=L​e​b​a​n​e​s​e\]\[x\_\{11\}=Lebanese\]\.

### 3\.2\.Defining TOD Constraints

We have established a set𝒳\\mathcal\{X\}of variablesx1,x2,…,xnx\_\{1\},x\_\{2\},\.\.\.,x\_\{n\}, where each variablexix\_\{i\}can assume a value either from the slot values or from amounts of instances inK​BKB\. Moving to step 2 in Figure[2](https://arxiv.org/html/2607.09338#S2.F2), we now define the set of constraints𝒞\\mathcal\{C\}over the values that can be assigned to𝒳\\mathcal\{X\}variables\. We consider the three categories of constraints introduced in Section[2\.2](https://arxiv.org/html/2607.09338#S2.SS2): linguistic, dialogic, and domain\-based constraints, and for each category we define a set of domain independent patterns, which are then instantiated as actual constraints on a TOD\.

##### Patterns for linguistic constraints\.

We model linguistic constraints as the need for a variable derived from a slot value to match the semantic type of its slot type\. For instance, given the utteranceI am looking for a restaurant atx1x\_\{1\}, the value of the variablex1x\_\{1\}must belong to theAreatype\. More precisely,C​1C1is defined as follows:

whereVVis the set of values belonging to the same slot type as the original value\. ConstraintC​1C1, is meant to avoid that a variable can assume values that are semantically non valid\. For instance, avoiding thatx1x\_\{1\}=northcan be assigned to aFood, as inI am looking for a restaurant atindian, which is ungrammatical in English\.

##### Patterns for dialogic constraints\.

We consider two dialogic constraints\.C​2C2ensures that variables referring to the same slot\-name and slot\-value inddare assigned to the same value\.C​3C3ensures that variables with the same semantic type \(i\.e\., same slot\-name\) occurring in the same utterance are assigned to different values\. Given the turnU: I want anx1x\_\{1\}restaurant\. S: There are 3 restaurant that servex2x\_\{2\}, we defineC​2C2as follows:

C2:x1=x2C\_\{2\}:x\_\{1\}=x\_\{2\}
where the aim is to keep internal coherence across the dialogue turns\. Given the utteranceWe havex1x\_\{1\},x2x\_\{2\}, orx3x\_\{3\}restaurants\., we defineC​3C3as:

C3:x1≠x2,x1≠x3,x2≠x3C\_\{3\}:x\_\{1\}\\neq x\_\{2\},\\quad x\_\{1\}\\neq x\_\{3\},\\quad x\_\{2\}\\neq x\_\{3\}
which captures non redundancy at the utterance level\.

##### Patterns for domain\-based constraints\.

We consider three domain\-based constraints\. All of them are meant to guarantee consistency between the number of instances mentioned inddand the actual number of instances present inK​BKB\. We distinguish three cases:C​4C4covers the cases when an utterance inddstates that there are no instances inK​BKB;C​5C5covers the cases where it is stated that there is at least one instance; andC​6C6the cases where there are exactlynninstances\.

As forC​4C4, consider an utterance indicating no results for a search:There are no restaurants servingx1x\_\{1\}food, assuming that there are no restaurants with\[Food=x1x\_\{1\}\]inK​BKB\. For this utterance,C​4C4is defined as:

C​4:¬∃i∈K​B​with values​x1C4:\\neg\\exists i\\in KB\\text\{ with values \}x\_\{1\}implying that the variablex1x\_\{1\}can not assume a value that is present in an instance of theK​BKB\.

As forC​5C5, consider the utterance:We have manyx1x\_\{1\}restaurants atx2x\_\{2\}, where at least one restaurant with\[Food=x1x\_\{1\}\]and\[Area=x2x\_\{2\}\]is supposed to exist inK​BKB\. For this utterance,C​5C5is defined as:

C​5:∃i∈K​B​with values​x1,x2C5:\\exists i\\in KB\\text\{ with values \}x\_\{1\},x\_\{2\}imposing the existence of at least one instance with valuesx1x\_\{1\}andx2x\_\{2\}\. Finally, forC​6C6, consider the utteranceThere arex1x\_\{1\}restaurants atx2x\_\{2\}\. We define the constraint as:

C​6:\|\{i∈K​B​with value​x2\}\|=x1C6:\|\\\{i\\in KB\\text\{ with value \}x\_\{2\}\\\}\|=x\_\{1\}to check that the number of instances with valuex2x\_\{2\}is exactly equal tox1x\_\{1\}\.

To sum up, we have defined six general, domain independent \(i\.e\., in principle they can be applied to any TOD\), constraint patterns over the variable of a TOD\.

### 3\.3\.Assessing Dialogue Consistency

After identifying all variables and constraints for a dialoguedd, a CSP solver computes all possible solutions for the variables inddbased on the knowledge base \(K​BKB\) \(step 3 in Figure[2](https://arxiv.org/html/2607.09338#S2.F2)\)\. If one of these solutions matches the variable assignments indd, the dialogue is consistent withK​BKB\(step 4 in Figure[2](https://arxiv.org/html/2607.09338#S2.F2)\)\. For example, in Figure[1](https://arxiv.org/html/2607.09338#S1.F1), the assignment\[x2=three\]\[x\_\{2\}=\\text\{three\}\]violatesC​6C6\(incorrect count of Spanish instances inK​BKB\), while\[x11=Lebanese\]\[x\_\{11\}=\\text\{Lebanese\}\]violatesC​2C2\(lack of coherence with prior turns\)\. If the CSP solver finds at least one solution, the variable assignments in the dialogue must match one of those solutions to ensure all constraints are satisfied\. Conversely, if no solution exists with respect toK​BKB, the variable assignments should either remain empty or include values not present inK​BKBto maintain consistency\. When at least one solution is found but none matches the variable assignments, the solver identifies the most similar solution and determines the minimal changes required to make the dialogue consistent\. This process provides a detailed report highlighting specific inconsistencies and suggesting corrections\.

## 4\.Validating CSP Performance

We evaluate the ability of the CSP\-based approach to detect inconsistencies in task\-oriented dialogues \(TODs\) through a controlled experiment \(Figure[2](https://arxiv.org/html/2607.09338#S2.F2)\)\. We construct a balanced dataset of dialogue–knowledge base pairs\[d,K​B\]\[d,KB\], with equal proportions ofconsistentandnot\-consistentinstances\. Each pair is processed through the pipeline, and the CSP component outputs a binary decision, which is compared against ground truth\.

### 4\.1\.Experimental Setup

We construct a balanced dataset of 108 dialogue–KB pairs from MultiWOZ 2\.3\. While the dataset is assumed to be consistent, manual inspection combined with CSP verification revealed that approximately 10% of dialogues exhibit inconsistencies due to annotation errors; these were removed\. The remaining dialogues were split evenly intoconsistentandnot\-consistentsubsets\. Inconsistent dialogues were generated by randomly modifying slot values to violate KB constraints\. For each dialogue, a tailoredK​BKBwas constructed by selecting relevant entities from the global MultiWOZ knowledge base\.

CSP variables are identified either from MultiWOZ annotations or via GPT\-4o, and constraints are instantiated using the six patterns introduced in Section[3\.2](https://arxiv.org/html/2607.09338#S3.SS2)\. We model the problem using MiniZinc\(Nethercoteet al\.,[2007](https://arxiv.org/html/2607.09338#bib.bib171)\), a declarative constraint programming language\. We use the Chuffed solver\(Chuet al\.,[2018](https://arxiv.org/html/2607.09338#bib.bib185)\), which is optimized for constraint satisfaction problems and supports efficient search and propagation strategies\. The solver determines whether a valid assignment exists; absence of a solution implies inconsistency\.

We compare four classification methods: a random baseline, two CSP\-based approaches using MultiWOZ annotations \(global and local\), and a fully automated pipeline using GPT\-4o for variable extraction\. Performance is measured using accuracy over\[d,K​B\]\[d,KB\]pairs\.

We consider four methods for classifying dialogue consistency:

- •Random Baseline: assigns labels randomly \(expected accuracy: 50%\)\.
- •MWoZ Global \+ CSP: variables are extracted from full\-dialogue annotations and evaluated globally\.
- •MWoZ Local \+ CSP: variables are evaluated independently per turn; a single inconsistent turn marks the dialogue as inconsistent\.
- •GPT\-4o Global \+ CSP: variables are extracted automatically using a two\-step prompt chain, then evaluated globally\.

The prompt chain used to annotate the dialogue turns consists of the following two prompts:

- •Prompt\-1:Analyze the given user utterance and extract any slot\-value pairs\. The possible slot types are: Area, Food, Price, Depart, Destination\. Return the output as JSON with the dialog\-act format\.
- •Prompt\-2:Refine the given annotation for the user utterance\. Ensure that only slots related to Area, Food, and Price are included\. Correct any errors in the provided annotation, add missing slots, and remove any irrelevant slots\. Return the output as JSON with the updated dialog\-act format\.

The GPT\-4o response to Prompt\-1 is used as input within Prompt\-2, and the final output is a JSON file containing annotations about slot variables\.

Thedialog\-actreferred to in the two prompts is a JSON schema that guides GPT\-4o in structured output mode and resembles the MultiWOZ JSON annotation schema\.

Methodaccuracy \(%\)Random Baseline50\.0MWoZ global variables \+ CSP91\.6MWoZ local variables \+ CSP79\.0GPT4\-o global variables \+ CSP75\.9Table 1\.Results on assessing CSP performance\.
### 4\.2\.Results

Table[1](https://arxiv.org/html/2607.09338#S4.T1)reports the results\. The upper\-bound method based on MultiWOZ global annotations achieves 91\.6% accuracy, confirming the effectiveness of the CSP formulation and the high coverage of the constraint patterns\. Performance drops to 79% when constraints are applied locally, highlighting the importance of global context\. The end\-to\-end pipeline using GPT\-4o reaches 75\.9%, with errors primarily due to imperfect variable extraction\. The results show that \(i\) CSP\-based modeling is effective for detecting dialogue inconsistencies, and \(ii\) the proposed constraint set provides strong coverage of relevant phenomena\.

## 5\.Analysing LLM Behavior

We investigate how large language models \(LLMs\) handle dialogue consistency under explicit constraints\. Specifically, we study \(i\) their ability to generate consistent TODs, \(ii\) the role of different constraint types, and \(iii\) the localization of inconsistency sources\.

### 5\.1\.Experimental Setup

We consider 950 MultiWOZ dialogues across multiple domains\. Each dialogue is de\-lexicalized by replacing slot values with placeholders\. Models are then prompted to reconstruct the original dialogue by filling these placeholders using the provided Knowledge Base \(K​BKB\), thereby generating a re\-lexicalized dialogue\.

All models operate in a zero\-shot setting without any fine\-tuning\. Closed\-source models are accessed via APIs, while open\-source models are run using HuggingFace checkpoints\.

The full prompt used in our experiments is reported below\.

System Prompt:

> You are given an instruction that outlines a task, a Knowledge Base containing domain\-specific information, and a dialogue to process\. Your goal is to fill in the\[MASK\]placeholders in the dialogue using only the information provided in the Knowledge Base\. Task:Replace each\[MASK\]with the most appropriate value from the Knowledge Base\. Preserve the original structure of the dialogue exactly\. If a turn does not contain any placeholders, leave it unchanged\. Each dialogue turn must start with eitherUserorSystem\. Maintain the original spacing and punctuation \(e\.g\., write “Hi ,” instead of “Hi,”\)\.

Input Format:

> ``` [Knowledge Base] <key-value pairs> [Dialogue] User: ... System: ... User: ... ... ```

Output:

> The same dialogue with all \[MASK\] tokens replaced accordingly\.

The generated dialogue is then evaluated using the CSP solver to verify constraint satisfaction\.

#### 5\.1\.1\.Models

We evaluate four language models: LLaMA\-3\.1 8B, GPT\-3\.5\-Turbo, GPT\-4o, and GPT\-o1\. LLaMA\-3\.1 8B is a large\-scale model fine\-tuned for handling complex dialogue contexts and maintaining coherence in text generation\(Dubeyet al\.,[2024](https://arxiv.org/html/2607.09338#bib.bib234)\)\. GPT\-3\.5\-Turbo is a model specifically designed for conversational tasks\(Achiamet al\.,[2023](https://arxiv.org/html/2607.09338#bib.bib186)\)\. GPT\-4o is an advanced language model recognized for its robust performance in various natural language processing tasks\(Hurstet al\.,[2024](https://arxiv.org/html/2607.09338#bib.bib233)\)\. GPT\-o1 is one of the latest update of the GPT series, designed to reason through complex tasks to solve harder problems111https://openai\.com/o1/\. All models were prompted with both the de\-lexicalized dialogue,dd​e​l​e​xd\_\{delex\}, and its associatedK​BKBas input, ensuring a comprehensive context for producing dialogues that adhered to implicit constraints\. Inference was conducted in zero\-shot mode without fine\-tuning, leveraging the respective APIs for closed source models and the huggingface checkpoints for the open ones: GPT\-3\.5\-Turbo \(2023\-05\-15\), GPT\-4o and GPT\-o1 \(2024\-05\-13\), and LLaMA\-3\.1 8B \(2023\-07\-10\)\.

#### 5\.1\.2\.Baselines

To comparison, we included four non\-trivial dialogue re\-lexicalization baselines:

- •Random\-ALLgenerates a re\-lexicalized dialoguedr​e​l​e​xd\_\{relex\}by randomly assigning variables indd​e​l​e​xd\_\{delex\}to any slot values present in theK​BKB, regardless of their slot type\.
- •Random\-SLOTalso assigns variables randomly but restricts the selection to values associated with the same slot type as the original\.
- •Most Frequent\-ALLbaseline assigns variables indd​e​l​e​xd\_\{delex\}to the most frequent slot values found across all slots in theK​BKB\.
- •Most Frequent\-SLOTbaseline selects the most frequent value from the same slot type as the original\.

#### 5\.1\.3\.Evaluation Metrics

We useGlobal Consistency Accuracy\(GCA\) andVariable Consistency Accuracy\(VCA\) as the metrics to evaluate the adherence of a dialogue to a specific set of constraints\. Given a re\-lexicalized dialoguedr​e​l​e​xd\_\{relex\}where CSP variables are assigned to values, GCA measures the overall accuracy of the assignments for each variable\. The average GCA is calculated as the proportion of dialogues that fully comply with all defined constraints:

G​C​A=∑i=1N\(∏j=1MSatisfies​\(Ai,Cj\)\)NGCA=\\frac\{\\sum\_\{i=1\}^\{N\}\\left\(\\prod\_\{j=1\}^\{M\}\\textit\{Satisfies\}\(A\_\{i\},C\_\{j\}\)\\right\)\}\{N\}whereNNis the total number of dialogues, andSatisfies​\(Ai,Cj\)\\textit\{Satisfies\}\(A\_\{i\},C\_\{j\}\)is a binary indicator function that returns 1 if and only if all variable assignments in dialoguedid\_\{i\}comply with the constraintjj, 0 otherwise\. On the other hand, VCA assesses the assignment accuracy on individual variables within the dialogue\. We compare the dialogue assignment to the solutions of the CSP solver and find the most similar solution; then, we count how many variable assignments coincide with the assignments of the most similar solution\. We formally define VCA as follows:

V​C​A=∑i=1N\|CorrectAssignments​\(di\)\|MVCA=\\frac\{\\sum\_\{i=1\}^\{N\}\\lvert\\textit\{CorrectAssignments\}\(d\_\{i\}\)\\rvert\}\{M\}
whereNNis the total number of dialogues,MMis the total number of variables in the dialogues, andCorrectAssignments​\(di\)\\textit\{CorrectAssignments\}\(d\_\{i\}\)are the variable assignments in dialoguedid\_\{i\}that coincide with the assignments of the most similar solution provided by the CSP solver\. GCA and VCA provide insights into the ability of the dialogue generation system to maintain coherence and fidelity to the underlying domain knowledge while generating responses\. Higher values of GCA and VCA indicate better performance in terms of dialogue quality and consistency, unlike traditional dialogue evaluation metrics \(e\.g\., BLEU, ROUGE, or perplexity\)\.

Additionally, the process used for computing VCA can be extended to identify specific errors within a dialogue\. In cases where a dialogue is not among the solutions identified by the CSP, the most similar solution can be used to detect erroneous slot\-value assignments\. Specifically, errors are defined as slot\-values that, if corrected, would result in a solution satisfying all constraints\. This enables the generation of detailed reports pinpointing the errors in the dialogue, facilitating more targeted improvements\.

MethodGCAVCARandom\-ALL0\.010\.02Random\-SLOT0\.010\.12Most Frequent\-ALL0\.010\.11Most Frequent\-SLOT0\.060\.23Llama\-3\.1 8B0\.030\.08GPT\-3\.5\-turbo0\.110\.37GPT\-4o0\.140\.41GPT\-o10\.140\.42Table 2\.Baselines and model performance on re\-lexicalizing TODs\.

### 5\.2\.Results

Table[2](https://arxiv.org/html/2607.09338#S5.T2)shows that GPT\-4o and GPT\-o1 outperform other models, although absolute performance remains moderate, indicating that constraint satisfaction is still challenging for LLMs\. Performance improves as the number of valid CSP solutions increases, suggesting that dialogues with higher flexibility are easier to generate correctly\. The constraint analysis \(Table[4](https://arxiv.org/html/2607.09338#S5.T4)\) highlights the importance of domain\-level constraints, particularlyC​6C6, in ensuring consistency\. These findings indicate that while modern LLMs can partially capture structured constraints, explicit modeling via CSP remains beneficial for enforcing consistency and diagnosing errors\.

#### 5\.2\.1\.Ablation Study

Table[3](https://arxiv.org/html/2607.09338#S5.T3)presents the results of an ablation study we conducted\. The ablation study removes one constraint at a time to measure impact on GCA and VCA\. Results indicate thatC​6C6\(exact match with KB instances\) is the most critical, followed byC​1C1\(hard constraints on slot values\)\.

ConstraintGCAVCAall except C10\.150\.45all except C20\.150\.42all except C30\.150\.45all except C40\.150\.46all except C50\.150\.45all except C60\.210\.48all exceptdialogic0\.150\.45all exceptdomain0\.230\.56Table 3\.Ablation study: global and variable consistency under different constraint configurations\.Constraint\# variables% coverageC19281100%C2608466%C3112412%C43013%C5236926%C6425746%Table 4\.Number and proportion of variables affected by each constraint\.
#### 5\.2\.2\.Additional Results: CSP Solution Distribution

Dataset\# dialogues\# variablesAll95092811 solution18542\-10 solutions13486811\-100 solutions2862332101\+ solutions3063151Table 5\.Dialogue distribution based on CSP solutions \(MiniZinc\)\.Table[5](https://arxiv.org/html/2607.09338#S5.T5)reports the distribution of dialogues based on the number of valid CSP solutions\. Dialogues with a higher number of solutions tend to allow more flexibility in variable assignments, which correlates with improved LLM performance\.

## 6\.Related Work

TOD systems have been extensively investigated in NLP\(Allenet al\.,[2001](https://arxiv.org/html/2607.09338#bib.bib169)\)\. Recent research has explored the use of neural network architectures for dialogue state tracking\(Wuet al\.,[2020](https://arxiv.org/html/2607.09338#bib.bib187); Zhaoet al\.,[2021](https://arxiv.org/html/2607.09338#bib.bib206); Labruna and Magnini,[2023](https://arxiv.org/html/2607.09338#bib.bib144)\)and policy learning\(Suet al\.,[2016](https://arxiv.org/html/2607.09338#bib.bib200); Liu and Lane,[2017](https://arxiv.org/html/2607.09338#bib.bib199)\)\. Several metrics have been proposed to assess the performance of TOD systems, including task completion rates, user satisfaction scores, and objective measures for system components, such as precision, recall, and F1\-score\(Chenet al\.,[2017](https://arxiv.org/html/2607.09338#bib.bib195); Santhanam and Shaikh,[2019](https://arxiv.org/html/2607.09338#bib.bib194); Deriuet al\.,[2021](https://arxiv.org/html/2607.09338#bib.bib193)\)\. Recent studies have emphasized the importance of holistic evaluation frameworks that consider multiple aspects of dialogue quality\(Zhanget al\.,[2021](https://arxiv.org/html/2607.09338#bib.bib192); Labrunaet al\.,[2024](https://arxiv.org/html/2607.09338#bib.bib232)\)\. Maintaining consistency and coherence in dialogues is essential for effective communication between users and dialogue systems\. Previous research has investigated various approaches to ensure dialogue coherence, including coherence modeling\(Cervoneet al\.,[2018](https://arxiv.org/html/2607.09338#bib.bib190)\), and coherence\-based response generation\(Cervone and Riccardi,[2020](https://arxiv.org/html/2607.09338#bib.bib189)\), aiming to enhance the naturalness and fluency of generated dialogues\. Finally, several studies have explored the application of CSPs to language\. These include early attempts to ensure coherence in generated text\(Kibble and Power,[2004](https://arxiv.org/html/2607.09338#bib.bib219)\), model preposition lexicalization using constraints\(Moriceau and Saint\-Dizier,[2004](https://arxiv.org/html/2607.09338#bib.bib223)\), guide lexical choices through constraints\(McKeownet al\.,[1997](https://arxiv.org/html/2607.09338#bib.bib220)\), and treat context\-sensitive utterance generation as a CSP\(Popescuet al\.,[2009](https://arxiv.org/html/2607.09338#bib.bib221)\)\. Differently to these works, our approach focuses on detecting inconsistencies in already generated TOD dialogues using CSP\.

## 7\.Conclusion

Generative LLMs may produce inconsistent TODs, due to misalignment between parametric memory and the TODK​BKB\. We have introduced a novel approach to detect TOD inconsistencies based on Constraint Satisfaction\. Several experiments demonstrate the feasibility of the approach, enabling to effectively identify and quantify inconsistencies present in TODs with high accuracy \(75\.9% with GPT\-4o and CSP solver\)\. We also analysed the LLM inconsistencies when tasked to re\-lexicalize TODs, finding that they primarily concern domain knowledge adherence, resulting in an overall accuracy of only 0\.14 at the dialogue level\. Our study highlights the potential of CSP\-based methodologies in evaluating dialogue consistency and identifying areas for improvement in automated dialogue generation systems\. Future research should further explore the application of CSP in task\-oriented dialogues and investigate strategies to enhance the coherence of LLM\-generated dialogues, particularly in applications with strong domain knowledge requirements\.

## Limitations

While the proposed Constraint Satisfaction Problem \(CSP\)\-based approach offers a novel and effective method for detecting inconsistencies in task\-oriented dialogues \(TODs\), it presents several limitations\.

The system relies on the explicit mapping of dialogues into variable\-constraint representations\. Although our method is domain\-independent in principle, the process of extracting variables and constraints from dialogues may require customization or adaptation for new domains or dialogue schemas\.

Our method focuses on identifying inconsistencies and suggesting minimal changes for correction, but it does not automatically regenerate fluent or user\-aligned responses after such modifications\. This leaves the generation of corrected natural language utterances as future work\.

Finally, although our experimental results are promising, they are based on controlled datasets and manually designed inconsistencies\. Further work is needed to assess robustness in more complex or organically generated dialogues\.

## Ethical Considerations

Use of Scientific Artifacts\.We used publicly available task\-oriented dialogue datasets for experimentation\. These datasets include MultiWOZ\(Budzianowskiet al\.,[2018](https://arxiv.org/html/2607.09338#bib.bib45)\)and variations based on it\. Additionally, we used off\-the\-shelf large language models \(LLMs\) to generate new dialogues with intentional inconsistencies for controlled evaluation\. All code developed for the CSP\-based inconsistency detection pipeline is our original contribution and will be made publicly available for research purposes under an open\-source license\.

Licensing and Intended Use\.All external datasets and models used in this work were employed in accordance with their licenses\. Our use was consistent with the intended purpose of the datasets \(research\), and we explicitly specify that the CSP\-based system and associated data artifacts are intended solely for research and educational use\. Any derivative dataset created using our framework also inherits this research\-only restriction\.

Privacy and Data Integrity\.The dialogue data used in this study does not include personally identifiable information \(PII\), and no effort was made to collect or infer such data\. We manually verified the synthetic and benchmark dialogues for inappropriate or offensive content, and none was found\. Our system does not involve any human annotation beyond the authors, so no consent or risk disclaimers were required\.

Documentation and Statistics\.All artifacts, including the experimental codebase, constraint templates, and synthetic dialogue generation scripts, are provided with the submission\. This includes coverage across domains, types of slot\-value inconsistencies, and linguistic patterns\. We report the number of dialogue examples used in each experiment, as well as their train/test splits, in the experimental section\. We also provide accuracy scores as descriptive statistics for the evaluation\.

Computational Resources\.Model generation and evaluation were conducted using a single NVIDIA A40 GPU, with a total budget of approximately 40 GPU hours\. We do not fine\-tune any large models; our work only uses them in inference mode\.

Use of Existing Software\.Our system uses standard NLP libraries such as Hugging Face Transformers and MiniZinc ‘constraint‘ solver library\. All packages were used with default or explicitly documented parameters\.

Human Participants\.No human participants were recruited for this study, and no user studies or annotation tasks involving external contributors were conducted\. Therefore, issues such as compensation or informed consent do not apply in our setting\.

## References

- J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)Gpt\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.09338#S5.SS1.SSS1.p1.2)\.
- J\. Allen, G\. Ferguson, and A\. Stent \(2001\)An architecture for more realistic conversational systems\.InProceedings of the 6th international conference on Intelligent user interfaces,pp\. 1–8\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- V\. Balaraman, S\. Sheikhalishahi, and B\. Magnini \(2021\)Recent neural methods on dialogue state tracking for task\-oriented dialogue systems: a survey\.InProceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2021, Singapore and Online, July 29\-31, 2021,pp\. 239–251\.External Links:[Link](https://aclanthology.org/2021.sigdial-1.25)Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- Y\. Bang, S\. Cahyawijaya, N\. Lee, W\. Dai, D\. Su, B\. Wilie, H\. Lovenia, Z\. Ji, T\. Yu, W\. Chung,et al\.\(2023\)A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity\.arXiv preprint arXiv:2302\.04023\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- S\. C\. Brailsford, C\. N\. Potts, and B\. M\. Smith \(1999\)Constraint satisfaction problems: algorithms and applications\.European Journal of Operational Research119\(3\),pp\. 557–581\.External Links:ISSN 0377\-2217,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S0377-2217%2898%2900364-6),[Link](https://www.sciencedirect.com/science/article/pii/S0377221798003646)Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p2.1),[§2\.3](https://arxiv.org/html/2607.09338#S2.SS3.p1.1)\.
- P\. Budzianowski, T\. Wen, B\. Tseng, I\. Casanueva, S\. Ultes, O\. Ramadan, and M\. Gašić \(2018\)MultiWOZ–a large\-scale multi\-domain wizard\-of\-oz dataset for task\-oriented dialogue modelling\.arXiv preprint arXiv:1810\.00278\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2),[Ethical Considerations](https://arxiv.org/html/2607.09338#Sx2.p1.1)\.
- A\. Cervone and G\. Riccardi \(2020\)Is this dialogue coherent? learning from dialogue acts and entities\.arXiv preprint arXiv:2006\.10157\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- A\. Cervone, E\. Stepanov, and G\. Riccardi \(2018\)Coherence models for dialogue\.arXiv preprint arXiv:1806\.08044\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- H\. Chen, X\. Liu, D\. Yin, and J\. Tang \(2017\)A survey on dialogue systems: recent advances and new frontiers\.Acm Sigkdd Explorations Newsletter19\(2\),pp\. 25–35\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- H\. Cho, C\. Sankar, C\. Lin, K\. R\. Sadagopan, S\. Shayandeh, A\. Celikyilmaz, J\. May, and A\. Beirami \(2022\)Know thy strengths: comprehensive dialogue state tracking diagnostics\.InFindings of the Association for Computational Linguistics: EMNLP 2022,Y\. Goldberg, Z\. Kozareva, and Y\. Zhang \(Eds\.\),Abu Dhabi, United Arab Emirates,pp\. 5345–5359\.External Links:[Link](https://aclanthology.org/2022.findings-emnlp.391/),[Document](https://dx.doi.org/10.18653/v1/2022.findings-emnlp.391)Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- G\. Chu, P\. J\. Stuckey, A\. Schutt, T\. Ehlers, G\. Gange, and K\. Francis \(2018\)Chuffed, a lazy clause generation solver\.Note:[https://github\.com/chuffed/chuffed](https://github.com/chuffed/chuffed)Cited by:[§4\.1](https://arxiv.org/html/2607.09338#S4.SS1.p2.1)\.
- J\. Deriu, A\. Rodrigo, A\. Otegi, G\. Echegoyen, S\. Rosset, E\. Agirre, and M\. Cieliebak \(2021\)Survey on evaluation methods for dialogue systems\.Artificial Intelligence Review54,pp\. 755–810\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan,et al\.\(2024\)The llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.09338#S5.SS1.SSS1.p1.2)\.
- M\. Hendersonet al\.\(2014\)The second dialog state tracking challenge\.InProceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue \(SIGDIAL\),Philadelphia, PA, U\.S\.A\.,pp\. 263–272\.External Links:[Link](https://www.aclweb.org/anthology/W14-4337),[Document](https://dx.doi.org/10.3115/v1/W14-4337)Cited by:[§2\.1](https://arxiv.org/html/2607.09338#S2.SS1.p1.3)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[§5\.1\.1](https://arxiv.org/html/2607.09338#S5.SS1.SSS1.p1.2)\.
- Z\. Jiet al\.\(2022\)Survey of hallucination in natural language generation\.ACM Computing Surveys\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- R\. Kibble and R\. Power \(2004\)Optimizing referential coherence in text generation\.Computational Linguistics30\(4\),pp\. 401–416\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- V\. Kumar \(1992\)Algorithms for constraint\-satisfaction problems: a survey\.AI magazine13\(1\),pp\. 32–32\.Cited by:[§2\.3](https://arxiv.org/html/2607.09338#S2.SS3.p1.1)\.
- T\. Labruna, S\. Brenna, G\. Bonetta, and B\. Magnini \(2024\)Are you a good assistant? assessing llm trustability in task\-oriented dialogues\.Clic\-It 2024\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2),[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- T\. Labruna and B\. Magnini \(2023\)Addressing domain changes in task\-oriented conversational agents through dialogue adaptation\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop,pp\. 149–158\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- T\. M\. Lai, G\. Castellucci, S\. Kuzi, H\. Ji, and O\. Rokhlenko \(2023\)External knowledge acquisition for end\-to\-end document\-oriented dialog systems\.InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics,pp\. 3633–3647\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- B\. Liu and I\. Lane \(2017\)Iterative policy learning in end\-to\-end trainable task\-oriented neural dialog models\.In2017 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 482–489\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- K\. McKeown, M\. Elhadad, and J\. Robin \(1997\)Floating constraints in lexical choice\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- M\. McTear \(2020\)Conversational ai: dialogue systems, conversational agents, and chatbots\.Synthesis Lectures on Human Language Technologies13\(3\),pp\. 1–251\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- V\. Moriceau and P\. Saint\-Dizier \(2004\)A constraint\-based model for preposition choice in natural language generation\.Constraint Solving and Language Processing,pp\. 124\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- N\. Nethercote, P\. J\. Stuckey, R\. Becket, S\. Brand, G\. J\. Duck, and G\. Tack \(2007\)MiniZinc: towards a standard cp modelling language\.InCP 2007,C\. Bessiere \(Ed\.\),LNCS, Vol\.4741,pp\. 529–543\.External Links:[Link](http://www.minizinc.org/)Cited by:[§4\.1](https://arxiv.org/html/2607.09338#S4.SS1.p2.1)\.
- V\. Popescu, J\. Caelen, and C\. Burileanu \(2009\)A constraint satisfaction approach to context\-sensitive utterance generation in multi\-party dialogue systems\.International Journal of Speech Technology12,pp\. 95–112\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- L\. Qin, W\. Pan, Q\. Chen, L\. Liao, Z\. Yu, Y\. Zhang, W\. Che, and M\. Li \(2023\)End\-to\-end task\-oriented dialogue: a survey of tasks, methods, and future directions\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 5925–5941\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.363/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.363)Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- S\. Santhanam and S\. Shaikh \(2019\)Towards best experiment design for evaluating dialogue system output\.arXiv preprint arXiv:1909\.10122\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- P\. Su, M\. Gasic, N\. Mrksic, L\. Rojas\-Barahona, S\. Ultes, D\. Vandyke, T\. Wen, and S\. Young \(2016\)On\-line active reward learning for policy optimisation in spoken dialogue systems\.arXiv preprint arXiv:1605\.07669\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- P\. Wu, B\. Zou, R\. Jiang, and A\. Aw \(2020\)GCDST: a graph\-based and copy\-augmented multi\-domain dialogue state tracking\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 1063–1073\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- S\. Young, M\. Gašić, B\. Thomson, and J\. D\. Williams \(2013\)Pomdp\-based statistical spoken dialog systems: a review\.Proceedings of the IEEE101\(5\),pp\. 1160–1179\.Cited by:[§1](https://arxiv.org/html/2607.09338#S1.p1.2)\.
- C\. Zhang, G\. Lee, L\. F\. D’Haro, and H\. Li \(2021\)D\-score: holistic dialogue evaluation without reference\.IEEE/ACM Transactions on Audio, Speech, and Language Processing29,pp\. 2502–2516\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.
- J\. Zhao, M\. Mahdieh, Y\. Zhang, Y\. Cao, and Y\. Wu \(2021\)Effective sequence\-to\-sequence dialogue state tracking\.arXiv preprint arXiv:2108\.13990\.Cited by:[§6](https://arxiv.org/html/2607.09338#S6.p1.1)\.