// Your LLM judge disagrees with the experts //

LLM Judges can be tricky to build.

Here is an interesting showcasing why:

There propose a reference-full benchmark of hundreds of complete human-to-human dialogues written by professional script writers, with realistic turn densities and more than 36,000 per-turn human annotations across over 30,000 expert-generated turns.

Conversational evaluation frameworks were mostly built for summarization, translation and short-form QA, and the metrics themselves are often derived and validated on synthetic data rather than human dialogue.

Tested against expert judgment at this scale, both classical automatic metrics and reference-free LLM-as-a-judge approaches turn out to be unreliable.

Their Mixture-of-Judges framework combines multiple evaluative signals and recovers roughly 30 percent better correlation with human assessment.

Paper: https://arxiv.org/abs/2608.26131

Chat with Paper: https://academy.dair.ai/papers/evaluating-language-models-in-realistic-conversational-contexts-2608.26131…

Evaluating Language Models in Realistic Conversational Contexts

Source: https://arxiv.org/html/2608.26131

Abstract

As Large Language Models (LLMs) are increasingly deployed to serve open-ended, multi-turn interactions, evaluating conversational quality at human scale has become a central challenge. Existing evaluation frameworks built for summarization, translation, or short-form QA tasks fall short of adequately measuring the consistency of human-scale dialogue, especially when derivation and validation of these metrics themselves often rely on synthetic rather than human sources. We fill the gap by introducingUPHELD(UPwork Human-Scale Evaluated Long Dialogues), a large, reference-full benchmark for evaluating human-scale conversational ability beyond factual correctness. UPHELD consists of hundreds of complete human-to-human dialogues authored byprofessional script writers, with realistic turn densities and36,000+ per-turn human annotationsacross30,000+ expert-generated dialogue turns. Using UPHELD, we systematically evaluate classical automatic metrics and reference-free LLM-as-a-judge approaches, and find them unreliable when correlated with expert human judgment. Building off this analysis, we use UPHELD to develop aMixture-of-Judgesframework that combines multiple evaluative signals and improves correlation with human assessments by approximately30%. Overall, UPHELD provides a robust, human-grounded foundation for evaluating human-scale conversational intelligence that fills a crucial gap in the pre-existing LLM dataset landscape.

1Introduction

The rapid advancement of Large Language Models (LLMs) has revolutionized text generation, particularly in focused, single-turn tasks like question answering. However, as these models are increasingly integrated into applications serving complex systems, the frontier of evaluation has shifted toward open-ended, multi-turn settings. In these environments, conversations are longer, less constrained, and more open-ended, and success is not merely defined by factual correctness but by the model’s ability to maintain a natural, coherent, and useful interaction with a human end-user over an extended dialogue. Existing evaluation metrics, largely inherited from pre-LLM natural language tasks such as machine translation, summarization, and question answering, focus on factual precision and lexical overlap. These metrics generally prove inadequate when assessing the nuanced qualities of a successful conversation, which depends on softer, long-range qualities like coherence, engagement, and the ability to maintain consistent tone over multiple turns. While recent benchmarks have evaluated longer dialogues through reference-free “LLM-as-a-judge” evaluations(Zhenget al.,2023; Liuet al.,2023; Duboiset al.,2023)or human preference platforms(Chianget al.,2024), these methods face critical limitations. LLM judge quality often fails to exceed that of non-expert human judges(Bavarescoet al.,2024)(Krumdicket al.,2025), and such benchmarks are increasingly susceptible to training data leakage(Mirzadehet al.,2024)and poor reliability on out-of-distribution tasks(Krumdicket al.,2025). Additionally, many of the metrics are tested primarily on synthetic datasets, which can introduce significant biases in metric results due to empirically verified limits in LLM dialogue generation(Wanget al.,2025)and persistent biases in LLM-simulated data used for both training and evaluation(Rahmaniet al.,2025).

The work presented here mitigates these issues through a specialized benchmark evaluating models on conversations, which are designed to be:

• •Multi-turn and human-to-human.Emphasizing naturalistic flow of real interaction rather than simulated or distilled outputs.
• •Not anchored to specific external knowledge.Shifting the focus from retrieval-based accuracy to conversational intelligence.
• •Not anchored to a specific outcome.Allowing for fluid, open-ended dialogue without focusing on a “right answer“ (e.g. coding, summarization), but still having an overall goal (e.g., “tell me how to make a pizza”).

We introduceUPHELD (UPwork Human-Scale Evaluated Long Dialogues), a novel collection of conversations that fulfill these criteria. UPHELD contains over 30,000 high-quality human-written turns of dialogue across hundreds of novel conversations, and for each turn we present multiple human labels along important dialogue coherence criteria like stylistic and content consistency. Unlike datasets extracted from noisy sources(Lison and Tiedemann,2016)or those relying on crowdsourced platforms where annotator backgrounds are unknown(Zhanget al.,2018), UPHELD utilizedprofessional script writersto guarantee high conversational quality, which has been shown to lead to notable improvements in the quality and authenticity of dataset collection(Pilanet al.,2024; Elishaet al.,2024).

Our strategy for data collection and general approach allows UPHELD clear divergence from pre-existing dialogue benchmarks in three fundamental ways (a more detailed point-by-point overview is given in AppendixA):

• •Task-oriented, natural dialogue:UPHELD focuses on task-oriented dialogues (e.g., math tutoring or trip planning), but all professional script writers were instructed to write dialogues in a natural, casual way. This deviates significantly from standard task-oriented datasets where agents follow a rigid “set” target or are constrained by encoded model knowledge. By providing writers the freedom to determine how the task is navigated, we capture a more naturalistic intersection of utility and human-like interaction.
• •Realistic conversational turn density:While some datasets focus on extremely long conversations, such as interviews or LoCMo(Maharanaet al.,2024), most human-to-human interactions are more concise and especially in the guided task-oriented space. Our analysis of human-to-human interaction in available datasets shows that real-world dialogue rarely exceeds 20 turns. AppendixAshows that the average of fully human-collected datasets is 9.3 turns, while the Topical-Chat dataset(Gopalakrishnanet al.,2019)maxes at 21 turns. Our writers were given the freedom to determine conversation lengths and end at natural stopping points to keep the interactions as natural as possible, which means UPHELD better reflects real-world turn distributions.
• •Emphasis on conversational fluency versus information retrieval:We deliberately avoid tasks requiring complex context, such as complex mathematics or coding. Instead, UPHELD focuses on the model’s ability to hold a coherent and reasonable conversation throughout an entire dialogue, and in service of everyday tasks that do not require heavy context engineering. This distinguishes our work from benchmarks like Wizard of Wikipedia(Dinanet al.,2018), which prioritize knowledge-grounding over pure conversational dynamics.

In order to further illustrate the critical gaps that UPHELD was designed to bridge, we include a deep analysis of existing methods like LLM-as-a-judge or classical NLP metrics, and where they fall short on evaluating LLMs in conversational settings when tested against UPHELD. Our analysis shows that these metrics perform poorly for the task in question, and we demonstrate how to use UPHELD to generate better evaluation metrics through a simple machine-learned mixture-of-judges (ensemble) metric. Across evaluations, the proposed metrics demonstrate higher correlation with human judgments than single LLM-as-a-judge metrics as well as established syntactic and semantic scoring methods.

Our main contributions are as follows:

• •We present UPHELD, a novel dataset of 10,000+ dialogue turns crafted byprofessional script writers. With153,578 dense, per-turn human labelsthat rate turn quality along various coherence criteria, UPHELD offers an evaluative scale nearly 10x that of comparable human-verified benchmarks(Zhanget al.,2023), effectively bridging the quality and volume relative to crowdsourced or synthetic datasets like in(Elishaet al.,2024; Rahmaniet al.,2025).
• •We focus on realistic task-oriented dialogues tied to natural dialogue length distributions rather than purely casual, open-ended dialogue. In doing so, UPHELD effectively evaluates functional utility within naturalistic human cognitive boundaries, avoiding the artificial turn-length inflation common in synthetic datasets(Wanget al.,2025).
• •We introduce a multi-turn consistency metric that evaluates dialogue through turn-level coherence and reasonableness. We demonstrate that while standard metrics exhibit a significant performance gap on this task, our proposed Mixture-of-Judges ensemble metric is more successful at predicting expert human judgment by∼\sim30% over individual LLM judges.

2Related Work

Evaluating the performance of large language models (LLMs) is a critical area of research. Current benchmarks, such as those on the Open LLM leaderboard (now archived)(Myrzakhanet al.,2024), often focus on task-based evaluations using input-output metrics. These metrics include task-based generation evaluations like IFEval(Zhouet al.,2023)which evaluates the ability of LLMs to follow instructions and BBH (Big Bench Hard(Srivastavaet al.,2022)), which comprises 23 groups of tasks such asword sorting,casual judgment,navigate. These and similar task-with-answer evaluations utilize metrics such as Exact Match (EM), Precision/Recall/F1, and substring-based accuracy. Specific examples include MATH for mathematical problems(Hendryckset al.,2021), and GPQA for multi-choice question answering(Reinet al.,2023). Unlike benchmarks focused on specific tasks (MATH, GPQA) or conversation quality, HELM(Lianget al.,2023)provides a unified framework evaluating models across different scenarios and metrics, emphasizing evaluation methodology rather than a specific task domain. However, it is still single turn based and is built on top of right/wrong factual answer comparison.

Question answering (QA) benchmarks are also prevalent. MUSR(Spragueet al.,2023)is a narrative-based QA evaluation where the input is a paragraph and the output is an answer with an evaluation score. MMLU-PRO(Wanget al.,2024)combines task-based evaluation with QA and chain-of-thought prompting. Recently, using an LLM-as-a-judge for evaluation is another approach which enables evaluating without having a ground truth or a reference answer(Zhenget al.,2023)(Zhanget al.,2023),(Duanet al.,2023),(Adlakhaet al.,2024). While the aforementioned benchmarks are prominent, several related areas of work are not typically included in leaderboards focused on single-turn evaluations, particularly those concerning longer conversations. These include approaches similar to Chatbot Arena(Chianget al.,2024), MultiHop QA(Schnitzleret al.,2024)which involves answering questions over multiple documents or turns, and MT-eval which focuses on evaluating conversations directly and multi-dimensionally(Kwanet al.,2024). BotChat(Duanet al.,2023)is also noted as a method for conversation generation evaluation.

The above works often rely on input-output pairs, exact matches, or scores tied to specific correct answers, which are well-suited for evaluating the ability to perform discrete tasks or extract factual information. However, they may not be appropriate for evaluating long, non-topical conversations. Such interactions involve sustained coherence, context management over many turns, engaging dialogue, and the ability to handle subjective or open-ended discussions that do not have a clear and concise “correct” answer. Current human-scale conversation datasets focusing on non-factual conversations, including MuTal(Cuiet al.,2020), Topical-chat(Gopalakrishnanet al.,2019), LLM-arena(Zhenget al.,2023), and DailyDialogue (Multi-turn)(Liet al.,2017), often lack authentic human-to-human interaction or are constrained to predefined topics. Recognizing these limitations, and to construct a robust benchmark, we identified the need for a dataset characterized by comprehensive curation, expert authorship, and human annotation. This paper introduces such a dataset, which we have meticulously collected and organized, with details provided in the ensuing sections.

3Datasets

3.1UPHELD dataset

We first collected conversational data by tasking a panel of professional writers hired from the Upwork freelancer marketplace to create open-ended, naturalistic dialogues across a diverse range of topics like customer service and education. The writers were explicitly instructed to create longer dialogues characterized by conversational complexity and exploratory interaction, rather than producing conversations with predetermined or narrow outcomes. For example, the writers developed dialogues exploring scenarios such as mobile phone selection, travel planning, or problem-solving discussions that were goal-oriented and where multiple nuanced exchanges would be appropriate. Writers were also instructed to avoid overly scripted or linear communication patterns. This approach ensures that the dataset captures the diverse and intricate nonlinear nature of human conversation. We also took steps to alleviate any potential writer bias (see details in AppendixC). The UPHELD dataset is publicly available under a permissivehttps://creativecommons.org/licenses/by/4.0/CC-BY-4.0 license and downloadable fromhttps://github.com/upwork/upwork-ai-research/tree/master/data/upheld.

3.1.1Input data

Given our initial set of rich natural language conversations, various LLM models were then used to output candidate completions at every turn of every conversation. Specifically, models were presented with conversation history up to a specific point, with the next human-written turn withheld. Models then generated a predicted next turn. The following models were used to generate the dataset: a)GPT-3.5(OpenAI,2023); b)GPT-4o(OpenAI,2024); c)GPT-5(Singhet al.,2025); d)Llama-3.1-70bandLlama-3.3-70b(Grattafioriet al.,2024); e)Kimi-K2.5(Bai and others,2026); f)Qwen3-32B(Yanget al.,2025); g)DeepSeek v3(DeepSeek-AIet al.,2025))custom modelwe callUma-V0.5(a fine-tuned version of Llama-3.1: see Section5.1). In addition to these, we added two baseline candidate completions: a) a version of the ground truth (human) output rephrased using GPT-4o and b) a random completion. These additional baselines provide an important calibration signal for the dataset, and as expected we found that the rephrased dataset received the highest marks while the random dataset received the lowest marks when evaluated by a human labeler. These checks provide us with additional confidence that humans are fair judges of our evaluation axes – style, content, and reasonableness.

At every possible conversational turn, annotators were then given (1) the chat history up to that point, (2) the ground truth human completion (Option A), and (3) one of the LLM-generated candidate completions (Option B). The provenance of options A and B were not disclosed to the annotators. Their task was to compare these continuations on content consistency, style consistency, and general reasonableness. The tasks, metric scales, and instructions were fine-tuned through two paid pilot studies. Example data-points within UPHELD are given in Table1, and full instructions provided to the annotators can be found in AppendixB.

Table 1:Input conversations and associated annotator ratings from a single annotator (all scores are out of 5). More examples can be found in AppendixL.chat_historyuser: I’m shopping for a new car and can’t decide what to get. assistant: I can help with that. What size car are you looking for?user: I’m really frustrated. I’ve been trying to get this printer to work all morning and it just won’t print anything! assistant: I understand that must be really annoying. Let’s work together to fix this. Can you tell me if you see any error messages on your printer or computer? user: No error messages. It just sits there doing nothing whenever I try to print! assistant: Alright, let’s try a few steps to see if we can get it responding. First, could you check whether the printer is set as the default printer on your computer? user: How do I check that?Option AThe Ford F-150 is one of the top selling trucks in the country. Have you taken one of those for a test drive?If you’re using a Windows PC, you can go to ’Devices and Printers’ in your control panel. For Mac, you can check under ’System Preferences’ and then ’Printers & Scanners’.Option BTrucks are great for hauling and towing. Here are a few questions to help narrow down your options: 1. What will you be using the truck for most often? (e.g., work, off-roading, daily driving, towing a trailer) 2. How many passengers will the truck need to seat? 3. Do you have a preferred fuel type (gasoline, diesel, hybrid)? 4. What’s your budget for the truck? 5. Do you have a specific brand or model in mind? 6. Do you need any specific features, such as 4-wheel drive, a certain bed size, or advanced safety featuresIt’s easy. On a Windows computer, you can go to the Control Panel, then Devices and Printers (or Printers and Devices in some versions). Look for your printer’s name and see if it has a checkmark next to it that says “Default printer”. If it doesn’t, right-click on the printer and select “Set as default printer”. On a Mac, you can go to System Preferences, then Printers & Scanners. Look for your printer’s name and make sure it’s selected as the default printer. If you’re still having trouble, we can try some other steps. Have you tried restarting the printerscore_task_1_content1: Strongly Disagree – no content overlap3: Neutral – some overlap of contentscore_task_2_style1: Different styles1: Different stylesscore_task_3_reasonableness1: Not a reasonable continuation5: A reasonable continuation

3.1.2Annotation dimensions

Across all candidate models, each model-generated turn was evaluated by five independent human annotators using the following scoring criteria (see full instructions in AppendixB).

• •Content Equivalence: A 5-point Likert scale (1-5) assessing semantic similarity between generated and reference turns.
• •Style Equivalence: A 3-point Likert scale (1-3) measuring linguistic and stylistic correspondence.
• •Utterance Reasonableness: A binary scale (1: unreasonable, 5: reasonable) evaluating the contextual appropriateness and coherence of the generated response,

The Style Equivalence scale we used is on a three level scale with scores (1 - 3 - 5) based on the initial pilot studies were participants suggested that 1-5 “standard” scale for “Style Equivalence” was more confusing compared to “Content Equivalence” The annotators were provided with the explanation of these three levels as well. For the reasonableness scale we explored different options including non-binary and pairwise comparisons (e.g., “Answer A is more reasonable than Answer B”). Based on the same pilot study we opted for a binary “Reasonableness”. During this phase, we found that because reasonableness is often perceived by humans as an absolute quality—either a response makes sense within the dialogue flow or it does not—annotators struggled to consistently rank one reasonable response over another. Our conclusion was that if humans primarily evaluate reasonableness as a binary attribute, forcing them to compare two such responses introduces significant subjective noise and reduces inter-annotator agreement.

3.1.3Data statistics

In total we collected complete evaluation labels for 800 conversations made up out of 9,203 turns/utterances, with with an average of 11.29 turns per conversation collected for this study and additional conversations from existing datasets(Cuiet al.,2020)to serve as control points. Each predicted turn was evaluated by five human annotators, and each annotator judge labeled at most 2,000 conversation continuations. Overall, we generated 30,715 sets of labels, or 153,578 labels (one set was annotated by 5 annotators). The ground truth conversations consist of 5.6 turn pairs (user-assistant) or 11.29 dialogue turns on average. The average length of the conversation history annotators analyzed was 560 characters and the length of the judged turns was on average 245 characters.

3.1.4Dataset content and quality checks

The dataset was curated to include a wide range of topics and situations. The writers were given high-level scenario descriptions and stylistic guidelines and were instructed to produce naturalistic, human-like conversations without being constrained to specific task templates. While some conversations contain elements of reasoning or factual exchange, these emerged organically rather than being explicitly assigned tasks. The dialogues in UPHELD cover a diverse range of human-scale situational contexts, including:

• •Decision Making: (e.g., A gym owner helping a client decide on a personal trainer).
• •Customer Service: (e.g., Navigating a website to cancel a subscription service).
• •General Life Advice: (e.g., An academic advisor helping a student choose a major).
• •Tutoring: (e.g., Solving multi-step math word problems involving currency and change).
• •Task Assistance: (e.g., A chef guiding a novice through a recipe for the first time).
• •Debate: (e.g., Speculative discussion on the state of the world in 50 years).

To minimize bias and ensure high quality of the collected data, we added several layers of checks and run several paid pilot studies (see AppendixC).

• •Writer Selection: All writers possessed high success rates on Upwork and passed a rigorous paid pilot phase evaluated by professional UX researchers for diversity and guideline adherence.
• •Diverse Expertise: Writers were recruited from varied backgrounds (novel writing, education, copywriting) to ensure situational and stylistic diversity.
• •Multi-Stage Review: Each conversation underwent independent checks by professional proofreaders for coherence, and naturalness and expert peer review for stylistic diversity.
• •*Enforcement:*Low-quality, inconsistent, or repetitive conversations were actively filtered and revised; each dialogue was reviewed by multiple independent professionals (typically 2-3 reviewers per conversation).

All writers were recruited via Upwork under transparent contracts and compensated at professional market rates. They were explicitly informed that their work would be used for AI research and provided informed consent. No personally identifiable information (PII) is included in the dataset. All annotators were recruited via an Upwork as a part of an agency and were compensated at a professional market rate including a paid pilot study to calibrate the amount of work and fair compensation.

3.1.5Verification datasets

To further validate our findings, we construct additional verification datasets by augmenting LLM-Arena(Myrzakhanet al.,2024)and Topical-Chat(Gopalakrishnanet al.,2019). The overall procedure consisted of three steps: (i) deriving a single “ground truth answer” from each data point of each existing dataset (see below), (ii) generating an alternative continuation with GPT-4o, and (iii) collecting human judgments following the UPHELD annotation protocol. We include all additional verification labels within our dataset for reproducibility.

Derivation of Ground Truth:

• a.LLM-Arena: - –Data Point Description: each dialogue is accompanied by two model-generated continuations plus a human preference label. - –Ground Truth Extraction: for every conversation we enumerated all candidate pairs, tallied human preferences, and chose the majority-preferred continuation as the reference (provided a clear winner existed).
• b.Topical-Chat: - –Data Point Description: human‐to‐human dialogues grounded in specific topic selections. - –Ground Truth Extraction: to mitigate cold-start artifacts, we extracted segments spanning turns 5–7. We then treated turns 1–5 (or 1–7) as the model input and selected the next human turn (turn 6 or 8) as the ground truth continuation.

After standard quality control (i.e. filtering for missing data and badly formatted inputs), we obtained 12,305 pairwise preference judgments. We note that both these verification datasets, although useful for verification, are still relatively deficient in freeform human-to-human interaction and focus on a limited set of pre-defined topics. As such they should be treated as verification datasets only and not as valid replacements for UPHELD.

4Metrics

In total, 12 candidate metrics were assessed for their ability to evaluate longer conversations via correlation with UPHELD labels. These metrics were grouped into 3 distinct groups: 1) token-based – metrics quantifying similarity based on exact overlap of tokens 2) semantic-based – metrics quantifying similarity based on semantic overlap (e.g. embedding models); and 3) LLM-based – metrics employing some form of the LLM-as-a-judge paradigm.

Recall-Oriented Understudy for Gisting Evaluation (ROUGE(Lin,2004)) is a set of standard language metrics that compare automatically produced summaries or translations against a set of reference summaries or translations. Specifically, ROUGE-N measures the overlap of n-grams between the system-generated text and the reference text. ROUGE-L measures the longest common subsequence, which accounts for sentence-level structure similarity.

We also explored cosine similarity between message embeddings as a measure of semantic similarity between the generated text and reference text. This approach is rooted in the work byReimers and Gurevych (2019)on Sentence-BERT embeddings, which have shown effectiveness in capturing semantic similarities in text data. BERTScore leverages the pre-trained contextual embeddings from BERT to evaluate text generation by matching words in candidate and reference sentences. It computes precision, recall, and F1 score, providing a more nuanced evaluation than traditional n-gram-based metrics.Zhanget al.(2020)introduced BERTScore as a robust metric for evaluating generated text.

We also tested LLM-as-a-judge metrics(Zhenget al.,2023)against UPHELD. This approach involves using a separate LLM to score the outputs based on various criteria, such as coherence, relevance, and overall quality. We used both binary (yes/no) and Likert scale (1-5) judgments, with and without explanations. Prompts for judges can be found in AppendixF. In addition to these LLM-judge metrics, we also ran experiments using the same prompts as those given to human raters (AppendixF) and observed lower scores than with independently created LLM-judge prompts.

4.1Reference-free vs human-grounded metrics

Fundamentally, UPHELD uses a ground truth reference to generate our human labels. In contrast, reference-free evaluation of LLMsLiuet al.(2023)relies on human preference, and LLM outputs can reliably reproduce these preferences, indicating their performance is consistent with human judgments(Zhanget al.,2023). However, reference-free preference datasets also incur significant limitations, such as poor performance at judging long task-oriented dialogues and weakened judgment reliability on out-of-distribution tasks(Krumdicket al.,2025). UPHELD is designed specifically to tackle these limitations by adopting a multi-prong approach via reference-full content similarity annotations,while still enabling reference-free evaluation through the reasonableness annotations and implicit stylistic evaluation through the style annotations.

The reference-full approach raises a question around dialogue multiplicity: a single input might incur multiple valid outputs, so how are we sure our ground truths are well defined? UPHELD’s design reduces susceptibility to this issue in two ways: (1) two of our key label categories (style and reasonableness) are well-defined even with dialogue multiplicity, and (2) UPHELD dialogues primarily revolve around task-oriented settings, which means content accuracy is a well-defined metric. For example, while opinion-oriented conversations (e.g.Who makes the best Caesar salad?) are susceptible to dialogue multiplicity, our task-oriented dialogues (e.g.How to make a Caesar salad?) are not. To quantify this effect, we ran the following experiment exploring multiplicity.

4.2UPHELD and Conversational Multiplicity

A valid concern may be that direct comparisons to a reference human answer may be inappropriate in settings when a particular prefix can lead to a multiplicity of valid responses. This effect may be prevalent especially when the prefix is asking for an opinion (e.g. “What is your favorite animal?“). We, however, observe that UPHELD dialogues avoid this potential pitfall as they are not strictly freeform, but are all targeted towards completion of a specific well-defined task. In this context, there is some notion of correct ground truth, and we specifically hired professionals who are experts at these tasks (see AppendixC). To put it simply, our task setting is analogous to the difference betweenwhat kind of salads do you like?(which has ambiguity and dialogue multiplicity) andhow do I make a Caesar salad?(which is much more constrained and has a more well-defined ground truth). To quantify this, we generated 100 open-ended questions (GPT-4o) and then generated two possible completions with GPT-4o at moderately high temperature (τ=1\tau=1) to those questions. We did the same with 100 UPHELD turn completions. We then asked GPT-4o to judge whether the two possible completions contain similar content. The results are as follows:

Table 2:Semantic Consistency Performance across Different DatasetsUPHELD exhibits much higher semantic consistency in the output, which means that UPHELD dialogues admit much less conversational multiplicity than more freeform datasets. The results show that UPHELD dialogues admit significantly higher output consistency (93%) compared to freeform dialogues (74%), demonstrating that our reference-full approach still allows us to collect meaningful labels on ground truth content overlap. This supports our hypothesis that targeted task-focused conversations like those in UPHELD admit well-defined “ground truth” references. Interestingly, UPHELD maintains high output consistency even when we only analyze the first turn, which is where we would expect more branching/multiplicity during a dialogue. Note that these results are likely an underestimate of the true consistency, since sampling multiple LLM outputs would induce additional randomness that likely would not exist within natural human dialogue.

5Experiments

To demonstrate the value of human-scale long conversational evaluation, we present a series of experiments showing that (a) UPHELD base conversations substantially improve LLM conversational fidelity, and (b) naive evaluation metrics degrade on human-scale long dialogues, motivating development of simple ensemble metrics that outperform baseline approaches. These metrics also perform well on our validation datasets (Topical and LLM Arena), showing that a method developed with UPHELD is transferrable to other contexts. We also include discussion on user disagreement within UPHELD.

5.1UPHELD as a Fine-Tuning Dataset

Figure 1:Aggregate human labeler scores as a share of the perfect score (see AppendixD.1) for each model on the UPHELD dataset within each label category. The custom model, which was fine-tuned on top of Llama3.1-70b using a held-out set of training data within the UPHELD dataset, performs significantly better than its baseline model rivaling or beating the SOTA model and demonstrating that UPHELD contains information that can greatly improve conversational quality of base models.For our experiments we report the numbers on a subset of UPHELD consisting of 756 label sets spread across conversations, each labeled by up to five annotators. The total number of unique utterances were 4,777 averaging at 6.3 pairs of user-assistant turns. The remaining conversations were used exclusively for fine-tuning and are therefore excluded from evaluation to avoid data leakage. An important way to validate the quality of data within the UPHELD dataset is to assess how the UPHELD scores differ between thecustommodel, which was fine-tuned on the base data, and the other baseline models. To do so, we directly plot the mean content, style, and reasonableness scores within the UPHELD dataset for the different models in Figure1. Older base models (GPT-3.5, GPT-4o, and Llama3.1-70b-base) exhibit lower performance in both content accuracy and style accuracy. This indicates a tendency for these models to deviate from the intended conversational style and introduce content inconsistencies. Specifically, the base models demonstrate a substantial deficit in maintaining the stylistic integrity and topical coherence of the conversation, diverging from the trajectory established by human writers.

In contrast, the custom model, fine-tuned for extended dialogue on a held-out set, shows a marked improvement and achieves approximately a 40% increase in both content and style accuracy compared to the base models and reaches comparable scores to newer, best-in class, SOTA models like GPT-5 and Deepseek v3. The trained model achieves this while retaining the core functionality of its model base (Llama-3.1-70b), which we validated at test time and also at training time where we observed minimal overfitting within loss curves.

Reasonableness is fairly flat across all models, which is not surprising as LLMs tend to output reasonable results regardless of style or content consistency. These results highlight that the UPHELD dataset’s conversations are both learnable and encode behavior that is not well-exposed within these models’ pre-training datasets. our decision to use a single-model judge for “Reasonableness” was intended to specifically test the performance of reference-free versus reference-full evaluation. The resulting low variance between model scores serves as a strong signal that reference-free metrics currently struggle to capture the nuances of human-scale dialogue. This outcome reinforces our thesis that a reference-full approach, supported by high-quality human data like UPHELD, remains a vital and more reliable methodology for the field.

5.2Evaluation Metric Performance on UPHELD

Given that UPHELD is designed to help develop novel evaluation metrics for long conversations, it is instructive to see how this development works in practice. We start this section with an analysis on how traditional metrics perform poorly on UPHELD, and how simple modifications to the traditional metrics provide a significant boost in performance. All results in this section are presented as 5-fold cross validation results on a 20% held-out set. For numerical values, we calculated Pearson correlation between the scores and human judges; for categorical metrics we calculated Cramér’s V correlation coefficients; and for binary metrics we report point-biserial correlation coefficient.

5.2.1Traditional Metrics

Table 3:Results on the UPHELD datasets for various candidate evaluation metrics on the first evaluation task (content accuracy). Ensemble metrics clearly perform better for both the UPHELD dataset and both verification datasets.Table 4:Results on the UPHELD datasets for various candidate evaluation metrics on the second evaluation task (style accuracy).The main results for traditional metrics (as defined in Section4) are shown in the first three rows of Tables3and4. The results reveal a weak to moderate correlation between metrics and human ratings. This suggests that no single metric captures the nuances of human assessment well within the UPHELD dataset. All definitions for metrics can be found in the AppendixD.2. Interestingly, the semantic metrics (like bert_score_F1) demonstrated the highest correlation with human judgments for both content and style aspects. The token-based metrics showed stronger correlations than LLM-as-a-judge as well. These findings suggest that LLM-as-a-judge, despite being increasingly explored in the literature (e.g.,(Maet al.,2019)(Zhanget al.,2020)), is a weak evaluator of human-scale longer conversations. Despite the observed strength of the semantic metrics, the results for traditional metrics show weak to moderate performance on UPHELD. This implies that relying solely on any single traditional metric inadequately captures the complexities of content and style quality in conversations, and we demonstrate that ensemble metrics can bridge this gap.

5.2.2Ensemble Metrics

We hypothesize that individual metrics attend to distinct facets of text quality, and so a learned ensemble will perform better on the UPHELD dataset. We train linear regression, SVM, and random forest models to predict human scores using the individual automatic metric scores as input features. Using this approach, we derive new hybrid metrics and assess their correlation with human judgments.

The last 3 rows of Tables3and4show that the learned ensemble metrics exhibit higher correlations with human scores compared to any single metric in isolation. Notably, the random forest model yields substantial improvements of 30-40% relative to the best-performing individual metrics on UPHELD, but an SVM ensemble produces consistently higher correlations to human judges across all datasets. Note that the ensemble metrics were trained only on the UPHELD data and then applied to the LLM Arena and Topical-Chat data. Although the transferability of the random forest model was poor, the SVM and linear regression ensembles indicate that remarkably, trained metrics developed just on the UPHELD dataset have exceptional transferability to other out-of-domain datasets.

The success of the ensemble metrics likely stems from their ability to integrate diverse signals captured by the individual metrics, mirroring the multifaceted nature of human evaluation. These findings strongly suggest that within complex settings like human-scale dialogue, learning to ensemble multiple automatic metrics offers a promising avenue for developing evaluation frameworks that more closely align with human judgments than relying on a single metric.

5.3Discussion Around Annotator Agreement

Because each conversation turn was independently labeled by five human annotators, we also analyze model responses that elicited stronger human-human agreements. We observed moderate levels of consensus at average Cohen’sκ=0.33\kappa=0.33. As expected, human-human agreement is highest on therandomandgpt4_\textunderscorerephrasebaselines confirming the quality of the annotations, while being consistent across all other models. Further analysis on labeler agreement can be found in AppendixH.

Table 5:LLM-as-a-judge performance on the UPHELD dataset for different dataset splits, grouped by the level of human annotator agreement within each bucket.We observed that approximately 25% of data points had full agreement across all 5 human judges. Otherwise, we bin the level of agreement as follows: agreement across 2 out of 5 labels represents a “plurality,“ while agreement across 3 or 4 labels represents a “majority.“ We further quantify how well an LLM-as-a-judge evaluator agrees with this winning score relative to the agreement bin. Table5shows that LLM-as-a-judge performance is heavily correlated to agreement level amongst the human labelers. This result demonstrates that human-human disagreement is a valid measure of data difficulty, and this uncertainty signal present in UPHELD may be integral in further evaluation metric development.

6Conclusion

In this work, we introduced UPHELD: a dataset designed to evaluate LLMs in human-scale conversational settings. We collected tens of thousands high-quality human-annotated labels on crucial consistency metrics within human-scale conversations, along with the high-quality conversations themselves. Analysis of existing evaluation metrics on UPHELD reveals that they do not effectively capture the nuances of human judgment for assessing conversational quality. We further demonstrated that simple-to-learn ensemble metrics result in substantially improved correlations with human evaluations. Taken in aggregate, our findings highlight the potential for developing robust evaluation frameworks that better align with human perceptions of effective conversation using UPHELD.

Impact statement

This study provides a valuable new dataset UPHELD and associated insights into its utility. As is, we believe it is already of significant interest to the machine learning and language modeling community, but we also believe it will help researchers to use the dataset to investigate more conversational verticals, especially those that are particularly relevant to enterprise applications such as targeted customer service and dialogues around more technical topics.

Acknowledgment

We would like to thank the Upwork team members whose contributions made this research possible: Jonathan Shen for his measurable contributions to Upwork’s internal LLM stack which enables easy and scalable LLM training and evaluation, as well as Brett Levert who managed recruitment and quality controls for writers and annotators used in collecting the Upheld dataset.

References

• V. Adlakha, P. BehnamGhader, X. H. Lu, N. Meade, and S. Reddy (2024)Evaluating correctness and faithfulness of instruction-following models for question answering.Transactions of the Association for Computational Linguistics12,pp. 681–699.External Links:Document,LinkCited by:§2.
• G. Bai, J. Liu, X. Bu, Y. He, J. Liu, Z. Zhou, Z. Lin, W. Su, T. Ge, B. Zheng, and W. Ouyang (2024)MT-bench-101: a fine-grained benchmark for evaluating large language models in multi-turn dialogues.arXiv preprint arXiv:2402.14762.External Links:2402.14762,LinkCited by:Table 6.
• K. T. T. Bai and 3. others (2026)Kimi k2.5: visual agentic intelligence.External Links:LinkCited by:§3.1.1.
• A. Bavaresco, R. Bernardi, L. Bertolazzi, D. Elliott, R. Fernández, A. Gatt, E. Ghaleb, M. Giulianelli, M. Hanna, A. Koller, A. F. T. Martins, P. Mondorf, V. Neplenbroek, S. Pezzelle, B. Plank, D. Schlangen, A. Suglia, A. K. Surikuchi, E. Takmaz, and A. Testoni (2024)LLMs instead of human judges? a large scale empirical study across 20 nlp evaluation tasks.arXiv preprint arXiv:2406.18403.External Links:2406.18403,LinkCited by:§1.
• C. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu (2023)ChatEval: towards better llm-based evaluators through multi-agent debate.arXiv preprint arXiv:2308.07201.External Links:2308.07201,LinkCited by:Table 6.
• W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, B. Zhu, H. Zhang, M. Jordan, J. E. Gonzalez, and I. Stoica (2024)Chatbot arena: an open platform for evaluating LLMs by human preference.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol.235,pp. 8359–8388.External Links:LinkCited by:§1,§2.
• L. Cui, Y. Wu, S. Liu, Y. Zhang, and M. Zhou (2020)MuTual: a dataset for multi-turn dialogue reasoning.arXiv preprint arXiv:2004.04494.External Links:2004.04494,LinkCited by:Table 6,§2,§3.1.3.
• DeepSeek-AI, A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Lu, C. Zhao, and others.. (2025)DeepSeek-v3.2: pushing the frontier of open large language models.External Links:2512.02556,LinkCited by:§3.1.1.
• K. Deshpande, V. Sirdeshmukh, J. B. Mols, L. Jin, E. Hernandez-Cardona, D. Lee, J. Kritz, W. E. Primack, S. Yue, and C. Xing (2025)MultiChallenge: a realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs.InFindings of the Association for Computational Linguistics: ACL 2025,Vienna, Austria,pp. 18632–18702.External Links:Link,DocumentCited by:Table 6.
• E. Dinan, S. Roller, K. Shuster, A. Fan, M. Auli, and J. Weston (2018)Wizard of wikipedia: knowledge-powered conversational agents.arXiv preprint arXiv:1811.01241.External Links:1811.01241,LinkCited by:Table 6,3rd item.
• H. Duan, J. Wei, C. Wang, H. Liu, Y. Fang, S. Zhang, D. Lin, and K. Chen (2023)BotChat: evaluating llms’ capabilities of having multi-turn dialogues.arXiv preprint arXiv:2310.13650.External Links:2310.13650,LinkCited by:§2.
• Y. Dubois, X. Li, R. Taori, T. Zhang, I. Gulrajani, J. Ba, C. Guestrin, P. Liang, and T. Hashimoto (2023)AlpacaFarm: a simulation framework for methods that learn from human feedback.arXiv preprint arXiv:2305.14387.External Links:LinkCited by:§1.
• S. Elisha, A. McDowell, M. Beguerisse-Díaz, and E. Benetos (2024)Classification of spontaneous and scripted speech for multilingual audio.arXiv preprint arXiv:2412.11896.External Links:2412.11896,LinkCited by:1st item,§1.
• K. Gopalakrishnan, B. Hedayatnia, Q. Chen, A. Gottardi, S. Kwatra, A. Venkatesh, R. Gabriel, and D. Hakkani-Tür (2019)Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations.InProc. Interspeech 2019,pp. 1891–1895.External Links:Document,LinkCited by:Table 6,2nd item,§2,§3.1.5.
• A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle,et al.(2024)The llama 3 herd of models.arXiv preprint arXiv:2407.21783.External Links:2407.21783,LinkCited by:§3.1.1.
• D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt (2021)Measuring mathematical problem solving with the math dataset.arXiv preprint arXiv:2103.03874.External Links:2103.03874,LinkCited by:§2.
• H. Kim, J. Hessel, L. Jiang, P. West, X. Lu, Y. Yu, P. Zhou, R. Bras, M. Alikhani, G. Kim, M. Sap, and Y. Choi (2023)SODA: million-scale dialogue distillation with social commonsense contextualization.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,Singapore,pp. 12930–12949.External Links:Link,DocumentCited by:Table 6.
• M. Krumdick, C. Lovering, V. Reddy, S. Ebner, and C. Tanner (2025)No free labels: limitations of llm-as-a-judge without human grounding.External Links:2503.05061,LinkCited by:§1,§4.1.
• W. C. Kwan, X. Zeng, Y. Jiang, Y. Wang, L. Li, L. Shang, and K. Wong (2024)MT-eval: a multi-turn capabilities evaluation benchmark for large language models.arXiv preprint arXiv:2401.16745.External Links:2401.16745,LinkCited by:Table 6,§2.
• Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu (2017)DailyDialog: a manually labelled multi-turn dialogue dataset.arXiv preprint arXiv:1710.03957.External Links:1710.03957,LinkCited by:Table 6,§2.
• P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar,et al.(2023)Holistic evaluation of language models.Transactions on Machine Learning Research.Note:arXiv:2211.09110External Links:LinkCited by:§2.
• C. Lin (2004)ROUGE: a package for automatic evaluation of summaries.InText Summarization Branches Out,Barcelona, Spain,pp. 74–81.External Links:LinkCited by:§4.
• P. Lison and J. Tiedemann (2016)OpenSubtitles2016: extracting large parallel corpora from movie and TV subtitles.InProceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16),pp. 923–929.External Links:LinkCited by:Table 6,§1.
• Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023)G-eval: nlg evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634.External Links:2303.16634,LinkCited by:§1,§4.1.
• R. Lowe, N. Pow, I. V. Serban, and J. Pineau (2015)The ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems.arXiv preprint arXiv:1506.08909.External Links:1506.08909,LinkCited by:Table 6.
• Q. Ma, J. Wei, O. Bojar, and Y. Graham (2019)Results of the WMT19 metrics shared task: segment-level and strong MT systems pose big challenges.InProceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1),Florence, Italy,pp. 62–90.External Links:Link,DocumentCited by:§5.2.1.
• A. Maharana, D. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y. Fang (2024)Evaluating very long-term conversational memory of llm agents.arXiv preprint arXiv:2402.17753.External Links:2402.17753,LinkCited by:Table 6,2nd item.
• I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar (2024)GSM-symbolic: understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229.External Links:2410.05229,LinkCited by:§1.
• A. Myrzakhan, S. M. Bsharat, and Z. Shen (2024)Open-llm-leaderboard: from multi-choice to open-style questions for llms evaluation, benchmark, and arena.arXiv preprint arXiv:2406.07545.External Links:2406.07545,LinkCited by:§2,§3.1.5.
• OpenAI (2023)ChatGPT (gpt-3.5) [large language model].Note:https://openai.com/chatgptCited by:§3.1.1.
• OpenAI (2024)GPT-4o system card.Note:https://openai.com/index/gpt-4o-system-card/Cited by:§3.1.1.
• I. Pilan, L. Prévot, H. Buschmeier, and P. Lison (2024)Conversational feedback in scripted versus spontaneous dialogues: a comparative analysis.InProceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue,Kyoto, Japan,pp. 440–457.External Links:Link,DocumentCited by:§1.
• H. A. Rahmani, V. Ramineni, E. Yilmaz, N. Craswell, and B. Mitra (2025)Towards understanding bias in synthetic data for evaluation.arXiv preprint arXiv:2506.10301.External Links:2506.10301,LinkCited by:1st item,§1.
• H. Rashkin, E. M. Smith, M. Li, and Y. Boureau (2019)Towards empathetic open-domain conversation models: a new benchmark and dataset.InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics,Florence, Italy,pp. 5370–5381.External Links:Document,LinkCited by:Table 6.
• N. Reimers and I. Gurevych (2019)Sentence-bert: sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084.External Links:1908.10084,LinkCited by:§4.
• D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman (2023)GPQA: a graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022.External Links:2311.12022,Document,LinkCited by:§2.
• A. Ritter, C. Cherry, and W. B. Dolan (2011)Data-driven response generation in social media.InProceedings of the 2011 Conference on Empirical Methods in Natural Language Processing,Edinburgh, Scotland, UK.,pp. 583–593.External Links:LinkCited by:Table 6.
• J. Schnitzler, X. Ho, J. Huang, F. Boudin, S. Sugawara, and A. Aizawa (2024)MoreHopQA: more than multi-hop reasoning.arXiv preprint arXiv:2406.13397.External Links:2406.13397,LinkCited by:Table 6,§2.
• A. K. Singh, A. Fry, A. Perelman, A. Tart,et al.(2025)OpenAI gpt-5 system card.External Links:LinkCited by:§3.1.1.
• E. M. Smith, M. Williamson, K. Shuster, J. Weston, and Y. Boureau (2020)Can you put it all together: evaluating conversational agents’ ability to blend skills.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,Online,pp. 2021–2030.External Links:Document,LinkCited by:Table 6.
• Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett (2023)MuSR: testing the limits of chain-of-thought with multistep soft reasoning.arXiv preprint arXiv:2310.16049.External Links:2310.16049,LinkCited by:§2.
• A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, and G. Wang (2022)Beyond the imitation game: quantifying and extrapolating the capabilities of language models.arXiv preprint arXiv:2206.04615.External Links:2206.04615,LinkCited by:§2.
• Q. Wang, J. Wu, Z. Jiang, Z. Tang, B. Luo, N. Chen, W. Chen, and B. He (2025)LLM-based human simulations have not yet been reliable.arXiv preprint arXiv:2501.08579.External Links:2501.08579,LinkCited by:2nd item,§1.
• Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen (2024)MMLU-pro: a more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574.External Links:2406.01574,LinkCited by:Table 6,§2.
• X. Xu, Z. Gou, W. Wu, Z. Niu, H. Wu, H. Wang, and S. Wang (2022)Long time no see! open-domain conversation with long-term persona memory.arXiv preprint arXiv:2203.05797.External Links:2203.05797,LinkCited by:Table 6.
• A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv,et al.(2025)Qwen3 technical report.Cited by:§3.1.1.
• C. Zhang, L. F. D’Haro, Y. Chen, M. Zhang, and H. Li (2023)A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators.arXiv preprint arXiv:2312.15407.External Links:2312.15407,LinkCited by:Table 6,Table 6,1st item,§2,§4.1.
• S. Zhang, E. Dinan, J. Urbanek, A. Szlam, D. Kiela, and J. Weston (2018)Personalizing dialogue agents: i have a dog, do you have pets too?.arXiv preprint arXiv:1801.07243.External Links:1801.07243,LinkCited by:Table 6,§1.
• T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi (2020)BERTScore: evaluating text generation with bert.arXiv preprint arXiv:1904.09675.External Links:1904.09675,LinkCited by:§4,§5.2.1.
• L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023)Judging llm-as-a-judge with mt-bench and chatbot arena.External Links:2306.05685,LinkCited by:Table 6,Table 6,§1,§2,§2,§4.
• J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou (2023)Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911.External Links:2311.07911,LinkCited by:Table 6,§2.

Appendix ARelated Work comparison table

This section formally defines the key dimensions and associated metrics used to characterize and compare dialogue benchmark datasets. These dimensions establish thedata provenance, scale, task focus, and evaluation rigorof a benchmark, which are critical for assessing its suitability for training and evaluating large language models (LLMs).

A.1Data Statistics

These dimensions shown in Table6quantify the scale and density of the linguistic resources within the benchmark.

A.1.1Scale Metrics

• •Total Conversations:Represents the absolute count of distinct conversational threads or sessions included in the dataset.
• •Total Utterances:Indicates the total number of individual speech acts or turns produced by all participants across the entire dataset.

A.1.2Density Metric

• •Total Comparison Labels:Denotes the aggregate number of specific human or machine judgments collected for the purpose of comparing the quality of model-generated responses (e.g., A/B test results, preference rankings, or score annotations). This metric is a key indicator of the dataset’s utility forpairwise evaluationand reinforcement learning from human feedback (RLHF).

A.2Collection Information

This dimension pertains to thedata generation processand the level of human involvement in its creation and quality control.

A.2.1Data Provenance

• •Human (is the data generated by a human):A binary metric indicating whether the dialogue turns were initially composed by human actors (yes) or whether they were generated synthetically, typically by an LLM (no).
• •Direct (are the conversations directly obtained from the source):A binary metric indicating whether the data was captured raw from its source (e.g., direct transcription, human writing) (yes) or whether it underwent substantial intermediate processing, such as speech-to-text conversion or LLM-based reformulation (no).

A.2.2Quality Control

• •Verified (is there human verification on the quality of the conversation):A binary metric indicating whether a human performed explicit quality checks, validation, or post-hoc filtering on the conversational data to ensure coherence, safety, or adherence to task instructions (yes).

A.3Task Information

These dimensions shown in Table7describe thefunctional objectiveandknowledge requirementsimposed by the dataset’s dialogue prompts.

A.3.1Task Type

• •Factual QA:The task is constrained to generating a single, verifiably correct answer to a question.
• •Open Dialogue (Chit-Chat):The dialogue is non-goal-oriented, focused on maintaining conversational flow, persona consistency, and engaging interaction.
• •Summarization:The task requires the model to produce a concise abstract or summary of provided source material.
• •Specialized Tasks:The benchmark encompasses a heterogeneous set of specific, domain-specific tasks, such as mathematical problem solving, code generation, or complex instruction following.
• •Context Retrieval:The primary conversational goal is to retrieve or locate a specific piece of information from a provided or externally accessible knowledge base.

A.3.2Context Utilization

This sub-dimension describes how external information is introduced to support the dialogue.

• •RAG (Retrieval-Augmented Generation):Context is inserted into the conversation using an algorithmic retrieval system, typically based on vector similarity.
• •Grounded Knowledge:Context is inserted into the conversation in a deterministic or pre-defined manner, ensuring the LLM has access to a specific piece of knowledge for its response.
• •None:No external context is provided; the model relies solely on the preceding dialogue history and its parametric knowledge.

A.3.3Prediction Format

• •Is Conversation:A binary metric indicating whether the dataset is entirely formatted as a sequence of alternating user/assistant turns (yes) or if it includes substantial non-dialogue components, such as initial instructions or descriptive context (no).

A.4Results Verification

The dimensions shown in Table8assess themethodology and validityof the benchmark’s evaluation process.

A.4.1Evaluation Basis

• •Explicit:Evaluation relies on a direct, measurable comparison of the model’s output against a predefined, canonical “ground-truth” reference answer (e.g., F1, exact match).
• •Reference-Free:Evaluation is performed without a canonical target answer, relying instead on subjective judgment, typically from LLM-as-a-Judge systems or human preference rankings.

A.4.2Annotation Quality and Transparency

• •Uses Human Annotators:A binary metric indicating whether human annotators were employed to subjectively rate or score the quality of the model’s generated outputs (yes/no).
• •Correlation with Human:A binary metric indicating whether the benchmark report provides an explicit measure of the agreement (e.g., Pearson’srr) between the reported automatic evaluation metrics and the parallel human evaluation results (yes/no).
• •Provides Author/Creator Selection Details:A binary metric indicating whether the documentation includes detailed information regarding the qualifications, recruitment, or background of the human writers and annotators employed for data generation and quality evaluation (yes/no). This speaks directly todata quality and reproducibility.

Table 6:Comparison of Dataset Properties (Part 1: Data Statistics)Dataset/BenchmarkLinkTotal Conv.Avg Utt.ComparisonsUPHELD5310.236,873MuTal(Cuiet al.,2020)6,7314.736,371Topical-Chat(Gopalakrishnanet al.,2019)9,05821.9150 (human)LLM Arena (crowd-sourced)(Zhenget al.,2023)33,0001.233,000LLM Arena (annotated)(Zhenget al.,2023)3,00023,000DailyDialogue(Liet al.,2017)13,1187.911,118IFEval(Zhouet al.,2023)2502250MT-bench(Kwanet al.,2024)80280MMLU-pro(Wanget al.,2024)12,032212,032HopQA(Schnitzleret al.,2024)7,40527,405MT-bench-101(Baiet al.,2024)1,3883.031,388(100 human)MultiChallenge(Deshpandeet al.,2025)2735273LoCMo(Maharanaet al.,2024)50304.950(?)LongTimeNoSee(Xuet al.,2022)27,50116.34200COMPREHENSIVE(Zhanget al.,2023)2,03013.32,030COMPREHENSIVE-turn(Zhanget al.,2023)4177.43,901ChatEval(Chanet al.,2023)80(open QA)+60(dialogue)2440Persona-chat(Zhanget al.,2018)10,907(1000 test/100 human)14.8515,602 (test)Wizards of Wikipedia(Dinanet al.,2018)18,4309.05166,787(300 human)SODA(Kimet al.,2023)∼\sim1.3m7.6300(human evaluated sample)Empathetic Dialogues(Rashkinet al.,2019)25,0004.025,000Blended Skill Talk(Smithet al.,2020)6,8104.56,810plain datasets (no benchmarks)Ubuntu dialogue context(Loweet al.,2015)∼\sim1 million7.7n/aOpen Subtitles(Lison and Tiedemann,2016)3.7 million16.75 (sentences)n/aTwitter-dataset(Ritteret al.,2011)4,32322161Table 7:Comparison of Dataset Properties (Part 2: Collection & Task Information)Dataset/BenchmarkHumanVerifiedDirectType of PredictionsIs Conv.Task TypeUses ContextUPHELDyyyPer-turnyestask-oriented dialoguenoMuTalynyEntire conversationyesopen dialogue/chit chatnoTopical-ChatynyEntire conversationyesopen dialogue/chit chatgrounded contextLLM Arena (crowd-sourced)nnyPer-turnyesopen dialogue/chit chatnoLLM Arena (annotated)nyyPer-turnmixedspecialized tasksnoDailyDialogueynnPer-turnyestask-oriented dialoguenoIFEvalyyyEntire conversationnospecialized tasksnoMT-benchynyEntire conversationnospecialized tasksnoMMLU-pronynEntire conversationnofactual QAnoHopQAynyEntire conversationnofactual QAgrounded contextMT-bench-101nynEntire conversationyesspecialized tasksgrounded contextMultiChallengenynEntire conversationyesspecialized tasksgrounded contextLoCMonynEntire conversationyesspecialized tasksRAG insertedLongTimeNoSeenynEntire conversationyesopen dialogue/chit chatnoCOMPREHENSIVEnny/nEntire conversationmixedspecialized tasksgrounded contextCOMPREHENSIVE-turnnny/nPer-turnmixedspecialized tasksgrounded contextChatEvalnyy/nEntire conversationmixedspecialized tasksgrounded contextPersona-chatynyPer-turnyesopen dialogue/chit chatgrounded contextWizards of WikipediayyyEntire conversationyescontext retrievalRAG insertedSODAny/nnPer-turnyesspecialized tasksgrounded contextEmpathetic DialoguesyyyPer-turnyesopen dialogue/chit chatnoBlended Skill TalkyyyEntire conversationyesopen dialogue/chit chatgrounded contextplain datasets (no benchmarks)Ubuntu dialogue contextynyn/ayesopen dialogue/chit chatnoOpen Subtitlesnnnn/amixedopen dialogue/chit chatnoTwitter-datasetyynPer-turnmixedopen dialogue/chit chatgrounded contextTable 8:Comparison of Dataset Properties (Part 3: Evaluation and Verification)Dataset/BenchmarkComp. to Ground TruthUses Human AnnotatorsCorr. with HumanProvides Author DetailsUPHELDexplicityyyMuTalexplicitnnnTopical-Chatexplicity (limited)nyLLM Arena (crowd-sourced)reference-freennnLLM Arena (annotated)reference-freeyynDailyDialogueexplicitnnnIFEvalexplicitnnnMT-benchreference-freennnMMLU-proexplicitnnnHopQAexplicitnnnMT-bench-101reference-freenynMultiChallengereference-freennnLoCMomixednnnLongTimeNoSeeexplicitnnnCOMPREHENSIVEreference-freenynCOMPREHENSIVE-turnreference-freenynChatEvalmixedyynPersona-chatexplicitynnWizards of WikipediaexplicitynnSODAexplicitn (sample y)nnEmpathetic DialoguesexplicitynyBlended Skill Talkexplicitynyplain datasets (no benchmarks)Ubuntu dialogue contextn/ann/anOpen Subtitlesn/an/an/anTwitter-datasetexplicitynn

Appendix BAnnotation Materials and Instructions

Here we provide a complete description of the materials provided to data annotators along with associated instructions.

B.1Materials

Users are given access to a spreadsheet file with three sheets – the first sheet contains the instructions, the second contains sample annotated examples and the third one contains a formatted table with 6 columns with the following labels:

• •chat history – (the conversation up to one point)
• •Option A – (one possible continuation to the conversation within chat history)
• •Option B – (alternative possible continuation to the conversation within chat history)
• •score_task_1_content – (a dropdown menu to select the content consistency score)
• •score_task_2_style (a dropdown menu to select the style consistency score)
• •score_task_3_reasonableness (a dropdown menu for choosing the reasonbleness score)

A screenshot of the interface annotators are given is shown in Figure2.

Figure 2:Screenshot of the interface annotators were provided to collect UPHELD data.

B.2Instructions

This section contains the verbatim instructions provided to the annotators in the data annotation spreadsheet. We start with the overall instructions and then reprint the granular instructions for each of the three types of labels within UPHELD.

==BEGIN INSTRUCTIONS==

Given a chat history, you will be presented two options for how to continue the conversation: Option A and Option B. You will be asked to rate these options by answering a number of questions.

Task 1 (Content Equivalence): Do you agree that the general information presented in Option A is roughly the same as the general information presented in Option B?

Task 2 (Stylistic Equivalence): Do you agree that the style of Option A is the same as the style of Option B? Put another way, does it feel like Option A and Option B are being spoken by the same person?

Task 3 (Reasonableness of Option B): For Option B only, do you agree that Option B is a reasonable way to continue the conversation, given the chat history?

Enter your score in the column corresponding to the task in the annotations tab/sheet (e.g. score_task_2_style).

Please read below for specific instructions and tips on each individual task.

B.2.1Task 1 – Content Equivalence

Check the examples tab for some already annotated data and additional explanation (note that you are not expected to provide explanations of your scores.)“ “Provide one of the following scores on a scale of 1-5 where a 1 reflects a strong DISAGREE and a 5 reflects a strong AGREE:

• •1: Strongly Disagree (that the content conveys equal information in both options)
• •2: Disagree
• •3: Neutral
• •4: Agree
• •5: Strongly Agree

Use the following criteria to help you determine if the two message options have equivalent content:

• •Information conveyed by Option B contains all information that is conveyed by Option A.
• •Using either Option A or Option B to continue the conversation would not change the flow of the conversation.
• •You can replace Option A with Option B, or replace Option B with Option A, without appreciably changing the content of the conversation.
• •Both Option A and Option B mean the same thing.

Tips:

• •Keep the chat history in mind when considering the content equivalence of Option A and Option B.
• •If one of the options seems incomplete or cut short, still try to evaluate the option as is.
• •If Option B is wordier or contains more details than Option A, but it still contains all the information in Option A and is relevant given the chat history, lower the score to at most a 3.
• •Do not lower the score if Option B contains AI self identification phrases such as (““As an AI agent…”“, ““I am a trained model..) and similar. Focus on the other information within Option B.
• •If Option B is not readable and/or contains non-coherent language give a score of 3.
• •Lower the score if Option B contains more details that are (1) not an expansion of the information in Option A and (2) are not relevant to the messages in the chat history.

B.2.2Task 2 - Style Equivalence

Provide one of the following scores on a scale of 1-5 where a 1 reflects a strong DISAGREE that styles are the same and a 5 reflects a strong AGREE that styles are the same:

• •1: Different styles (that the content conveys equal information in both options).
• •3: Somewhat same styles.
• •5: Same styles

Use the flowing instructions to help you determine if the two message options are stylistically equivalent

• •After reading them out loud, both options A and B feel like they are written by the same person in the same mood.
• •There is no noticeable change in sentiment or tone between the two options.
• •Even if one of the message options is longer than the other, they can still be considered stylistically similar if the content is expressed in similar ways.
• •If it sounds like option A and option B were written by different people, assign a low score.
• •If you believe that both options are written by the same person in the same mood but the content of the two options are different, still assign a high score.

Tips:

• •It may be useful to consider the context of chat history as a reference and seeing whether either option deviates from a natural continuation of the chat history, given the personality of the “assistant” in the chat history.
• •For this task you’re highly encouraged to read both options out loud as it may be helpful in forming the comparison.
• •Consider differences in vocabulary, tone, and syntax when making your decision.

B.2.3Task 3 – Reasonableness

Provide one of the following scores:

• •1: Not a reasonable continuation (to chat_history)
• •5: A reasonable continuation (to chat_history)

Guidelines:

• •This task ONLY applies to Option B. The task it to determine whether Option B is a reasonable way to continue the conversation from the chat history.
• •Ignore Option A in your judgment; Option B may be completely different from Option A but still score highly in this task as long as it is on topic.
• •If Option B seems cut short assess the text up to the cutoff point.
• •If Option B is not readable and/or sounds incoherent, assign a score of 1.

Tips:

• •We encourage you to read the chat history out loud as well as the message in Option B directly afterwards. If it sounds like a natural conversation flow out loud the score is likely a high score.
• •If you were the “user” in this scenario and received Option B as the next response, would you be generally happy with the state of the conversation? If the answer is yes, the score is likely 5. If not the score is likely 1.
• •Do not lower your score if Option B contains any model self identification (e.g. As an AI model….) but is still a viable continuation of chat history.
• •All of the following reasons are valid for assigning a low score of 1: - *Option B is excessively wordy and/or provides too much information. - *Option B is incoherent.Option B seems random and gets off topic. - *Option B is excessively rude or aggressive. - *Option B has an inappropriate tone or uses inappropriate language. - *Option B does not add any additional helpful information to the conversation or prompt the user to provide additional relevant information.
• •Check the examples tab for some already annotated data and additional explanation (note that you are not expected to provide explanations of your scores.)

==END INSTRUCTIONS==

B.3Instructions as LLM judge prompts

We initially used the above instructions as prompts for the LLM-judge evaluation. Our analysis of these results when compared to the “free” instructions showed that using human instructions as prompts provides comparable, but lower correlation scores –0.360.36(vs0.40.4) for the content equivalence,0.210.21(vs0.240.24) for style equivalence; and0.080.08(vs0.120.12) for reasonableness score. Due to this we have removed these metrics from further analysis, as inclusion would have only increased the strength of the signal for LLM judges effectively doubling it.

Appendix CNotes on writer and annotator selection and bias mitigation

Our contracted writers were required to have a high job success rate on Upwork and all were first evaluated through a rigorous initial (paid) pilot phase where their written conversations were evaluated by a professional user experience team for diversity and faithfulness to our prompts. Writers were also selected from diverse professional backgrounds: we employed writers with backgrounds from novel writing to education to copywriting. Prompts were selected for diversity of tasks and diverse defined styles that had to adhere to a number of user personas and styles. All conversations were quality-checked by a separate set of experienced proofreaders to explicitly ensure style diversity and consistency. We also acknowledge that our current focus is primarily on English language conversations, but also plan to eventually incorporate multilingual UPHELD additions.

All conversations were further quality-checked by another professional writer to ensure situational and stylistic diversity. We were admittedly limited to English-speaking writers, which may introduce some clustering of labeler backgrounds. Because each conversation went through multiple rounds of checks from different professionals (including both user research professionals, other writing professionals, and machine learning professionals) who were explicitly instructed to check for diversity and to eliminate bias, we hope that any effects of geographical/linguistic clustering are mitigated by our rigorous process.

All data labelers also participated in an initial (paid) pilot program that was carefully evaluated by internal user research professionals before being selected to write conversations at scale. The scenarios the writers built were evaluated by the same user research professionals to ensure they covered a wide variety of scenario types and user behavioral/personality patterns which were representative of what chat agents might encounter in a customer-facing context.

Appendix DMetrics

D.1Aggregation metric

The aggregate scores in Figure1represent the total score of a given response divided by the maximum possible sum score. If33judges score a turn55,33, and22with a maximum score of55, the aggregate score is(5+3+2)/(5+5+5)=10/15(5+3+2)/(5+5+5)=10/15. More formally, Let a response be scored byJJjudges. Judgejjgives a scoresjs_{j}with a per-judge maximumMjM_{j}(often allMj=MM_{j}=M).

score=∑j=1Jsj∑j=1JMj,where​0≤score≤1\text{score}=\frac{\sum_{j=1}^{J}s_{j}}{\sum_{j=1}^{J}M_{j}},\quad\text{where }0\leq\text{score}\leq 1

D.2Experimental metric

We include all metrics used within the experimental studies tabulated in Tables3and4.

• •Message Embedding Cosine Similarity: CosineSim​(𝐮,𝐯)=𝐮⋅𝐯‖𝐮‖​‖𝐯‖\text{CosineSim}(\mathbf{u},\mathbf{v})=\frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}where𝐮\mathbf{u}and𝐯\mathbf{v}are the embedding vectors of the reference and generated messages.
• •BERTScore Precision: P=1|x|​∑i=1|x|maxj⁡sim​(xi,yj)P=\frac{1}{|x|}\sum_{i=1}^{|x|}\max_{j}\text{sim}(x_{i},y_{j})
• •BERTScore Recall: R=1|y|​∑j=1|y|maxi⁡sim​(xi,yj)R=\frac{1}{|y|}\sum_{j=1}^{|y|}\max_{i}\text{sim}(x_{i},y_{j})
• •BERTScore F1: F​1=2​P​RP+RF1=\frac{2PR}{P+R}wherexxandyyare the sets of tokens from the candidate and reference texts respectively, andsim​(xi,yj)\text{sim}(x_{i},y_{j})denotes cosine similarity between contextual embeddings of tokensxix_{i}andyjy_{j}.
• •ROUGE-1 (Unigram Overlap): ROUGE-1=∑w∈Refmin⁡(Countgen​(w),Countref​(w))∑w∈RefCountref​(w)\text{ROUGE-1}=\frac{\sum_{w\in\text{Ref}}\min(\text{Count}_{\text{gen}}(w),\text{Count}_{\text{ref}}(w))}{\sum_{w\in\text{Ref}}\text{Count}_{\text{ref}}(w)}
• •ROUGE-2 (Bigram Overlap): ROUGE-2=∑b∈Refmin⁡(Countgen​(b),Countref​(b))∑b∈RefCountref​(b)\text{ROUGE-2}=\frac{\sum_{b\in\text{Ref}}\min(\text{Count}_{\text{gen}}(b),\text{Count}_{\text{ref}}(b))}{\sum_{b\in\text{Ref}}\text{Count}_{\text{ref}}(b)}
• •ROUGE-L (Longest Common Subsequence - LCS): ROUGE-L=LCS​(X,Y)Length​(Y)\text{ROUGE-L}=\frac{\text{LCS}(X,Y)}{\text{Length}(Y)}whereXXandYYare sequences of tokens in the generated and reference texts respectively.
• •ROUGE-Lsum (LCS over multiple sentences): ROUGE-Lsum=∑iLCS​(Xi,Yi)∑iLength​(Yi)\text{ROUGE-Lsum}=\frac{\sum_{i}\text{LCS}(X_{i},Y_{i})}{\sum_{i}\text{Length}(Y_{i})}whereXiX_{i}andYiY_{i}are sentence-level pairs from the candidate and reference summaries.

To compute all metrics, we used the ground truth next turn (Option A) as a reference data point and the model-generated next turn (Option B) as a candidate data point. We use themixedbread-ai/mxbai-embed-large-v1111https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1embedding model for all metrics that required a calculated similarity score.

All prompts associated with thellm-as-a-judgemetrics can be found in AppendixF.

Appendix EHow did GPT-4o Underperform GPT-3.5?

In Figure1, we revealed a somewhat surprising result: humans tended to prefer the output of GPT-3.5 over that of GPT-4o. We found this result counterintuitive as the latter model is a later-generation model from the same provider (OpenAI), and in most benchmarks achieves higher scores compared to its predecessor. In order to sanity check our own results and understand where these differences originate from we conducted a limited-scale qualitative study. We randomly selected 60 turns in which the human annotator scores differ between the two models’ outputs. These were then additionally judged across two dimensions: (a) general phrasing quality (evaluators could select betweenhumanortemplatesounding, and (b) perceived differences between the GPT-x output and the reference ground-truth output (evaluators were free to enter any difference). We then aggregated the results of this exercise, extracted major difference axes as described in the (b) labels, and we display the results in Figure3. The Figure suggests thatGPT-4otend to be more verbose and less engaging in a conversation than GPT-3.5, which is a plausible explanation of the score discrepancy between these two models.

Figure 3:Human-perceived differences between the outputs ofGPT3.5andGPT4oto the reference answer.Samemeans there was no perceived difference to the reference.

Appendix FLLM-as-a-judge prompts

In this section we reproduce the prompts used by thellm-as-a-judgemetrics for our experiments that generated Tables3and4.

• •llm judge - yes/no: Conversation: {chat_history} Continuation: {Option A} Prediction: {Option B} For the above your job is to compare the continuation and reference response as being equivalent in regards to the conversation. Output "Yes" if you think the continuation contains the same information as reference, otherwise output "No".
• •llm judge - yes/no explain: Conversation: {chat_history} Continuation: {Option A} Prediction: {Option B} For the above your job is to compare the continuation and reference response as being equivalent in regards to the conversation. Output ”Yes” if you think the continuation seems natural and human generated, otherwise output ”No”. Also output the explanation of why you made the judgment.
• •llm judge - likert 1-5: Conversation: {conversation} Reference: {reference} Prediction: {prediction} For the above your job is to compare the prediction and reference responses. Score whether the prediction conveys the same information as the reference the on a likert scale of 1 to 5. 1 means none of the reference information is conveyed by the prediction; and 5 means reference and prediction are semantically equivalent. Output only scores from 1 to 5 (integer)
• •llm judge - likert 1-5 explain: Conversation: {chat_history} Reference: {Option A} Prediction: {Option B} For the above your job is to compare the prediction and reference responses. Score whether the prediction conveys the same response as the reference the on a score of 1 to 5 and give a reason as to why.

Appendix GMachine learning metrics

Our ensemble metrics are in essence standard machine learning models. We trained these models by using all other metrics presented in Tables3and4as input features for a data point and a human label as a target variable (the variable we are predicting). The training was in a cross-validation setting and we held out 20% of the input for validation. We used standard hyper-parameters for all models. We used scikit-learn Python library (https://scikit-learn.org/) for training the models. In addition to models presented in the paper we trained additional models (or same models with different parameters) but the validation results were very low and we discarded them from further analysis. An example of a decision tree ensemble model is given in Figure4to illustrate the power of combining different metrics.

Figure 4:One of the decision trees in an ensemble model used as an example.

Appendix HAnnotator agreements levels

In addition to the Kappa scores that we presented in Section5.3, we assessed annotator agreement through categorical bins to further analyze our dataset statistics. We quantified agreement at three distinct levels:

• •perfect— where all annotators assign the same score to the same data point.
• •majority— where more than half of the annotators assign the same score to the same data point.
• •lead/plurality— where there is a score assigned more frequently than others to the same data point.

The results, as depicted in Figure5, indicate that a substantial dataset can be retained even when considering only those data points on which all five annotators agree. Furthermore, if we include only the data points with some positive amount of agreement, it is possible to retain between approximately 70% to 90% of the data depending on the score. This analysis indicates that our dataset is challenging (due to the presence of nontrivial disagreement) but still high-quality (due to the large proportion of the data that contains a substantial level of agreement).

Additionally, we can show that more agreement between humans leads to less difficult tasks for the metrics. This is clearly shown in Table9where we show how the correlation between human and automatic metrics increases as the human agreement level increases.

Table 9:Comparison of content and style scores with relative increases between agreement levels. TheΔ\Deltavalues show the relative improvement on the previous level of human agreement.Figure 5:Annotator agreement for the three tasks at different categorical levels of agreement: plurality, majority, and perfect agreement.

Appendix IAdditional Linear Regression Analysis

In Tables3and4we showed that an ensembled linear regression classifier readily wins against single metrics. Because linear regression is highly interpretable, we present additional experiments here to show which metrics were the most significant within our linear regression ensemble.

Figure 6:Coefficients for Ensembled Linear Regression (Content Accuracy)Figure 7:p-values for Ensembled Linear Regression (Content Accuracy)Figure 8:Coefficients for Ensembled Linear Regression (Style Accuracy)Figure 9:p-values for Ensembled Linear Regression (Style Accuracy)We see in Figure7and9the p-value significance of each metric included in the ensemble. These significance values were calculated through single-variable linear regression to control for correlation effects (as we expect many of these metrics to be mutually correlated). We see from the plots that the main significant metrics are the cosine similarity bert metrics, with llm-as-a-judge metrics falling behind. Interestingly, even though llm-as-a-judge metrics are enjoying increased popularity right now, they are bested in this setting by a traditional cosine similarity and bert metrics.

From the actual coefficient values as shown in Figures6and8for content consistency, we see that other than the reasonableness llm-as-a-judge metric (which performed poorly and we omitted from most analysis within this work), all metrics have strong positive correlations with the UPHELD labels.

Appendix JThe Reasonableness Label

In the main paper, we provided extensive analysis of the accuracy and style UPHELD label sets, but UPHELD also contains a third set of labels around reasonableness. For completeness, we include the same analysis for the reasonableness label here, in Table10. We also provide the same linear regression analysis as in SectionIfor the reasonableness label in Figures10and11.

Table 10:Reasonableness results on the UPHELD dataset and verification datasets.MetricUPHELDLLM ArenaTopical ChatSemantic Metricsmessage_embedding_cos_sim0.010.280.01bert_score_precision0.170.150.13bert_score_recall0.070.150.04bert_score_F10.140.190.10LLM-as-a-judge Metricsllm_judge_yes_no0.130.160.09llm_judge_yes_no_explain0.060.160.00llm_judge_likert_1_50.030.260.00llm_judge_likert_1_5_explain0.000.260.00Token-based Metricsrouge10.120.310.06rouge20.110.240.04rougeL0.120.180.07rougeLsum0.110.280.06Ensembled ML Metrics (Ours)Linear Regression0.16-0.160.01SVM0.010.200.03Random Forest0.520.180.04Figure 10:Coefficients for Ensembled Linear Regression (Reasonableness).Figure 11:p-values for Ensembled Linear Regression (Reasonableness).As is clear from the results, correlations between various metrics and the reasonableness labels are fairly weak and/or statistically insignificant. Even though our ensemble tree model still performs admirably in this setting, the labels themselves have a very lopsided distribution with most labels being in the positive class (see Figure1).

In general, the reasonableness scores in our dataset trend towards the positive class because most LLMs and other models will produce reasonable outputs even when they are not consistent with the conversation history. As in Figure1, one can see that all models (except for the random model baseline) produce reasonableness scores that are substantially greater than 80%.

Due to both of these effects (the lopsidedness of the data and the lack of statistical significance in the regression results), we generally consider the reasonableness score as a sanity check label and a good filter for data that is out of distribution. It is for this reason that we decided to not analyze the reasonableness labels at length within the main paper. However, the reasonableness scores are still informative and we look forward to followup work to analyze this signal as a potential uncertainty or out-of-distribution feature.

Appendix KMore Dataset Statistics: Score Distributions

We provide more granular breakdowns of score distributions within the UPHELD dataset within this section, in Figures12(a)-(c).

(a)Content scores distribution (b)Style scores distribution (c)Reasonability scores distribution

Figure 12:Annotator Scores Distribution, averaged agreement rounded to the nearest score.

Appendix LData examples

On the following pages we now provide a number of other examples from the UPHELD dataset.

Table 11:Annotator Scores Distribution (Examples 1-2)Table 12:Annotator Scores Distribution (Examples 3-4)Table 13:Annotator Scores Distribution (Examples 5-6)Table 14:Annotator Scores Distribution (Examples 7-8)Table 15:Annotator Scores Distribution (Examples 9-10)Table 16:Annotator Scores Distribution (Examples 11-12)Table 17:Annotator Scores Distribution (Examples 13-14)Table 18:Annotator Scores Distribution (Examples 15-16)

Appendix MLLM Usage Disclosure

We used LLMs to check for spelling errors but did not use LLMs to directly generate any text that appear in this paper. We also used LLMs for help in minor formatting fixes of the tables.