Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations

# Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations
Source: [https://arxiv.org/html/2607.12884](https://arxiv.org/html/2607.12884)
\\theorembodyfont\\theoremheaderfont\\theorempostheader

:\\theoremsep \\jmlrvolume340\\jmlryear2026\\jmlrworkshopMachine Learning for Healthcare

\\NameMonica Munnangi\\Emailmunnangi\.m@northeastern\.edu \\addrKhoury College of Computer Sciences Northeastern University \\NameSaiph Savage\\Emails\.savage@northeastern\.edu \\addrKhoury College of Computer Sciences Northeastern University

###### Abstract

Patients seeking medical information often ask questions that embed incorrect assumptions or misconceptions\. In such cases, safe medical communication requires not only answering the question, but identifying and correcting the underlying false belief\. These interactions naturally unfold over multiple turns, a pattern now mirrored in interactions with LLMs\. Yet current evaluation frameworks do not capture model behavior in these settings, where misconceptions can emerge, persist, or evolve over the course of a conversation\. Whether LLMs can reliably correct such misconceptions over time remains largely unexamined\. To study this, we introduceThReadMed\-QA, a multi\-turn medical dialogue dataset of 2,437 patient–physician conversation threads comprising 8,204 question–answer pairs, derived from real patient interactions on r/AskDocs\. This dataset enables systematic evaluation of whether models can detect and correct misconceptions under a multi\-turn context\. We evaluate five state\-of\-the\-art LLMs using a rubric\-based LLM\-as\-a\-Judge framework that scores responses based on their ability to identify and correct misconceptions\. Our experiments reveal a consistent pattern: even frontier models that can address misconceptions in a single interaction degrade substantially over subsequent turns\. GPT\-5 and Claude\-Haiku correct these false presuppositions around 85% on initial questions but drop to approximately 50% within two follow\-ups\. Additionally, GPT\-4o exhibits a sharper decline, falling from 65% to 21%, indicating a failure to sustain safe reasoning across dialogue\. An oracle analysis replacing prior model outputs with physician responses shows that much of the degradation is driven by error propagation, while performance remains imperfect even under correct context\. These findings reveal a critical reliability gap in LLMs\. Even when models tend to correct misconceptions initially, their performance degrades substantially over subsequent turns, leading to inconsistent and potentially unsafe guidance in patient\-facing settings and highlighting the need for evaluation frameworks that capture multi\-turn behavior\.

## 1Introduction

LLMs are increasingly used by patients as a source of medical information, offering unprecedented accessibility to medical guidance outside traditional clinical settings\(Costa\-Gomes et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib9)\)\. Surveys indicate that a substantial and growing fraction of patients now seek health advice from LLMs, particularly where access to clinical care is limited\(Montero et al\.,[2026](https://arxiv.org/html/2607.12884#bib.bib23)\)\. In these interactions, patients often ask questions that embed incorrect assumptions or misconceptions about their condition, symptoms, or treatment\(Srikanth et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib39)\)\. These misconceptions are often implicit and shaped by incomplete knowledge or prior beliefs\. Safe medical communication, therefore, requires not only answering the question but also identifying and correcting the underlying false belief rather than accepting the premise as stated\.

Table 1:Comparison of medical QA datasets\. Our datasetThReadMed\-QAis the first to have patient\-authored questions and natural follow\-ups, along with verified physician responses\.Moreover, in clinical practice, patient–physician interactions are inherently conversational, unfolding over multiple turns as patients refine their concerns, introduce new information, or reinterpret prior guidance\(Heritage and Maynard,[2006](https://arxiv.org/html/2607.12884#bib.bib15)\)\. This pattern is increasingly reflected in interactions with LLMs, where users engage in iterative dialogue rather than isolated queries, and misconceptions may arise at any point in the conversation \(Figure[1](https://arxiv.org/html/2607.12884#S1.F1)\)\. However, existing evaluation frameworks do not assess model behavior in these settings\. Early medical QA datasets primarily emphasize factual recall and exam\-style reasoning\. MedQA\(Jin et al\.,[2020](https://arxiv.org/html/2607.12884#bib.bib17)\), MedMCQA\(Pal et al\.,[2022](https://arxiv.org/html/2607.12884#bib.bib30)\), and PubMedQA\(Jin et al\.,[2019](https://arxiv.org/html/2607.12884#bib.bib18)\)are constructed from licensing examinations or biomedical literature, offering limited insight into real\-world patient interaction\(Raji et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib32); Agrawal et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib3)\)\. While later datasets incorporate consumer\-facing questions, including presuppositions\(Sambara et al\.,[2026](https://arxiv.org/html/2607.12884#bib.bib34); Zhu et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib43)\), they continue to evaluate responses in isolation\(Abacha et al\.,[2019](https://arxiv.org/html/2607.12884#bib.bib2); Singhal et al\.,[2023a](https://arxiv.org/html/2607.12884#bib.bib37); Nguyen et al\.,[2023](https://arxiv.org/html/2607.12884#bib.bib27)\)\. More recent efforts probe multi\-turn robustness through synthetic or adversarial follow\-ups, yet they do not capture how misconceptions arise and evolve in authentic multi\-turn interactions that real patients have when asking medical questions around their conditions\(Manczak et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib22); Laban et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib21)\)\. Consequently, a critical question remains unanswered:Can LLMs reliably identify and correct patient misconceptions as conversations unfold over multiple turns of patient\-authored interactions?

To address this question, we introduceThReadMed\-QA, a multi\-turn medical dialogue dataset derived from real patient–physician interactions on r/AskDocs\. The dataset includes 2,437 conversation threads, comprising a total of 8,204 question–answer pairs\. Unlike prior resources,ThReadMed\-QAcaptures how real patients iteratively seek medical guidance, preserving the natural flow of follow\-up questions, clarifications, and evolving concerns across turns\. As shown in Table[1](https://arxiv.org/html/2607.12884#S1.T1), it is the first dataset to combine patient\-authored questions, multi\-turn interactions, and physician\-verified responses within a single evaluation setting\. This structure enables systematic evaluation of whether LLMs can identify and correct misconceptions not only at initial contact, but as they emerge, persist, or evolve throughout a conversation\.

We evaluate five state\-of\-the\-art LLMs on their ability to identify and correct patient misconceptions in multi\-turn conversations\. Specifically, we assess whether models detect false assumptions embedded in patient questions and appropriately challenge or correct them as they arise throughout the interaction\. To enable scalable evaluation, we employ a rubric\-based LLM\-as\-a\-Judge framework, validated against expert evaluations, to score how effectively models handle misconceptions\. Our results reveal a consistent failure mode: models do not reliably sustain this behavior over time, with performance deteriorating as conversations unfold\. This instability indicates that misconception correction by LLMs is not a persistent capability, but one that breaks down under realistic multi\-turn interactions\.

### Generalizable Insights about Machine Learning in the Context of Healthcare

Our study highlights an important gap in how patient\-facing medical LLMs are currently evaluated\. We show that the ability to identify and correct patient misconceptions is not a stable capability, but one that degrades as interactions progress across turns\. This highlights the need to evaluate how models handle misconceptions as they emerge and evolve across interactions\. By enabling evaluation on real patient\-authored questions and follow\-ups with physician responses,ThReadMed\-QAprovides a foundation for studying model behavior under realistic conditions\. Beyond misconception handling, our proposed setup can support the evaluation of other safety\-critical behaviors that emerge during interaction\. Our findings suggest that future benchmarks and model development must explicitly account for interaction dynamics when evaluating patient\-facing systems\. To facilitate future research in this direction, we release the code111[https://github\.com/monicamunnangi/ThReadMed\-QA\-Misc](https://github.com/monicamunnangi/ThReadMed-QA-Misc)and the dataset222[https://huggingface\.co/datasets/monicamunnangi23/threadmed\-qa](https://huggingface.co/datasets/monicamunnangi23/threadmed-qa)\.

![Refer to caption](https://arxiv.org/html/2607.12884v1/x1.png)Figure 1:A representative conversation thread fromThReadMed\-QA\. Misconception can emerge early in the conversation \(left\) or in the later turns \(right\)\. Physicians correct these and provide safe answers irrespective of where they appear\.

## 2Related Work

##### Evolution of medical QA datasets

Early benchmarks for medical question answering primarily emphasize factual recall and exam\-style reasoning\. Datasets such as MedQA\(Jin et al\.,[2020](https://arxiv.org/html/2607.12884#bib.bib17)\), MedMCQA\(Pal et al\.,[2022](https://arxiv.org/html/2607.12884#bib.bib30)\), and PubMedQA\(Jin et al\.,[2019](https://arxiv.org/html/2607.12884#bib.bib18)\)are constructed from medical licensing examinations or biomedical literature, enabling controlled evaluation of medical knowledge but offering limited insight into real\-world patient interaction\(Raji et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib32); Agrawal et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib3)\)\. Subsequent efforts sought to better reflect consumer\-facing use cases by incorporating health\-related search queries and community\-authored questions, including Medication QA\(Abacha et al\.,[2019](https://arxiv.org/html/2607.12884#bib.bib2)\), and HealthSearchQA\(Singhal et al\.,[2023a](https://arxiv.org/html/2607.12884#bib.bib37)\)\. There are several datasets constructed from online health forums, search logs, and tele\-health consultations\(Nguyen et al\.,[2023](https://arxiv.org/html/2607.12884#bib.bib27); Abacha and Demner\-Fushman,[2019](https://arxiv.org/html/2607.12884#bib.bib1)\)\. These datasets improve coverage of lay language and consumer concerns, yet they continue to evaluate models largely through single\-turn settings\. To our knowledge,ThReadMed\-QAis the first multi\-turn medical dataset with patient\-authored questions and follow\-ups\.

##### Medical misconceptions and LLM Sycophancy

Questions that contain misconceptions have long been studied in linguistics\(Kaplan,[1978](https://arxiv.org/html/2607.12884#bib.bib20); Duží and Číhalová,[2015](https://arxiv.org/html/2607.12884#bib.bib12)\), where the appropriate response is often to challenge or negate the incorrect presupposition\. However, LLMs are known to align with users even when they are factually incorrect\(Perez et al\.,[2023](https://arxiv.org/html/2607.12884#bib.bib31)\), termed assycophancy\. This tendency is particularly concerning in medical contexts, where incorrect assumptions may involve diagnoses, medication safety, or symptom severity\. Recent work introduced datasets to explore misconception handling in both general domain\(Yu et al\.,[2023](https://arxiv.org/html/2607.12884#bib.bib42)\)and specific medical settings, including pregnancy and maternal health\(Srikanth et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib39)\)and cancer care\(Zhu et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib43)\)\. However, most prior work focuses on single\-turn settings\(Sambara et al\.,[2026](https://arxiv.org/html/2607.12884#bib.bib34)\)or relies on synthetically generated questions\(Zhu et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib43)\), which lack patient\-specific context and underestimate the difficulty of real interactions where misconceptions emerge implicitly and evolve over time\. Our dataset addresses this gap by enabling evaluation on authentic multi\-turn patient interactions\.

##### Multi\-turn LLM Interaction in Medicine

Recent work highlights a disconnect between how medical LLMs are evaluated and how they are used in practice\(Agrawal et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib3); Stanwyck et al\.,[2026](https://arxiv.org/html/2607.12884#bib.bib40)\)\. Systematic reviews show that the majority of medical LLM evaluations rely on synthetic or exam\-style data, with minimal use of real clinical or patient\-generated text\(Bedi et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib5)\)\. On the other hand, research reports LLMs having near\-clinician accuracy in single\-turn settings\(Singhal et al\.,[2023b](https://arxiv.org/html/2607.12884#bib.bib38)\)in some tasks, their performance degrades sharply when moving to multi\-turn interactions\(Laban et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib21)\)\. Error propagation is also a pattern observed in prior work on sequential and dialogue systems, showing that earlier model outputs can influence later predictions, producing compounding failures across an interaction\(Chen et al\.,[2017](https://arxiv.org/html/2607.12884#bib.bib8); Ranzato et al\.,[2016](https://arxiv.org/html/2607.12884#bib.bib33)\)\. In medical settings this brittleness is especially concerning: models that appear competent on static questions may fail to maintain consistency, challenge incorrect premises, or preserve appropriate safety guidance as a conversation unfolds\. However,\(Manczak et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib22)\)experimented with adversarial set\-ups in examination\-style MCQs and does not model real patient information\-seeking trajectories\. Our work addresses this gap by explicitly evaluating multi\-turn patient–LLM interactions grounded in authentic patient behavior\.

## 3Dataset

![Refer to caption](https://arxiv.org/html/2607.12884v1/x2.png)Figure 2:Dataset construction and filtering process\. We selected posts from r/AskDocs from Jan 2015 to Jun 2023, totaling approximately 1\.8 million posts\. After preprocessing and keeping the thread with the original poster \(OP\) and physician, we have the final setcomprising 8,204 question\-answer pairs, within the 2,437 conversation threads\.We introduceThReadMed\-QA, a multi\-turn medical dialogue dataset derived from real patient–physician interactions on r/AskDocs, a Reddit community where patients seek medical advice from healthcare professionals\. Unlike prior QA datasets,ThReadMed\-QAcaptures extended patient–physician dialogues in which patients ask follow\-up questions to clarify, refine, or challenge initial answers\.

### 3\.1r/AskDocs

We collect data from the subredditr/AskDocs333[https://www\.reddit\.com/r/AskDocs/](https://www.reddit.com/r/AskDocs/), an online forum where lay users seek medical advice from a community of physicians and healthcare professionals\. In this setting, users post detailed descriptions of their symptoms, medical history, and concerns, and may engage in follow\-up questions as the conversation evolves\. Responses are often provided by community\-verified medical professionals, whose credentials are reviewed by subreddit moderators and indicated through user flairs or tags \(e\.g\., “physician”, “doctor”\)\. This structure enables multi\-turn, patient\-physician dialogues grounded in realistic clinical communication, where initial questions are frequently refined or expanded through subsequent interaction\. The forum enforces structured posting guidelines that encourage users to include relevant demographic and clinical context, leading to detailed and consistent patient queries\. As a result,r/AskDocsprovides a natural source of longitudinal patient–physician exchanges suitable for studying how misconceptions arise and are addressed by humans over the course of a conversation\.

### 3\.2Dataset Construction

Similar to prior workNguyen et al\. \([2023](https://arxiv.org/html/2607.12884#bib.bib27)\), we collected posts from January 2015 through June 2023 fromr/AskDocs\. We then apply standard pre\-processing, followingNguyen et al\. \([2023](https://arxiv.org/html/2607.12884#bib.bib27)\)to remove low\-quality or incomplete content, including deleted posts and bot\-generated responses\. We also collected all comments from each post to reconstruct patient–physician exchanges\. The same cleaning pipeline is applied to comments\. An overview of the construction process is outlined in Figure[2](https://arxiv.org/html/2607.12884#S3.F2)and more details about pre\-processing and dataset construction are in Appendix[A](https://arxiv.org/html/2607.12884#A1)\.

Table 2:Comparison of the complete dataset and the fully answered subset\. Both datasets are derived from r/AskDocs threads\. The complete dataset includes all threads with three or more QA turns \(9,741 threads; 37,191 questions, 58\.1% answered\)\. The fully answered subset retains only threads where every question has a physician response \(2,437 threads; 8,204 questions\)\. Token counts use tiktokenc​l​100​k​\_​b​a​s​ecl100k\\\_base\.
### 3\.3Multi\-turn Construction

We construct linear patient–physician conversation threads by retaining interactions between the original poster \(OP\) and verified medical professionals, identified via r/AskDocs flair\. Follow\-up questions are restricted to the OP to preserve coherent patient narratives\. For posts with multiple comment branches, we select the branch most relevant to the original question using a word\-overlap heuristic\(Kannan et al\.,[2016](https://arxiv.org/html/2607.12884#bib.bib19)\), favoring on\-topic discussions\. We require at least two follow\-up questions per thread to ensure meaningful multi\-turn structure and filter out threads with incomplete or low\-quality responses\.

##### Orphan Handling:

Some threads contain unanswered follow\-up questions, which we refer to asorphans\. To enable evaluation with complete ground\-truth responses, we construct a fully answered subset by retaining only threads in which every question has a physician response\. After filtering, we obtain our primary dataset of 2,437 conversation threads comprising 8,204 question–answer pairs\.

Table[2](https://arxiv.org/html/2607.12884#S3.T2)summarizes key statistics of the dataset\. The complete collection contains 9,741 conversation threads \(37,191 question–answer pairs\), while the fully answered subset used for evaluation contains 2,437 threads \(8,204 pairs\)\. The data span 2015\-2023, covering a period of increasing reliance on online medical advice\. Conversations often involve patients providing detailed context \(mean of 129 tokens\) and refining their concerns through follow\-up questions, while physician responses tend to be more concise \(mean of 75 tokens\) and targeted\. This asymmetry reflects realistic patient–physician communication and creates a setting where misconceptions by patients may be introduced, clarified, or persist across turns\.

## 4Methods

### 4\.1Misconception Identification

We first identify questions that contain medical misconceptions\. Each question is labeled using an LLM\-based classifier as containing a misconception if it embeds an incorrect assumption or false presupposition about a medical condition, treatment, or outcome\. LLM\-as\-a\-Judge was validated on a subset of 310 questions against physician judgment with an agreement rate of 90% and Cohen’sκ\\kappa= 0\.72 \(substantial agreement, followingZhu et al\. \([2025](https://arxiv.org/html/2607.12884#bib.bib43)\); Munnangi et al\. \([2025](https://arxiv.org/html/2607.12884#bib.bib26)\)in the clinical domain\)\. We provide detailed information on validation in Appendix[B](https://arxiv.org/html/2607.12884#A2)\. We retain all conversation threads containing at least one such question, yielding a subset of 505 conversation threads used for evaluation\. We provide the full prompt in the Appendix[E](https://arxiv.org/html/2607.12884#A5)\.

### 4\.2Multi\-Turn Inference

For each thread, models generate responses sequentially with full conversational context\. At turntt, the model receives all prior patient questions and its own previous responses:

> \[User\]:⟨q0⟩\\langle q\_\{0\}\\rangle \[Assistant\]:⟨a0⟩\\langle a\_\{0\}\\rangle \[User\]:⟨q1⟩\\langle q\_\{1\}\\rangle \[Assistant\]:⟨a1⟩\\langle a\_\{1\}\\rangle \[User\]:⟨q2⟩\\langle q\_\{2\}\\rangle ⋮\\vdots

Each response conditions on all prior model outputs, such that the model’s own previous answers form part of the context for subsequent turns\.

### 4\.3Models

We evaluate five state\-of\-the\-art LLMs spanning the proprietary frontier and open\-source categories: GPT\-5\(OpenAI,[2025](https://arxiv.org/html/2607.12884#bib.bib29)\)\(snapshot:gpt\-5\-2025\-08\-07\), GPT\-4o\(OpenAI,[2024](https://arxiv.org/html/2607.12884#bib.bib28)\), Claude Haiku 4\.5\(Anthropic,[2026](https://arxiv.org/html/2607.12884#bib.bib4)\)\(snapshot:claude\-haiku\-4\-5\-20251001\), Gemini 2\.5 Flash\(et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib13)\), and Llama 3\.3–70B\-Instruct\(Grattafiori et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib14)\)\. All models are evaluated zero\-shot with no specific system prompt, no few\-shot examples, or medical\-domain specialization are applied, assessing out\-of\-the\-box capability\. Temperature is set to 0 for all models that support it\. We only experiment with general\-domain state\-of\-the\-art models, as numerous studies, includingJeong et al\. \([2024](https://arxiv.org/html/2607.12884#bib.bib16)\)andCeballos\-Arroyo et al\. \([2024](https://arxiv.org/html/2607.12884#bib.bib7)\), have shown that domain\-specific models often underperform and are rarely deployed in patient\-facing interfaces\.

### 4\.4Evaluation Framework

We evaluate model responses using a rubric\-based LLM\-as\-a\-judge framework, followingZhu et al\. \([2025](https://arxiv.org/html/2607.12884#bib.bib43)\)\. Each response is scored on a three\-point scale \(full prompt in Appendix[E](https://arxiv.org/html/2607.12884#A5)\):

- •Score \-1:The answer fails to recognize or acknowledge false presuppositions in the questions
- •Score 0:The answer appears aware of false presuppositions but often struggles to identify them clearly, or does not fully address them with the correct information
- •Score 1:The answer accurately addresses the false presuppositions, providing comprehensive responses that clarify misunderstandings or question the presuppositions

We validate our LLM\-as\-a\-Judge with physician judgment, with an agreement rate of 87\.14% and Cohen’sκ\\kappa= 0\.68 \(substantial agreement\)\. We provide detailed information on validation in Appendix[B](https://arxiv.org/html/2607.12884#A2)\. We also adapt multiple models as LLM judges and aggregate the scores to de\-bias the judgments \(more information in Appendix[B](https://arxiv.org/html/2607.12884#A2)\) following prior workWataoka et al\. \([2025](https://arxiv.org/html/2607.12884#bib.bib41)\)\.

### 4\.5Metrics

#### 4\.5\.1Single\-turn Metrics:

We adopt single\-turn metrics fromZhu et al\. \([2025](https://arxiv.org/html/2607.12884#bib.bib43)\)\. PCR \(Presupposition Correction Rate\) measures the proportion of responses that correctly address the misconception \(score=1=1\), while PCS \(Presupposition Correction Score\) is the average score over\{−1,0,1\}\\\{\-1,0,1\\\}, capturing partial and incorrect responses\.

#### 4\.5\.2Multi\-turn Metrics:

To analyze model behavior over the course of a conversation, we extend single\-turn metrics to the multi\-turn setting\.

##### Per\-turn Performance:

We compute the Presupposition Correction Rate \(PCR\) at each turntt:

PCRt=1Nt​∑i=1Nt𝕀​\[si,t=1\]\\text\{PCR\}\_\{t\}=\\frac\{1\}\{N\_\{t\}\}\\sum\_\{i=1\}^\{N\_\{t\}\}\\mathbb\{I\}\[s\_\{i,t\}=1\]\(1\)wheresi,t∈\{−1,0,1\}s\_\{i,t\}\\in\\\{\-1,0,1\\\}is the score for exampleiiat turntt, andNtN\_\{t\}is the number of evaluated instances at that turn and𝕀​\(⋅\)\\mathbb\{I\}\(\\cdot\)is the indicator function, equal to 1 if the condition holds and 0 otherwise\.

##### Recovery Rate:

We measure the ability of models to recover from prior failures in misconception correction across turns\. Specifically, we compute the conditional probability that a model correctly resolves a misconception at turnt\+1t\+1given that it failed to do so at turntt:

RR=P​\(st\+1=1∣st≠1\)\\text\{RR\}=P\(s\_\{t\+1\}=1\\mid s\_\{t\}\\neq 1\)\(2\)wherest∈\{−1,0,1\}s\_\{t\}\\in\\\{\-1,0,1\\\}denotes the score at turntt\. Higher values indicate that a model is more likely to recover and correctly address a misconception after previously failing\. We estimate this probability over all consecutive turn pairs\(t,t\+1\)\(t,t\+1\)where both turns are labeled as containing misconceptions\.

![Refer to caption](https://arxiv.org/html/2607.12884v1/x3.png)Figure 3:Example of multi\-turn misconception handling under two inference settings\. The same patient\-authored conversation is shown with different model responses\. On the left, the model generates responses conditioned on its own previous outputs, leading to failure to explicitly address the misconception \(“I most definitely have a concussion”\) and resulting in incomplete guidance \(score: 0\)\. On the right, \(oracle \-physician\)the model produces a response that correctly identifies and challenges the underlying false presupposition while providing appropriate medical advice \(score: 1\)\.

### 4\.6Oracle Physician Ablation

To separate the effect of question difficulty and embedded false presuppositions from errors introduced by prior model\-generated context, we conduct an oracle\-style ablation in which all preceding turns use verified physician responses from the r/AskDocs thread, rather than the model’s own prior outputs\.

For each multi\-turn thread, let patient turns be indexed ast=0,1,…,T−1t=0,1,\\ldots,T\-1\. For each target turnt≥1t\\geq 1, we construct the input by concatenating the dialogue history up to that turn\. Specifically, for eachi<ti<t, we include the patient questionQiQ\_\{i\}and the corresponding physician responseAiphysA^\{\\mathrm\{phys\}\}\_\{i\}\. For the target turn, we append only the patient questionQtQ\_\{t\}, which the model must answer\. Under this setup, the model conditions exclusively on the oracle \(physician\) context and never on its own prior generations\. Since turnt=0t=0has no preceding in\-thread physician responses, oracle prompts are defined only fort≥1t\\geq 1\. We report PCR \(%\) and Uncorrected \(%\) in this setting to assess the impact of oracle context on model behavior\. We illustrate this setup in Figure[3](https://arxiv.org/html/2607.12884#S4.F3)\.

## 5Results

![Refer to caption](https://arxiv.org/html/2607.12884v1/x4.png)

\(a\)

![Refer to caption](https://arxiv.org/html/2607.12884v1/x5.png)

\(b\)

Figure 4:Overall misconception correction performance across models\. \(a\) PCR \(%\) shows correction ratesaggregated across all turns\. \(b\) Uncorrected response rates \(%\), including both partially correct \(0\) and incorrect \(−1\-1\) responses,aggregated across all turns, indicate how often models fail to fully resolve misconceptions\.Figure[4](https://arxiv.org/html/2607.12884#S5.F4)summarizes overall misconception correctionperformance aggregated across all turns\. As shown in Figure[4](https://arxiv.org/html/2607.12884#S5.F4)\(a\), frontier models achieve higher correction rates, with Claude Haiku and GPT\-5 exceeding 60% PCR, while Gemini 2\.5 Flash performs moderately \(43\.0%\)\. In contrast, GPT\-4o and Llama 3\.3–70B show substantially lower correction rates\. Figure[4](https://arxiv.org/html/2607.12884#S5.F4)\(b\) provides a complementary view by reporting the proportion of responses that fail to fully correct misconceptions\. These uncorrected rates remain high across all models, exceeding 35% even for the strongest systems and rising above 65% for weaker models\. This indicates that even when evaluated across all turns, models frequently provide incomplete or incorrect responses to misconception\-laden questions\.

![Refer to caption](https://arxiv.org/html/2607.12884v1/x6.png)Figure 5:Multi\-turn Presupposition Correction Rate \(PCR\) across models with confidence intervals \(CI\)\. Performance consistently degrades as conversations progress, with all models exhibiting substantial drops after the initial turn\. Even frontier models such as Claude Haiku and GPT\-5 show marked declines, while weaker models degrade further and remain at low performance across turns\.### 5\.1Observed Correction Rates Decline at Later Turns

We next analyze model performance across turns within a conversation by measuring PCR at each turn\. As shown in Figure[5](https://arxiv.org/html/2607.12884#S5.F5), performance decreases consistently as conversations progress\. Even frontier models that achieve strong initial performance fail to sustain this behavior over time\. GPT\-5 starts at 88\.1% PCR at the first turn but drops to 48\.0% by turn 2, a reduction of over 40 percentage points\. Similarly, Claude Haiku declines from 84\.4% to 53\.5% over the same span, indicating that initial success does not translate into stable multi\-turn performance\. This effect is more pronounced for weaker models\. GPT\-4o drops from 65\.1% at the first turn to 21\.2% by turn 2, while Llama 3\.3–70B falls from 55\.0% to 17\.7%\. At these levels, most misconception\-laden queries are no longer correctly addressed\. We report detailed results in Appendix[C](https://arxiv.org/html/2607.12884#A3)\.

Across all models, the largest drop occurs within the first few turns, after which performance stabilizes at substantially lower levels\. Overall, these results show that misconception correction is not a stable capability, and models struggle to maintain it as conversations unfold\.

Table 3:Recovery rate \(RR\) across models with 95% bootstrap confidence intervals \(1000 bootstrap iterations\)\. RR measures the probability that a model correctly resolves a misconception at turnt\+1t\+1given failure at turntt\. \# Failure\-Start Pairs denotes the number of consecutive turn pairs where the model failed at turntt\.
### 5\.2Limited Recovery Across Turns

While the previous analysis shows that performance declines across turns, we next examine whether models can recover after an initial failure\. Table[3](https://arxiv.org/html/2607.12884#S5.T3)reports recovery rates \(RR\) with 95% confidence intervals estimated via bootstrap resampling \(1,000 iterations\)\. Because RR is conditioned on prior failure, the evaluated instances differ across models; consequently, RR should not be interpreted as a strict ranking of model capability, but rather as a diagnostic measure of how difficult it is for models to recover once an incorrect conversational trajectory has been established\.

Recovery remains limited across all systems, indicating that once a model fails to correct a misconception, it is unlikely to recover in subsequent turns\. Even stronger models show only partial recovery\. Claude Haiku achieves the highest recovery rate at 51\.9% \(33\.3–69\.6\), while GPT\-5 recovers in only 25\.9% \(9\.5–45\) of cases following a failure\. Recovery is substantially lower for weaker models, with GPT\-4o and Llama 3\.3–70B below 15%, and confidence intervals reflecting high uncertainty due to the limited number of failure\-start pairs\.

Taken together, these results suggest that once models fail to identify or correct a misconception, recovery is uncommon\. Although RR is conditioned on model\-specific failures and is not intended for direct cross\-model comparison, it highlights a shared vulnerability where models struggle to recover from erroneous conversational trajectories\.

![Refer to caption](https://arxiv.org/html/2607.12884v1/x7.png)

\(a\)

![Refer to caption](https://arxiv.org/html/2607.12884v1/x8.png)

\(b\)

Figure 6:Oracle evaluation using physician\-provided context\. \(a\) PCR \(%\) aggregated over all turns and \(b\) uncorrected response rates \(%\), aggregated over all turns, when prior turns are replaced with gold physician answers\. Improvements relative to the standard setting indicate the impact of error propagation from model\-generated context\.
### 5\.3Oracle Physician Context Substantially Improves Performance

To isolate the effect of conversational context, we evaluate models in an oracle setting where prior turns \(LLMs’ responses\) are replaced with physician\-provided responses\. This removes error accumulation from earlier model outputs and allows us to assess performance when conditioned on the correct context\. Figure[6](https://arxiv.org/html/2607.12884#S5.F6)shows that all models improve substantially under the oracle context\. Correction rates increase across the board, indicating that a significant portion of multi\-turn degradation arises from error propagation\. For example, GPT\-5 achieves 87\.4% PCR, up from 62\.4%, while Claude Haiku reaches 75\.8% PCR, up from 60\.6%\. Even weaker models show notable gains, with GPT\-4o and Llama 3\.3–70B approaching 50% PCR under oracle context\.

Overall, models continue to exhibit high rates of uncorrected responses, particularly for weaker systems, suggesting that correct context alone is insufficient to fully resolve misconception handling\. Additionally, these results indicate that error propagation is a major contributor to multi\-turn degradation, but intrinsic limitations in identifying and correcting misconceptions remain an important source of failure\.

##### Turn Difficulty & Oracle Style Shifts:

To fully isolate error propagation from both question difficulty and style shifts, we perform another ablation "target\-turn", and summarize the results in the following table[4](https://arxiv.org/html/2607.12884#S5.T4)\. This setting provides the patient’s prior questions but removes all prior responses \(model or physician\)\. Results show that removing prior model responses significantly improves performance \(e\.g\., GPT\-5 improves from 60\.6% to 76\.3%\), confirming that models actively propagate their own errors, though a gap below the Oracle remains due to intrinsic difficulty\.

Table 4:Presupposition Correction Rate \(PCR, %\) across conversational settings\. Multi\-turn uses model\-generated conversation history, Target\-Turn evaluates questions independently without prior context \(responses\), and Oracle conditions on physician responses\. Higher Oracle performance suggests that error propagation from previous model responses is a major contributor to multi\-turn degradation\.

## 6Discussion

Patients often ask questions built on incorrect assumptions, requiring AI systems not just to answer but to identify and correct the underlying misconception\(Srikanth et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib39)\)\. These misconceptions can evolve across multi\-turn conversations, as patients reinterpret prior guidance from the AI system\. Yet existing evaluation frameworks mostly test isolated questions, missing how misconceptions emerge, persist, and shift over the course of a dialogue\(Agrawal et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib3); Bedi et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib5)\)\.

Our results show that strong single\-turn performance does not guarantee reliability over the course of a conversation\. While several models demonstrate strong correction ability at the initial turn, this ability degrades rapidly as conversations progress, with substantial declines within the first few follow\-ups\. As performance drops, misconceptions increasingly go unresolved, and errors tend to persist rather than being corrected in later responses\. This pattern suggests that turns are not independent: a missed correction shapes the context for everything that follows, producing path\-dependent failures rather than isolated ones\. This is consistent with prior work on sequential and dialogue systems\(Bigham et al\.,[2017](https://arxiv.org/html/2607.12884#bib.bib6)\), where earlier outputs influence later predictions and compound into larger failures over an interaction\(Chen et al\.,[2017](https://arxiv.org/html/2607.12884#bib.bib8); Ranzato et al\.,[2016](https://arxiv.org/html/2607.12884#bib.bib33)\)\.

The oracle\-physician ablation supports this interpretation: replacing prior model\-generated responses with physician answers consistently improves performance across all models, indicating that some of the decline is driven by earlier model outputs rather than the difficulty of any single turn\. However, performance still falls short of perfect for some models even with this oracle context, suggesting that clean prior context alone is not sufficient\. Together, these results point to two distinct contributors to failure: propagation of earlier errors, and the underlying difficulty of catching misconceptions at all\.

These findings carry implications for how LLMs are evaluated and deployed in patient\-facing settings\. Benchmarks based on isolated queries miss the temporal dynamics observed here and can overestimate reliability in real interactions\. Multi\-turn evaluation reveals failure modes hidden in single\-turn settings, including performance decline, persistence of errors, and limited recovery after mistakes\(Laban et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib21); Manczak et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib22); Dongre et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib10)\)\. Evaluation should account for interaction dynamics, not just single\-turn accuracy\.

This also points to a design implication\. Since much of the decline traces back to uncorrected errors compounding over time, brief points of human input could break that chain\(Morris et al\.,[2017](https://arxiv.org/html/2607.12884#bib.bib24)\)\. One approach is lightweight micro\-interactions, where a physician or other healthcare worker flags a response mid\-conversation or can manually provide a correction\(Savage et al\.,[2016](https://arxiv.org/html/2607.12884#bib.bib35)\), giving the model a chance to correct course before the error shapes later turns\. The oracle\-physician ablation supports the potential value of this: even partial, intermittent physician input could meaningfully improve responses over time\.

More broadly, this also suggests a role for multi\-stakeholder system design\(Savage et al\.,[2022](https://arxiv.org/html/2607.12884#bib.bib36)\), where different healthcare workers contribute different kinds of oversight: physicians catching clinical errors, while nurses or care navigators watch for communication breakdowns or signs that a patient has misunderstood guidance\. Distributing oversight this way could be more scalable than relying on physicians alone, since not every error requires the same level of expertise to catch, and it better reflects how care is delivered across multiple roles\.

Beyond misconception handling,ThReadMed\-QAoffers a realistic testbed for studying broader phenomena in longitudinal interactions, including error accumulation, safety drift, and sensitivity to evolving user input\. Future work could use the dataset to develop and evaluate models built for multi\-turn reliability\. For instance, approaches that explicitly incorporate feedback, correction, and consistency across turns, or to fine\-tune and align models to better detect and resolve misconceptions in realistic patient interactions\. This matters most in clinical contexts, where uncorrected early errors can compound into larger misinformation over the course of an interaction\.

##### Limitations

Several limitations of this work warrant consideration\. First, the ground truth responses are drawn from r/AskDocs, an online forum, and are not equivalent to formal clinical documentation\. Although responses are provided by verified physicians and subject to moderation, they may vary in completeness and reflect differences in clinical judgment, introducing potential noise in the reference answers\. We only use the responses for oracle\-physician ablation in this work\. Second, the dataset is derived from Reddit and may reflect platform\-specific biases\. The user population is not representative of the general patient population, and interactions often follow informal, iterative patterns shaped by the platform\. In addition, our filtering procedure retains only multi\-turn threads with follow\-up questions and a fully answered subset, which may over\-represent more complex or engaged interactions and under\-represent simpler, single\-turn queries\. Third, misconception identification and response evaluation rely on an LLM\-as\-a\-Judge framework\. Despite strong agreement with physician annotations, this approach may introduce labeling noise that can affect downstream metrics\. Fourth, the number of failure\-start pairs used to estimate recovery rates is limited, resulting in wide confidence intervals and restricting the strength of conclusions regarding recovery behavior\. Fifth, comparisons between GPT\-5 and other models should be interpreted cautiously, as GPT\-5 outputs are stochastic while other models are evaluated deterministically\. Finally, the dataset is restricted to English\-language text and does not include multi\-modal inputs, which are common in real clinical settings\.

##### Ethical Considerations

While Reddit posts are publicly available, we recognize that users may not have originally intended their queries for large\-scale research\. To prioritize privacy, we strictly adhered to Reddit’s API terms of use and performed rigorous de\-identification\. We removed all usernames, post IDs, and specific timestamps, replacing them with unique anonymized identifiers \(e\.g\., THREAD\_001\)\. No metadata that could lead to the re\-identification of specific Reddit threads or users was included in our analysis\.

\\acks

This work was supported in part by National Science Foundation \(NSF\) awards 2339443 and 2403252\. We also thank Monica’s thesis committee \(Byron Wallace, Malihe Alikhani, Aakanksha Naik\), the anonymous reviewers, Christopher Curtis, Hunjun Shin, and Smit Kiri for their valuable feedback\.

## References

- Abacha and Demner\-Fushman \(2019\)Asma Ben Abacha and Dina Demner\-Fushman\.On the summarization of consumer health questions\.In*Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 2228–2234, 2019\.
- Abacha et al\. \(2019\)Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan, and Dina Demner\-Fushman\.Bridging the gap between consumers’ medication questions and trusted answers\.In*MEDINFO 2019: Health and Wellbeing e\-Networks for All*, pages 25–29\. IOS Press, 2019\.
- Agrawal et al\. \(2025\)Manish Agrawal, Irene Y Chen, Freya Gulamali, and Shalmali Joshi\.The evaluation illusion of large language models in medicine\.*NPJ Digital Medicine*, 8, 2025\.URL[https://api\.semanticscholar\.org/CorpusID:281889723](https://api.semanticscholar.org/CorpusID:281889723)\.
- Anthropic \(2026\)Anthropic\.Anthropic\.claude 4\.5 haiku\.[https://www\.anthropic\.com/claude/haiku](https://www.anthropic.com/claude/haiku), 2026\.Accessed: 2025\-10\-15\.
- Bedi et al\. \(2024\)Suhana Bedi, Yutong Liu, Lucy Orr\-Ewing, Dev Dash, Sanmi Koyejo, Alison Callahan, Jason A\. Fries, Michael Wornow, Akshay Swaminathan, Lisa Soleymani Lehmann, Hyo Jung Hong, Mehr Kashyap, Akash R\. Chaurasia, Nirav R\. Shah, Karandeep Singh, Troy Tazbaz, Arnold Milstein, Michael A\. Pfeffer, and Nigam H\. Shah\.A systematic review of testing and evaluation of healthcare applications of large language models \(llms\)\.*medRxiv*, 2024\.[10\.1101/2024\.04\.15\.24305869](https://arxiv.org/doi.org/10.1101/2024.04.15.24305869)\.URL[https://www\.medrxiv\.org/content/early/2024/04/18/2024\.04\.15\.24305869](https://www.medrxiv.org/content/early/2024/04/18/2024.04.15.24305869)\.
- Bigham et al\. \(2017\)Jeffrey P Bigham, Raja Kushalnagar, Ting\-Hao Kenneth Huang, Juan Pablo Flores, and Saiph Savage\.On how deaf people might use speech to control devices\.In*Proceedings of the 19th international ACM SIGACCESS conference on computers and accessibility*, pages 383–384, 2017\.
- Ceballos\-Arroyo et al\. \(2024\)Alberto Mario Ceballos\-Arroyo, Monica Munnangi, Jiuding Sun, Karen Zhang, Jered McInerney, Byron C\. Wallace, and Silvio Amir\.Open \(clinical\) LLMs are sensitive to instruction phrasings\.In Dina Demner\-Fushman, Sophia Ananiadou, Makoto Miwa, Kirk Roberts, and Junichi Tsujii, editors,*Proceedings of the 23rd Workshop on Biomedical Natural Language Processing*, pages 50–71, Bangkok, Thailand, August 2024\. Association for Computational Linguistics\.[10\.18653/v1/2024\.bionlp\-1\.5](https://arxiv.org/doi.org/10.18653/v1/2024.bionlp-1.5)\.URL[https://aclanthology\.org/2024\.bionlp\-1\.5/](https://aclanthology.org/2024.bionlp-1.5/)\.
- Chen et al\. \(2017\)Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang\.A survey on dialogue systems: Recent advances and new frontiers\.*ACM SIGKDD Explorations Newsletter*, 19\(2\):25–35, November 2017\.ISSN 1931\-0153\.[10\.1145/3166054\.3166058](https://arxiv.org/doi.org/10.1145/3166054.3166058)\.URL[http://dx\.doi\.org/10\.1145/3166054\.3166058](http://dx.doi.org/10.1145/3166054.3166058)\.
- Costa\-Gomes et al\. \(2025\)Beatriz Costa\-Gomes, Sophia Chen, Connie Hsueh, Deborah Morgan, Philipp Schoenegger, Yash Shah, Sam Way, Yuki Zhu, Timothé Adeline, Michael Bhaskar, Mustafa Suleyman, and Seth Spielman\.It’s about time: The temporal and modal dynamics of copilot usage, 2025\.URL[https://arxiv\.org/abs/2512\.11879](https://arxiv.org/abs/2512.11879)\.
- Dongre et al\. \(2025\)Vardhan Dongre, Ryan A Rossi, Viet Dac Lai, David Seunghyun Yoon, Dilek Hakkani\-Tür, and Trung Bui\.Drift no more? context equilibria in multi\-turn llm interactions\.*arXiv preprint arXiv:2510\.07777*, 2025\.
- Dunn et al\. \(2022\)Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain\.Structured information extraction from complex scientific text with fine\-tuned large language models\.*arXiv preprint arXiv:2212\.05238*, 2022\.
- Duží and Číhalová \(2015\)Marie Duží and Martina Číhalová\.Questions, answers, and presuppositions\.*Computación y Sistemas*, 19\(4\):647–659, 2015\.
- et al\. \(2025\)Gheorghe Comanici et al\.Gemini 2\.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025\.URL[https://arxiv\.org/abs/2507\.06261](https://arxiv.org/abs/2507.06261)\.
- Grattafiori et al\. \(2024\)Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al\-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al\.The llama 3 herd of models\.*arXiv preprint arXiv:2407\.21783*, 2024\.
- Heritage and Maynard \(2006\)John Heritage and Douglas W Maynard\.*Communication in Medical Care: Interaction between Primary Care Physicians and Patients*, volume 20 of*Studies in Interactional Sociolinguistics*\.Cambridge University Press, Cambridge, 2006\.ISBN 9780521628990\.[10\.1017/CBO9780511607172](https://arxiv.org/doi.org/10.1017/CBO9780511607172)\.
- Jeong et al\. \(2024\)Daniel P Jeong, Saurabh Garg, Zachary Chase Lipton, and Michael Oberst\.Medical adaptation of large language and vision\-language models: Are we making progress?In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 12143–12170, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.[10\.18653/v1/2024\.emnlp\-main\.677](https://arxiv.org/doi.org/10.18653/v1/2024.emnlp-main.677)\.URL[https://aclanthology\.org/2024\.emnlp\-main\.677/](https://aclanthology.org/2024.emnlp-main.677/)\.
- Jin et al\. \(2020\)Di Jin, Eileen Pan, Nassim Oufattole, Wei\-Hung Weng, Hanyi Fang, and Peter Szolovits\.What disease does this patient have? a large\-scale open domain question answering dataset from medical exams, 2020\.URL[https://arxiv\.org/abs/2009\.13081](https://arxiv.org/abs/2009.13081)\.
- Jin et al\. \(2019\)Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W\. Cohen, and Xinghua Lu\.Pubmedqa: A dataset for biomedical research question answering, 2019\.URL[https://arxiv\.org/abs/1909\.06146](https://arxiv.org/abs/1909.06146)\.
- Kannan et al\. \(2016\)S\. Kannan, S\. Karuppusamy, A\. Nedunchezhian, P\. Venkateshan, P\. Wang, N\. Bojja, and A\. Kejariwal\.Chapter 3 \- big data analytics for social media\.In Rajkumar Buyya, Rodrigo N\. Calheiros, and Amir Vahid Dastjerdi, editors,*Big Data*, pages 63–94\. Morgan Kaufmann, 2016\.ISBN 978\-0\-12\-805394\-2\.[https://doi\.org/10\.1016/B978\-0\-12\-805394\-2\.00003\-9](https://arxiv.org/doi.org/https://doi.org/10.1016/B978-0-12-805394-2.00003-9)\.URL[https://www\.sciencedirect\.com/science/article/pii/B9780128053942000039](https://www.sciencedirect.com/science/article/pii/B9780128053942000039)\.
- Kaplan \(1978\)S Jerrold Kaplan\.Indirect responses to loaded questions\.In*Theoretical issues in natural language processing\-2*, 1978\.
- Laban et al\. \(2025\)Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville\.Llms get lost in multi\-turn conversation\.*arXiv preprint arXiv:2505\.06120*, 2025\.
- Manczak et al\. \(2025\)Blazej Manczak, Eric Lin, Francisco Eiras, James O’ Neill, and Vaikkunth Mugunthan\.Shallow robustness, deep vulnerabilities: Multi\-turn evaluation of medical llms\.*arXiv preprint arXiv:2510\.12255*, 2025\.
- Montero et al\. \(2026\)Alex Montero, Julian Montalvo III, Audrey Kearney, Isabelle Valdes, Ashley Kirzinger, and Liz Hamel\.KFF Tracking Poll on Health Information and Trust: Use of AI For Health Information and Advice \| KFF — kff\.org\.[https://www\.kff\.org/public\-opinion/kff\-tracking\-poll\-on\-health\-information\-and\-trust\-use\-of\-ai\-for\-health\-information\-and\-advice/](https://www.kff.org/public-opinion/kff-tracking-poll-on-health-information-and-trust-use-of-ai-for-health-information-and-advice/), 2026\.\[Accessed 04\-04\-2026\]\.
- Morris et al\. \(2017\)Meredith Ringel Morris, Jeffrey P Bigham, Robin Brewer, Jonathan Bragg, Anand Kulkarni, Jessie Li, and Saiph Savage\.Subcontracting microwork\.In*Proceedings of the 2017 CHI conference on human factors in computing systems*, pages 1867–1876, 2017\.
- Munnangi et al\. \(2024\)Monica Munnangi, Sergey Feldman, Byron Wallace, Silvio Amir, Tom Hope, and Aakanksha Naik\.On\-the\-fly definition augmentation of LLMs for biomedical NER\.In Kevin Duh, Helena Gomez, and Steven Bethard, editors,*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 3833–3854, Mexico City, Mexico, June 2024\. Association for Computational Linguistics\.[10\.18653/v1/2024\.naacl\-long\.212](https://arxiv.org/doi.org/10.18653/v1/2024.naacl-long.212)\.URL[https://aclanthology\.org/2024\.naacl\-long\.212/](https://aclanthology.org/2024.naacl-long.212/)\.
- Munnangi et al\. \(2025\)Monica Munnangi, Akshay Swaminathan, Jason Alan Fries, Jenelle A Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi Omiye, Mehr Kashyap, and Nigam Shah\.FactEHR: A dataset for evaluating factuality in clinical notes using LLMs\.In Monica Agrawal, Kaivalya Deshpande, Matthew Engelhard, Shalmali Joshi, Shengpu Tang, and Iñigo Urteaga, editors,*Proceedings of the 10th Machine Learning for Healthcare Conference*, volume 298 of*Proceedings of Machine Learning Research*\. PMLR, 15–16 Aug 2025\.URL[https://proceedings\.mlr\.press/v298/munnangi25a\.html](https://proceedings.mlr.press/v298/munnangi25a.html)\.
- Nguyen et al\. \(2023\)Vincent Nguyen, Sarvnaz Karimi, Maciej Rybinski, and Zhenchang Xing\.Medredqa for medical consumer question answering: Dataset, tasks, and neural baselines\.In*Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia\-Pacific Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 629–648, 2023\.
- OpenAI \(2024\)OpenAI\.Gpt\-4o system card, 2024\.URL[https://arxiv\.org/abs/2410\.21276](https://arxiv.org/abs/2410.21276)\.
- OpenAI \(2025\)OpenAI\.GPT\-5 system card, August 2025\.URL[https://cdn\.openai\.com/gpt\-5\-system\-card\.pdf](https://cdn.openai.com/gpt-5-system-card.pdf)\.Accessed: 2026\-01\-05\.
- Pal et al\. \(2022\)Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu\.Medmcqa : A large\-scale multi\-subject multi\-choice dataset for medical domain question answering, 2022\.URL[https://arxiv\.org/abs/2203\.14371](https://arxiv.org/abs/2203.14371)\.
- Perez et al\. \(2023\)Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al\.Discovering language model behaviors with model\-written evaluations\.In*Findings of the association for computational linguistics: ACL 2023*, pages 13387–13434, 2023\.
- Raji et al\. \(2025\)Inioluwa Deborah Raji, Roxana Daneshjou, and Emily Alsentzer\.It’s time to bench the medical exam benchmark, 2025\.
- Ranzato et al\. \(2016\)Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba\.Sequence level training with recurrent neural networks, 2016\.URL[https://arxiv\.org/abs/1511\.06732](https://arxiv.org/abs/1511.06732)\.
- Sambara et al\. \(2026\)Sraavya Sambara, Yuan Pu, Ayman Ali, Vishala Mishra, Lionel Wong, and Monica Agrawal\.Medredflag: Investigating how llms redirect misconceptions in real\-world health communication, 2026\.URL[https://arxiv\.org/abs/2601\.09853](https://arxiv.org/abs/2601.09853)\.
- Savage et al\. \(2016\)Saiph Savage, Andres Monroy\-Hernandez, and Tobias Höllerer\.Botivist: Calling volunteers to action using online bots\.In*Proceedings of the 19th ACM Conference on Computer\-Supported Cooperative Work & Social Computing*, pages 813–822, 2016\.
- Savage et al\. \(2022\)Saiph Savage, Claudia Flores\-Saviaga, Rachel Rodney, Liliana Savage, Jon Schull, and Jennifer Mankoff\.The global care ecosystems of 3d printed assistive devices\.*ACM Transactions on Accessible Computing*, 15\(4\):1–29, 2022\.
- Singhal et al\. \(2023a\)Karan Singhal, Shekoofeh Azizi, Tao Tu, Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole\-Lewis, Perry Payne, Stephen Pfohl, Martin Seneviratne, Paul Gamble, Christopher Kelly, Abubakr Abdelrazig Hassan Babiker, Nathanael Schaerli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner\-Fushman, Blaise Aguera\-Arcas, Dale Webster, Greg Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomašev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan\.Large language models encode clinical knowledge\.*Nature*, 2023a\.URL[https://www\.nature\.com/articles/s41586\-023\-06291\-2](https://www.nature.com/articles/s41586-023-06291-2)\.
- Singhal et al\. \(2023b\)Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole\-Lewis, Darlene Neal, et al\.Towards expert\-level medical question answering with large language models\.*arXiv preprint arXiv:2305\.09617*, 2023b\.
- Srikanth et al\. \(2024\)Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh Nguyen, Rachel Rudinger, and Jordan Boyd\-Graber\.Pregnant questions: The importance of pragmatic awareness in maternal health question answering\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 7253–7268, 2024\.
- Stanwyck et al\. \(2026\)Chloe Stanwyck, Amin Adibi, Paz Dozie\-Nnamah, and Emily Alsentzer\.Boards\-style benchmarks overestimate prior\-chat bias in large language models: a factorial evaluation study\.*medRxiv*, 2026\.[10\.64898/2026\.02\.12\.26346164](https://arxiv.org/doi.org/10.64898/2026.02.12.26346164)\.URL[https://www\.medrxiv\.org/content/early/2026/02/14/2026\.02\.12\.26346164](https://www.medrxiv.org/content/early/2026/02/14/2026.02.12.26346164)\.
- Wataoka et al\. \(2025\)Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri\.Self\-preference bias in llm\-as\-a\-judge, 2025\.URL[https://arxiv\.org/abs/2410\.21819](https://arxiv.org/abs/2410.21819)\.
- Yu et al\. \(2023\)Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi\.CREPE: Open\-domain question answering with false presuppositions\.In Anna Rogers, Jordan Boyd\-Graber, and Naoaki Okazaki, editors,*Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 10457–10480, Toronto, Canada, July 2023\. Association for Computational Linguistics\.[10\.18653/v1/2023\.acl\-long\.583](https://arxiv.org/doi.org/10.18653/v1/2023.acl-long.583)\.URL[https://aclanthology\.org/2023\.acl\-long\.583/](https://aclanthology.org/2023.acl-long.583/)\.
- Zhu et al\. \(2025\)Wang Bill Zhu, Tianqi Chen, Xinyan Velocity Yu, Ching Ying Lin, Jade Law, Mazen Jizzini, Jorge J\. Nieva, Ruishan Liu, and Robin Jia\.Cancer\-myth: Evaluating large language models on patient questions with false presuppositions, 2025\.URL[https://arxiv\.org/abs/2504\.11373](https://arxiv.org/abs/2504.11373)\.

## Appendix AAdditional Dataset Details

### A\.1Source and Format Conversion

We constructThReadMed\-QAfrom publicly available Reddit data fromr/AskDocs, where users seek medical advice from healthcare professionals\. The raw data consist of JSONL files containing one record per post and comment, with metadata fields includingid,link\_id,parent\_id,body,author,created\_utc,author\_flair\_text, andis\_submitter\.

To reconstruct conversational structure, we group comments bylink\_idand build a tree for each post, where top\-level comments correspond to replies to the original post\. This tree representation is used to extract linear multi\-turn conversation threads\.

### A\.2Filtering and Preprocessing

We restrict posts to a fixed time window \(January 2015 to June 2023\) and retain only those with at least one comment to ensure engagement\. We apply standard preprocessing, followingNguyen et al\. \([2023](https://arxiv.org/html/2607.12884#bib.bib27)\), to remove low\-quality or incomplete content\. This includes:

- •deleted or moderator\-removed posts and comments,
- •content from banned users or known bots,
- •posts with minimal textual content \(fewer than five words\),
- •image\-only or non\-textual submissions\.

URLs are removed from both posts and comments\. The same cleaning procedure is applied to comment trees, and only threads associated with retained posts are preserved\.

### A\.3Multi\-turn Thread Construction

We extract linear patient–physician conversations from the comment trees using the following procedure\.

##### Participant roles\.

The original post is treated as the first patient question\. Subsequent messages authored by the original poster \(OP\) are treated as follow\-up questions\. Responses from users with verified medical flair \(e\.g\., Physician, Doctor\) are treated as answers\. Comments from non\-clinicians are ignored when constructing the question–answer sequence\.

##### Branch selection\.

Each post may contain multiple comment branches\. For each top\-level reply, we construct a candidate linear conversation by traversing its descendants in chronological order\. Among candidate branches, we select the one most relevant to the original question using a word\-overlap heuristic between the post text and the first physician response\. This favors coherent, on\-topic exchanges\.

##### Multi\-turn requirement\.

We retain only threads with at least two follow\-up questions beyond the initial post, ensuring a minimum of three question–answer pairs per thread\.

##### Quality filtering\.

We remove threads with incomplete or low\-quality responses, including those with excessive\[deleted\]or\[removed\]content or broken formatting\. If the OP deletes their content mid\-thread, the entire conversation is discarded\.

### A\.4Orphan Handling and Dataset Variants

Some conversations contain unanswered follow\-up questions, which we refer to asorphans\. These arise when a patient’s follow\-up receives no physician response or when responses are removed during preprocessing\.

We construct two dataset variants:

- •Complete Dataset:all extracted multi\-turn threads, including those with unanswered follow\-ups\.
- •Fully Answered Subset:a filtered subset in which every question has a corresponding physician response\.

To construct the fully answered subset, we only retain the threads where all questions are paired with physician responses and discard threads with fewer than three question\-answer pairs\. We then retain only threads with complete coverage across all turns\.

### A\.5Dataset Statistics

The Complete Dataset contains 9,741 threads and 37,191 question–answer pairs\. The Fully Answered Subset contains 2,437 threads and 8,204 pairs, with physician responses for all questions\.

Conversations exhibit varying depth, with most threads containing multiple follow\-up questions and a long tail of deeper interactions\. This structure enables analysis of model behavior across extended, realistic patient interactions\.

### A\.6Reproducibility

To facilitate reproducibility, we provide scripts for preprocessing, thread construction, and dataset filtering\. All preprocessing steps and filtering criteria described above can be reproduced using the provided code444[https://anonymous\.4open\.science/r/ThReadMed\-QA\-Misc\-6732/](https://anonymous.4open.science/r/ThReadMed-QA-Misc-6732/)\.

## Appendix BLLM\-as\-a\-Judge Validation

We use LLM\-as\-a\-Judge in two distinct roles: \(i\)*misconception identification*: binary detection of false presuppositions in a patient question and \(ii\)*scoring*: ordinal assessment of whether a model response identifies and corrects those presuppositions\. In both cases, we report agreement with expert labels and cohen’sκ\\kappa\.

### B\.1Misconception Identification

##### Gold standard and Sample:

We treat expert annotations \(collected from ECFMG\-certified M\.D\.\) as the reference standard on a stratified multi\-turn sample of patient questionsN=310N=310rows with expert labels in our annotations\. Experts labeled each target patient turn for the presence of clinically relevant false presuppositions \(YES/NO\)\.

##### Judge setup and agreement:

The identification judge receives the patient question \(and, when applicable, prior questions per our task definition\) and must return a single binaryLABEL∈\{YES,NO\}\\in\\\{\\text\{YES\},\\text\{NO\}\\\}\. Table[5](https://arxiv.org/html/2607.12884#A2.T5)summarizes exact agreement and Cohen’sκ\\kappafor GPT\-4o run with the rubric and prompt version used in the paper\. Following common benchmarks for Cohen’sκ\\kappa, the coefficient falls in the*substantial*range \(0\.610\.61–0\.800\.80\)\.

Table 5:LLM misconception identification vs\. expert labels \(N=310N=310\)\. Cohen’sκ\\kappais computed for binary YES/NO agreement\.

### B\.2Misconception Scoring and Debiasing

##### Gold Standard and Sample:

We curated a small held\-out validation set ofN=72N=72instances with expert judgment scores\. We use JSON as the output format, following previous work which showed better performance with structured outputs\(Munnangi et al\.,[2024](https://arxiv.org/html/2607.12884#bib.bib25); Dunn et al\.,[2022](https://arxiv.org/html/2607.12884#bib.bib11)\)\.

Table 6:LLM misconception judgment vs\. expert labels \(N=72N=72\)\. Cohen’sκ\\kappais computed for \{\-1, 0 1\} scoring\.
##### Judge Setup and Agreement:

Given a patient turn that contains false presuppositions and a candidate model answer, a sharpness judge assigns an integer score in\{−1,0,1\}\\\{\-1,0,1\\\}reflecting whether the answer fails to recognize the misconception \(−1\-1\), shows partial awareness or incomplete correction \(0\), or clearly identifies and corrects it \(11\), per the rubric in the appendix[E](https://arxiv.org/html/2607.12884#A5)\. We report the agreement in Table[6](https://arxiv.org/html/2607.12884#A2.T6)\.

##### De\-biasing LLM Judgments:

We use multiple LLMs as judges \(GPT\-4o, Claude\-Sonnet, and Llama3\.3\-70B\) and aggregate their scores to reduce bias\. GPT\-4o serves as the primary judge \(for tie\-breaking\) due to its strong agreement with expert judgment, supported by prior work\(Munnangi et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib26); Zhu et al\.,[2025](https://arxiv.org/html/2607.12884#bib.bib43)\)\. This reduces variance from any single model’s positional or stylistic biases and avoids optimistic pooling rules \(e\.g\., always taking the most frequent score\)\.

To provide greater transparency regarding judge behavior, we add per\-class precision, recall, F1 scores in Table[7](https://arxiv.org/html/2607.12884#A2.T7)\.

Table 7:LLM\-as\-a\-Judge Performance for Misconception Scoring\. Expert validation results comparing model\-assigned scores against physician ground truth\(N=72\)\(N=72\)\. Overall accuracy reaches87\.1%87\.1\\%, with a macro\-F1 score of0\.800\.80, demonstrating substantial agreement for evaluating misconception identification and correction in multi\-turn patient dialogues\.

## Appendix CAdditional Results

##### Turn\-Wise Performance with Confidence Intervals:

In this section, we provide the detailed turn\-wise performance metrics corresponding to Figure 5 in the main text\. As multi\-turn interactions progress, the number of eligible misconception\-bearing questions naturally decreases\. To make the evaluation sample size and statistical uncertainty explicit, the table below reports the exact number of question\-answer pairs evaluated at each turn \(n\), alongside the Presupposition Correction Rate \(PCR\) and its 95% percentile bootstrap confidence intervals across all models\. This ensures that performance estimates at later turns are properly contextualized by their underlying support\.

Table 8:Turn\-wise PCR \(%\) with 95% CIs\. “n” is the number of misconception\-bearing QA pairs\.

## Appendix DCost

### D\.1Misconception Identification

We report the cost of running GPT\-4o for misconception identification\. This was run on 8,204 questions\. We report this in Table[9](https://arxiv.org/html/2607.12884#A4.T9)\.

Table 9:LLM\-as\-a\-Judge cost for running GPT\-4o for misconception identification\.
### D\.2Inference

After identifying the questions with misconceptions \(604 question\-answer pairs across 505 conversation threads\), we run inference on them as mentioned in Section[4](https://arxiv.org/html/2607.12884#S4)\. We also run oracle inference for all questions except the ones in the first turn\. We used 5 LLMs to generate responses\. We report the cost in Table[10](https://arxiv.org/html/2607.12884#A4.T10)\.

Table 10:Inference cost for 5 models on 505 conversation threads\.
### D\.3LLM\-as\-Judge Evaluation

We report API costs for running LLM\-as\-a\-Judge evaluation for our 3 judge models for each of the 5 generations\. We summarize these numbers in Table[11](https://arxiv.org/html/2607.12884#A4.T11)\.

Table 11:LLM\-as\-a\-Judge cost for judging with 3 models on outputs from 5 models\.

## Appendix EPrompts

Evaluation Prompt## Misconception Identification #### System Prompt Your task is to review medical questions from users and identify if the question contains a false medical assumption or misconceptions based on incorrect medical information\. You are shown a question\. Choose one of the following: 1. 1\.Yesif the question contains a false medical assumption or misconception and write a short explanation of the misconception\. 2. 2\.Noif the question does not contain false medical assumptions or misconceptions or if you are unsure\. #### User Prompt Now evaluate the following question: Question: \[Question to Evaluate\] Label: ``` {"Score": <0, 1>, "Explanation": [str]} ```

Evaluation Prompt## LLM\-as\-a\-Judge Evaluation Prompt555Adopted fromZhu et al\. \([2025](https://arxiv.org/html/2607.12884#bib.bib43)\) #### System Prompt You are provided with a medical patient question containing some false presuppositions, and an answer to this patient question\. Please evaluate the answer with the following criteria: Does the answer identify and address false presuppositions in the questions? 1. 1\.Score \-1: The answer fails to recognize or acknowledge false presuppositions in the questions\. 2. 2\.Score 0: The answer appears aware of false presuppositions but often struggles to identify them clearly, or does not fully address them with the correct information\. 3. 3\.Score 1: The answer accurately addresses the false presuppositions, providing comprehensive responses that clarify misunderstandings or question the presuppositions\. #### User Prompt Now evaluate the following question: Question: \[Question to Evaluate\] Answer: \[Answer to Evaluate\] Score: ``` {"Score": <-1, 0, 1>, "Explanation": [str]} ```