Context-Aware Multimodal Claim Verification in Spoken Dialogues

# Context-Aware Multimodal Claim Verification in Spoken Dialogues
Source: [https://arxiv.org/html/2606.11420](https://arxiv.org/html/2606.11420)
Delvin Ce Zhang2Dongwon Lee1 1The Pennsylvania State University, USA 2University of Sheffield, UK czc5884@psu\.edu, delvin\.ce\.zhang@sheffield\.ac\.uk, dongwon@psu\.edu

###### Abstract

Every day, millions absorb claims from podcasts and streams that no fact\-checker ever sees\. Spoken misinformation is built through conversation, where credibility comes not from facts alone but from how claims are framed, reinforced, or left unchallenged across turns\. Yet fact\-checking has focused on isolated text, leaving dialogue audio under\-studied\. We introduce MAD2, a new Multi\-turn Audio Dialogues benchmark for spoken claim verification, containing 1,000 two\-speaker dialogues with 3,368 check\-worthy claims and approximately 10 hours of audio, and propose calibrated multimodal fusion of a context\-aware audio encoder and a dialogue\-aware text model\. Across settings, adding dialogue context improves verification, but the gains depend on scenario type\. Using only preceding context often matches offline performance, supporting live\-moderation settings, and audio contributes most when transcript\-based models are destabilized by additional context\. Overall, conversational structure matters more for verification than misinformation framing\.

Context\-Aware Multimodal Claim Verification in Spoken Dialogues

Chaewan Chun1and Delvin Ce Zhang2and Dongwon Lee11The Pennsylvania State University, USA2University of Sheffield, UKczc5884@psu\.edu, delvin\.ce\.zhang@sheffield\.ac\.uk, dongwon@psu\.edu

## 1Introduction

Millions hear factual claims in podcasts every day, yet most of those claims never enter the fact\-checking pipeline\. Audiences increasingly shift toward audio platforms for news and informationWhittle \([2024](https://arxiv.org/html/2606.11420#bib.bib39)\), yet most verification pipelines still assume written textZhouet al\.\([2019](https://arxiv.org/html/2606.11420#bib.bib34)\); Liuet al\.\([2020](https://arxiv.org/html/2606.11420#bib.bib24)\)\. In text, claims can be linked to sources and cross\-referenced via hyperlinksZhaoet al\.\([2020](https://arxiv.org/html/2606.11420#bib.bib36)\); Zhang and Lee \([2025](https://arxiv.org/html/2606.11420#bib.bib35)\); in podcasts and live streams, claims are embedded in multi\-turn conversation without citations or attributions, and relevant context may be distributed across turns\. Existing text benchmarksThorneet al\.\([2018](https://arxiv.org/html/2606.11420#bib.bib12)\); Wang \([2017](https://arxiv.org/html/2606.11420#bib.bib4)\); Guptaet al\.\([2022](https://arxiv.org/html/2606.11420#bib.bib5)\)are not designed to capture this, leaving spoken, conversational misinformation underexplored\.

What makes this problem particularly hard is that spoken misinformation is often*constructed*through interaction\. A statement that appears innocuous in isolation may become misleading through how it is framed, reinforced, or left unchallenged across turns\. The conversational context surrounding a claim—preceding exchanges, agreements, rebuttals, and elaborations—may carry signal for verification that is simply absent when claims are evaluated in isolation\. Prior work on dialogue fact\-checking has focused on adapting evidence retrieval pipelines to better exploit textual dialogue contextGuptaet al\.\([2022](https://arxiv.org/html/2606.11420#bib.bib5)\); Chamounet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib7)\), but has not examined whether the spoken signal itself carries veracity\-relevant information\. This leaves a basic empirical question open: does audio contribute anything to conversational claim verification beyond what a transcript provides, and under what context conditions does conversational context help?

Script:Exactly\. It’s a real problem for the county budget\. I mean, at some point, you have to ask if it’s really …

ASR:real problem for the county budget i mean at some point you have to ask if it s really …

0 s3838s21\.9621\.96s–29\.7329\.73s

Figure 1:Example of precise claim–audio alignment inMAD2, where a check\-worthy claim is linked to its spoken segment using WhisperXBainet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib3)\)word\-level timestamps\.Answering it has been held back by two gaps\. First, no benchmark has coupled realistic conversational audio with fine\-grained, claim\-level veracity annotations at scale\. Early efforts on political debatesKopevet al\.\([2019](https://arxiv.org/html/2606.11420#bib.bib9)\)and audio check\-worthiness detectionIvanovet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib15)\)demonstrated that acoustic features can complement text for factuality prediction, but operated on monologue\-style debate speech rather than natural conversational audio\. Existing systems for live audio fact\-checkingVenktesh and Setty \([2025](https://arxiv.org/html/2606.11420#bib.bib6)\)operate on broadcast\-style streams and route all verification through Automatic Speech Recognition \(ASR\)\-derived transcripts fed into text\-based Natural Language Inference \(NLI\) pipelines, with no use of acoustic features and no modeling of conversational context\. On the multimodal side, text–image verification has a mature benchmark ecosystemYaoet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib25)\); Huet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib26)\), yet no equivalent line of work exists for spoken claims in conversational settings\. Second, there has been no systematic study of how dialogue context affects verification across modalities — whether context helps audio and text equally, whether the two modalities capture different aspects of the conversational signal, and how much each modality benefits from seeing more of the dialogue\. Without this, we cannot know whether audio and text are complementary, redundant, or modality\-specific in what they extract from conversational context\.

To address both gaps, we introduceMAD2, a synthetic Multi\-turn Audio Dialogues benchmark of 1,000 two\-speaker dialogues \(8,192 sentences; 3,368 check\-worthy claims\) with approximately 10 hours of audio\. Extending MADChunet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib1)\), MAD2 adds two capabilities that the original benchmark lacked: higher\-fidelity audio synthesis via MoonCastJuet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib2)\)and precise claim\-to\-audio alignment via WhisperXBainet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib3)\)word\-level timestamps \(Figure[1](https://arxiv.org/html/2606.11420#S1.F1)\)\. We also propose a calibrated conditional fusion of context\-aware audio and text encoders\.

Using MAD2, we systematically evaluate how conversational context affects verification across text\-only, audio\-only, and fusion models under matched sentence\-based context windows\. We focus on three questions: \(i\) how verification changes as context expands, \(ii\) whether past\-only context can match full\-dialogue context, and \(iii\) when audio provides complementary signal beyond ASR transcripts\. Our contributions are: \(1\)MAD2, a spoken\-dialogue claim verification benchmark with paired audio, ASR transcripts, and claim\-to\-audio alignments, \(2\) a calibrated conditional fusion method together with an analysis showing that audio is not a uniform additive boost but a*selective*corrective signal, helping precisely where transcript\-only verification is destabilized by noisy context, and \(3\) a controlled evaluation protocol that systematically varies dialogue context under matched sentence\-based windows across text\-only, audio\-only, and fusion models\. Source code and the MAD2 benchmark will be released upon publication\.

## 2Related Work

Most fact\-checking research operates over written claims, supported by benchmarks such as FEVERThorneet al\.\([2018](https://arxiv.org/html/2606.11420#bib.bib12)\), LIARWang \([2017](https://arxiv.org/html/2606.11420#bib.bib4)\), and DialFactGuptaet al\.\([2022](https://arxiv.org/html/2606.11420#bib.bib5)\), with dialogue work adapting evidence\-retrieval pipelines to exploit textual contextGuptaet al\.\([2022](https://arxiv.org/html/2606.11420#bib.bib5)\); Chamounet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib7)\)\. We inherit veracity labels from one such resource \(LIAR\) but depart from this line in modality: our task verifies claims from conversational*audio*, where acoustic cues, disfluencies, timing, and prosody are available yet absent from transcript\-only pipelines\.

Work on spoken misinformation is comparatively limited\. Studies on political debates and audio check\-worthiness show that speech can carry useful signalKopevet al\.\([2019](https://arxiv.org/html/2606.11420#bib.bib9)\); Ivanovet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib15)\), and recent systems verify spoken claims in streaming settingsVenktesh and Setty \([2025](https://arxiv.org/html/2606.11420#bib.bib6)\)\. Unlike audio check\-worthiness work, which decides whether a segment*deserves*verification, our task predicts claim veracity in two\-speaker dialogue and systematically tests how surrounding conversational context affects it; and unlike live systems that route verification entirely through ASR text, we model the acoustic signal directly\. Large podcast corpora offer rich audio–text dataCliftonet al\.\([2020](https://arxiv.org/html/2606.11420#bib.bib13)\); Littereret al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib14)\)but lack claim\-level check\-worthiness labels, veracity annotations, and aligned claim spans, making them unsuitable for this evaluation\. Finally, multimodal fact\-checking has matured in text–image settingsYaoet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib25)\); Huet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib26)\), but spoken dialogue remains underexplored\. MAD2 fills this gap by pairing conversational audio with claim\-level veracity annotations and aligned transcript spans, enabling controlled evaluation of audio, text, and dialogue context in spoken claim verification\.

Table 1:Key extensions from MAD to MAD2\.
## 3Dataset Creation: MAD2

MAD2 is a synthetic English benchmark of two\-speaker dialogues with fine\-grained claim\-level veracity annotations and high\-fidelity conversational audio\. We adopt a synthetic construction pipeline because obtaining expert veracity labels aligned to long\-form podcast audio at scale is prohibitively expensive, and a controlled pipeline lets us vary key misinformation factors such as how claims enter conversations and how speakers respond\. We build on the construction pipeline of our prior work, MADChunet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib1)\), and substantially extend it, as summarized in Table[1](https://arxiv.org/html/2606.11420#S2.T1)\.

Each dialogue is generated by conditioning Gemini 2\.5 ProTeam \([2025](https://arxiv.org/html/2606.11420#bib.bib32)\)on a human fact\-checked political claim from the LIARWang \([2017](https://arxiv.org/html/2606.11420#bib.bib4)\)benchmark, inheriting its true/false label directly\. We sample 1,000 claims \(500 true, 500 false\) and generate one two\-speaker script per claim, conditioned on fictional speaker profiles, a spread style, and a dialogue scenario\.Spread stylescapture how a claim enters conversation \(*Consequential Storytelling*,*Casual Rumor\-Mongering*,*Ironic Dismissal*\) anddialogue scenariosdefine the interactional dynamic \(*Collaborative Agreement*,*Collaborative Skepticism*,*The Classic Debate*,*The Persuader and the Questioner*,*The Unresolved Argument*\)\. Each sentence is annotated with a binary check\-worthiness label\. Automated quality control with ChatGPTOpenAI \([2024](https://arxiv.org/html/2606.11420#bib.bib33)\)filters each dialogue for coherence and consistency with the seed claim\. MAD2 comprises 1,000 dialogues \(8,192 sentences; 3,368 check\-worthy claims: 1,516 true / 1,852 false\) with about 10 hours of audio \(avg\. 36s/dialogue\)\.

Audio generation\.MAD used XTTS\-v2111[https://huggingface\.co/coqui/XTTS\-v2](https://huggingface.co/coqui/XTTS-v2), producing flat, monotone synthetic speech\. MAD2 replaces it with MoonCastJuet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib2)\), a two\-speaker podcast synthesis model that produces natural prosody, richer conversational flow, and realistic disfluencies\.Juet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib2)\)report that MoonCast outperforms concatenation\-based Text\-to\-Speech \(TTS\) by\+0\.68\+0\.68in spontaneity and\+0\.62\+0\.62in coherence, with a word error rate of 1\.81%Juet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib2)\), making our audio substantially closer to real\-world podcast conditions\.

Transcript alignment\.MAD2 provides WhisperXBainet al\.\([2023](https://arxiv.org/html/2606.11420#bib.bib3)\)transcriptions with word\-level timestamps\. Since ASR output does not preserve sentence\-level labels, we align each transcript back to the generated script to recover check\-worthy annotations and timestamped claim spans; low\-confidence alignments are flagged for manual review\. These ASR transcripts are used as text input in all experiments, matching realistic conditions where transcripts are automatically and imperfectly obtained\. To quantify transcription noise, we compare WhisperX transcripts against the generated scripts and observe non\-zero WERs at the sentence, dialogue, and corpus levels: 11\.3%, 11\.6%, and 11\.9%, respectively\. Thus, the text\-only and fusion models operate on imperfect ASR\-derived transcripts rather than gold scripts\.

Claim\-level spoken instances\.While the script\-generation protocol follows MADChunet al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib1)\), MAD2 turns each check\-worthy script sentence into a timestamped spoken claim instance\. Each aligned example links a check\-worthy sentence to its speaker, turn position, ASR transcript, veracity label, scenario type, spread style, and start–end time span in the waveform\. This representation is what enables matched multimodal evaluation: the text model receives the ASR sentence window with the target claim marked, while the audio model receives the waveform crop spanning the same sentence window and retains the claim’s relative timestamp for claim\-aware pooling\. Thus, differences between text\-only, audio\-only, and fusion models reflect modality and context use rather than mismatched input scopes\.

## 4Methodology

Given a dialogue and a candidate claim sentence aligned to its position and timestamp span in the audio, the task is to predict a binary veracity labely∈\{0,1\}y\\in\\\{0,1\\\}\(11= True,0= False\)\. We develop three model variants \(audio\-only, text\-only, and fusion\), all trained with cross\-entropy loss; the best checkpoint is determined by validation AUC, and the decision threshold is tuned on validation data by F1 scoreZeng and Gao \([2023](https://arxiv.org/html/2606.11420#bib.bib40)\)and applied to test data without further adjustment\.

### 4\.1Context Configurations

We vary the scope and direction of dialogue context using sentence\-window neighborhoods around each claim\. Dialogues are segmented into ordered sentences by turn and position\. For a claim at positionii, we select a window of up toNNneighboring sentences on each side, preserving speaker turn structure\. Here,NNindexes neighboring sentences rather than speaker turns, so even small windows such asN=2N=2can include adjacent cross\-speaker context when claims occur near turn boundaries\. All three model variants operate over*identical*sentence windows, enabling a fair cross\-modal comparison\.

\(1\) Claim only\(N=0N=0\): the claim sentence alone, mimicking traditional fact\-checking setupsLiuet al\.\([2020](https://arxiv.org/html/2606.11420#bib.bib24)\)where claims are verified in isolation\.

\(2\) Real\-time\(−N\-N\): precedingNNsentences only, simulating live or streaming moderation where future turns are unavailable\. We evaluateN∈\{1,2\}N\\in\\\{1,2\\\}\.

\(3\) Offline\(±N\\pm N\):NNsentences on each side of the claim, reflecting post\-hoc analysis of recorded audio\. We evaluateN∈\{1,2\}N\\in\\\{1,2\\\}, with natural truncation at dialogue boundaries\.

\(4\) Full dialogue: the entire relevant dialogue to the claim, serving as an upper\-bound oracle under offline conditions\.

### 4\.2Audio\-only Claim Verifier

WavLM\-base\+ \(94\.68M parameters\)Chenet al\.\([2022](https://arxiv.org/html/2606.11420#bib.bib18)\)is used as the speech encoder, supported by prior benchmarkingYanget al\.\([2024](https://arxiv.org/html/2606.11420#bib.bib22)\); Penget al\.\([2025](https://arxiv.org/html/2606.11420#bib.bib23)\)and preliminary runs showing HuBERTHsuet al\.\([2021](https://arxiv.org/html/2606.11420#bib.bib19)\), wav2vec2Baevskiet al\.\([2020](https://arxiv.org/html/2606.11420#bib.bib20)\), and Data2VecBaevskiet al\.\([2022](https://arxiv.org/html/2606.11420#bib.bib21)\)yield weaker results\. We extract a single contiguous audio crop spanning the sentence window using WhisperX word\-level timestamps, with a0\.150\.15s padding margin on each side to avoid hard boundary effects, resampled to 16 kHz and loaded on\-the\-fly during training\. The context crop and the claim span serve different roles\. The waveform crop gives WavLM access to surrounding speech, pauses, speaker transitions, and local conversational context, while the claim timestamps identify which portion of that crop should drive the final decision\. Thus, the model does not discard context before encoding, but it also does not ask the classifier to summarize the entire dialogue window uniformly\. This mirrors the text model’s claim boundary tokens: both modalities receive the same surrounding sentences, but both are given an explicit pointer to the target claim\.

Rather than pooling uniformly over all frames, we apply*claim\-aware attention pooling*to focus the representation on the claim region while still letting the full context inform the encoder\. We compute a binary mask over WavLM frame representations by mapping the claim’s sample\-level timestamps to frame indices via WavLM’s convolutional downsampling ratio, then apply a learned attention pooler: a two\-layer MLP with tanh activationBishop and Nasrabadi \([2006](https://arxiv.org/html/2606.11420#bib.bib37)\)that produces a scalar score per frame, followed by softmaxBishop and Nasrabadi \([2006](https://arxiv.org/html/2606.11420#bib.bib37)\)over the masked frames to yield attention weights\. The resulting pooled vector passes through LayerNorm and a two\-layer classification head \(Linear–ReLU–Dropout\(0\.1\)–Linear\)Vaswaniet al\.\([2017](https://arxiv.org/html/2606.11420#bib.bib38)\)to produce binary logits\.

![Refer to caption](https://arxiv.org/html/2606.11420v1/x1.png)Figure 2:Architecture of the proposed multimodal claim verification system\.
### 4\.3Text\-only Claim Verifier

The text\-only model encodes the sentence window as a single serialized sequence using RoBERTa\-base \(125M parameters\)Liuet al\.\([2019](https://arxiv.org/html/2606.11420#bib.bib29)\)\. To mark claim identity within the context, we wrap the claim sentence with special boundary tokens\[CLAIM\_START\]and\[CLAIM\_END\], and prepend speaker tokens \(\[SPK\_A\],\[SPK\_B\]\) and turn separators \(\[TURN\_SEP\]\) to preserve dialogue structure\. All special tokens are added to the tokenizer vocabulary before fine\-tuning\. We obtain a sentence\-window representation via masked mean pooling over the final\-layer hidden states, then pass it through LayerNorm and the same two\-layer MLP head as the audio model to produce binary logits\.

Table 2:Claim verification results across context configurations and model variants, reported as mean±\\pmstandard deviation over three seeds\.
### 4\.4Calibrated Conditional Fusion

Rather than jointly training a fusion model from scratch, we use a post\-hoc fusion approach, motivated by a complementarity analysis of the independently trained unimodal models\. Preliminary joint\-fusion variants, including a two\-stage freeze\-then\-joint setup, did not outperform post\-hoc calibrated fusion on validation data, likely because MAD2 remains limited in supervised scale for jointly fine\-tuning two large encoders\. Across context\-aware settings, the two modalities disagree on 30–39% of instances \(where one model is correct and the other is not\), and audio correctly resolves 29–49% of those disagreements, indicating genuine complementary signal\. This benefit is most pronounced in the symmetric offline \(±\\pm1\) and full\-dialogue settings, where audio has sufficient acoustic context to offer independent signal\. This directly motivates a fusion strategy that defers to audio selectively rather than uniformly, and a calibration step to bring both modalities onto a common probability scale before combining them\. See Figure[2](https://arxiv.org/html/2606.11420#S4.F2)\.

Building on this observation, we treat audio as a selective corrective signal rather than as an always\-useful second view\. In preliminary experiments, uniformly weighting audio and text often diluted the stronger text signal, especially in settings where ASR context was already sufficient\. Conditional fusion instead asks whether the audio model should alter the text prediction only under specific confidence patterns, such as low text confidence or high audio confidence\. Accordingly, the fusion search is framed as validation\-selected decision calibration rather than as evidence that audio should receive a fixed global weight\.

Letpt∈\[0,1\]p\_\{t\}\\in\[0,1\]denote the raw softmax probability of the True class output by the text model, andpap\_\{a\}the corresponding probability from the audio model\. Raw softmax probabilities are often poorly calibratedGuoet al\.\([2017](https://arxiv.org/html/2606.11420#bib.bib30)\): text models tend toward overconfidence, audio toward underconfidence\. We therefore apply Platt scalingPlatt \([2000](https://arxiv.org/html/2606.11420#bib.bib31)\)as part of the grid search, fitting a single sigmoid to the model’s output log\-odds on held\-out validation predictions to map them to a common probability scale; no calibration \(identity\) is included as a baseline option\. Letp^t\\hat\{p\}\_\{t\}andp^a\\hat\{p\}\_\{a\}denote the probabilities after the selected calibration is applied\.

Givenp^t\\hat\{p\}\_\{t\}andp^a\\hat\{p\}\_\{a\}, we search over four fusion strategies: \(1\)*late fusion*:pf=α⋅p^t\+\(1−α\)⋅p^ap\_\{f\}=\\alpha\\cdot\\hat\{p\}\_\{t\}\+\(1\-\\alpha\)\\cdot\\hat\{p\}\_\{a\}, a fixed\-weight average controlled by a grid\-searched scalarα\\alpha; \(2\)*recall boost*:pf=p^t\+β⋅max⁡\(0,p^a−p^t\)p\_\{f\}=\\hat\{p\}\_\{t\}\+\\beta\\cdot\\max\(0,\\hat\{p\}\_\{a\}\-\\hat\{p\}\_\{t\}\), which selectively raises text probability when audio is more confident; \(3\)*audio override*: replacep^t\\hat\{p\}\_\{t\}with a blend when text confidence falls belowτt\\tau\_\{t\}and audio confidence exceedsτa\\tau\_\{a\}, deferring to audio under high text uncertainty; and \(4\)*conditional alpha*:pf=α​\(p^a\)⋅p^t\+\(1−α​\(p^a\)\)⋅p^ap\_\{f\}=\\alpha\(\\hat\{p\}\_\{a\}\)\\cdot\\hat\{p\}\_\{t\}\+\(1\-\\alpha\(\\hat\{p\}\_\{a\}\)\)\\cdot\\hat\{p\}\_\{a\}, whereα\\alphatakes a lower value whenp^a\\hat\{p\}\_\{a\}exceeds an audio confidence threshold, increasing audio’s weight dynamically\. The best\-performing strategy, calibration method, and hyperparameters are jointly selected by validation AUC\.

To prevent leakage, calibration and strategy selection follow a leave\-one\-seed\-out \(LOSO\) protocol: for each held\-out seed, calibrators are fitted on the other seeds’ validation predictions, and the decision threshold is tuned on those same calibrated outputs before evaluating on the held\-out test set\.

## 5Experiments

### 5\.1Implementation Details

All models are fine\-tuned with AdamWLoshchilov and Hutter \([2019](https://arxiv.org/html/2606.11420#bib.bib27)\)\. We search over eight hyperparameter combinations: learning rate∈\{2×10−5,3×10−5\}\\in\\\{2\{\\times\}10^\{\-5\},3\{\\times\}10^\{\-5\}\\\}, warmup ratio∈\{0\.05,0\.1\}\\in\\\{0\.05,0\.1\\\}, and weight decay∈\{0\.01,0\.03\}\\in\\\{0\.01,0\.03\\\}, with dropout fixed at 0\.1 throughout\. For each run, the best checkpoint is determined by validation AUC; the decision threshold is then tuned on the same validation predictions by F1 score\. The hyperparameter configuration with the highest validation AUC is selected for final evaluation\. The text model uses batch size 32 with bf16 mixed precision, while the audio model uses batch size 8 with gradient checkpointing to handle longer waveform crops\. Both are trained for up to 30 epochs with early stopping \(patience 5\) on NVIDIA H100 GPUs\. A full text model sweep \(three configurations in parallel\) takes approximately 1\.5h wall\-clock time, and audio model training takes approximately 5\.7h per configuration\.

Dialogue\-level splits ensure no dialogue appears in more than one partition\. Splits are stratified by a composite label of scenario type and spread style, so that each combination is proportionally represented across train, validation, and test; any combination with fewer than three dialogues is assigned entirely to train\. Across seeds \(42/43/44\), train/val/test contain 588–591/127–128/126–130 dialogues \(788–797/162–180/163–173 claims\), and results are averaged over three seeds with mean and standard deviation reported\. The same quality filters \(alignment match score≥0\.8\\geq 0\.8, ASR confidence≥0\.8\\geq 0\.8\) and sentence\-window definitions apply to all model variants\. Statistical significance is assessed via bootstrap confidence intervals on AUC differences \(n=10,000n=10\{,\}000resamples\), and a difference is considered significant when the 95% CI excludes zero\. Bootstrap tests pool instances across seeds, whereas Table[2](https://arxiv.org/html/2606.11420#S4.T2)reports mean±\\pmstd AUC across seeds\.

### 5\.2Results and Analysis

We treat AUC as the primary reported metric, as it is threshold\-independent, and report F1 for completeness\. Table[2](https://arxiv.org/html/2606.11420#S4.T2)reports aggregate results across all context configurations and modalities, while Table[3](https://arxiv.org/html/2606.11420#S5.T3)breaks these down by scenario type and spread style to understand where context and modality matter most\. Four findings emerge\.

\(1\) Context helps all modalities, and audio benefits most from it\.AUC improves with context across all modalities: audio\-only from0\.6690\.669\(claim only\) to0\.7800\.780\(full dialogue\); text\-only from0\.7470\.747to0\.8410\.841; fusion from0\.7570\.757to0\.8520\.852\. These gains are statistically significant for both audio\-only \(Δ\\DeltaAUC=\+0\.108=\+0\.108, 95% CI\[\+0\.057,\+0\.158\]\[\+0\.057,\\,\+0\.158\],p<0\.001p<0\.001\) and text\-only \(Δ\\DeltaAUC=\+0\.069=\+0\.069, 95% CI\[\+0\.021,\+0\.118\]\[\+0\.021,\\,\+0\.118\],p<0\.01p<0\.01\), with predictions pooled across all three seeds \(M=508M=508claim instances, whereMMdenotes the total test set size summed across seeds\)\. The clearest pattern is not that any one modality gains more in absolute terms—the three gains are comparable—but that audio is the modality most reliant on context: it is substantially weaker than text without context \(AUC=0\.669=0\.669\), and the text–audio gap narrows at full dialogue to its smallest value \(0\.0610\.061, compared to0\.0780\.078at claim only\)\. This suggests that speech\-derived cues are most useful when interpreted relative to surrounding context; in isolation, audio provides limited veracity signal\.

Context effects are broadly positive across scenario types in Table[3](https://arxiv.org/html/2606.11420#S5.T3)\. Moving from claim\-only \(C\) to full dialogue \(F\) increases AUC for*audio*and*fusion*in all five scenarios, and for*text*in four of five\. Audio and fusion benefit uniformly from additional context across dialogue structures, whereas text has one clear exception\. The largest text\-only gain occurs in Unresolved Argument, where AUC rises from0\.6940\.694to0\.8970\.897\(Δ=\+0\.203\\Delta=\+0\.203\)\. These dialogues end without resolution, so the conversational arc—who introduced the claim, who challenged it, and whether any consensus emerged—carries veracity signal that the claim lacks in isolation\. The main exception is Collaborative Skepticism: while audio and fusion improve sharply with full dialogue \(0\.676→0\.8520\.676\\rightarrow 0\.852and0\.826→0\.9100\.826\\rightarrow 0\.910\), text\-only shows a small*decline*\(0\.826→0\.8080\.826\\rightarrow 0\.808\)\. While modest in magnitude, this is the only scenario in which added context does not help text, and one plausible explanation is that when both speakers already question the claim, the claim is often self\-evident in isolation, so additional turns may introduce distracting or redundant text context that the text model struggles to filter\.

\(2\) Real\-time performance is competitive with offline\.Models using only preceding sentences perform comparably to their offline counterparts\. Audio\-only at Real\-time \(−2\-2\) achieves AUC=0\.726=0\.726vs\.0\.7130\.713for Offline \(±2\\pm 2\); text\-only achieves0\.7900\.790vs\.0\.8010\.801; fusion achieves0\.7950\.795vs\.0\.8120\.812\. Bootstrap tests find no significant AUC difference between Real\-time \(−2\-2\) and Offline \(±2\\pm 2\) in text\-only or audio\-only \(0/30/3seeds in both cases; audio\-only: medianp=0\.696p=0\.696; text\-only: medianp=0\.743p=0\.743\)\. For audio, the real\-time setting slightly*outperforms*offline at both window sizes \(−1\-1:0\.7040\.704vs\.±1\\pm 1:0\.6970\.697;−2\-2:0\.7260\.726vs\.±2\\pm 2:0\.7130\.713\), hinting that future acoustic context may sometimes introduce as much noise as signal for the audio encoder\. These results suggest that the preceding discourse has already established the context needed to assess a claim by the time it is made, making real\-time moderation nearly equivalent to offline analysis without architectural concessions\.

\(3\) Fusion helps selectively, not uniformly: audio appears most useful where text is sensitive to noisy context\.Across all settings in Table[2](https://arxiv.org/html/2606.11420#S4.T2), fusion’s AUC advantage over text\-only is consistent in sign \(\+0\.005\+0\.005to\+0\.016\+0\.016\) but not statistically significant \(full\-dialogue: medianΔ\\DeltaAUC=\+0\.011=\+0\.011, 95% CI\[−0\.012,\+0\.030\]\[\-0\.012,\\,\+0\.030\],0/30/3seeds; claim\-only: medianΔ\\DeltaAUC=\+0\.000=\+0\.000,1/31/3seeds significant\)\. This indicates that audio does not provide a uniform additive boost over text\.

The strata where fusion helps most are those where text is weakest\. In Collaborative Skepticism, where text\-only declines with full dialogue context \(0\.826→0\.8080\.826\\rightarrow 0\.808\), fusion reaches0\.9100\.910—the largest fusion\-over\-text advantage \(\+0\.102\+0\.102\) in our stratified results\. Among spread styles, a parallel pattern appears: Casual Rumor\-Mongering, where text has its lowest full\-dialogue AUC \(0\.7910\.791\), shows the largest fusion lift \(\+0\.044\+0\.044over text, reaching0\.8350\.835\)\. These are a small number of strata, so we read the pattern as associational rather than conclusive, but it is consistent across both the scenario\-type and spread\-style breakdowns: where text struggles with noisy or unstructured conversational context, audio appears to supply an independent signal\.

The converse also holds: when text is already strong, fusion adds little\. In Unresolved Argument, text dominates at full dialogue \(0\.8970\.897\) after the largest context gain in the table \(\+0\.203\+0\.203\); fusion is nearly identical \(0\.8980\.898\)\. Beyond AUC, fusion also shows a modest F1 advantage at full dialogue \(0\.7640\.764vs\.0\.7400\.740\), consistent with audio contributing more to stable threshold behavior than to ranking when both modalities are available and context is rich\.

\(4\) Conversational structure matters more than framing strategy\.Full\-dialogue AUC varies far more across scenario types than across spread styles in Table[3](https://arxiv.org/html/2606.11420#S5.T3)\. The scenario\-type range spans0\.1280\.128for audio \(0\.7240\.724–0\.8520\.852\),0\.1300\.130for text \(0\.7670\.767–0\.8970\.897\), and0\.1470\.147for fusion \(0\.7630\.763–0\.9100\.910\)\. For example, under full dialogue, text\-only reaches0\.8970\.897in Unresolved Argument but only0\.7670\.767in Classic Debate\. The corresponding spread\-style ranges are22–6×6\{\\times\}narrower:0\.0280\.028,0\.0630\.063, and0\.0230\.023, respectively\. This gap widens at full dialogue relative to claim only, where the ratio is1\.31\.3–1\.6×1\.6\{\\times\}, suggesting that conversational dynamics amplify scenario differences as more of the dialogue becomes visible\.

One interpretation is that context helps less by simply adding more words and more by exposing the interactional roles around the claim\. In Unresolved Argument, the absence of consensus leaves the model to infer veracity from how the claim is challenged, defended, or left standing across turns\. In Collaborative Skepticism, by contrast, the local claim may already contain enough lexical signal, and additional skeptical turns can introduce semantically similar but non\-decisive context\. These differences help explain why scenario type produces larger variation than spread style: the model is sensitive to the conversational trajectory surrounding the claim, not only to the rhetorical frame used to introduce it\.

Table 3:Stratified AUC under claim\-only \(C\) and full\-dialogue \(F\) settings\.

## 6Conclusion

We presented MAD2 and a calibrated multimodal fusion framework for spoken claim verification\. Conversational context improves verification across modalities, but its benefit is strongly scenario\-dependent; in many cases, past\-only context approaches offline performance, supporting real\-time moderation\. Audio provides a selective rather than uniform benefit, contributing mainly when transcript\-only models struggle under noisy context, and our analyses suggest that conversational structure is more predictive for verification than framing alone\.

## Limitations

MAD2 is synthetic and restricted to two\-speaker English dialogues, so even with high\-fidelity TTS it cannot fully capture the diversity, noise, speaker overlap, and interaction patterns of real podcast audio\. In addition, while audio improves performance in targeted cases, this paper does not isolate which acoustic cues drive those gains\. Future work should extend the benchmark to longer, natural, and multilingual conversations, and should include deeper audio\-centered analyses\.

## Ethical Considerations

MAD2 is intended as a research benchmark for studying spoken claim verification, not as a deployable moderation system or an automated truth arbiter\. Because the dataset contains synthetic dialogues derived from fact\-checked political claims, release should include clear documentation of data provenance, generation procedures, and intended use\. Models trained on synthetic audio may not generalize reliably to real speakers, accents, dialects, or noisy recording environments\. Any real\-world deployment would require human oversight, careful calibration, and evaluation on natural conversational audio\.

## References

- A\. Baevski, W\. Hsu, Q\. Xu, A\. Babu, J\. Gu, and M\. Auli \(2022\)Data2vec: a general framework for self\-supervised learning in speech, vision and language\.InInternational conference on machine learning,pp\. 1298–1312\.Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p1.1)\.
- A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli \(2020\)Wav2vec 2\.0: a framework for self\-supervised learning of speech representations\.Advances in neural information processing systems33,pp\. 12449–12460\.Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p1.1)\.
- M\. Bain, J\. Huh, T\. Han, and A\. Zisserman \(2023\)WhisperX: Time\-Accurate Speech Transcription of Long\-Form Audio\.InInterspeech 2023,pp\. 4489–4493\.External Links:[Document](https://dx.doi.org/10.21437/Interspeech.2023-78),ISSN 2958\-1796Cited by:[Figure 1](https://arxiv.org/html/2606.11420#S1.F1),[§1](https://arxiv.org/html/2606.11420#S1.p4.1),[§3](https://arxiv.org/html/2606.11420#S3.p4.1)\.
- C\. M\. Bishop and N\. M\. Nasrabadi \(2006\)Pattern recognition and machine learning\.Vol\.4,Springer\.Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p2.1)\.
- E\. Chamoun, M\. Saeidi, and A\. Vlachos \(2023\)Automated fact\-checking in dialogue: are specialized models needed?\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 16009–16020\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p2.1),[§2](https://arxiv.org/html/2606.11420#S2.p1.1)\.
- S\. Chen, C\. Wang, Z\. Chen, Y\. Wu, S\. Liu, Z\. Chen, J\. Li, N\. Kanda, T\. Yoshioka, X\. Xiao, J\. Wu, L\. Zhou, S\. Ren, Y\. Qian, Y\. Qian, J\. Wu, M\. Zeng, X\. Yu, and F\. Wei \(2022\)WavLM: large\-scale self\-supervised pre\-training for full stack speech processing\.IEEE Journal of Selected Topics in Signal Processing16\(6\),pp\. 1505–1518\.External Links:[Document](https://dx.doi.org/10.1109/JSTSP.2022.3188113)Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p1.1)\.
- C\. Chun, L\. Terrisse, D\. C\. Zhang, and D\. Lee \(2025\)MAD: a benchmark for multi\-turn audio dialogue fact\-checking\.arXiv preprint arXiv:2508\.12186\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p4.1),[Table 1](https://arxiv.org/html/2606.11420#S2.T1.1.1.1.2.1.1.1),[§3](https://arxiv.org/html/2606.11420#S3.p1.1),[§3](https://arxiv.org/html/2606.11420#S3.p5.1)\.
- A\. Clifton, A\. Pappu, S\. Reddy, Y\. Yu, J\. Karlgren, B\. Carterette, and R\. Jones \(2020\)The spotify podcast dataset\.arXiv preprint arXiv:2004\.04270\.Cited by:[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- C\. Guo, G\. Pleiss, Y\. Sun, and K\. Q\. Weinberger \(2017\)On calibration of modern neural networks\.InProceedings of the 34th International Conference on Machine Learning \- Volume 70,ICML’17,pp\. 1321–1330\.Cited by:[§4\.4](https://arxiv.org/html/2606.11420#S4.SS4.p3.4)\.
- P\. Gupta, C\. Wu, W\. Liu, and C\. Xiong \(2022\)DialFact: a benchmark for fact\-checking in dialogue\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 3785–3801\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1),[§1](https://arxiv.org/html/2606.11420#S1.p2.1),[§2](https://arxiv.org/html/2606.11420#S2.p1.1)\.
- W\. Hsu, B\. Bolte, Y\. H\. Tsai, K\. Lakhotia, R\. Salakhutdinov, and A\. Mohamed \(2021\)Hubert: self\-supervised speech representation learning by masked prediction of hidden units\.IEEE/ACM transactions on audio, speech, and language processing29,pp\. 3451–3460\.Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p1.1)\.
- X\. Hu, Z\. Guo, J\. Chen, L\. Wen, and P\. S\. Yu \(2023\)Mr2: a benchmark for multimodal retrieval\-augmented rumor detection in social media\.InProceedings of the 46th international ACM SIGIR conference on research and development in information retrieval,pp\. 2901–2912\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p3.1),[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- P\. Ivanov, I\. Koychev, M\. Hardalov, and P\. Nakov \(2023\)Detecting check\-worthy claims in political debates, speeches, and interviews using audio data\.ICASSP 2024 \- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 12011–12015\.External Links:[Link](https://api.semanticscholar.org/CorpusID:259129320)Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p3.1),[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- Z\. Ju, D\. Yang, S\. Kai, Y\. Leng, Z\. Wang, S\. Liu, X\. Zhou, T\. Qin, X\. Li, J\. Yu, and X\. Tan \(2025\)MoonCast: high\-quality zero\-shot podcast generation\.InAdvances in Neural Information Processing Systems,D\. Belgrave, C\. Zhang, H\. Lin, R\. Pascanu, P\. Koniusz, M\. Ghassemi, and N\. Chen \(Eds\.\),Vol\.38,pp\. 6283–6317\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2025/file/095a6917768712b7ccc61acbeecad1d8-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p4.1),[§3](https://arxiv.org/html/2606.11420#S3.p3.2)\.
- D\. Kopev, A\. Ali, I\. Koychev, and P\. Nakov \(2019\)Detecting deception in political debates using acoustic and textual features\.In2019 IEEE Automatic Speech Recognition and Understanding Workshop \(ASRU\),pp\. 652–659\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p3.1),[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- B\. Litterer, D\. Jurgens, and D\. Card \(2025\)Mapping the podcast ecosystem with the structured podcast research corpus\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Cited by:[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: a robustly optimized bert pretraining approach\.External Links:1907\.11692,[Link](https://arxiv.org/abs/1907.11692)Cited by:[§4\.3](https://arxiv.org/html/2606.11420#S4.SS3.p1.1)\.
- Z\. Liu, C\. Xiong, M\. Sun, and Z\. Liu \(2020\)Fine\-grained fact verification with kernel graph attention network\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,pp\. 7342–7351\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.11420#S4.SS1.p2.1)\.
- I\. Loshchilov and F\. Hutter \(2019\)Decoupled weight decay regularization\.InInternational Conference on Learning Representations,Cited by:[§5\.1](https://arxiv.org/html/2606.11420#S5.SS1.p1.3)\.
- OpenAI \(2024\)GPT\-4 technical report\.External Links:2303\.08774,[Link](https://arxiv.org/abs/2303.08774)Cited by:[§3](https://arxiv.org/html/2606.11420#S3.p2.1)\.
- J\. Peng, T\. Ashihara, M\. Delcroix, T\. Ochiai, O\. Plchot, S\. Araki, and J\. Černockỳ \(2025\)TS\-superb: a target speech processing benchmark for speech self\-supervised learning models\.InICASSP 2025\-2025 IEEE International Conference on Acoustics, Speech and Signal Processing \(ICASSP\),pp\. 1–5\.Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p1.1)\.
- J\. Platt \(2000\)Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods\.Adv\. Large Margin Classif\.10,pp\.\.Cited by:[§4\.4](https://arxiv.org/html/2606.11420#S4.SS4.p3.4)\.
- G\. Team \(2025\)Gemini: a family of highly capable multimodal models\.External Links:2312\.11805,[Link](https://arxiv.org/abs/2312.11805)Cited by:[§3](https://arxiv.org/html/2606.11420#S3.p2.1)\.
- J\. Thorne, A\. Vlachos, C\. Christodoulopoulos, and A\. Mittal \(2018\)FEVER: a large\-scale dataset for fact extraction and verification\.In2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018,pp\. 809–819\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1),[§2](https://arxiv.org/html/2606.11420#S2.p1.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in neural information processing systems30\.Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p2.1)\.
- V\. Venktesh and V\. Setty \(2025\)LiveFC: a system for live fact\-checking of audio streams\.InProceedings of the Eighteenth ACM International Conference on Web Search and Data Mining,pp\. 1060–1063\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p3.1),[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- W\. Y\. Wang \(2017\)“Liar, liar pants on fire”: a new benchmark dataset for fake news detection\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),pp\. 422–426\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1),[§2](https://arxiv.org/html/2606.11420#S2.p1.1),[§3](https://arxiv.org/html/2606.11420#S3.p2.1)\.
- C\. Whittle \(2024\)Why america is downloading the news: a study on daily news podcasts and why u\.s\. audiences listen\.Electronic News18\(3\),pp\. 169–182\.External Links:[Document](https://dx.doi.org/10.1177/19312431241238452),[Link](https://doi.org/10.1177/19312431241238452)Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1)\.
- S\. Yang, H\. Chang, Z\. Huang, A\. T\. Liu, C\. Lai, H\. Wu, J\. Shi, X\. Chang, H\. Tsai, W\. Huang, T\. Feng, P\. Chi, Y\. Y\. Lin, Y\. Chuang, T\. Huang, W\. Tseng, K\. Lakhotia, S\. Li, A\. Mohamed, S\. Watanabe, and H\. Lee \(2024\)A large\-scale evaluation of speech foundation models\.IEEE/ACM Transactions on Audio, Speech, and Language Processing32\(\),pp\. 2884–2899\.External Links:[Document](https://dx.doi.org/10.1109/TASLP.2024.3389631)Cited by:[§4\.2](https://arxiv.org/html/2606.11420#S4.SS2.p1.1)\.
- B\. M\. Yao, A\. Shah, L\. Sun, J\. Cho, and L\. Huang \(2023\)End\-to\-end multimodal fact\-checking and explanation generation: a challenging dataset and models\.InProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval,pp\. 2733–2743\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p3.1),[§2](https://arxiv.org/html/2606.11420#S2.p2.1)\.
- F\. Zeng and W\. Gao \(2023\)Prompt to be consistent is better than self\-consistent? few\-shot and zero\-shot fact verification with pre\-trained language models\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 4555–4569\.Cited by:[§4](https://arxiv.org/html/2606.11420#S4.p1.3)\.
- D\. C\. Zhang and D\. Lee \(2025\)Correct: context\-and reference\-augmented reasoning and prompting for fact\-checking\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),pp\. 3007–3019\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1)\.
- C\. Zhao, C\. Xiong, C\. Rosset, X\. Song, P\. Bennett, and S\. Tiwary \(2020\)Transformer\-xh: multi\-evidence reasoning with extra hop attention\.InInternational conference on learning representations,Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1)\.
- J\. Zhou, X\. Han, C\. Yang, Z\. Liu, L\. Wang, C\. Li, and M\. Sun \(2019\)GEAR: graph\-based evidence aggregating and reasoning for fact verification\.InProceedings of the 57th annual meeting of the association for computational linguistics,pp\. 892–901\.Cited by:[§1](https://arxiv.org/html/2606.11420#S1.p1.1)\.