On Improving Faithfulness of Podcasts from Documents

# On Improving Faithfulness of Podcasts from Documents
Source: [https://arxiv.org/html/2607.21961](https://arxiv.org/html/2607.21961)
Soumya Dutta1,2,Tejas Indulal Dhamecha2,Pannaga Shivaswamy2 1Indian Institute of Science,2Adobe Research [soumyad@adobe\.com](https://arxiv.org/html/2607.21961v1/mailto:[email protected])

###### Abstract

Large language models \(LLMs\) are increasingly used to generate long\-form conversational content such as podcasts from textual sources\. While these systems produce fluent and engaging narratives, they often introduce ungrounded information\. In this work, we present the first systematic study of faithfulness in document\-grounded podcast generation, where grounding must be maintained across conversational turns in long\-form, multi\-speaker transcripts\. We construct a dataset of over1,5001\{,\}500documents spanning five domains and generate podcast transcripts using multiple LLMs\. We introduce a turn\-level LLM\-as\-a\-judge framework for evaluating whether conversational turns are supported by the source document, and validate its reliability through human studies\. Our analysis shows that even state\-of\-the\-art models, includingGPT\-4o, frequently generate ungrounded content\. To mitigate this issue, we proposecatch\-n\-repair, a model\-agnostic framework that detects and rewrites unfaithful conversational turns while preserving conversational flow\. Experiments demonstrate consistent improvements in faithfulness across both in\-domain and out\-of\-domain settings\.

On Improving Faithfulness of Podcasts from Documents

Soumya Dutta1,2, Tejas Indulal Dhamecha2, Pannaga Shivaswamy21Indian Institute of Science,2Adobe Research[soumyad@adobe\.com](https://arxiv.org/html/2607.21961v1/mailto:[email protected])

## 1Introduction

Podcasts have emerged as one of the most popular means of sharing information, spanning domains such as news reporting, education, entertainment, and lifestyle\. A recent Pew Research Center report highlights this growth: while only12%12\\%of Americans aged1212and older listened to a podcast in a typical month in20132013, the proportion rose to42%42\\%by20232023\(Pew Research Center,[2023](https://arxiv.org/html/2607.21961#bib.bib1)\)\. This surge in adoption reflects not only the familiarity of audiences with the medium but also the increasing trust placed in podcasts as a source of information\. Until recently, however, podcasting largely remained a medium of consumption, where listeners tuned in to content created and curated by human hosts\. The paradigm is now shifting with the advent of generative AI systems, such as Google’s NotebookLM111[https://notebooklm\.google\.com/](https://notebooklm.google.com/), which can produce podcast\-style narratives directly from source materials\. This shift opens up the possibility of personalized podcast creation at scale, where individuals and organizations can generate podcasts tailored to their own needs and interests\.

![Refer to caption](https://arxiv.org/html/2607.21961v1/x1.png)Figure 1:Example of unfaithfulness in generated podcast transcripts\. The transcript generated from ‘Attention Is All You Need’ paper mentions ChatGPT which did not exist in 2017\.This development builds on a broader line of work in generative AI applications such as summarization\(Zhanget al\.,[2024](https://arxiv.org/html/2607.21961#bib.bib10); Kim and Kim,[2025](https://arxiv.org/html/2607.21961#bib.bib9); Liuet al\.,[2025a](https://arxiv.org/html/2607.21961#bib.bib2)\)and question answering \(QA\)\(Zaibet al\.,[2022](https://arxiv.org/html/2607.21961#bib.bib11); Huet al\.,[2025](https://arxiv.org/html/2607.21961#bib.bib13); Zhanget al\.,[2025](https://arxiv.org/html/2607.21961#bib.bib12)\), where models transform source material into more accessible forms\. A central challenge in these tasks is*faithfulness*—ensuring that generated outputs remain grounded in the source without introducing distortions or hallucinations\(Chiesurinet al\.,[2023](https://arxiv.org/html/2607.21961#bib.bib16); Jiaet al\.,[2023](https://arxiv.org/html/2607.21961#bib.bib15); Minget al\.,[2024](https://arxiv.org/html/2607.21961#bib.bib14)\)\.

While AI\-generated podcasts inherit these concerns, they introduce a distinct set of challenges\. Unlike summaries or answers, podcasts are typically long\-form, narrative\-driven, and multi\-speaker, requiring models to maintain coherence and factual accuracy across extended spans\(Xiaoet al\.,[2025](https://arxiv.org/html/2607.21961#bib.bib17)\)\. Moreover, faithfulness in this setting must be preserved at the level of individual turns, where hallucinations can be localized yet propagate through the narrative\. Despite recent efforts in podcast generation\(Juet al\.,[2025](https://arxiv.org/html/2607.21961#bib.bib18); Xiaoet al\.,[2025](https://arxiv.org/html/2607.21961#bib.bib17); Penget al\.,[2026](https://arxiv.org/html/2607.21961#bib.bib3)\), prior work has primarily focused on fluency, structure, and audio delivery, leaving the question of grounding to the source document largely unexplored\.

To illustrate this challenge, consider the example in Figure[1](https://arxiv.org/html/2607.21961#S1.F1)\. The source is the Attention Is All You Need paper222[https://arxiv\.org/pdf/1706\.03762](https://arxiv.org/pdf/1706.03762)\.NotebookLMgenerated a transcript mentioning ChatGPT as part of the paper’s legacy; while true in hindsight, ChatGPT did not exist in 2017 and there is no mention of ChatGPT in the source paper making this an unfaithful statement\. This example highlights a subtle but critical distinction between factual correctness and faithfulness to the provided context\.

Motivated by this observation, we undertake the first systematic study of faithfulness in*document\-grounded podcast generation*\. In contrast to prior work, we focus on long\-form, multi\-speaker transcripts and analyze faithfulness at the level of individual conversational turns\. To enable this study, we collect a dataset of15201520documents spanning five domains and generate corresponding podcast transcripts using multiple large language models \(LLMs\)\. We introduce aturn\-level LLM\-as\-a\-judge evaluation protocolthat assesses the degree to which each conversational turn is supported by the source document\. This approach captures nuanced cases such as partial grounding and subtle hallucinations, which are common in long\-form generation\. We further validate the reliability of this evaluation framework through human studies\. Building on this evaluation framework, we proposecatch\-n\-repair, a simple yet effective, model\-agnostic mitigation strategy\. Our approach operates at the level of individual conversational turns: it detects unfaithful generations using a lightweight classifier and rewrites them to ensure grounding in the source document while preserving conversational flow\. Importantly,catch\-n\-repaircan be applied to both open\-source and closed\-source LLMs, making it broadly applicable in practical settings\. Our contributions are summarized as follows:

- •Evaluation protocol:We introduce a turn\-level LLM\-as\-a\-judge framework for measuring faithfulness in long\-form, multi\-speaker podcast transcripts\. The protocol captures fine\-grained grounding at the level of individual conversational turns and is validated against human annotations\.
- •Benchmarking and analysis:We conduct a systematic empirical study across multiple LLMs and domains, providing the first analysis of faithfulness in document\-grounded podcast generation\.
- •Mitigation method:We proposecatch\-n\-repair, a model\-agnostic strategy that detects unfaithful conversational turns and rewrites them to ensure grounding in the source document, improving faithfulness while preserving conversational flow\.

## 2Related Work

![Refer to caption](https://arxiv.org/html/2607.21961v1/x2.png)Figure 2:Faithfulness evaluation: Once these transcripts are generated, LLM\-as\-a\-judge \(GPT4o\) evaluation is used to classify each turn of the podcast on a scale of11to55based on its faithfulness to the source document\.Mitigation: The proposed mitigation strategy \-catch\-n\-repair\- is used in conjunction with the LLM to generate transcripts that are faithful to the source document\.Generation of Podcasts from Documents: The task of generating podcasts directly from source documents has received relatively little attention in the research community\.Juet al\.\([2025](https://arxiv.org/html/2607.21961#bib.bib18)\)proposed Mooncast, which leverages existing podcast scripts to produce long\-form, engaging audio content\.Xiaoet al\.\([2025](https://arxiv.org/html/2607.21961#bib.bib17)\)proposed PodAgent, an LLM agent that generates podcast scripts based on a user\-specified topic, without relying on a source document\. In contrast, our work focuses on*document\-grounded podcast generation*and systematically studies the faithfulness of generated transcripts to the underlying source material\. Faithfulness in LLM Generation: Ensuring that model outputs remain faithful to the provided context has been widely studied in tasks such as summarization and question answering\(Chiesurinet al\.,[2023](https://arxiv.org/html/2607.21961#bib.bib16); Jiaet al\.,[2023](https://arxiv.org/html/2607.21961#bib.bib15); Minget al\.,[2024](https://arxiv.org/html/2607.21961#bib.bib14)\)\. Existing metrics, including token\-overlap measures such as Knowledge\-F1\(Shusteret al\.,[2021](https://arxiv.org/html/2607.21961#bib.bib31)\), and learned evaluators such as FaithCritic\(Dziriet al\.,[2022](https://arxiv.org/html/2607.21961#bib.bib33)\), have been proposed to detect hallucinations\. More recently, LLM\-as\-a\-judge frameworks\(Zhenget al\.,[2023](https://arxiv.org/html/2607.21961#bib.bib34); Adlakhaet al\.,[2024](https://arxiv.org/html/2607.21961#bib.bib35)\)have demonstrated strong performance in evaluating correctness and faithfulness\. However, prior work has largely focused on short\-form generation, where outputs are typically single\-turn and concise\. In contrast, podcast transcripts are long\-form, multi\-speaker, and narrative\-driven, requiring faithfulness to be maintained at the level of individual conversational turns\. Our work extends the study of faithfulness to this more complex setting\. Evaluation of Podcast Transcripts: Recent work has begun to explore the evaluation of podcast\-style outputs generated by LLMs, often using LLM\-as\-a\-judge frameworks to assess properties such as coherence, fluency, and engagementXiaoet al\.\([2025](https://arxiv.org/html/2607.21961#bib.bib17)\); Xuet al\.\([2026](https://arxiv.org/html/2607.21961#bib.bib4)\)\. However, these approaches primarily operate at the level of the entire transcript and do not explicitly evaluate grounding with respect to a source document\. As a result, they are not designed to detect unfaithful content that may arise during generation\. In contrast, our work focuses on*faithfulness to the source document*as a primary evaluation objective and introduces a turn\-level evaluation protocol that enables fine\-grained assessment of grounding within long\-form, multi\-speaker transcripts\. Mitigating Ungroundedness in LLMs: A wide range of approaches have been proposed to mitigate hallucinations in LLM outputs, where models generate content that is either factually incorrect or not grounded in the provided context\. In this work, we focus on the latter—*ungroundedness*with respect to a source document\. Notably, ungrounded statements may be factually correct in the real world but remain unsupported by the input document, making them particularly challenging to detect in long\-form generation\. Existing approaches mitigate hallucinations from open\-weight LLMs through several directions\. Decoding\-based approaches adjust token probabilities to reduce ungrounded generations\(Shiet al\.,[2024](https://arxiv.org/html/2607.21961#bib.bib29); Huanget al\.,[2025](https://arxiv.org/html/2607.21961#bib.bib30)\)\. Other methods perform post\-hoc correction using auxiliary models or knowledge graphs\(Dziriet al\.,[2021](https://arxiv.org/html/2607.21961#bib.bib27); Jiet al\.,[2023a](https://arxiv.org/html/2607.21961#bib.bib28)\)\. However, many of these approaches require access to model internals \(e\.g\., logits\), limiting their applicability in black\-box settings\. For such settings, prior work has largely relied on external knowledge or self\-consistency signals\. For instance, Ji et al\.Jiet al\.\([2023b](https://arxiv.org/html/2607.21961#bib.bib7)\)formulate validation as a question\-answering task over domain\-specific knowledge bases, while Manakul et al\.Manakulet al\.\([2023](https://arxiv.org/html/2607.21961#bib.bib8)\)detect hallucinations through inconsistencies across multiple sampled generations\. Gao et al\.Gaoet al\.\([2023](https://arxiv.org/html/2607.21961#bib.bib5)\)further propose correcting outputs by retrieving supporting evidence from the web\. In contrast, our proposedcatch\-n\-repairframework is model\-agnostic and operates at the level of individual conversational turns, enabling faithful generation without requiring access to model internals or external retrieval systems\.

## 3Problem Statement

### 3\.1Problem Setup

Let𝒟=d1,d2,…,dn\\mathcal\{D\}=\{d\_\{1\},d\_\{2\},\\dots,d\_\{n\}\}denote a collection of source documents\. For each documentdi∈𝒟d\_\{i\}\\in\\mathcal\{D\}, a large language model \(LLM\), denoted byℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}, generates a podcast\-style conversational transcript:\[𝒯i=ti​1,ti​2,…,ti​mi\]\[\\mathcal\{T\}\_\{i\}=\{t\_\{i1\},t\_\{i2\},\\dots,t\_\{im\_\{i\}\}\}\]where eachti​jt\_\{ij\}represents thejthj^\{\\text\{th\}\}conversational turn in the transcript\.

Our objective is to evaluate and improve the faithfulness of generated podcast transcripts with respect to their source documents\. To this end, we define a faithfulness evaluation functionℱeval\\mathcal\{F\}\_\{\\text\{eval\}\}that assigns a faithfulness score to each conversational turnti​jt\_\{ij\}conditioned on the source documentdid\_\{i\}\. The overall faithfulness scorefif\_\{i\}for transcript𝒯i\\mathcal\{T\}\_\{i\}is obtained by averaging across itsmim\_\{i\}turns:

fi=1mi​∑j=1miℱeval​\(ti​j,di\)\.f\_\{i\}=\\frac\{1\}\{m\_\{i\}\}\\sum\_\{j=1\}^\{m\_\{i\}\}\\mathcal\{F\}\_\{\\text\{eval\}\}\(t\_\{ij\},d\_\{i\}\)\.\(1\)Unlike prior settings that evaluate grounding at the response or document level, we formulate faithfulness at the level of individual conversational turns\. This granularity is particularly important in long\-form podcast generation, where ungrounded content is often localized to specific turns while the remainder of the transcript remains faithful to the source\. Moreover, turn\-level assessment enables targeted mitigation without requiring regeneration of the entire transcript\.

Our goal is to design a mitigation method,catch\-n\-repair, which can be applied to the generating modelℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}in order to increase the faithfulness score; i\.e\. to obtain improved transcript𝒯i′\\mathcal\{T\}^\{\\prime\}\_\{i\}such that the corresponding faithfulness score improvesfi′\>fif^\{\\prime\}\_\{i\}\>f\_\{i\}\. An overview of the proposed framework is shown in Figure[2](https://arxiv.org/html/2607.21961#S2.F2)\.

### 3\.2Creation of Podcast Transcripts

We consider a podcast generation setup in which transcripts are generated sequentially in a turn\-by\-turn manner\. Specifically, we focus on a two\-speaker conversational structure commonly observed in real\-world podcasts, where a host guides the discussion and an expert provides detailed, document\-grounded responses\. The host turns are expected to contain natural and engaging questions that facilitate conversational flow, while the expert turns are expected to remain faithful to the source document\. The transcript generation process is driven by an LLMℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}, whose sequential nature enables turn\-level monitoring and intervention during generation\.

### 3\.3Evaluating Faithfulness of Generated Podcast Transcripts

Designing the faithfulness scoring functionℱeval\\mathcal\{F\}\_\{\\text\{eval\}\}is non\-trivial due to two key challenges\. First, conversational turns often express information through paraphrasing, making lexical overlap insufficient for evaluating grounding\. Pure token\-overlap metrics may therefore incorrectly penalize semantically faithful generations\. Second, the LLMℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}, may introduce content drawn from its parametric memory rather than fromdid\_\{i\}itself\. Such hallucinated statements can be factually correct in the real world yet remain ungrounded with respect to the provided document\.

To address these challenges, we adopt anLLM\-as\-a\-judgeframework for computingℱeval\\mathcal\{F\}\_\{\\text\{eval\}\}\. Given a source documentdid\_\{i\}and a transcript turnti​jt\_\{ij\}, an LLM evaluator modelℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}is prompted to assess the degree to which the factual content ofti​jt\_\{ij\}is supported bydid\_\{i\}\. Rather than issuing a binary verdict,ℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}assigns a*Likert\-style*scoreri​j∈\{1,2,3,4,5\}r\_\{ij\}\\in\\\{1,2,3,4,5\\\}, where11indicates that the turn is entirely unfaithful and55indicates a faithful generation\. This score directly definesℱeval​\(ti​j,di\)=ri​j\\mathcal\{F\}\_\{\\text\{eval\}\}\(t\_\{ij\},d\_\{i\}\)=r\_\{ij\}, and the document\-level faithfulnessfif\_\{i\}is computed as the mean of the turn\-level scores \(see Eq\.[1](https://arxiv.org/html/2607.21961#S3.E1)\)\. Using a graded scale allows the evaluator to capture nuanced cases, such as partially faithful turns or minor omissions, while remaining sensitive to hallucinated content\. The evaluation prompt instructs the model to assess whether the response is explicitly grounded in the provided context, penalizing plausible but unsupported claims\. Coverage Score: While faithfulness captures the reliability of generated transcripts, podcast usability also depends on how comprehensively the transcript reflects the key content of the source document\. We therefore additionally evaluate*coverage*, which measures the extent to which important information fromdid\_\{i\}is represented in𝒯i\\mathcal\{T\}\_\{i\}\. Unlike faithfulness, which is evaluated at the level of individual turns, coverage is assessed at the transcript level\. The evaluator modelℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}is prompted to assign a score indicating how comprehensively the generated transcript covers the major points contained in the source document\. Although we report coverage scores for completeness, improving coverage is not the primary focus of this work and is left as an important direction for future research\.

![Refer to caption](https://arxiv.org/html/2607.21961v1/x3.png)Figure 3:Step 1: Using an LLM to create training data for thecatchmodel\.Step 2: Training another LLM for predicting faithful/unfaithful given any podcast turn\.Step 3: Thecatch\-n\-repairmechanism for mitigating the unfaithful generations\. Thecatchmodel flags each generation as faithful or unfaithful\. The unfaithful generations are replaced by the same LLM with the old unfaithful generation as a negative example\.

## 4Catch\-n\-Repair

We now presentcatch\-n\-repair, our proposed method for mitigating unfaithful generations in podcast transcripts\. The design of this strategy is guided by two key goals: \(i\) applicability to both open\-source and closed\-source models, and \(ii\) improvement of the faithfulness scorefif\_\{i\}with minimal impact on the coverage score, as evaluated byℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}\(Sec\.[3\.3](https://arxiv.org/html/2607.21961#S3.SS3)\)\. To achieve these goals, we train an auxiliary model that approximates the judgment ofℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}and deploy it during transcript generation to identify unfaithful conversational turns\.

### 4\.1Training Data

To train the detection component ofcatch\-n\-repair, we construct a synthetic dataset of grounded and ungrounded conversational turns\. For this stage, we employ theLLaMA\-3\.1\-70Bmodel as both the generation and evaluation model\. Using an open\-weight model ensures reproducibility and avoids practical and legal restrictions associated with creating supervision data using proprietary APIs\. Importantly, the resulting mitigation framework is evaluated using a different evaluator model\. To generate ungrounded examples, we modify the generation instructions to explicitly encourage the model to introduce information beyond the source document\. Specifically, the model is encouraged to draw from general world knowledge, make tangential associations, and introduce plausible but unsupported details\. The resulting transcript𝒯ihallu\\mathcal\{T\}\_\{i\}^\{\\text\{hallu\}\}is then evaluated using the Likert\-scale protocol described in Sec\.[3\.3](https://arxiv.org/html/2607.21961#S3.SS3)\. Turns receiving a faithfulness score≥4\\geq 4are labeled*faithful*\(grounded\), while all others are labeled*unfaithful*\(ungrounded\), yielding a binary classification dataset for training the detection component ofcatch\-n\-repair\. This results in17,56117,561faithfulturns and9,4369,436unfaithfulturns\.

### 4\.2Training thecatchmodel

With the synthetic dataset constructed in Sec\.[4\.1](https://arxiv.org/html/2607.21961#S4.SS1), we now describe how to train the model that detects unfaithful generations fromℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}\. Each training instance is a tuple\(ti​jhallu,di,li​j\)\(t^\{\\text\{hallu\}\}\_\{ij\},d\_\{i\},l\_\{ij\}\), whereti​jhallut^\{\\text\{hallu\}\}\_\{ij\}is a hallucinated turn,did\_\{i\}is the source document, andli​j∈\{0,1\}l\_\{ij\}\\in\\\{0,1\\\}denotes whether the turn is faithful\.

Because this is a binary classification task, we initially considered encoder\-only architectures\. However, modern encoders such as ModernBERTWarneret al\.\([2025](https://arxiv.org/html/2607.21961#bib.bib36)\)still have a maximum context length of only81928192tokens, while many source documentsdid\_\{i\}in our dataset are substantially longer\. To accommodate these long contexts, we instead perform instruction fine\-tuning of a large language model, specificallyLLaMA\-3\.1\-8B, using LoRAHuet al\.\([2022](https://arxiv.org/html/2607.21961#bib.bib37)\)to predict either “faithful” or “unfaithful\.” The model is trained with an instruction format that presents the context document and the conversational response, then asks whether the response is faithful to the provided context\. For efficiency, the context documentdid\_\{i\}is truncated so that the complete instruction does not exceed32,76832\{,\}768tokens\. We further balance the training dataset between the two classes while training thecatchmodel\. Once trained, this model can be seamlessly integrated into the generation pipeline to flag unfaithful turns\.

### 4\.3Repair Strategy

We now describe the repair component ofcatch\-n\-repair\. Our goal is to correct unfaithful generations while maintaining applicability to both open\-source and black\-box LLMs\. This requirement rules out approaches that depend on access to model internals, such as activation steeringTurneret al\.\([2023](https://arxiv.org/html/2607.21961#bib.bib39)\); Rimskyet al\.\([2024](https://arxiv.org/html/2607.21961#bib.bib38)\)or confidence\-based decoding strategiesBiet al\.\([2025](https://arxiv.org/html/2607.21961#bib.bib40)\); Liuet al\.\([2025b](https://arxiv.org/html/2607.21961#bib.bib42)\)\.

While thecatchmodel is trained to predict whether each generated turn is faithful, our repair mechanism follows a simple but effective strategy\. Whenever a conversational turnti​jt\_\{ij\}is flagged as unfaithful,ℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}is re\-prompted to reviseti​jt\_\{ij\}to produce a corrected turn that is semantically faithful to the provided context while preserving conversational flow\. The regeneration prompt instructs the model to rewrite the turnti​jt\_\{ij\}using only facts explicitly stated in the source document, while maintaining natural conversational flow\. Importantly, the repair process is integrated into the sequential generation pipeline, allowing unfaithful turns to be corrected before subsequent turns are generated\. This reduces propagation of ungrounded content across the conversation\. The repaired turn is then appended directly to the transcript\. The final transcript produced byℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}under thecatch\-n\-repairmitigation strategy is denoted by𝒯ifaith\\mathcal\{T\}\_\{i\}^\{\\text\{faith\}\}\. The workflow is shown in Fig\.[3](https://arxiv.org/html/2607.21961#S3.F3)\.

## 5Experiments

### 5\.1Doc\-to\-Podcast

We collected a total of15001500documents to create ourDoc\-to\-Podcastdataset\. The corpus spans five domains—academic,legal,policy,financial, andmedical—with300300documents sampled from each domain\.Further details are in Appendix[A\.1](https://arxiv.org/html/2607.21961#A1.SS1)\.

To capture a broad range of input complexities, we deliberately varied the length of the collected documents\. The distribution of document sizes across domains is shown in Appendix Fig\.[6](https://arxiv.org/html/2607.21961#A1.F6); all documents contain between22and5050pages\. The dataset serves two purposes: \(i\) it is used to evaluate the faithfulness of podcasts generated by multiple LLMs \(ℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}\), and \(ii\) it provides training and validation material for our proposedcatchmodel\.

For testing thecatch\-n\-repairmitigation strategy, we use4040held\-out documents\. Of these,2020are from theDoc\-to\-Podcastdataset \(44per domain\), forming the*in\-domain*test set\. The remaining2020documents originate from unrelated areas such asexam evaluationsandbusiness practices, and constitute the*out\-of\-domain*test set\.

![Refer to caption](https://arxiv.org/html/2607.21961v1/x4.png)Figure 4:The average faithfulness score as evaluated withℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}set asGPT\-4o\. The scores for differentℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}are shown for the different domains\.
### 5\.2Faithfulness across LLMs

Figure[4](https://arxiv.org/html/2607.21961#S5.F4)shows the average faithfulness scores across all documents in each domain for different generating LLMs\. All podcast turns are evaluated byGPT\-4oacting as the judge LLM\. For transcript generation, we use three variants from theQwen2\.5family \(1414B,3232B, and7272B\), the7070B model from theLLaMA3\.3family, andGPT\-4oitself\. TheLLaMA\-3\.1\-70Bmodel is not used in this case in order to avoid overlap with the training paradigm of thecatchmodel \(Sec\.[4\.1](https://arxiv.org/html/2607.21961#S4.SS1)\)\.

- •GPT\-4o achieves the highest faithfulness: Among all models, transcripts generated byGPT\-4oreceive the highest average faithfulness score of3\.893\.89on the Likert scale, followed byQwen2\.5\-32Bwith a score of3\.753\.75\. Notably, even the strongest model fails to consistently achieve fully grounded generation, indicating that maintaining faithfulness remains challenging in long\-form conversational settings despite substantial advances in LLM capabilities\.
- •Bigger open models aren’t automatically more faithful: While increasing size generally helps, our results show diminishing returns beyond the mid\-scale range\. For instance,LLaMA3\.3\-70Bperforms on par withQwen2\.5\-72BandQwen2\.5\-32B, while the smallerQwen2\.5\-14Blags behind at3\.553\.55\.
- •Faithfulness varies by domain: Scores are not uniform across domains\. The academic domain records the lowest average faithfulness \(3\.383\.38across all models\), while the policy domain achieves the highest \(4\.054\.05\)\. This suggests that the intrinsic knowledge distribution of LLMs influences their ability to remain faithful to the document\.
- •Cross\-LLM agreement alleviates bias concerns: To further assess evaluator reliability, we additionally replaceGPT\-4owithQwen2\.5\-72Bas the evaluator model\. The results mirrored those ofGPT\-4o:Qwenalso ratedGPT\-4o’s generations as more faithful than its own \(Fig\.[5](https://arxiv.org/html/2607.21961#S5.F5)\)\. This agreement across independent evaluator models reduces concerns regarding model\-specific judge bias\.

![Refer to caption](https://arxiv.org/html/2607.21961v1/x5.png)Figure 5:The average faithfulness score as evaluated with the Qwen model when the generating LLM is either Qwen or GPT\-4o\. The evaluations with GPT\-4o model are also shown for reference\.We present the coverage scores and observations from them in Appendix[A\.3](https://arxiv.org/html/2607.21961#A1.SS3)\.

### 5\.3Reliability of the Evaluator

In order to assess the reliability of our evaluator, we conduct a human annotation study using3030randomly selected documents \(66from each domain\) from theDoc\-to\-Podcastdataset and annotate a total of328328conversational turns\. The annotations were collected from1010annotators \(Sec\.[A\.8](https://arxiv.org/html/2607.21961#A1.SS8)\)\. We measure inter\-annotator agreement using Krippendorff’sα\\alpha, obtaining a value of0\.690\.69, indicating substantial agreement\. We compare the scores assigned by our evaluator model \(GPT\-4o\) against human ratings, and also benchmark the evaluator against existing automatic faithfulness metrics\. K\-Prec\., K\-Rec\., and K\-F1 compute lexical overlap between the podcast turn and the source documentShusteret al\.\([2021](https://arxiv.org/html/2607.21961#bib.bib31)\), while K\-BERTs\-F1 and K\-BERTs\-Prec\. extend this approach using BERTScore\-based semantic similarityAdlakhaet al\.\([2024](https://arxiv.org/html/2607.21961#bib.bib35)\); Zhanget al\.\([2019](https://arxiv.org/html/2607.21961#bib.bib6)\)\. The Pearson correlations for each domain are shown in Table[1](https://arxiv.org/html/2607.21961#S5.T1)\.GPT\-4oachieves the highest correlation with human judgments across all domains, with an average correlation of0\.630\.63\. In contrast, existing automatic faithfulness metrics exhibit substantially weaker alignment\. These results suggest that LLM\-based evaluation provides a more reliable proxy for human assessment in long\-form conversational generation\. Interestingly, model\-free lexical grounding metrics such asKK\-Precision consistently outperform learned evaluators such asFaithCritic, with the exception ofGPT\-4o\. This observation is consistent with prior findings reported by Adlakha et al\.\(Adlakhaet al\.,[2024](https://arxiv.org/html/2607.21961#bib.bib35)\)\. Notably, these correlations are obtained in a more challenging setting than conventional short\-form evaluation\.

Table 1:Pearson correlation with human faithfulness scores across domains\. GPT\-4o shows consistently higher alignment than existing metrics\.Source Document SummaryVanilla GenerationAftercatch\-n\-repairThe source document is an environmental science coursework handout focused on topics such as BPA exposure, recycling, environmental health, and sustainability\. It contains educational explanations, reflective discussion prompts, and assessment\-style questions intended to encourage students to think critically about environmental risks and personal choices related to health and pollution\.Environmental science education seems like such a critical topic these days\. Can you explainwhy understanding human impacts on the environment is so important?I’ve been reading about environmental science education focusing on human impacts\. It’s fascinating how it integrates assessments and activities related to pollutants and toxic chemicals\. Could you share more about the types of assessments or activities mentioned in the document?Table 2:Example of correcting an ungrounded conversational turn usingcatch\-n\-repair\.
### 5\.4Training and Evaluation of thecatchModel

Implementation Details: We fine\-tune aLLaMA\-3\.1\-8Bmodel to classify each transcript turn asfaithfulorunfaithfulgiven the source document as context\. To respect computational limits, each document is truncated so that the combined input remains below32,76832,768tokens\. Fine\-tuning is performed with a batch size of88and gradient accumulation of88using LoRAHuet al\.\([2022](https://arxiv.org/html/2607.21961#bib.bib37)\)\(rank =88,α=16\\alpha=16, dropout =0\.050\.05\)\. Training uses90%90\\%of the documents from the synthetic dataset \(Sec\.[4\.1](https://arxiv.org/html/2607.21961#S4.SS1)\), while turns from the remaining10%10\\%form the validation set\. Performance: The validation split contains3,0393\{,\}039turns, including1,9731\{,\}973faithful and1,0661\{,\}066unfaithful examples\. The trainedcatchmodel achieves F1\-scores of79\.9%79\.9\\%and67\.3%67\.3\\%on the faithful and unfaithful classes, respectively\. The lower performance on unfaithful examples reflects the difficulty of detecting subtle yet plausible ungrounded content\. Despite being substantially smaller than the original evaluator \(LLaMA\-3\.1\-70B\), the resulting88B model remains lightweight and suitable for runtime deployment\.

### 5\.5Effectiveness ofcatch\-n\-repair

Table[3](https://arxiv.org/html/2607.21961#S5.T3)presents the faithfulness scores for both in\-domain \(ID\) and out\-of\-domain \(OOD\) settings, comparing vanilla generation with ourcatch\-n\-repair\(CnR\) strategy\.

- •catch\-n\-repairconsistently boosts faithfulness: Across all models and both ID and OOD settings, CnR improves faithfulness scores\. The largest relative gain for the ID test set is seen for theQwen2\.5\-32BandQwen2\.5\-72Bmodels \(∼7\.5%\\sim 7\.5\\%\)\. For the OOD test set, the largest gain of12\.8%12\.8\\%is seen for theLLaMA\-3\.3\-70Bmodel\.
- •Robust to domain shift: Faithfulness gains hold for the OOD test set\. On average, across all models, the relative increase in faithfulness scores are more for the OOD set \(∼10%\\sim 10\\%\) versus the ID set \(∼5%\\sim 5\\%\)\.
- •GPT\-4oremains the overall leader:GPT\-4oachieves the highest faithfulness in both ID and OOD settings, with an average relative increase of7\.4%7\.4\\%after the mitigation strategy\. The consistent gains obtained fromcatch\-n\-repairindicate that even state\-of\-the\-art proprietary models continue to benefit from explicit turn\-level grounding interventions\.

Table 3:Faithfulness scores for in\-domain \(ID\) and out\-of\-domain \(OOD\) test sets\. Van\. denotes vanilla prompting \(Sec\.[3\.2](https://arxiv.org/html/2607.21961#S3.SS2)\), CnR indicates thecatch\-n\-repairstrategy,Δ\\Deltarepresents the absolute improvement, and Avg\.Δ\\Deltais the mean improvement across ID and OOD\.#### Qualitative Analysis:

Table[2](https://arxiv.org/html/2607.21961#S5.T2)presents an example of conversational turns before and after applyingcatch\-n\-repair\. The source document primarily consists of environmental science coursework and assessment instructions related to BPA exposure, recycling, and environmental health\. In the vanilla generation setting, the conversation deviates from discussing the specific content in the document to broader discussions about the importance of assessing the impact that humans have on the environment\. While these additions are plausible, they are not explicitly grounded in the source document\. After applyingcatch\-n\-repair, the generated response adheres more closely to the scope and content of the original document\.

The performance of CnR withQwen\-2\.5\-72Bas judge is shown in Appendex[A\.2](https://arxiv.org/html/2607.21961#A1.SS2)\. The impact of CnR on coverage is shown in Appendix[A\.4](https://arxiv.org/html/2607.21961#A1.SS4)\. The performance of thecatchmodel with a text classifier instead of a LLM is shown in Sec\.[A\.5](https://arxiv.org/html/2607.21961#A1.SS5)\.

## 6Conclusion

We present the first study of faithfulness in document\-grounded podcast generation, which involves long\-form, multi\-speaker conversations\. Our analysis across multiple LLMs and domains shows that even state\-of\-the\-art models frequently generate ungrounded content, highlighting the difficulty of maintaining faithfulness in long\-form conversational generation\. To address this, we introduce a turn\-level LLM\-as\-a\-judge evaluation framework and validate its reliability against human judgments\. We further proposecatch\-n\-repair, a model\-agnostic mitigation strategy that detects and corrects unfaithful conversational turns\. Experiments demonstrate consistent improvements in faithfulness across both in\-domain and out\-of\-domain settings\. Overall, our work highlights faithfulness as an important challenge in AI\-generated podcasts and demonstrates the effectiveness of turn\-level grounding mechanisms for trustworthy conversation generation systems\.

## 7Limitations

While our work provides a first benchmark for faithfulness in document\-to\-podcast generation, several limitations remain\. First, our evaluation relies primarily on English\-language documents; extending the dataset to multilingual and culturally diverse sources is an important future step\. Second, although ourcatch\-n\-repairmethod improves faithfulness, it depends on the accuracy of thecatchclassifier, which may occasionally mislabel subtle paraphrases as unfaithful\. Third, our faithfulness assessment is based on prompts that were designed for this task\. Different prompts might result in different results\. Another limitation of our work is the potential misuse, even after our attempts some ungrounded generations may still occur\. Finally, the podcasts in our dataset are text\-based transcripts rather than full audio renderings — future work could explore faithfulness in end\-to\-end audio generation pipelines, where prosody and delivery interact with factual grounding\.

## References

- Evaluating correctness and faithfulness of instruction\-following models for question answering\.Transactions of the Association for Computational Linguistics12,pp\. 681–699\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1),[§5\.3](https://arxiv.org/html/2607.21961#S5.SS3.p1.8)\.
- B\. Bi, S\. Liu, Y\. Wang, Y\. Xu, J\. Fang, L\. Mei, and X\. Cheng \(2025\)Parameters vs\. context: fine\-grained control of knowledge reliance in language models\.arXiv preprint arXiv:2503\.15888\.Cited by:[§4\.3](https://arxiv.org/html/2607.21961#S4.SS3.p1.1)\.
- S\. Chiesurin, D\. Dimakopoulos, M\. A\. S\. Cabezudo, A\. Eshghi, I\. Papaioannou, V\. Rieser, and I\. Konstas \(2023\)The dangers of trusting stochastic parrots: faithfulness and trust in open\-domain conversational question answering\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 947–959\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1),[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- N\. Dziri, E\. Kamalloo, S\. Milton, O\. Zaiane, M\. Yu, E\. M\. Ponti, and S\. Reddy \(2022\)Faithdial: a faithful benchmark for information\-seeking dialogue\.Transactions of the Association for Computational Linguistics10,pp\. 1473–1490\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- N\. Dziri, A\. Madotto, O\. R\. Zaiane, and A\. J\. Bose \(2021\)Neural path hunter: reducing hallucination in dialogue systems via path grounding\.InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp\. 2197–2214\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- L\. Gao, Z\. Dai, P\. Pasupat, A\. Chen, A\. T\. Chaganty, Y\. Fan, V\. Zhao, N\. Lao, H\. Lee, D\. Juan,et al\.\(2023\)Rarr: researching and revising what language models say, using language models\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 16477–16508\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, W\. Chen,et al\.\(2022\)Lora: low\-rank adaptation of large language models\.\.ICLR1\(2\),pp\. 3\.Cited by:[§4\.2](https://arxiv.org/html/2607.21961#S4.SS2.p2.4),[§5\.4](https://arxiv.org/html/2607.21961#S5.SS4.p1.14)\.
- Z\. Hu, H\. Yan, Q\. Zhu, Z\. Shen, Y\. He, and L\. Gui \(2025\)Beyond prompting: an efficient embedding framework for open\-domain question answering\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 19975–19990\.External Links:[Link](https://aclanthology.org/2025.acl-long.981/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.981),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1)\.
- Y\. Huang, Y\. Zhang, N\. Cheng, Z\. Li, S\. Wang, and J\. Xiao \(2025\)Dynamic attention\-guided context decoding for mitigating context faithfulness hallucinations in large language models\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 5174–5193\.External Links:[Link](https://aclanthology.org/2025.findings-acl.269/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.269),ISBN 979\-8\-89176\-256\-5Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- Z\. Ji, Z\. Liu, N\. Lee, T\. Yu, B\. Wilie, M\. Zeng, and P\. Fung \(2023a\)RHO: reducing hallucination in open\-domain dialogues with knowledge grounding\.InFindings of the Association for Computational Linguistics: ACL 2023,pp\. 4504–4522\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- Z\. Ji, T\. Yu, Y\. Xu, N\. Lee, E\. Ishii, and P\. Fung \(2023b\)Towards mitigating llm hallucination via self reflection\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 1827–1843\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- Q\. Jia, S\. Ren, Y\. Liu, and K\. Zhu \(2023\)Zero\-shot faithfulness evaluation for text summarization with foundation language model\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,pp\. 11017–11031\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1),[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- Z\. Ju, D\. Yang, J\. Yu, K\. Shen, Y\. Leng, Z\. Wang, X\. Tan, X\. Zhou, T\. Qin, and X\. Li \(2025\)MoonCast: high\-quality zero\-shot podcast generation\.arXiv preprint arXiv:2503\.14345\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p3.1),[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- H\. Kim and B\. Kim \(2025\)NexusSum: hierarchical LLM agents for long\-form narrative summarization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 10120–10157\.External Links:[Link](https://aclanthology.org/2025.acl-long.500/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.500),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1)\.
- N\. Liu, J\. Zhu, Y\. Ma, Z\. Lu, W\. Xu, Y\. Yang, J\. Zhong, and K\. Wei \(2025a\)SARA: Salience\-Aware Reinforced Adaptive Decoding for Large Language Models in Abstractive Summarization\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 25450–25463\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1)\.
- Z\. Liu, R\. A\. Amjad, R\. Adkathimar, T\. Wei, and H\. Tong \(2025b\)SelfElicit: your language model secretly knows where is the relevant evidence\.arXiv preprint arXiv:2502\.08767\.Cited by:[§4\.3](https://arxiv.org/html/2607.21961#S4.SS3.p1.1)\.
- P\. Manakul, A\. Liusie, and M\. Gales \(2023\)Selfcheckgpt: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 conference on empirical methods in natural language processing,pp\. 9004–9017\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- Y\. Ming, S\. Purushwalkam, S\. Pandit, Z\. Ke, X\. Nguyen, C\. Xiong, and S\. Joty \(2024\)FaithEval: can your language model stay faithful to context, even if" the moon is made of marshmallows"\.arXiv preprint arXiv:2410\.03727\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1),[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- Z\. Peng, J\. Yu, W\. Wang, Y\. Chang, Y\. Sun, L\. Dong, Y\. Zhu, W\. Xu, H\. Bao, Z\. Wang,et al\.\(2026\)VibeVoice: expressive podcast generation with next\-token diffusion\.InThe Fourteenth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p3.1)\.
- Pew Research Center \(2023\)Audio and podcasting fact sheet\.Note:Accessed: 2025\-09\-02External Links:[Link](https://www.pewresearch.org/journalism/fact-sheet/audio-and-podcasting/)Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p1.5)\.
- N\. Rimsky, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. Turner \(2024\)Steering llama 2 via contrastive activation addition\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 15504–15522\.Cited by:[§4\.3](https://arxiv.org/html/2607.21961#S4.SS3.p1.1)\.
- W\. Shi, X\. Han, M\. Lewis, Y\. Tsvetkov, L\. Zettlemoyer, and W\. Yih \(2024\)Trusting your evidence: hallucinate less with context\-aware decoding\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 783–791\.External Links:[Link](https://aclanthology.org/2024.naacl-short.69/),[Document](https://dx.doi.org/10.18653/v1/2024.naacl-short.69)Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- K\. Shuster, S\. Poff, M\. Chen, D\. Kiela, and J\. Weston \(2021\)Retrieval augmentation reduces hallucination in conversation\.InFindings of the Association for Computational Linguistics: EMNLP 2021,pp\. 3784–3803\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1),[§5\.3](https://arxiv.org/html/2607.21961#S5.SS3.p1.8)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2023\)Steering language models with activation engineering\.arXiv preprint arXiv:2308\.10248\.Cited by:[§4\.3](https://arxiv.org/html/2607.21961#S4.SS3.p1.1)\.
- B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, G\. T\. Adams, J\. Howard, and I\. Poli \(2025\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 2526–2547\.External Links:[Link](https://aclanthology.org/2025.acl-long.127/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.127),ISBN 979\-8\-89176\-251\-0Cited by:[§A\.5](https://arxiv.org/html/2607.21961#A1.SS5.p1.4),[§4\.2](https://arxiv.org/html/2607.21961#S4.SS2.p2.4)\.
- Y\. Xiao, L\. He, H\. Guo, F\. Xie, and T\. Lee \(2025\)PodAgent: a comprehensive framework for podcast generation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 23923–23937\.External Links:[Link](https://aclanthology.org/2025.findings-acl.1226/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.1226),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p3.1),[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- C\. Xu, M\. Zheng, M\. Zheng, and M\. Song \(2026\)PodBench: a comprehensive benchmark for instruction\-aware audio\-oriented podcast script generation\.arXiv preprint arXiv:2601\.14903\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.
- M\. Zaib, W\. E\. Zhang, Q\. Z\. Sheng, A\. Mahmood, and Y\. Zhang \(2022\)Conversational question answering: a survey\.Knowledge and Information Systems64\(12\),pp\. 3151–3195\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1)\.
- T\. Zhang, D\. Li, Q\. Chen, C\. Wang, and X\. He \(2025\)BELLE: a bi\-level multi\-agent reasoning framework for multi\-hop question answering\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 4184–4202\.External Links:[Link](https://aclanthology.org/2025.acl-long.211/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.211),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1)\.
- T\. Zhang, V\. Kishore, F\. Wu, K\. Q\. Weinberger, and Y\. Artzi \(2019\)Bertscore: evaluating text generation with bert\.arXiv preprint arXiv:1904\.09675\.Cited by:[§5\.3](https://arxiv.org/html/2607.21961#S5.SS3.p1.8)\.
- T\. Zhang, F\. Ladhak, E\. Durmus, P\. Liang, K\. McKeown, and T\. B\. Hashimoto \(2024\)Benchmarking large language models for news summarization\.Transactions of the Association for Computational Linguistics12,pp\. 39–57\.Cited by:[§1](https://arxiv.org/html/2607.21961#S1.p2.1)\.
- L\. Zheng, W\. Chiang, Y\. Sheng, S\. Zhuang, Z\. Wu, Y\. Zhuang, Z\. Lin, Z\. Li, D\. Li, E\. Xing,et al\.\(2023\)Judging llm\-as\-a\-judge with mt\-bench and chatbot arena\.Advances in neural information processing systems36,pp\. 46595–46623\.Cited by:[§2](https://arxiv.org/html/2607.21961#S2.p1.1)\.

## Appendix AAppendix

### A\.1Dataset details

![Refer to caption](https://arxiv.org/html/2607.21961v1/x6.png)Figure 6:The number of documents in each page range for the different domains\.We provide the links from which the documents are collected for each of the domains below:

- •
- •
- •
- •
- •

### A\.2Evaluator Robustness Across Judge Models

One potential concern in LLM\-as\-a\-judge evaluation is whether the observed improvements are specific to a particular evaluator model\. To assess the robustness of our findings, we repeat the evaluation ofcatch\-n\-repairusingQwen2\.5\-72Bas the evaluator in place ofGPT\-4o\. The resulting faithfulness scores are shown in Table[4](https://arxiv.org/html/2607.21961#A1.T4)\. Across all generator models and both in\-domain \(ID\) and out\-of\-domain \(OOD\) settings,catch\-n\-repaircontinues to improve faithfulness scores relative to vanilla generation\. Moreover, the overall ranking of generator models remains broadly consistent across evaluators\. While the absolute scores differ slightly between evaluators, the qualitative trends remain stable\. In particular, both evaluators consistently identify improvements in faithfulness after applyingcatch\-n\-repair, suggesting that the observed gains are not artifacts of a specific judge model\. These results reduce concerns regarding evaluator\-specific bias and support the robustness of our conclusions across independent LLM evaluators\.

Table 4:Faithfulness scores for in\-domain \(ID\) and out\-of\-domain \(OOD\) test sets\. Van\. denotes vanilla prompting \(Sec\.[3\.2](https://arxiv.org/html/2607.21961#S3.SS2)\), CnR indicates thecatch\-n\-repairstrategy,Δ\\Deltarepresents the absolute improvement, and Avg\.Δ\\Deltais the mean improvement across ID and OOD\. The judge isQwen\-2\.5\-72B\.Table 5:Faithfulness and coverage for in\-domain \(ID\) and out\-of\-domain \(OOD\) test sets\. Van\. stands when the LLMs are prompted as described in Sec\.[3\.2](https://arxiv.org/html/2607.21961#S3.SS2), while CnR represents the scores when unfaithfulness is mitigated using thecatch\-n\-repairstrategy\.
### A\.3Evaluating Coverage inDoc\-to\-Podcast

![Refer to caption](https://arxiv.org/html/2607.21961v1/x7.png)Figure 7:The average coverage score as evaluated withℒeval\\mathcal\{L\}\_\{\\text\{eval\}\}set asGPT\-4o\. The scores for differentℒgen\\mathcal\{L\}\_\{\\text\{gen\}\}are shown for the different domains\.The coverage scores for the different domains and the different generating LLMs are shown in Fig\.[7](https://arxiv.org/html/2607.21961#A1.F7)\.

- •GPT\-4o achieves the highest coverage: Among all models, transcripts generated byGPT\-4oreceive the highest average coverage score of4\.814\.81on the Likert scale, followed byQwen2\.5\-72Bwith a score of4\.374\.37\.
- •Coverage scores of LLaMA are strikingly low: The coverage scores assigned toLLaMAare strikingly low at2\.752\.75\. The similar sizedQwen2\.5\-72Bgets a score of4\.374\.37\. This highlights a potential issue with usingLLaMAas the generating LLM\.
- •Bigger models are better in the Qwen family: As the model size increases in theQwen2\.5family, the coverage score increases\. While the1414B model has an average coverage score of3\.53\.5, the3232B model is much better at3\.973\.97\. The7272B model is the best of the three, with an average coverage score of4\.374\.37\.
- •Coverage varies by domain: Like faithfulness, the coverage score also varies according to the domain\. It is highest for the medical domain \(4\.134\.13\) and lowest for the legal domain \(3\.673\.67\)

### A\.4Impact ofcatch\-n\-repairon Coverage

The results for faithfulness and coverage are shown in Table[5](https://arxiv.org/html/2607.21961#A1.T5)\. The following are the key takeaways\.

- •Minimal impact on coverage: Coverage remains largely unchanged, with only small decreases \(≤0\.2\\leq 0\.2in most cases\), indicating that the mitigation strategy does not significantly reduce transcript completeness\.
- •GPT\-4o is the best model for faithfulness and coverage:GPT\-4oremains the best model in terms of balancing faithfulness and coverage in the generated podcasts\.

### A\.5Are LLMs really required for thecatchmodel?

Thecatchcomponent can be viewed as a text classification problem: given a podcast turn and its corresponding source document, the model predicts whether the turn isfaithfulorunfaithful\. An attentive reader might question the need for using an LLM for this classification objective\. To investigate this, we replace theLLaMA\-3\.1\-8Bbackbone with the encoder\-basedModernBERTmodelWarneret al\.\([2025](https://arxiv.org/html/2607.21961#bib.bib36)\)\. We observe that the F1\-score on faithful validation examples decreases from79\.9%79\.9\\%to76\.1%76\.1\\%, while performance on unfaithful examples drops more substantially from67\.3%67\.3\\%to60\.9%60\.9\\%\. These results suggest that long\-context LLMs provide a significant advantage for detecting subtle forms of ungroundedness in long\-form conversational generation\. In particular, identifying unsupported yet factually plausible statements often requires reasoning over large portions of the source document, which remains challenging for shorter\-context encoder architectures\. Since the effectiveness of thecatch\-n\-repairpipeline depends critically on accurately identifying unfaithful turns, stronger long\-context reasoning substantially improves the reliability of downstream repair\.

### A\.6License

- •
- •

### A\.7Use of LLMs

LLMs were used for polishing the writing and also for help during coding\.

### A\.8Human annotation setup

The human evaluation study was conducted using1010student annotators recruited from the university community\. Annotators were presented with a source document and a generated podcast turns from the system described in Sec\.[3\.2](https://arxiv.org/html/2607.21961#S3.SS2), and were asked to rate the faithfulness of the podcast turns with respect to the source document on a Likert scale from11to55\.

Annotators were specifically instructed to evaluate whether the information in the conversational turn was supported by the source document, rather than whether the content was factually plausible in general\. In particular, they were asked to penalize unsupported extrapolations, policy\-oriented additions, or conversational embellishments not grounded in the document\. Each example was independently annotated by multiple annotators, and the final score was obtained by averaging the collected ratings\. Annotators were financially compensated in accordance with local standards\. The instruction for the human annotation is shown here\.

#### Instruction to humans:

Please go through the PDF\. Grounding means explicit support from the context\. Plausible or theoretically possible claims that are not mentioned in the context should be penalized\. Please assign a score from 1–5 using the following guidelines: \(1\) 5\(Completely Grounded\): ALL claims/questions are explicitly supported by the context\. No speculation or extrapolation beyond what is directly stated\. \(2\) 4\(Mostly Grounded\): Core claims are explicitly supported, with only very minor inferences that stay within the scope of the provided material\. \(3\) 3\(Partially Grounded\): Some connection to context but includes unsupported claims, applications, or examples not mentioned in the source material\. \(4\) 2\(Mostly Ungrounded\): Relates to the context but makes several significant unsupported claims or extrapolations beyond what is stated, even if plausible\. \(5\) 1\(Completely Ungrounded\): Contradicts the context directly OR is completely unrelated/irrelevant to the source material\.