SHAM-Bench：跨多任务的片段级幻觉基准

# A Span-level Hallucination Benchmark Across Multiple Tasks
Source: [https://arxiv.org/html/2610.27370](https://arxiv.org/html/2610.27370)
Ian LaneAffiliation:shaun213@ucsc\.edu,ialane@ucsc\.edu

###### Abstract

Span\-level hallucination detection pinpoints unsupported content in LLM outputs, enabling actionable feedback and targeted mitigation\. Existing evaluations for span\-level hallucination detection methods are fragmented across tasks, annotation formats, and metrics, and most span\-level datasets cover only a single NLG domain, making it difficult to assess whether a detection method actually generalizes across tasks and domains\. In this paper, we introduceSHAM\-Bench\(Span\-levelHallucination benchmarkAcrossMultiple tasks\), a unified benchmark for span\-level hallucination detection spanning six NLG tasks, aggregated from existing human\-annotated datasets totaling 6,461 examples under a common schema and evaluation protocol\. Using SHAM\-Bench, we evaluate existing hallucination detection methods and find that fine\-tuned detectors often perform well in\-domain but degrade under distribuiton shift, whereas prompt\-based methdos with reasoning are more robust across tasks\. Our analysis further shows that subtle unsupported content remains difficult to detect and span\-level evaluation is still challenged by ambiguous span boundaries and noisy labels\. We release the dataset, evaluation code, and a public leaderboard to support future research on broadly generalizable hallucination detection\.

## 1Introduction

While Large Language Models \(LLMs\) are extremely effective at generating fluent text, they often produce hallucinations, generating content that isfactually incorrect with respect to world knowledge or is unsupported by the available evidence\([Huang et al\., 2025a](https://arxiv.org/html/2610.27370#bib.bib27);[Zhang et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib28)\)\. Detecting hallucinations in LLM outputs is important for applications ranging from dialog assistants\([Li et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib25)\)and decision\-support systems\([Singhal et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib37);[Brynjolfsson et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib38)\)to complex agentic workflows\([Lin et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib29)\)\. Early work in hallucination detection focused on coarse\-grained, response\-level decisions\([Wang et al\., 2020](https://arxiv.org/html/2610.27370#bib.bib32);[Fabbri et al\., 2021](https://arxiv.org/html/2610.27370#bib.bib33);[Adlakha et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib34);[Healy et al\., 2026](https://arxiv.org/html/2610.27370#bib.bib39)\)\. While useful for screening, response\-level detection does not identifywherehallucination errors occur, limiting its value for actionable feedback\.

Span\-level hallucination detection addresses this limitation by localizing the specific words or phrases in a generation that are unsupported or factually incorrect\([Liu et al\., 2022](https://arxiv.org/html/2610.27370#bib.bib35)\)\. Rather than rejecting an entire response, a span\-level detector can preserve supported content while identifying hallucinated segments\. This is especially valuable for post\-editing workflows\([Tonmoy et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib30)\)and for providing fine\-grained training signals for hallucination mitigation\([Wen et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib31)\)\.

Despite its practical importance, span\-level hallucination detection remains difficult to evaluate systematically\. Most annotated resources provide only sentence\- or response\-level labels\([Lin et al\., 2022](https://arxiv.org/html/2610.27370#bib.bib26)\), too coarse to test whether methods identify precise unsupported spans\. Datasets with span\-level annotations remain scarce and typically focus on a single task\([Tang et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib8);[Vazquez et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib6)\), limiting how far their conclusions generalize across domains\. Moreover, differences in annotation formats and evaluation metrics further complicate direct comparisons of detectors\.

To address these challenges, we introduceSHAM\-Bench, theSpan\-levelHallucination benchmarkAcrossMultiple tasks\. SHAM\-Bench is a unified benchmark for span\-level hallucination detection\. It consolidates nine task partitions aggregated from six human\-annotated datasets into a single standardized multitask benchmark\. The benchmark spans six NLG tasks: open\-domain QA, context\-based QA, data\-to\-text generation, biographical generation, summarization, and machine translation, totaling 6,461 test set examples\. By normalizing annotation formats and using standard evaluation metrics across all datasets and tasks, SHAM\-Bench allows a meaningful comparison of hallucination detection methods across a wide variety of tasks\.

Using SHAM\-Bench, we perform a comprehensive evaluation of seven existing hallucination detection methods and analyze why they succeed or fail across tasks\. Our results show that fine\-tuned detectors often perform well in narrow domains but struggle under distribution shift, while prompt\-based methods with reasoning are more robust across domains\. Furthermore, our analysis shows that both annotation and modeling challenges exist\. Span annotations can suffer from ambiguous span boundaries and noisy labels, and subtle unsupported content remains difficult to detect reliably even for the most capable model\.

Taken together, our findings suggest that advancing span\-level hallucination detection requires both accurrate and reliable annotations across tasks to support evaluation and more sensitive detection methods to identify subtle errors\. We hope SHAM\-Bench will serve as a valuable resource for the community, allowing practitioners to understand the pros and cons of different hallucination detection methods, and to advance the state\-of\-the\-art in span\-level hallucination detection\. Further, we hope that future work would build on SHAM\-Bench to incorporate more tasks and datasets capturing the fuller spectrum of NLG tasks to support more generalizable hallucination detection\.

## 2Related Works

### 2\.1Detecting Hallucinations in LLMs

Most early approaches to detect hallucinations in LLM outputs aim to provide response level judgements\. Starting with a reference and a response, BERTScore\([Zhang\* et al\., 2020](https://arxiv.org/html/2610.27370#bib.bib41)\)judges a response by scoring the embedding similarity between a reference and the response while BARTScore\([Yuan et al\., 2021](https://arxiv.org/html/2610.27370#bib.bib40)\)uses the log\-likelihood of the response directly as the score\. Taking a reference\-free approach, SelfCheckGPT\([Manakul et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib20)\)estimates factuality by sampling multiple responses and measuring their mutual inconsistency\.

Later approaches aim to provide more fine\-grained hallucination judgements\. For instance, AlignScore\([Zha et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib2)\)and MiniCheck\([Tang et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib17)\)fine\-tune models to determine whether a sentence aligns or is supported by a given context\. RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib3)\)and FAVA\([Mishra et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib9)\)use span\-annotated data to fine\-tune models to achieve span\-level hallucination detection\. QAGS\([Wang et al\., 2020](https://arxiv.org/html/2610.27370#bib.bib32)\)and FactScore\([Min et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib10)\)operate at the fact level by either asking and answering factual questions about the response or by decomposing the response into individual facts and verifying each one\.

Another class of methods leverage model\-internal signals\. LLM\-Check\([Sriramanan et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib21)\)uses internal attention maps, hidden activations, and output probabilities to detect hallucinations within a single response, while[Ogasa and Arase \(2025\)](https://arxiv.org/html/2610.27370#bib.bib22)exploits attention irregularities across informative layers to localize hallucinated spans\. Leveraging uncertainty,[Farquhar et al\. \(2024\)](https://arxiv.org/html/2610.27370#bib.bib23)introduces semantic entropy as a hallucination signal, showing hallucinated claims exhibit higher entropy\.

### 2\.2Evaluating and Benchmarking Hallucination Detection Methods

Most prior benchmarks for hallucination detection are limited in one of two ways: they either provide only coarse\-grained annotations at the response or sentence level, or they provide fine\-grained span annotations but only for a single task\.

TruthfulQA\([Lin et al\., 2022](https://arxiv.org/html/2610.27370#bib.bib26)\)evaluate response\-level hallucinations on QA task and find that larger models are more prone to misconceptions\. AggreFact\([Tang et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib8)\)aggregates nine summarization datasets for response\-level evaluation, finding that no single metric performs best across the individual datasets and HADES\([Liu et al\., 2022](https://arxiv.org/html/2610.27370#bib.bib35)\)gathers token\-level annotations for free\-form generation, showing that surrounding context length greatly affects how well hallucinations can be detected\. While these approaches offer valuable insight when evaluating hallucination detection on these tasks, the evaluations in each work are limited to a single task\.

HaluEval\([Li et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib25)\), though spans multiple tasks, only perform hallucination detection at the response level\. The same is true for LLM\-AggreFact\([Tang et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib17)\)which aggregates 10 datasets for sentence\-level factual consistency evaluation\. Mu\-SHROOM\([Vazquez et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib6)\)and PsiloQA\([Rykov et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib24)\), despite providing span\-level annotations for multiple languages, focus exclusively on the QA task\. To our knowledge, RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib3)\)and[Zhou et al\. \(2021\)](https://arxiv.org/html/2610.27370#bib.bib7)are the only two benchmark with human labeled span\-label annotations of hallucinations that span multiple tasks\.

## 3SHAM\-Bench

SHAM\-Bench, theSpan\-levelHallucination BenchmarkAcrossMultiple Tasks is a reference\-based benchmark for span\-level hallucination detection in LLM outputs\. It combines all currently available human\-annotated span\-labeled hallucination detection datasets, covering nine task\-specific test sets from six datasets, into a single unified multitask benchmark\. SHAM\-Bench spans six NLG tasks: open\-domain QA, context\-based QA, data\-to\-text, biographical generation, summarization, and machine translation\. The responses in the benchmark are generated by modern LLMs, allowing SHAM\-Bench to reflect the real\-world conditions and LLM\-generated outputs which hallucination detection systems are likely to encounter\. Additionally, the summarization and QA tasks include multiple dataset partitions, enabling SHAM\-Bench to identify generalization failures across datasets that single\-dataset benchmarks cannot expose\.

### 3\.1Benchmark Datasets

FaithBench\([Bao et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib5)\)is a summarization faithfulness benchmark containing 750 summaries \(75 passages×\\times10 LLMs\), where annotators label spans as*questionable*,*benign*, or*unwanted*\. We convert these into binary hallucinated/non\-hallucinated span labels\.FAVABench\([Mishra et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib9)\)contains∼\\sim1k model responses to open\-domain queries with human annotations that localize factual errors at the span level with a fine\-grained error\-type taxonomy\. After processing, we retain 266 examples, each augmented with GPT\-5\-retrieved context for use as reference evidence\.HalluEntity\([Yeh et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib4)\)is an entity\-level hallucination dataset built from GPT\-3\.5\-generated biographies\. Following the FActScore pipeline\([Min et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib10)\), atomic facts are verified against Wikipedia and mapped to entity mentions, yielding 157 manually verified examples\. Each is paired with GPT\-5\-retrieved context\.Mu\-SHROOM\([Vazquez et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib6)\)provides human\-annotated span\-level hallucination labels for LLM\-generated answers to open\-domain questions across 14 languages\. We use the English subset of 150 examples, each augmented with GPT\-5\-retrieved context\.RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib3)\)is a span\-annotated RAG benchmark covering question answering, data\-to\-text, and summarization\. Each of the 2,700 test examples pairs a user query and evidence context with an LLM\-generated response\. Human annotators highlight spans unsupported by or contradicted by the evidence\.Zhou\([Zhou et al\., 2021](https://arxiv.org/html/2610.27370#bib.bib7)\)provides token\-level hallucination annotations for summarization and machine translation tasks\. It includes 2,000 span\-annotated summarization outputs adapted from[Maynez et al\. \(2020\)](https://arxiv.org/html/2610.27370#bib.bib11), generated by PtGen, TConvS2S, TransS2S, and BERTS2S\([See et al\., 2017](https://arxiv.org/html/2610.27370#bib.bib12);[Narayan et al\., 2018](https://arxiv.org/html/2610.27370#bib.bib13);[Vaswani et al\., 2017](https://arxiv.org/html/2610.27370#bib.bib14);[Rothe et al\., 2020](https://arxiv.org/html/2610.27370#bib.bib15)\), and∼\\sim500 MT outputs from TransS2S and mBART\([Vaswani et al\., 2017](https://arxiv.org/html/2610.27370#bib.bib14);[Liu et al\., 2020](https://arxiv.org/html/2610.27370#bib.bib16)\)\.

### 3\.2Dataset Schema, Tasks, and Statistics

Table 1:SHAM\-Bench statistics in both task aggregates and dataset breakdowns\. Hallucination rate is the proportion of reponse words that are hallucinated\. Word counts and hallucination rate are averaged across all examples\.#### Standardized schema\.

To reconcile differences in dataset formats, we convert every instance to a unified schema with four main fields, enabling consistent preprocessing and evaluation across tasks:

- •question: the original user query or instruction that elicited the response\.
- •context: auxiliary input required to answer the query \(e\.g\., evidence passage, structured record, source document, or target\-language text\)\. For tasks that do not provide additional input \(e\.g\. open\_qa, open\_biography\), this field contains retrieved context that answers the question, so that the hallucination detection methods that require context can use it\.
- •response: the model\-generated response to be evaluated for hallucinations\.
- •hallu\_spans: a list of human\-annotated hallucinated spans in the response text\.

#### Tasks\.

We assign each instance from the six source datasets \(§[3\.1](https://arxiv.org/html/2610.27370#S3.SS1)\) to one of six tasks that cover common NLG use cases:

- •open\_qa: open\-domain QA where the answering model is given only a question, without any supporting evidence or context\.
- •context\_qa: QA conditioned on an explicit context passage; the model receives both a question and a provided context and is expected to answer using that context\.
- •data\-to\-text: data\-grounded generation where the query targets information in structured data \(e\.g\., a JSON record\) and the model produces a natural\-language output\.
- •open\_biography: biographical generation where the model is prompted to write a biography of a real person\.
- •summarization: summarization of an input document; the model is given a source text and asked to produce a concise summary\.
- •machine\_translation: translation of a source text from one language to another\.

We provide a small validation set by sampling five instances for each \(dataset, task\)\-pair, yielding 45 instances in total\. This validation set is used to explore the effectiveness of prompt optimization explored in section §[4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px3)of this paper\.

SHAM\-Bench contains 6,461 examples spanning six tasks sourced from six datasets \(Table[1](https://arxiv.org/html/2610.27370#S3.T1)\)\. Summarization is the largest category, consisting of 56% of the total examples, while open\_biography is comparatively small, being only 2\.4% of the total\. Response lengths differ markedly by task; machine\_translation averages 27\.5 words in response while open\_qa averages 192 words\. Hallucination rate is likewise highly non\-uniform across tasks and datasets\. At the task level, average hallucination rates range from∼\\sim0\.04 in data2txt and context\_qa to 0\.227 in open\_biography and 0\.294 in summarization\. The dataset\-level breakdown further reveals within\-task differences; summarization contains hallucinated content in 3% of token on RAGTruth but 46\.6% on Zhou, and open\_qa hallucinates 14\.2% of tokens on FAVABench vs\. 32\.6% on Mushroom\. Differences in annotation guidelines and the model used to generate responses likely contribute to the variation in hallucination rates across tasks and datasets\.

### 3\.3Metrics and Evaluation

We use F1score as the primary evaluation metric\. This choice follows prior span\-level hallucination detection work including[Mishra et al\. \(2024\)](https://arxiv.org/html/2610.27370#bib.bib9);[Niu et al\. \(2024\)](https://arxiv.org/html/2610.27370#bib.bib3);[Zhou et al\. \(2021\)](https://arxiv.org/html/2610.27370#bib.bib7)\. Moreover, since SHAM\-Bench standardizes detector outputs into binary token\-level labels, hallucination detection can be naturally evaluated as a binary classification problem\. F1score balances precision and recall and is therefore not dominated by the majority non\-hallucinated class\. Appendix[B](https://arxiv.org/html/2610.27370#A2)provides further discussion of alternative metrics

SHAM\-Bench supports evaluation at two levels of granularity: token\-level and sentence\-level\. Each sample in the dataset consists of a queryqq, a model\-generated responserr, and human\-labeled hallucinated spansS⊂rS\\subset r\. A detector predicts either hallucinated spansS^\\hat\{S\}or sentence\-level labels𝐳^\\hat\{\\mathbf\{z\}\}\. We convert all outputs to token\-level binary labels\. Given tokenization𝒯⁡\(r\)=\(t1,…,tn\)\\mathcal\{T\}\(r\)=\(t\_\{1\},\\dots,t\_\{n\}\), we define ground\-truth labels𝐲∈\{0,1\}n\\mathbf\{y\}\\in\\\{0,1\\\}^\{n\}and predicted labels𝐲^∈\{0,1\}n\\hat\{\\mathbf\{y\}\}\\in\\\{0,1\\\}^\{n\}as:

yi=𝕀\[∃s∈S:tioverlaps withs\]\.y\_\{i\}=\\mathbb\{I\}\\\!\\left\[\\exists s\\in S:t\_\{i\}\\mbox\{ overlaps with \}s\\right\]\.y^i=𝕀\[∃s^∈S^:tioverlaps withs^\]\.\\hat\{y\}\_\{i\}=\\mathbb\{I\}\\\!\\left\[\\exists\\hat\{s\}\\in\\hat\{S\}:t\_\{i\}\\mbox\{ overlaps with \}\\hat\{s\}\\right\]\.
For sentence\-level evaluation, given segmentation𝒰⁡\(r\)=\(u1,…,um\)\\mathcal\{U\}\(r\)=\(u\_\{1\},\\dots,u\_\{m\}\), we project span annotations onto sentences to obtain ground\-truth labels𝐳∈\{0,1\}m\\mathbf\{z\}\\in\\\{0,1\\\}^\{m\}and predicted labels𝐳^∈\{0,1\}m\\hat\{\\mathbf\{z\}\}\\in\\\{0,1\\\}^\{m\}:

zj=𝕀\[∃s∈S:ujoverlaps withs\]z\_\{j\}=\\mathbb\{I\}\\\!\\left\[\\exists s\\in S:u\_\{j\}\\text\{ overlaps with \}s\\right\]z^j=𝕀\[∃s^∈S^:ujoverlaps withs^\]\\hat\{z\}\_\{j\}=\\mathbb\{I\}\\\!\\left\[\\exists\\hat\{s\}\\in\\hat\{S\}:u\_\{j\}\\text\{ overlaps with \}\\hat\{s\}\\right\]Methods that directly output sentence labels use them as𝐳^\\hat\{\\mathbf\{z\}\}without projection\. We report F1at both the span and sentence\-level, aggregated by task and by dataset\. We additionally report precision and recall scores in Appendix[D](https://arxiv.org/html/2610.27370#A4)\.

### 3\.4The SHAM\-Bench Leaderboard

The SHAM\-Bench leaderboard is publicly available at:[tbd](https://tbd/)111Submissions must not include links to non\-anonymous repositories\. Will add in the camera\-ready version\., allowing research groups to submit their systems for automated evaluation\. As aliving benchmark, SHAM\-Bench will evolve with new tasks and annotated datasets contributed by the community\. Its open\-source dataset and evaluation code provide a unified platform for tracking progress in hallucination detection across diverse NLG tasks and domains\.

## 4Hallucination Detection Methods

Using SHAM\-Bench, we evaluate seven existing high\-performing hallucination detection methods in three major categories, namely discriminatively trained, generatively trained, and prompt\-based methods\. Appendix[A](https://arxiv.org/html/2610.27370#A1)provides detailed method descriptions\.

#### Discriminatively trained methods\.

MiniCheck\([Tang et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib17)\), which is trained on synthetic data to judge whether claims are supported, andAlignScore\([Zha et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib2)\), which is trained on multiple datasets to score alignment, both detect hallucination at the sentence level\.PsiloQA\([Rykov et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib24)\), trained on synthetic data, andLettuceDetect\([Kovács and Recski, 2025](https://arxiv.org/html/2610.27370#bib.bib1)\), trained on human\-annotated data, instead detect hallucination at the token level\. These methods fundamentally treat hallucination detection as a binary classification problem whether at the sentence or token level\.

#### Generatively trained methods\.

FAVA\([Mishra et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib9)\)andRAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib3)\)are train using the language modeling objective with FAVA using synthetic data and RAGTruth using human annotated data\. They both identify hallucinated content by generating the unsupported content as text spans\.

#### Prompt\-based methods\.

Prompt\-based detectors use an LLM as a direct hallucination span extractor by providing the relevant information and instruction to the LLM as prompt\. We evaluate both hand\-crafted and automatically optimized prompts\([Huang et al\., 2025b](https://arxiv.org/html/2610.27370#bib.bib18)\)\. Prompt templates are provided in Appendix[F](https://arxiv.org/html/2610.27370#A6)\.

Table 2:Average span\- and sentence\-level F1scores on SHAM\-Bench by task\. Prompt method usesgpt\-5\-miniwith reasoning but no prompt optimization\.Overallreports the unweighted mean and standard deviation of six task scores\. Highest scores are inbold; second highest areunderlined\. The full results can be found in Appendix[D](https://arxiv.org/html/2610.27370#A4)\.Table 3:Average span\- and sentence\-level F1scores on SHAM\-Bench summarization and open/context\_qa tasks by dataset\. Prompt method usesgpt\-5\-miniwith reasoning but no prompt optimization\. Highest scores are inbold; second highest areunderlined\.

## 5Benchmarking Results

### 5\.1Prompt\-based methods are robust across different tasks

Results in Table[2](https://arxiv.org/html/2610.27370#S4.T2)indicate promptedGPT\-5\-mini\(with reasoning, no prompt optimization\) is the most balanced method across all tasks\. At span\-level F1, the prompted method achieves an overall score of 0\.498, placing second only to RAGTruth \(0\.518\)\. It ranks first on three of the six tasks: open\_qa \(0\.402\), open\_bio \(0\.525\), and mt \(0\.435\), while falling behind RAGTruth or LettuceDetect on the three tasks that those models were directly trained on, which is further discussed in §[5\.2](https://arxiv.org/html/2610.27370#S5.SS2)\. At sentence level, it achieves the highest overall score \(0\.691\), surpassing all discriminatively and generatively fine\-tuned methods\. It outperforms all discriminatively trained methods on four tasks, falling slightly behind MiniCheck on summarization and LettuceDetect on context\_qa\. The prompt\-based method also shows the most consistent performance across tasks, with the lowest standard deviation in task\-level F1scores \(0\.07\), suggesting that prompting is less prone to large task\-specific performance swings\.

Table 4:Average span\-level F1scores of prompt\-based detectors on SHAM\-Bench by task\.Reasonandoptindicate whether reasoning and prompt optimization are used\.Overallreports the unweighted mean and standard deviation of six task scores\. Highest scores are inbold; second highest areunderlined\. Complete span\- and sentence\-level results are in Table[5](https://arxiv.org/html/2610.27370#A4.T5)\.
### 5\.2Finetuned methods can perform well in narrow domains but struggle to generalize

#### PsiloQA: strong where hallucinations are frequent, but over\-predicts elsewhere\.

PsiloQA obtains its best span\-F1on Mushroom QA \(0\.556\) and Zhou summarization \(0\.547\), two datasets with high hallucination rates, but it performs poorly on context\_qa \(0\.072\) and data2txt \(0\.089\), where hallucinations are rare\. The same pattern holds for sentence\-level F1: PsiloQA achieves its highest score on Zhou summarization, where 91% of responses are labeled as containing hallucinations\. However, this over\-prediction leads it to classify nearly all examples as hallucinated, dragging down its overall performance\.

#### RAGTruth and LettuceDetect: dominant in\-domain, weak elsewhere\.

Although RAGTruth achieves the highest overall span\-F1\(0\.518\), this aggregate masks a sharp split between its training tasks and everything else\. Table[3](https://arxiv.org/html/2610.27370#S4.T3)makes this split explicit: RAGTruth scores 0\.835 on the RAGTruth summarization partition and 0\.865 on RAGTruth QA at the span level, but is only mediocre on other datasets\. The same pattern appears at sentence level, where it leads on its own partitions yet ranks among the worst on summarization Zhou and QA Mushroom\. LettuceDetect, fine\-tuned on the same data, behaves similarly: strong on RAGTruth partitions but weak elsewhere, except on Zhou summarization, where over\-flagging is less costly because most responses contain annotated hallucinations\.

#### FAVA: surprisingly weak on its own dataset\.

Table[3](https://arxiv.org/html/2610.27370#S4.T3)shows FAVA performs competitively on datasets where the evidence–response structure closely resembles FAVA’s Wikipedia\-grounded training format\. FAVA achieves span\-F1of 0\.761 on RAGTruth summarization and 0\.695 on RAGTruth context\_qa\. However, it performs unexpectedly poorly on FAVABench \(span\-F1of 0\.235\), the benchmark derived from the same work that introduced FAVA\.

#### MiniCheck and AlignScore: strong on text\-only tasks, brittle on structured data\.

Table[2](https://arxiv.org/html/2610.27370#S4.T2)shows the two sentence\-level methods, MiniCheck and AlignScore, exhibit similar performance profiles despite the differences in architecture and training data\. Similar to the PsiloQA, AlignScore is weak on data2txt, with a precision of 0\.247 and recall of 0\.429\. On this dataset it over\-predicts hallucinated sentences when the input context contains structured JSON records, likely because it hasn’t seen any JSON record grounding during claim\-verification training, thus flagging response sentences as unsupported when JSON records are encountered\. MiniCheck performs better on data2txt likely because it is fine\-tuned from a larger base model which brings broader pre\-training coverage of structured data formats though still lags behind prompted method by a significant margin\.

### 5\.3Reasoning plays a significant role in prompt\-based hallucination detection

#### Reasoning is the single most impactful factor\.

Enabling reasoning in prompt\-based approaches improves accuracy across every model and task\. The gain is largest forQwen3\-4B, whose overall span\-level F1jumps from 0\.298 to 0\.442 \(\+0\.144\) and sentence\-level F1from 0\.504 to 0\.631 \(\+0\.127\)\.GPT\-5\-miniimproves more modestly \(0\.425→\\to0\.498 span\-F1\), consistent with its stronger no\-reasoning baseline;OLMo\-3\-7Bsimilarly gains \+0\.042 span\-F1\.

Reasoning substantially reduces false positives and false negatives\. On examples*without*annotated hallucinations,Qwen3\-4Bwith no reasoning averages only 0\.13 span\-F1because it often marks spans as hallucinated when they are not; reasoning raises this to 0\.67\.OLMo\-3\-7B\-Instructwithout reasoning is instead over\-conservative\. It only scores 0\.28 span\-F1on all\-hallucinated samples by rarely predicting any spans, leaving most hallucinations undetected\. Reasoning raises it to 0\.39\. Reasoning better balances over\- and under\-prediction, reducing both false positives and missed hallucinations\.

#### Prompt optimization has mixed effects\.

MiPROv2 optimization improves performance on ctx\_qa and summarization, but it also consistently degrades open\_qa performance\. With only five examples per task used during optimization, the optimizer appears to over\-adapt to these samples and generalize poorly\. This highlights the risk of prompt optimization with very small development sets\.

## 6Challenges in Hallucination Detection

Figure 1:Distribution of hallucination rates as predicted by RAGTruth on summarization \(top\) and PsiloQA on QA \(bottom\)\. The first distribution in each plot is the distribution of the corresponding model’s training set\. To the right of each distribution is the proportion of examples that contain any hallucinated spans\.To better understand the system behavior observed in §[5](https://arxiv.org/html/2610.27370#S5), we examine two complementary sources of difficulty\. First, using RAGTruth and PsiloQA as representative cases, we analyze how the distribution of fine\-tuning data shapes detector predictions\. Second, we analyze failure cases from prompt\-based methods and identify several factors that continue to limit span\-level hallucination detection\.

### 6\.1Fine\-tuned models tend to follow their training data distribution\.

As shown in Figure[1](https://arxiv.org/html/2610.27370#S6.F1), hallucination rates in the RAGTruth summarization training set are concentrated near the low end\. The model fine\-tuned on this data closely follows the training distribution, regardless of the distribution of the evaluation set\. This behavior of output distribution aligns well with the RAGTruth test set, but not with FaithBench, whose distribution is less skewed, or Zhou et al\., which has an additional mode at the high end\. As a result, the RAGTruth\-trained model places disproportionate weight on low hallucination rates and tends to under\-predict hallucinations\. The PsiloQA model exhibits the opposite bias\. Although hallucination rates in the PsiloQA training set are more evenly distributed, 96% of its training examples contain hallucinations\. Consequently, when evaluated on QA datasets whose hallucination rates are concentrated near the low end, the model tends to over\-predict hallucinations, resulting in systematically inflated hallucination estimates\.

Fundamentally, these results reflect a train\-test distribution shift, in which the training distribution does not match the evaluation setting\. For practitioners, this suggests that class balancing alone is insufficient\. Instead, training data should be curated to ensure adequate representation of hallucination cases, especially rare error types, while preserving or explicitly modeling the expected deployment distribution\.

Figure 2:Top: ground\-truth span recovery by GPT\-5\-mini reasoning prompt\-based detector\. Bottom: error categories for spans received no overlapping prediction\.
### 6\.2Prompt\-based methods exhibit mixed failure modes

To characterize prompt\-based failure modes, we divide ground\-truth spans into two groups: those that overlap with at least one prediction and those with no overlapping prediction\. The first group captures cases in which the detector identifies the relevant factual issue, even if its predicted boundary differs from the human annotation\. The second captures hallucinations that are missed entirely\. For these missed spans, we useGPT\-5\.4to assign one of five error categories:labeling error,overgeneralization,unsupported detail,misattribution, andsubjective language; category definitions appear in Appendix[C](https://arxiv.org/html/2610.27370#A3)\.

Figure[2](https://arxiv.org/html/2610.27370#S6.F2)shows that the prompt\-based detector recovers 7,542 of 10,066 ground\-truth spans \(74\.9%\) with at least some overlap\. This suggests that many apparent span\-level errors arise from boundary disagreement rather than complete failure to recognize the underlying hallucination\. Among overlapping spans, predicted spans are on average 23 words longer than the corresponding ground\-truth span when they over\-predict, and 31 words shorter when they under\-predict\. Over\-prediction distribution shown in Appendix Figure[3](https://arxiv.org/html/2610.27370#A3.F3)\.

For the 2,524 ground\-truth spans with no overlapping prediction, human labeling error accounts for the largest share \(64%\)\. The remaining missed spans correspond to more substantive detector failures: overgeneralization accounts for 14%, unsupported details for 11%, misattribution for 7%, and subjective language for 3%\. Except for misattribution, these categories generally involve subtle departures from the reference document, which remain difficult for prompt\-based detectors to identify reliably\.

These results suggest that improving span\-level hallucination detection requires progress on annotation and modeling\. Accurate annotations and standardized span definitions would reduce evaluation noise, while greater emphasis on subtle semantic departures from the reference could help detectors address the remaining missed hallucinations\.

## 7Conclusion

We presented SHAM\-Bench, a unified benchmark for span\-level hallucination detection across diverse generation tasks\. Our evaluation shows that fine\-tuned detectors often perform well in\-domain but degrade sharply outside their training distributions, whereas prompt\-based methods with reasoning are more robust and consistent across tasks\. At the same time, ambiguous span boundaries, noisy labels, and subtle unsupported content remain open challenges for reliable span\-level evaluation\. SHAM\-Bench provides a foundation for future research on accurate, broadly generalizable hallucination detection\.

## 8Limitations

#### Task coverage\.

SHAM\-Bench covers a diverse set of NLG tasks, but it does not exhaustively represent all useful language generation settings\. Many important tasks are still missing because the benchmark is built from currently available human\-annotated span\-level hallucination resources\. Our goal is therefore not to provide a complete taxonomy of NLG tasks, but to establish a first extensible multi\-task benchmark for span\-level hallucination detection\. As more human\-annotated span\-level datasets become available, additional tasks and domains can be incorporated into SHAM\-Bench\.

#### RAGTruth over\-representation\.

RAGTruth contributes 2,675 examples to SHAM\-Bench across context QA, data\-to\-text, and part of the summarization task, accounting for approximately 41% of the benchmark\. This over representation provides an advantage to detectors fine\-tuned on the RAGTruth training set, meaning that strong overall performance does not necessarily reflect broad generalization\. As additional tasks and datasets are contributed to SHAM\-Bench in the future this imbalance will diminish\.

#### Legacy model outputs\.

A substantial portion of the Zhou data was generated by pre\-LLM\-era models\. The resulting errors differ significantly from hallucinations produced by modern LLMs\. This limits the extent to which performance on this subset reflects detector behavior in contemporary settings\.

#### Annotation inconsistency\.

The source datasets used in SHAM\-Bench were annotated by different groups following different annotation guidelines, which leads to substantial variation in how hallucination spans are defined and marked\. Some datasets favor minimal spans, while others use broader spans\. In aggregate, however, we believe that the rank of the hallucination detection methods will not be affected by variation in annotation styles across the individual the datasets\.

#### Annotation quality\.

In this work, we did not perform quality control of the annotations provided in the source datasets\. The quality of annotations across the datasets are known to vary widely, which limits the extent to which the benchmark can faithfully measure a model’s ability to identify hallucinations\. However, again in aggregate, we believe that the relative ranking of hallucination detection methods is unlikely to be affected by the annotation quality of individual datasets\.

#### Automated failure classification\.

The error categories used in our prompt\-based failure analysis are assigned byGPT\-5\.4rather than manually verified by expert annotators\. As a result, the category proportions should be interpreted as an approximate diagnostic analysis rather than as definitive class labels\.

#### Coverage of evaluated methods\.

In this work, we did not evaluate methods that require knowledge of or internal access to the generating model, such as hidden\-state\-based detectors or self\-consistency methods\. To ensure faithful implementation and accurate evaluation, we encourage researchers developing such techniques to evaluate them on SHAM\-Bench and submit their results to the SHAM\-Bench Leaderboard\.

#### Potential risks\.

Although SHAM\-Bench is intended as a research benchmark, it may still introduce downstream risks if used incautiously\. Imperfect hallucination detectors can be over\-trusted in deployment; false negatives may allow unsupported content to pass as reliable, while false positives may incorrectly flag factual content and reduce the usefulness of model outputs\. In addition, some benchmark instances involve generation about real people \(e\.g\., the open\_biography task\), so detector outputs may affect how biographical or factual claims are interpreted in downstream applications\. We therefore view SHAM\-Bench as a research evaluation resource and not as a sufficient basis for high\-stakes real\-world deployment decisions\.

## References

- Adlakhaet al\.\(2024\)V\. Adlakha, P\. BehnamGhader, X\. H\. Lu, N\. Meade, and S\. ReddyEvaluating correctness and faithfulness of instruction\-following models for question answering\.Transactions of the Association for Computational Linguistics12,pp\. 681–699\.External Links:[Link](https://aclanthology.org/2024.tacl-1.38/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00667)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Baoet al\.\(2025\)F\. S\. Bao, M\. Li, R\. Qu, G\. Luo, E\. Wan, Y\. Tang, W\. Fan, M\. S\. Tamber, S\. Kazi, V\. Sourabh, M\. Qi, R\. Tu, C\. Xu, M\. Gonzales, O\. Mendelevitch, and A\. AhmadFaithBench: a diverse hallucination benchmark for summarization by Modern LLMs\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 2: Short Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 448–461\.External Links:[Link](https://aclanthology.org/2025.naacl-short.38/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-short.38),ISBN 979\-8\-89176\-190\-2Cited by:[Appendix B](https://arxiv.org/html/2610.27370#A2.p2.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Brynjolfssonet al\.\(2023\)E\. Brynjolfsson, D\. Li, and L\. R\. RaymondGenerative ai at work\.Working PaperTechnical Report31161,Working Paper Series,National Bureau of Economic Research\.External Links:[Document](https://dx.doi.org/10.3386/w31161),[Link](http://www.nber.org/papers/w31161)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Fabbriet al\.\(2021\)A\. R\. Fabbri, W\. Kryściński, B\. McCann, C\. Xiong, R\. Socher, and D\. RadevSummEval: re\-evaluating summarization evaluation\.Transactions of the Association for Computational Linguistics9,pp\. 391–409\.External Links:[Link](https://aclanthology.org/2021.tacl-1.24/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00373)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Farquharet al\.\(2024\)S\. Farquhar, J\. Kossen, L\. Kuhn, and Y\. GalDetecting hallucinations in large language models using semantic entropy\.Nature630\(8017\),pp\. 625–630\.External Links:[Document](https://dx.doi.org/10.1038/s41586-024-07421-0),[Link](https://www.nature.com/articles/s41586-024-07421-0)Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p3.1)\.
- Healyet al\.\(2026\)K\. Healy, B\. Srinivasan, V\. Madathil, and J\. WuInternal representations as indicators of hallucinations in agent tool selection\.External Links:2601\.05214,[Link](https://arxiv.org/abs/2601.05214)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Huanget al\.\(2025a\)L\. Huang, W\. Yu, W\. Ma, W\. Zhong, Z\. Feng, H\. Wang, Q\. Chen, W\. Peng, X\. Feng, B\. Qin, and T\. LiuA survey on hallucination in large language models: principles, taxonomy, challenges, and open questions\.ACM Transactions on Information Systems43\(2\),pp\. 1–55\.External Links:[Document](https://dx.doi.org/10.1145/3703155),[Link](https://doi.org/10.1145/3703155)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Huanget al\.\(2025b\)S\. Huang, J\. He, S\. Huang, K\. R\. Anandan, A\. Chakraborty, and I\. LaneUCSC at SemEval\-2025 task 3: context, models and prompt optimization for automated hallucination detection in LLM output\.InProceedings of the 19th International Workshop on Semantic Evaluation \(SemEval\-2025\),S\. Rosenthal, A\. Rosá, D\. Ghosh, and M\. Zampieri \(Eds\.\),Vienna, Austria,pp\. 1981–1992\.External Links:[Link](https://aclanthology.org/2025.semeval-1.257/),ISBN 979\-8\-89176\-273\-2Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px3.p1.1)\.
- Kovács and Recski \(2025\)Á\. Kovács and G\. RecskiLettuceDetect: a hallucination detection framework for rag applications\.External Links:2502\.17125,[Link](https://arxiv.org/abs/2502.17125)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px1.p4.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px1.p1.1)\.
- Liet al\.\(2023\)J\. Li, X\. Cheng, X\. Zhao, J\. Nie, and J\. WenHaluEval: a large\-scale hallucination evaluation benchmark for large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 6449–6464\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.397/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.397)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p3.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 3214–3252\.External Links:[Link](https://aclanthology.org/2022.acl-long.229/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.229)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p2.1)\.
- Linet al\.\(2025\)X\. Lin, Y\. Ning, J\. Zhang, Y\. Dong, Y\. Liu, Y\. Wu, X\. Qi, N\. Sun, Y\. Shang, K\. Wang, P\. Cao, Q\. Wang, L\. Zou, X\. Chen, C\. Zhou, J\. Wu, P\. Zhang, Q\. Wen, S\. Pan, B\. Wang, Y\. Cao, K\. Chen, S\. Hu, and L\. GuoLLM\-based agents suffer from hallucinations: a survey of taxonomy, methods, and directions\.External Links:2509\.18970,[Link](https://arxiv.org/abs/2509.18970)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Liuet al\.\(2022\)T\. Liu, Y\. Zhang, C\. Brockett, Y\. Mao, Z\. Sui, W\. Chen, and B\. DolanA token\-level reference\-free hallucination detection benchmark for free\-form text generation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 6723–6737\.External Links:[Link](https://aclanthology.org/2022.acl-long.464/),[Document](https://dx.doi.org/10.18653/v1/2022.acl-long.464)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p2.1)\.
- Liuet al\.\(2020\)Y\. Liu, J\. Gu, N\. Goyal, X\. Li, S\. Edunov, M\. Ghazvininejad, M\. Lewis, and L\. ZettlemoyerMultilingual denoising pre\-training for neural machine translation\.Transactions of the Association for Computational Linguistics8,pp\. 726–742\.External Links:[Link](https://aclanthology.org/2020.tacl-1.47/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00343)Cited by:[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Manakulet al\.\(2023\)P\. Manakul, A\. Liusie, and M\. GalesSelfCheckGPT: zero\-resource black\-box hallucination detection for generative large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 9004–9017\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.557/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.557)Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p1.1)\.
- Maynezet al\.\(2020\)J\. Maynez, S\. Narayan, B\. Bohnet, and R\. McDonaldOn faithfulness and factuality in abstractive summarization\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 1906–1919\.External Links:[Link](https://aclanthology.org/2020.acl-main.173/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.173)Cited by:[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Minet al\.\(2023\)S\. Min, K\. Krishna, X\. Lyu, M\. Lewis, W\. Yih, P\. Koh, M\. Iyyer, L\. Zettlemoyer, and H\. HajishirziFActScore: fine\-grained atomic evaluation of factual precision in long form text generation\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 12076–12100\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.741/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.741)Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Mishraet al\.\(2024\)A\. Mishra, A\. Asai, V\. Balachandran, Y\. Wang, G\. Neubig, Y\. Tsvetkov, and H\. HajishirziFine\-grained hallucination detection and editing for language models\.InFirst Conference on Language Modeling,External Links:[Link](https://openreview.net/forum?id=dJMTn3QOWO)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.27370#S3.SS3.p1.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px2.p1.1)\.
- Narayanet al\.\(2018\)S\. Narayan, S\. B\. Cohen, and M\. LapataDon’t give me the details, just the summary\! topic\-aware convolutional neural networks for extreme summarization\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 1797–1807\.External Links:[Link](https://aclanthology.org/D18-1206/),[Document](https://dx.doi.org/10.18653/v1/D18-1206)Cited by:[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Niuet al\.\(2024\)C\. Niu, Y\. Wu, J\. Zhu, S\. Xu, K\. Shum, R\. Zhong, J\. Song, and T\. ZhangRAGTruth: a hallucination corpus for developing trustworthy retrieval\-augmented language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 10862–10878\.External Links:[Link](https://aclanthology.org/2024.acl-long.585/),[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.585)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px2.p2.1),[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p3.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.27370#S3.SS3.p1.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px2.p1.1)\.
- Ogasa and Arase \(2025\)Y\. Ogasa and Y\. AraseHallucinated span detection with multi\-view attention features\.InProceedings of the 14th Joint Conference on Lexical and Computational Semantics \(\*SEM 2025\),L\. Frermann and M\. Stevenson \(Eds\.\),Suzhou, China,pp\. 381–394\.External Links:[Link](https://aclanthology.org/2025.starsem-1.31/),[Document](https://dx.doi.org/10.18653/v1/2025.starsem-1.31),ISBN 979\-8\-89176\-340\-1Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p3.1)\.
- Opsahl\-Onget al\.\(2024\)K\. Opsahl\-Ong, M\. J\. Ryan, J\. Purtell, D\. Broman, C\. Potts, M\. Zaharia, and O\. KhattabOptimizing instructions and demonstrations for multi\-stage language model programs\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 9340–9366\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.525/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.525)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px3.p1.1)\.
- Rotheet al\.\(2020\)S\. Rothe, S\. Narayan, and A\. SeverynLeveraging pre\-trained checkpoints for sequence generation tasks\.Transactions of the Association for Computational Linguistics8,pp\. 264–280\.External Links:[Link](https://aclanthology.org/2020.tacl-1.18/),[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00313)Cited by:[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Rykovet al\.\(2025\)E\. Rykov, K\. Petrushina, M\. Savkin, V\. Olisov, A\. Vazhentsev, K\. Titova, A\. Panchenko, V\. Konovalov, and J\. BelikovaWhen models lie, we learn: multilingual span\-level hallucination detection with PsiloQA\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 11663–11682\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.626/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.626),ISBN 979\-8\-89176\-335\-7Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px1.p3.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p3.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px1.p1.1)\.
- Seeet al\.\(2017\)A\. See, P\. J\. Liu, and C\. D\. ManningGet to the point: summarization with pointer\-generator networks\.InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),R\. Barzilay and M\. Kan \(Eds\.\),Vancouver, Canada,pp\. 1073–1083\.External Links:[Link](https://aclanthology.org/P17-1099/),[Document](https://dx.doi.org/10.18653/v1/P17-1099)Cited by:[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Singhalet al\.\(2023\)K\. Singhal, S\. Azizi, T\. Tu, S\. S\. Mahdavi, J\. Wei, H\. W\. Chung, N\. Scales, A\. Tanwani, H\. Cole\-Lewis, S\. Pfohl, P\. Payne, M\. Seneviratne, P\. Gamble, C\. Kelly, N\. Scharli, A\. Chowdhery, P\. Mansfield, B\. A\. y\. Arcas, D\. Webster, G\. S\. Corrado, Y\. Matias, K\. Chou, J\. Gottweis, N\. Tomasev, Y\. Liu, A\. Rajkomar, J\. Barral, C\. Semturs, A\. Karthikesalingam, and V\. NatarajanLarge language models encode clinical knowledge\.Nature620\(7972\),pp\. 172–180\.External Links:[Document](https://dx.doi.org/10.1038/s41586-023-06291-2)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Sriramananet al\.\(2024\)G\. Sriramanan, S\. Bharti, V\. S\. Sadasivan, S\. Saha, P\. Kattakinda, and S\. FeiziLLM\-check: investigating detection of hallucinations in large language models\.InThe Thirty\-eighth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=LYx4w3CAgy)Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p3.1)\.
- Tanget al\.\(2023\)L\. Tang, T\. Goyal, A\. Fabbri, P\. Laban, J\. Xu, S\. Yavuz, W\. Kryscinski, J\. Rousseau, and G\. DurrettUnderstanding factual errors in summarization: errors, summarizers, datasets, error detectors\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 11626–11644\.External Links:[Link](https://aclanthology.org/2023.acl-long.650/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.650)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p2.1)\.
- Tanget al\.\(2024\)L\. Tang, P\. Laban, and G\. DurrettMiniCheck: efficient fact\-checking of LLMs on grounding documents\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 8818–8847\.External Links:[Link](https://aclanthology.org/2024.emnlp-main.499/),[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.499)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p2.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p3.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px1.p1.1)\.
- Tonmoyet al\.\(2024\)S\. M\. T\. I\. Tonmoy, S\. M\. M\. Zaman, V\. Jain, A\. Rani, V\. Rawte, A\. Chadha, and A\. DasA comprehensive survey of hallucination mitigation techniques in large language models\.External Links:2401\.01313,[Link](https://arxiv.org/abs/2401.01313)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p2.1)\.
- Vaswaniet al\.\(2017\)A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, L\. Kaiser, and I\. PolosukhinAttention is all you need\.InAdvances in Neural Information Processing Systems \(NeurIPS\),pp\. 5998–6008\.External Links:[Link](https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf)Cited by:[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Vazquezet al\.\(2025\)R\. Vazquez, T\. Mickus, E\. Zosa, T\. Vahtola, J\. Tiedemann, A\. Sinha, V\. Segonne, F\. Sanchez \- Vega, A\. Raganato, J\. Libovický, J\. Karlgren, S\. Ji, J\. Helcl, L\. Guillou, O\. De Gibert, J\. Bengoetxea, J\. Attieh, and M\. ApidianakiSemEval\-2025 task 3: mu\-SHROOM, the multilingual shared\-task on hallucinations and related observable overgeneration mistakes\.InProceedings of the 19th International Workshop on Semantic Evaluation \(SemEval\-2025\),S\. Rosenthal, A\. Rosá, D\. Ghosh, and M\. Zampieri \(Eds\.\),Vienna, Austria,pp\. 2472–2497\.External Links:[Link](https://aclanthology.org/2025.semeval-1.322/),ISBN 979\-8\-89176\-273\-2Cited by:[Appendix B](https://arxiv.org/html/2610.27370#A2.p2.1),[§1](https://arxiv.org/html/2610.27370#S1.p3.1),[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p3.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Wanget al\.\(2020\)A\. Wang, K\. Cho, and M\. LewisAsking and answering questions to evaluate the factual consistency of summaries\.InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics,D\. Jurafsky, J\. Chai, N\. Schluter, and J\. Tetreault \(Eds\.\),Online,pp\. 5008–5020\.External Links:[Link](https://aclanthology.org/2020.acl-main.450/),[Document](https://dx.doi.org/10.18653/v1/2020.acl-main.450)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p2.1)\.
- Warneret al\.\(2024\)B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. PoliSmarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.External Links:2412\.13663,[Link](https://arxiv.org/abs/2412.13663)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px1.p3.1),[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px1.p4.1)\.
- Wenet al\.\(2025\)X\. Wen, J\. Lou, X\. Lu, Y\. Ji, X\. Guan, Y\. Lu, H\. Lin, B\. He, X\. Han, D\. Zhang, and L\. SunOn\-policy self\-alignment with fine\-grained knowledge feedback for hallucination mitigation\.InFindings of the Association for Computational Linguistics: ACL 2025,W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 5215–5231\.External Links:[Link](https://aclanthology.org/2025.findings-acl.271/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-acl.271),ISBN 979\-8\-89176\-256\-5Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p2.1)\.
- Yehet al\.\(2025\)M\. Yeh, M\. Kamachee, S\. Park, and Y\. LiHalluEntity: benchmarking and understanding entity\-level hallucination detection\.Transactions on Machine Learning Research\.Note:External Links:ISSN 2835\-8856,[Link](https://openreview.net/forum?id=494k7e9R5D)Cited by:[Appendix B](https://arxiv.org/html/2610.27370#A2.p2.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1)\.
- Yuanet al\.\(2021\)W\. Yuan, G\. Neubig, and P\. LiuBARTScore: evaluating generated text as text generation\.InAdvances in Neural Information Processing Systems,A\. Beygelzimer, Y\. Dauphin, P\. Liang, and J\. W\. Vaughan \(Eds\.\),External Links:[Link](https://openreview.net/forum?id=5Ya8PbvpZ9)Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p1.1)\.
- Zhaet al\.\(2023\)Y\. Zha, Y\. Yang, R\. Li, and Z\. HuAlignScore: evaluating factual consistency with a unified alignment function\.InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),A\. Rogers, J\. Boyd\-Graber, and N\. Okazaki \(Eds\.\),Toronto, Canada,pp\. 11328–11348\.External Links:[Link](https://aclanthology.org/2023.acl-long.634/),[Document](https://dx.doi.org/10.18653/v1/2023.acl-long.634)Cited by:[Appendix A](https://arxiv.org/html/2610.27370#A1.SS0.SSS0.Px1.p2.1),[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p2.1),[§4](https://arxiv.org/html/2610.27370#S4.SS0.SSS0.Px1.p1.1)\.
- Zhanget al\.\(2025\)Y\. Zhang, Y\. Li, L\. Cui, D\. Cai, L\. Liu, T\. Fu, X\. Huang, E\. Zhao, Y\. Zhang, Y\. Chen, L\. Wang, A\. T\. Luu, W\. Bi, F\. Shi, and S\. ShiSiren’s song in the ai ocean: a survey on hallucination in large language models\.Computational Linguistics51\(4\),pp\. 1373–1418\.External Links:[Document](https://dx.doi.org/10.1162/COLI.a.16)Cited by:[§1](https://arxiv.org/html/2610.27370#S1.p1.1)\.
- Zhang\*et al\.\(2020\)T\. Zhang\*, V\. Kishore\*, F\. Wu\*, K\. Q\. Weinberger, and Y\. ArtziBERTScore: evaluating text generation with bert\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkeHuCVFDr)Cited by:[§2\.1](https://arxiv.org/html/2610.27370#S2.SS1.p1.1)\.
- Zhouet al\.\(2021\)C\. Zhou, G\. Neubig, J\. Gu, M\. Diab, F\. Guzmán, L\. Zettlemoyer, and M\. GhazvininejadDetecting hallucinated content in conditional neural sequence generation\.InFindings of the Association for Computational Linguistics: ACL\-IJCNLP 2021,C\. Zong, F\. Xia, W\. Li, and R\. Navigli \(Eds\.\),Online,pp\. 1393–1404\.External Links:[Link](https://aclanthology.org/2021.findings-acl.120/),[Document](https://dx.doi.org/10.18653/v1/2021.findings-acl.120)Cited by:[§2\.2](https://arxiv.org/html/2610.27370#S2.SS2.p3.1),[§3\.1](https://arxiv.org/html/2610.27370#S3.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.27370#S3.SS3.p1.1)\.

## Appendix AHallucination Detection Method Details

#### Discriminatively trained methods\.

MiniCheck\([Tang et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib17)\)frames hallucination detection as*evidence\-grounded claim verification*: given a generated sentence and supporting context, it predicts whether the sentence is supported\. The models are trained with two synthetic data pipelines: Claim\-to\-Doc \(C2D\), which decomposes claims into atomic facts and builds multi\-sentence evidence, and Doc\-to\-Claim \(D2C\), which derives summaries as claims from real documents and constructs unsupported pairs by removing sentences from the original document\. This yields sentence\-level fact\-checking models\.

AlignScore\([Zha et al\., 2023](https://arxiv.org/html/2610.27370#bib.bib2)\)frames hallucination detection as an*information alignment*problem: for a text pair\(a,b\)\(a,b\), a unified alignment functionf:\(a,b\)→yf:\(a,b\)\\rightarrow yestimates whether all information inbbis present in, and non\-contradictory with,aa\. The model is trained with multi\-task learning on 15 datasets spanning seven NLP task families, and training is further augmented with synthetic positives from back translation and synthetic negatives from masking/infilling\. At inference time, it splits the output into sentence\-level claims and chunks evidence into segments, computes the alignment score of each claim against all evidence chunks, and aggregates these scores into an overall score; sentences with low alignment are treated as likely hallucinated\.

PsiloQA\([Rykov et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib24)\)is an encoder token\-level hallucination detector based onModernBERT\([Warner et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib36)\)\. It takes in context–question–response triples and predicts a binary classification label for each response token\. The model is trained on the English subset of the PsiloQA\([Rykov et al\., 2025](https://arxiv.org/html/2610.27370#bib.bib24)\)training data, which is synthetically generated from a three\-stage pipeline: creating question\-answer pairs from Wikipedia, eliciting potentially hallucinated answers from diverse LLMs in a no\-context setting, and automatically annotating hallucinated spans by comparing responses to the golden answers and retrieved context using GPT\-4o\.

LettuceDetect\([Kovács and Recski, 2025](https://arxiv.org/html/2610.27370#bib.bib1)\)is a token\-level hallucination detection model built onModernBERT\([Warner et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib36)\)\. Similar to PsiloQA, it processes concatenated context–question–response triples and predicts whether each response token is hallucinated\. In contrast to PsiloQA, LettuceDetect is trained using RAGTruth’s dataset, which is labeled by human annotators\.

#### Generatively trained methods\.

FAVA\([Mishra et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib9)\)frames hallucination detection as a generation problem\. Given an LLM response and retrieved evidence, the model generates the response with markers that wrap unsupported content\. FAVA is trained on synthetic data generated by first creating factually grounded seed passages from Wikipedia and then injecting errors while marking corrupted spans\. An LLM is trained on the resulting input–output pairs \(erroneous input \+ evidence→\\rightarrowmarked output\)\. Hallucinated spans are identified by marking them as unsupported content during generation\.

RAGTruth\([Niu et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib3)\)adopts a simpler generative approach by fine\-tuning an LLM to directly output hallucinated text spans when given a context–response pair\. The LLM is fine\-tuned on a large\-scale training set \(∼\\sim18,000 examples\) including context\_qa, data2text, and summarization tasks, which consists of human\-annotated span\-level labels of hallucinated content\.

#### Prompt\-based methods\.

We also evaluate prompt\-based detection methods that treat an LLM as a direct hallucination span extractor\. Following[Huang et al\. \(2025b\)](https://arxiv.org/html/2610.27370#bib.bib18), given a task question, evidence context, and generated response, the prompt asks the LLM to compare the response against the evidence and output hallucinated text as explicit spans, i\.e\. substrings of the response\. This approach requires no additional model training while still producing span\-level outputs suitable for SHAM\-Bench evaluation\. Beyond a hand\-crafted instruction prompt, we apply prompt optimization\([Huang et al\., 2025b](https://arxiv.org/html/2610.27370#bib.bib18)\)using MiPROv2\([Opsahl\-Ong et al\., 2024](https://arxiv.org/html/2610.27370#bib.bib19)\)to maximizeF1F\_\{1\}scores\.

## Appendix BMetric Choice

F1score is intentionally simple, but it is well aligned with the span\-detection task evaluated in SHAM\-Bench\. Because gold annotations and model predictions are represented as sets of hallucinated response tokens, precision measures how many predicted hallucinated tokens are correct and recall measures how many gold hallucinated tokens are identified\. F1combines these quantities into a single positive\-class overlap score while avoiding the inflated performance that can arise from metrics dominated by the much larger set of non\-hallucinated tokens\.

Other overlap\-based metrics would capture closely related information\. For example,[Vazquez et al\. \(2025\)](https://arxiv.org/html/2610.27370#bib.bib6)uses intersection\-over\-union \(IoU\); for a fixed predicted token set and gold token set, IoU and F1are monotonic transformations of the same overlap counts, so they are expected to produce similar rankings\. Threshold\-independent metrics such as AUPRC and AUROC used in[Yeh et al\. \(2025\)](https://arxiv.org/html/2610.27370#bib.bib4)can be useful when systems output calibrated confidence scores, but several SHAM\-Bench systems, including prompt\-based and generative span\-extraction methods, output discrete spans without comparable scores\. Balanced accuracy and macro\-F1that provide balanced weighting of positive and negative classes reported in[Bao et al\. \(2025\)](https://arxiv.org/html/2610.27370#bib.bib5)are also less directly aligned with our goal of measuring detection quality for the positive hallucinated\-token class\. We therefore use token\-level F1as the primary span\-level metric and report sentence\-level F1as a complementary view when methods operate at sentence granularity\.

## Appendix CFailure Category Definitions

We classify missed ground\-truth spans into five categories for the prompt\-based failure analysis in Section[6](https://arxiv.org/html/2610.27370#S6)\. These categories are intended to distinguish likely annotation artifacts from substantive detector failures, and to separate different types of unsupported content\.

#### Labeling error\.

This category covers cases in which the missed “ground\-truth hallucination” is not clearly a detector failure because the gold annotation itself is questionable\. The annotated span may be directly supported by the source, may constitute an acceptable paraphrase or modest inference, or may reflect an annotation\-boundary artifact, such as a fragment, duplicate span, discourse marker, or non\-factual token\. In such cases, the detector may have appropriately ignored the span or identified the substantive hallucination with a different boundary\. This category therefore captures annotation\-side noise rather than a clear failure to detect unsupported content\.

#### Unsupported detail\.

This category covers genuine hallucination misses in which the response introduces concrete information that is not supported by the source\. The unsupported content may be an added attribute, procedural step, recommendation, purpose, causal explanation, exact descriptor, or schema\-derived claim\. It also includes cases in which missing or null metadata is converted into an affirmative factual assertion, such as treating an unknown ambience field as evidence that a restaurant is “casual” or treating an unspecified amenity as absent\. The defining property is that the response adds a specific fact\-like detail that is not licensed by the available evidence\.

#### Overgeneralization\.

Overgeneralization errors occur when the response remains loosely connected to the source but changes the strength, scope, certainty, or characterization of the supported claim\. The response may turn narrow evidence into a broader rule, extrapolate from a single example to a general conclusion, or replace a source\-supported description with a stronger or subtly different one\. Unlike unsupported\-detail errors, these cases often do not introduce a wholly fabricated fact; instead, they overextend or semantically drift from what the source establishes\.

#### Misattribution\.

This category covers cases in which content may be related to the available evidence but is assigned to the wrong source, entity, task direction, comparison frame, status, or evidential basis\. For example, a response may claim that a detail comes from a particular passage when it does not, merge information from separate cases, apply facts about one entity to another, or answer the reverse of the intended task\. The central error is therefore not merely a lack of support, but the placement of otherwise plausible information in the wrong evidential or contextual frame\.

#### Subjective language\.

This category covers unsupported evaluative, promotional, sentimental, or ambience\-related wording\. The response may add judgments such as “friendly,” “cozy,” “popular,” “beautiful,” or other claims about reception, service quality, or atmosphere that are not grounded in the source\. Although these spans may sound plausible and are often softer than concrete factual fabrications, they still introduce unsupported characterizations\. We separate this category because such language reflects a distinct form of embellishment rather than a specific factual addition\.

Figure 3:Distribution of over\-prediction lengths for overlapping spans predicted by prompting GPT\-5\-mini with reasoning, grouped by dataset and query type\. Negative values indicate under\-prediction\.

## Appendix DFull Experiment Results

The complete prompt\-based method results with sentence\-level F1scores is given in Table[5](https://arxiv.org/html/2610.27370#A4.T5)\. The full set of experimental results for span\-level labeling with dataset breakdown can be found in Table[6](https://arxiv.org/html/2610.27370#A4.T6), sentence\-level results in Table[7](https://arxiv.org/html/2610.27370#A4.T7), span\-level precision/recall results in Table[8](https://arxiv.org/html/2610.27370#A4.T8), and sentence\-level precision/recall results in Table[9](https://arxiv.org/html/2610.27370#A4.T9)\.

MethodReasonOptOverall \(std\)open\_qactx\_qadata2txtopen\_biosummmtSpan\-levelprompt \(GPT\-5\-mini\)✗✗0\.425 \(0\.09\)0\.3860\.5530\.3210\.4860\.4710\.331prompt \(GPT\-5\-mini\)✓✗0\.498\(0\.07\)0\.4020\.6030\.5280\.5250\.4950\.435prompt \(GPT\-5\-mini\)✓✓0\.497\(0\.09\)0\.3640\.6360\.5360\.4760\.5320\.437prompt \(Qwen3\-4B\)✗✗0\.298 \(0\.07\)0\.3430\.1760\.2550\.3620\.3060\.347prompt \(Qwen3\-4B\)✓✗0\.442 \(0\.11\)0\.3690\.6470\.5110\.3280\.4570\.338prompt \(Qwen3\-4B\)✓✓0\.446 \(0\.15\)0\.3290\.7480\.4900\.3280\.4620\.316prompt \(Olmo3\-7B\-Inst\)✗✗0\.383 \(0\.14\)0\.3220\.6700\.3460\.2610\.4220\.278prompt \(Olmo3\-7B\-Think\)✓✗0\.425 \(0\.07\)0\.3550\.5390\.4950\.3840\.4420\.338prompt \(Olmo3\-7B\-Think\)✓✓0\.435 \(0\.16\)0\.3480\.7900\.3600\.3170\.4510\.343Sentence\-levelprompt \(GPT\-5\-mini\)✗✗0\.641 \(0\.11\)0\.6200\.5610\.4470\.7720\.7050\.740prompt \(GPT\-5\-mini\)✓✗0\.691\(0\.07\)0\.6330\.6100\.6350\.7840\.7260\.759prompt \(GPT\-5\-mini\)✓✓0\.705\(0\.06\)0\.6310\.6400\.6620\.7870\.7310\.779prompt \(Qwen3\-4B\)✗✗0\.504 \(0\.16\)0\.5570\.1970\.3910\.6150\.5690\.693prompt \(Qwen3\-4B\)✓✗0\.631 \(0\.06\)0\.5730\.6590\.6250\.5410\.6670\.720prompt \(Qwen3\-4B\)✓✓0\.630 \(0\.09\)0\.5600\.7580\.5910\.5240\.6100\.736prompt \(Olmo3\-7B\-Inst\)✗✗0\.559 \(0\.12\)0\.5300\.6750\.3540\.4840\.6120\.697prompt \(Olmo3\-7B\-Think\)✓✗0\.620 \(0\.05\)0\.5660\.5540\.6160\.6090\.6730\.699prompt \(Olmo3\-7B\-Think\)✓✓0\.587 \(0\.14\)0\.5530\.7900\.3620\.5000\.6140\.702Table 5:Average span\- and sentence\-level F1scores of prompt\-based hallucination detection methods on SHAM\-Bench aggregated by task\.Reasonandoptindicate whether reasoning and prompt optimization are used\. Theoverallcolumn shows the unweighted mean and standard deviation of six task scores\. Highest score in each column inboldand the second highest score isunderlined\.MethodReasoningPromptOptimizationcontext\_qadata2txtmtopen\_bioopen\_qasummragtruthragtruthzhouhalluentityfavabenchmushroomfaithbenchragtruthzhouPsiloQA \(ModernBERT\-base\)––0\.0720\.0890\.2160\.3630\.2100\.5560\.2250\.0530\.547LettuceDetect \(lettuce\-large\)––0\.8330\.6140\.3820\.1980\.2420\.4630\.3070\.6950\.559RAGTruth \(ft\-llama2\-13B\)––0\.8650\.6540\.3980\.3540\.3270\.4350\.3230\.8350\.357FAVA \(ft\-llama2\-7B\)––0\.6950\.2430\.4110\.2440\.2350\.4630\.3100\.7610\.228Prompt \(GPT\-5\-mini\)✗✗0\.5530\.3210\.3310\.4860\.2610\.5970\.3530\.4970\.504Prompt \(GPT\-5\-mini\)✓✗0\.6030\.5280\.4350\.5250\.2600\.6420\.3870\.5410\.514Prompt \(GPT\-5\-mini\)✓✓0\.6360\.5360\.4370\.4760\.2710\.5200\.4220\.5750\.554Prompt \(Qwen3\-4B\)✗✗0\.1760\.2550\.3470\.3620\.1800\.6200\.1780\.1760\.413Prompt \(Qwen3\-4B\)✓✗0\.6470\.5110\.3380\.3280\.2340\.5970\.3330\.7200\.385Prompt \(Qwen3\-4B\)✓✓0\.7480\.4900\.3160\.3280\.2620\.4420\.3530\.7530\.372Prompt \(Olmo\-3\-7B\-Instruct\)✗✗0\.6700\.3460\.2770\.2610\.1990\.5320\.2510\.5920\.409Prompt \(Olmo\-3\-7B\-Think\)✓✗0\.5380\.4950\.3380\.3840\.2140\.5950\.3270\.5940\.416Prompt \(Olmo\-3\-7B\-Think\)✓✓0\.7900\.3600\.3430\.3170\.2750\.4710\.3290\.7460\.363Table 6:Average span\-level F1of hallucination detection methods on SHAM\-Bench grouped by task and dataset\. Highest score in each column inboldand the second highest score isunderlined\.
MethodReasoningPromptOptimizationcontext\_qadata2txtmtopen\_bioopen\_qasummragtruthragtruthzhouhalluentityfavabenchmushroomfaithbenchragtruthzhouMiniCheck \(Bespoke\-7B\)––0\.4660\.4930\.7150\.7770\.4130\.9260\.3780\.7300\.875AlignScore \(roberta\-large\)––0\.4920\.2900\.7340\.7450\.4280\.9150\.3480\.6260\.864PsiloQA \(ModernBERT\-base\)––0\.0940\.2140\.6710\.7540\.3790\.8980\.3890\.0920\.906LettuceDetect \(lettuce\-large\)––0\.8420\.7160\.6250\.3420\.4100\.8850\.3630\.7130\.845RAGTruth \(ft\-llama2\-13B\)––0\.8690\.7520\.5070\.5330\.4370\.8190\.3490\.8450\.566FAVA \(ft\-llama2\-7B\)––0\.7010\.2680\.7280\.4350\.3920\.7880\.3340\.7730\.570Prompt \(GPT\-5\-mini\)✗✗0\.5610\.4470\.7400\.7720\.4450\.9170\.4540\.5220\.882Prompt \(GPT\-5\-mini\)✓✗0\.6100\.6350\.7590\.7840\.4500\.9440\.4930\.5690\.885Prompt \(GPT\-5\-mini\)✓✓0\.6400\.6620\.7790\.7870\.4470\.9430\.4920\.6030\.878Prompt \(Qwen3\-4B\)✗✗0\.1970\.3910\.6930\.6150\.3700\.8750\.3410\.2150\.813Prompt \(Qwen3\-4B\)✓✗0\.6590\.6250\.7200\.5410\.4000\.8670\.3910\.7440\.735Prompt \(Qwen3\-4B\)✓✓0\.7580\.5910\.7360\.5240\.4140\.8080\.3910\.7740\.618Prompt \(Olmo\-3\-7B\-Instruct\)✗✗0\.6750\.3540\.6970\.4840\.3540\.8270\.3340\.6050\.719Prompt \(Olmo\-3\-7B\-Think\)✓✗0\.5540\.6160\.6990\.6090\.4020\.8430\.4240\.6180\.790Prompt \(Olmo\-3\-7B\-Think\)✓✓0\.7900\.3610\.7010\.5000\.4140\.7880\.3820\.7570\.636Table 7:Average sentence\-level F1of hallucination detection methods on SHAM\-Bench grouped by task and dataset\. Highest score in each column is inboldand the second highest score isunderlined\.
MethodReasoningPromptOptimizationcontext\_qadata2txtmtopen\_bioopen\_qasummragtruthragtruthzhouhalluentityfavabenchmushroomfaithbenchragtruthzhouPsiloQA \(ModernBERT\-base\)––0\.05 / 0\.170\.05 / 0\.580\.15 / 0\.650\.24 / 0\.990\.16 / 0\.620\.46 / 0\.920\.16 / 0\.620\.03 / 0\.220\.47 / 0\.89LettuceDetect \(lettuce\-large\)––0\.84 / 0\.840\.63 / 0\.670\.36 / 0\.530\.16 / 0\.310\.22 / 0\.450\.37 / 0\.880\.35 / 0\.310\.70 / 0\.720\.52 / 0\.75RAGTruth \(ft\-llama2\-13B\)––0\.87 / 0\.870\.72 / 0\.670\.42 / 0\.430\.39 / 0\.410\.34 / 0\.460\.37 / 0\.760\.35 / 0\.320\.85 / 0\.840\.40 / 0\.42FAVA \(ft\-llama2\-7B\)––0\.71 / 0\.690\.26 / 0\.250\.43 / 0\.470\.46 / 0\.230\.27 / 0\.350\.49 / 0\.630\.34 / 0\.310\.77 / 0\.760\.40 / 0\.22Prompt \(GPT\-5\-mini\)✗✗0\.55 / 0\.580\.27 / 0\.580\.26 / 0\.660\.42 / 0\.670\.26 / 0\.410\.51 / 0\.900\.40 / 0\.370\.49 / 0\.550\.50 / 0\.65Prompt \(GPT\-5\-mini\)✓✗0\.61 / 0\.620\.51 / 0\.640\.38 / 0\.660\.49 / 0\.640\.26 / 0\.420\.55 / 0\.940\.51 / 0\.360\.55 / 0\.560\.55 / 0\.61Prompt \(GPT\-5\-mini\)✓✓0\.63 / 0\.660\.50 / 0\.710\.37 / 0\.700\.40 / 0\.710\.23 / 0\.510\.41 / 0\.960\.51 / 0\.400\.57 / 0\.620\.51 / 0\.76Prompt \(Qwen3\-4B\)✗✗0\.21 / 0\.180\.32 / 0\.280\.31 / 0\.550\.58 / 0\.320\.28 / 0\.220\.75 / 0\.630\.32 / 0\.160\.20 / 0\.180\.51 / 0\.46Prompt \(Qwen3\-4B\)✓✗0\.66 / 0\.650\.56 / 0\.550\.28 / 0\.590\.45 / 0\.310\.31 / 0\.280\.65 / 0\.720\.40 / 0\.320\.73 / 0\.730\.47 / 0\.42Prompt \(Qwen3\-4B\)✓✓0\.76 / 0\.750\.51 / 0\.530\.25 / 0\.630\.36 / 0\.360\.30 / 0\.340\.40 / 0\.760\.39 / 0\.350\.76 / 0\.770\.37 / 0\.49Prompt \(Olmo\-3\-7B\-Instruct\)✗✗0\.67 / 0\.680\.35 / 0\.360\.22 / 0\.570\.31 / 0\.390\.21 / 0\.330\.52 / 0\.730\.28 / 0\.290\.59 / 0\.600\.42 / 0\.55Prompt \(Olmo\-3\-7B\-Think\)✓✗0\.56 / 0\.540\.57 / 0\.510\.29 / 0\.540\.53 / 0\.360\.29 / 0\.250\.64 / 0\.710\.43 / 0\.320\.61 / 0\.590\.51 / 0\.46Prompt \(Olmo\-3\-7B\-Think\)✓✓0\.79 / 0\.790\.36 / 0\.360\.29 / 0\.600\.39 / 0\.320\.33 / 0\.320\.44 / 0\.720\.40 / 0\.320\.75 / 0\.750\.40 / 0\.43Table 8:Average span\-level precision/recall \(P/R\) of hallucination detection methods on SHAM\-Bench grouped by task and dataset\. Each cell reports precision followed by recall, rounded to two decimal places\.
MethodReasoningPromptOptimizationcontext\_qadata2txtmtopen\_bioopen\_qasummragtruthragtruthzhouhalluentityfavabenchmushroomfaithbenchragtruthzhouMiniCheck \(Bespoke\-7B\)––0\.45 / 0\.520\.43 / 0\.670\.71 / 0\.730\.84 / 0\.770\.35 / 0\.640\.92 / 0\.960\.42 / 0\.370\.72 / 0\.750\.88 / 0\.88AlignScore \(roberta\-large\)––0\.48 / 0\.530\.25 / 0\.430\.74 / 0\.740\.79 / 0\.760\.36 / 0\.670\.91 / 0\.940\.38 / 0\.340\.62 / 0\.640\.86 / 0\.86PsiloQA \(ModernBERT\-base\)––0\.07 / 0\.180\.13 / 0\.640\.66 / 0\.690\.65 / 1\.000\.29 / 0\.710\.88 / 0\.950\.30 / 0\.680\.06 / 0\.230\.91 / 0\.91LettuceDetect \(lettuce\-large\)––0\.84 / 0\.850\.70 / 0\.770\.63 / 0\.620\.37 / 0\.340\.36 / 0\.590\.89 / 0\.910\.39 / 0\.360\.71 / 0\.720\.84 / 0\.84RAGTruth \(ft\-llama2\-13B\)––0\.88 / 0\.870\.78 / 0\.750\.51 / 0\.510\.78 / 0\.450\.45 / 0\.520\.85 / 0\.810\.36 / 0\.340\.85 / 0\.840\.57 / 0\.57FAVA \(ft\-llama2\-7B\)––0\.70 / 0\.700\.28 / 0\.270\.73 / 0\.730\.72 / 0\.340\.40 / 0\.490\.85 / 0\.760\.35 / 0\.330\.78 / 0\.770\.57 / 0\.57Prompt \(GPT\-5\-mini\)✗✗0\.56 / 0\.590\.37 / 0\.670\.73 / 0\.750\.82 / 0\.770\.42 / 0\.570\.92 / 0\.940\.46 / 0\.480\.50 / 0\.570\.88 / 0\.88Prompt \(GPT\-5\-mini\)✓✗0\.61 / 0\.630\.58 / 0\.770\.75 / 0\.770\.84 / 0\.780\.41 / 0\.610\.94 / 0\.970\.55 / 0\.470\.55 / 0\.610\.88 / 0\.88Prompt \(GPT\-5\-mini\)✓✓0\.64 / 0\.660\.62 / 0\.790\.78 / 0\.790\.86 / 0\.770\.39 / 0\.630\.94 / 0\.960\.55 / 0\.470\.59 / 0\.630\.88 / 0\.88Prompt \(Qwen3\-4B\)✗✗0\.21 / 0\.200\.42 / 0\.420\.69 / 0\.700\.81 / 0\.530\.44 / 0\.410\.94 / 0\.850\.41 / 0\.330\.21 / 0\.230\.81 / 0\.81Prompt \(Qwen3\-4B\)✓✗0\.67 / 0\.660\.65 / 0\.640\.72 / 0\.730\.81 / 0\.450\.48 / 0\.420\.94 / 0\.840\.43 / 0\.380\.75 / 0\.750\.74 / 0\.73Prompt \(Qwen3\-4B\)✓✓0\.77 / 0\.760\.62 / 0\.590\.73 / 0\.750\.78 / 0\.430\.47 / 0\.460\.88 / 0\.790\.43 / 0\.370\.78 / 0\.780\.62 / 0\.62Prompt \(Olmo\-3\-7B\-Instruct\)✗✗0\.68 / 0\.680\.35 / 0\.360\.69 / 0\.710\.62 / 0\.470\.35 / 0\.450\.88 / 0\.830\.35 / 0\.350\.60 / 0\.610\.72 / 0\.72Prompt \(Olmo\-3\-7B\-Think\)✓✗0\.56 / 0\.560\.62 / 0\.650\.69 / 0\.710\.80 / 0\.530\.45 / 0\.440\.93 / 0\.810\.47 / 0\.410\.62 / 0\.620\.79 / 0\.79Prompt \(Olmo\-3\-7B\-Think\)✓✓0\.79 / 0\.790\.36 / 0\.360\.70 / 0\.710\.77 / 0\.400\.48 / 0\.440\.87 / 0\.760\.42 / 0\.370\.76 / 0\.760\.64 / 0\.64Table 9:Average sentence\-level precision/recall \(P/R\) of hallucination detection methods on SHAM\-Bench grouped by task and dataset\. Each cell reports precision followed by recall, rounded to two decimal places\.

## Appendix EExample Labeled Outputs

QuestionSummarize the following textContextLake Providence is a town in and the parish seat of East Carroll Parish in northeastern Louisiana, United States\. University of Mississippi\. About 55 percent of its undergraduates and 60 percent overall come from Mississippi, and 23 percent are minorities; international students come from 90 nations\.Response with gold annotated spansLake Providence is a town in Louisiana that is the parish seat of East Carroll Parish\. The University of Mississippi has a student body that is55% undergraduateand 60% overall from Mississippi\. 23% of the student body are minorities and international students come from 90 nations\.Response with predicted spansLake Providence is a town in Louisiana that is the parish seat of East Carroll Parish\. The University of Mississippihas a student body that is 55% undergraduateand 60% overall from Mississippi\. 23% of the student body are minorities and international students come from 90 nations\.MetricsSpan: F1=0\.5000, Precision=0\.3333, Recall=1\.0000Sentence: F1=1\.0000, Precision=1\.0000, Recall=1\.0000

Figure 4:Example FaithBench summarization output labeled by promptedGPT\-5\-miniwith reasoning\. The box shows the task question, grounding context, model response, gold hallucinated spans, model predicted spans, and the resulting metrics\.QuestionTranslate the following text from Chinese to English:附连插口10具有两个保持段11和12（图6显示了它们的截面），两个升离斜坡13，以及在保持段11和升离斜坡13之间的接合处的旋转止动部14。Contextattachment socket 10 has two retaining segments 11 and 12 whose cross\-section is shown in fig\. 6, two lift\-off ramps 13 and a rotational stop 14 at the junction between retaining segment 11 and a lift\-off ramp 13\.Response with gold annotated spansthe attached plug 10 has two maintaining sections 11 and 12\(图\) 6 shows their intersections, two uphill slopes 13, and a rotating stop unit 14 in the joints between the maintaining sections 11 and uphill slopes 13\.Response with predicted spanstheattached plug 10has two maintaining sections 11 and 12 \(图 6 shows their intersections\),two uphill slopes 13, and a rotating stop unit 14in the joints between the maintaining sections 11 and uphill slopes 13\.MetricsSpan: F1=0\.0769, Precision=0\.0417, Recall=0\.5000Sentence: F1=1\.0000, Precision=1\.0000, Recall=1\.0000

Figure 5:Example Zhou machine\-translation output labeled by promptedGPT\-5\-miniwith reasoning\. The box shows the task question, grounding context, model response, gold hallucinated spans, model predicted spans, and the resulting metrics\.Questionhow to factory reset a htc one vxContextpassage 1: 1 In order to switch off the device, hold down the Power button for a while\. 2 Then press and hold the Volume Down \+ Power key for a couple of seconds\. 3 When you see Bootloader Mode menu release held buttons\. 4 From the menu choose option RECOVERY MODE by using Volume buttons to scroll and Power key to select options\. 4 Congratulations\! 5 You should be right now in the expected mode\.passage 2: 1 First of all, switch off the phone by using the Power rocker\. 2 After that, press and hold Volume Down and Power buttons together for a couple of seconds\. 3 You can release all keys when the Bootloader Mode appears on the screen\. 4 Choose the Factory Reset by pressing the Volume Down button to scroll and Power key to select\.passage 3: Be sure to back up any data and files you want to keep before you do a factory reset\. Hard reset your HTC One VX as follows: 1 Slide the Notifications panel open\. 2 Tap Storage\. 3 Tap Factory data reset\. 4 Tap Reset phone, and then tap Erase everything\.Response with gold annotated spansBased on the provided passages, here are the steps to factory reset an HTC One VX:1\. Slide the Notifications panel open\.2\. Tap on Storage\.3\. Tap on Factory data reset\.4\. Tap on Reset phone\.5\. Tap on Erase everything\.Alternatively, you can also perform a factory reset by following these steps:1\. Hold down the Power button until the device shuts down\.2\. Press and hold the Volume Down and Power buttons together for a few seconds until the bootloader mode appears\.3\. Use the Volume down button to scroll to “Recovery Mode” and press the Power button to select\.4\. From the recovery menu, select “wipe data/factory reset” and press the Power button to confirm\.5\. Wait for the process to complete\.6\. Once the reset is done, select “Reboot System” from the recovery menu and press the Power button to restart the device\.Please note that performing a factory reset will erase all data on the device, so make sure to back up any important data before doing so\.Response with predicted spansBased on the provided passages, here are the steps to factory reset an HTC One VX:1\. Slide the Notifications panel open\.2\. Tap on Storage\.3\. Tap on Factory data reset\.4\. Tap on Reset phone\.5\. Tap on Erase everything\.Alternatively, you can also perform a factory reset by following these steps:1\. Hold down the Power button until the device shuts down\.2\. Press and hold the Volume Down and Power buttons together for a few seconds until the bootloader mode appears\.3\. Use the Volume down button to scroll to “Recovery Mode” and press the Power button to select\.4\.From the recovery menu, select “wipe Data/Factory Reset” and press the Power button to confirm\.5\.Wait for the process to complete\.6\.Once the reset is done, select “Reboot System” from the recovery menu and press the Power button to restart the device\.Please note that performing a factory reset will erase all data on the device, so make sure to back up any important data before doing so\.MetricsSpan: F1=0\.6750, Precision=0\.5294, Recall=0\.9310Sentence: F1=0\.6667, Precision=0\.6667, Recall=0\.6667

Figure 6:Example RAGTruth context\_qa output labeled by promptedGPT\-5\-miniwith reasoning\. The box shows the task question, grounding passages, model response, gold hallucinated spans, model predicted spans, and the resulting metrics\.
## Appendix FPrompt Templates

System prompt

Based on the provided information, identify incorrect spans in the given answer text\.You will be provided with a question with some related information\. You will also be given an answer to the question\. Your task is to identify any specific parts of the answer that describes facts that are not supported by the provided information\. If there are multiple incorrect segments, report each one separately\.\# Steps1\. \*\*Read the Question and Information\*\*: Carefully read the provided question and information\.2\. \*\*Analyze the Answer\*\*: Carefully evaluate the given answer for accuracy regarding the question and the provided information\.3\. \*\*Identify Incorrect Spans\*\*: Mark the parts of the text that are incorrect, misleading, or irrelevant to the question and context provided in the question\.\# Output FormatThe output should be in JSON format as shown below:\`\`\`json\{"incorrect\_spans": \["identified incorrect span","another identified incorrect span"\]\}\`\`\`If no incorrect spans are identified, return an empty list:\`\`\`json\{"incorrect\_spans": \[\]\}\`\`\`\# Example 1\*\*Input\*\*:<question\>What is the capital of France?Answer the question based on the following information:Paris, the capital city of France, is a metropolis steeped in history, culture, and global significance\. This comprehensive analysis will delve into the city’s current status, basic information, and historical importance, providing a thorough understanding of why Paris is not just the capital of France, but also one of the world’s most influential cities\.</question\><answer\>The capital of France is Berlin\.</answer\>\*\*Output\*\*:\`\`\`json\{"incorrect\_spans": \["Berlin"\]\}\`\`\`\# Example 2\*\*Input\*\*:<question\>Answer the question: "What did Mara do?" based on the following paragraph:The train arrived at the platform without a sound, its windows glowing softly as if lit by bottled moonlight\. Mara hesitated before stepping aboard, aware that everyone else seemed to know exactly where they were going, while she alone carried a ticket with no destination printed on it\. As the doors closed, the city outside blurred into a ribbon of color and memory, and she realized with a quiet thrill that whatever waited at the end of the line would not be something she could ever return from\-\-only something she could become\.</question\><answer\>Mara got off a train that arrived without a sound while holding a ticket to Copenhagen\.</answer\>\*\*Output\*\*:\`\`\`json\{"incorrect\_spans": \["got off","Copenhagen"\]\}\`\`\`\# Example 3\*\*Input\*\*:<question\>Translate the following text from Chinese to English:请去帮我买点零食。The reference translation is:Please go and buy me buy some snacks\.</question\><answer\>Please help me buy groceries\.</answer\>\*\*Output\*\*:\`\`\`json\{"incorrect\_spans": \["groceries"\]\}\`\`\`\# Example 4\*\*Input\*\*:<question\>Summarize the following news:The Riverton City Council voted unanimously Tuesday night to approve a six\-month pilot program that will introduce late\-night bus service across five major routes\. The program, scheduled to begin in March, is intended to support shift workers, expand transportation access, and reduce nighttime traffic incidents linked to fatigue and impaired driving\.Under the plan, buses will run every 30 minutes between midnight and 4 a\.m\., connecting downtown Riverton with several residential neighborhoods and the industrial district on the city’s east side\. Transportation Director Lena Morales said the routes were selected based on employment data, hospital locations, and feedback from residents collected during a public survey last fall\.Funding for the pilot will come from a combination of state transit grants and unused funds from a postponed road expansion project\. City officials emphasized that ridership levels, operating costs, and safety outcomes will be closely monitored throughout the trial period\. At the end of six months, the council will vote on whether to make the service permanent, expand it, or discontinue the program altogether\.</question\><answer\>The Riverton City Council approved a six\-month pilot program to run late\-night buses on five major routes starting in March\. The service will operate every 30 minutes from midnight to 4 a\.m\., targeting shift workers and improving nighttime safety\. Officials will monitor costs, ridership, and safety before deciding whether to continue or expand the program\.</answer\>\*\*Output\*\*:\`\`\`json\{"incorrect\_spans": \[\]\}\`\`\`\# Notes\- It is possible for multiple incorrect spans to exist in the same answer; make sure to capture each one\.\- If the answer is fully correct, return \`"incorrect\_spans": \[\]\`\.\- Try to identify short spans as short as possible that captures the incorrect part of the answer\.\- The spans should appear in the same order as they appear in the original answer\.

User prompt

<question\>\{query\}</question\><answer\>\{answer\}</answer\>