Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

Hugging Face Daily Papers Papers
hallucination-detection llm-evaluation meta-study benchmarking span-level-evaluation factuality

Summary

A meta-study analyzing hallucination detection as a benchmark-conditioned measurement problem across provenance, typology, and granularity axes, proposing a Span Coverage F1 metric to address variability in span-level annotation boundaries.

Hallucination detection is increasingly important for evaluating and deploying large language models. However, existing methods are often compared across benchmarks that define and measure hallucination in substantially different ways, making reported scores difficult to interpret across settings. We study hallucination detection as a benchmark-conditioned measurement problem and organize existing benchmarks along three axes: provenance, which describes how model outputs and evidence conditions are obtained; typology, which captures the factuality and faithfulness failures represented by benchmark labels; and granularity, which specifies the unit at which hallucinations are annotated, predicted, and evaluated. Using this framework, we analyze how benchmark construction, annotation, and evaluation choices shape the underlying prediction task, and relate detector mechanisms and assumptions to the settings in which their performance is measured. We also examine span-level evaluation, where variation in annotation boundaries can substantially affect reported results, and propose Span Coverage F1 as a complementary containment-based metric. Our analysis clarifies what detector scores capture, highlights limitations of cross-benchmark comparisons, and motivates more explicit evaluation of generalization across benchmark conditions.
Original Article
View Cached Full Text

Cached at: 10/28/26, 01:58 AM

Paper page - Mycelium: A Meta-Study of Hallucination Detection Across Provenance, Typology, and Granularities

Source: https://huggingface.co/papers/2610.33608

Abstract

Hallucination detection is analyzed as a benchmark-conditioned measurement problem across provenance, typology, and granularity axes, with a proposed Span Coverage F1 metric to address span-level evaluation variability.

Hallucination detectionis increasingly important for evaluating and deploying large language models. However, existing methods are often compared across benchmarks that define and measure hallucination in substantially different ways, making reported scores difficult to interpret across settings. We studyhallucination detectionas a benchmark-conditioned measurement problem and organize existing benchmarks along three axes:provenance, which describes how model outputs and evidence conditions are obtained;typology, which captures the factuality and faithfulness failures represented by benchmark labels; andgranularity, which specifies the unit at which hallucinations are annotated, predicted, and evaluated. Using this framework, we analyze how benchmark construction, annotation, and evaluation choices shape the underlying prediction task, and relatedetector mechanismsand assumptions to the settings in which their performance is measured. We also examinespan-level evaluation, where variation in annotation boundaries can substantially affect reported results, and proposeSpan Coverage F1as a complementary containment-based metric. Our analysis clarifies what detector scores capture, highlights limitations ofcross-benchmark comparisons, and motivates more explicit evaluation of generalization across benchmark conditions.

View arXiv pageView PDFGitHub0Add to collection

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2610.33608 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2610.33608 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2610.33608 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

The Detectability Gap: Hidden Heterogeneity in Hallucination Detection Across Language Models

arXiv cs.CL

This paper identifies a 'detectability gap' in hallucination detection: hallucinations split into high-agreement (Ghost) and low-agreement (Flickering) regimes with a 0.35–0.46 AUC gap, persisting across four models and three factual QA datasets even after freezing regime assignments and using stricter trajectory-based tests. The authors argue aggregate detection metrics hide model-dependent heterogeneity and call for regime-conditioned evaluation.

PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts

arXiv cs.CL

This paper reveals that much of the reported progress in LLM hallucination detection is due to benchmark construction artifacts, where ground-truth answers are embedded in prompts, allowing a simple text-similarity baseline to achieve near-perfect scores. Through a large-scale controlled evaluation, the authors show that most methods perform near chance under proper controls, except for supervised probes on upper-layer hidden states such as SAPLMA and their proposed DRIFT.