Score-Control for Hallucination Reduction in Diffusion Models
Summary
This paper introduces Variance-Guided Score Modulation (VSM) to reduce hallucinations in diffusion models by controlling score function smoothness, achieving up to ~25% reduction while maintaining image quality.
View Cached Full Text
Cached at: 06/04/26, 03:41 AM
Paper page - Score-Control for Hallucination Reduction in Diffusion Models
Source: https://huggingface.co/papers/2606.00377
Abstract
Variance-Guided Score Modulation reduces hallucinations in diffusion models by controlling score function smoothness through Jacobian modulation while maintaining image quality.
Diffusion modelshave emerged as the backbone of modern generative AI, powering advances in vision, language, audio and other modalities. Despite their success, they suffer fromhallucinations, implausible samples that lie outside the support of true data distribution, which degrade reliability and trust. In this work, we first empirically confirm previously proposed hypothesis that score smoothness causeshallucinationsinImage Generationdiffusion modelsand provide a density-based perspective. We further formalize this notion by linking thehallucinationsprobability mass tolipschitz constantof the learnedscore function. Motivated by this, we introduce aVariance-Guided Score Modulation(VSM) strategy that controls thescore Jacobian, in turn reducing score smoothness and better approximating the ground truth score that decreaseshallucinations. Empirical results on synthetic and real-world datasets demonstrate that our approach reduceshallucinations(up to ~25%) while maintaining high fidelity and diversity, providing a principled step toward more reliable diffusion-basedimage generation. We also propose two benchmark datasets with extremesemantic variationfor systematichallucination evaluation. Code and Datasets are publicly available at https://github.com/bhosalems/VSM.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2606\.00377
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.00377 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.00377 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.00377 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Introduces HDSR and HDSR-PL, methods that use hallucination detectors to guide iterative self-refinement and preference learning, achieving up to 48% reduction in hallucinations for clinical summarization using Llama and Gemma models on MIMIC-IV-Note.
Mitigating Multimodal Hallucination via Phase-wise Self-reward
PSRD framework halves multimodal hallucination in LVLMs by using phase-wise self-reward decoding and a distilled lightweight reward model without extra supervision.
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
This paper demonstrates that Whisper's hallucination failures on silence, noise, or music can be detected and mitigated purely from internal activations using sparse autoencoders, achieving large reductions in hallucination rate without fine-tuning.
PARALLAX: Separating Genuine Hallucination Detection from Benchmark Construction Artifacts
This paper reveals that much of the reported progress in LLM hallucination detection is due to benchmark construction artifacts, where ground-truth answers are embedded in prompts, allowing a simple text-similarity baseline to achieve near-perfect scores. Through a large-scale controlled evaluation, the authors show that most methods perform near chance under proper controls, except for supervised probes on upper-layer hidden states such as SAPLMA and their proposed DRIFT.
Why DDIM Hallucinates More than DDPM: A Theoretical Analysis of Reverse Dynamics
This paper provides a theoretical analysis explaining why deterministic DDIM samplers hallucinate more than stochastic DDPM samplers in diffusion models, attributing it to getting stuck in mode-interpolation regions during reverse dynamics.