Score-Control for Hallucination Reduction in Diffusion Models
Summary
This paper introduces Variance-Guided Score Modulation (VSM) to reduce hallucinations in diffusion models by controlling score function smoothness, achieving up to ~25% reduction while maintaining image quality.
View Cached Full Text
Cached at: 06/04/26, 03:41 AM
Paper page - Score-Control for Hallucination Reduction in Diffusion Models
Source: https://huggingface.co/papers/2606.00377
Abstract
Variance-Guided Score Modulation reduces hallucinations in diffusion models by controlling score function smoothness through Jacobian modulation while maintaining image quality.
Diffusion modelshave emerged as the backbone of modern generative AI, powering advances in vision, language, audio and other modalities. Despite their success, they suffer fromhallucinations, implausible samples that lie outside the support of true data distribution, which degrade reliability and trust. In this work, we first empirically confirm previously proposed hypothesis that score smoothness causeshallucinationsinImage Generationdiffusion modelsand provide a density-based perspective. We further formalize this notion by linking thehallucinationsprobability mass tolipschitz constantof the learnedscore function. Motivated by this, we introduce aVariance-Guided Score Modulation(VSM) strategy that controls thescore Jacobian, in turn reducing score smoothness and better approximating the ground truth score that decreaseshallucinations. Empirical results on synthetic and real-world datasets demonstrate that our approach reduceshallucinations(up to ~25%) while maintaining high fidelity and diversity, providing a principled step toward more reliable diffusion-basedimage generation. We also propose two benchmark datasets with extremesemantic variationfor systematichallucination evaluation. Code and Datasets are publicly available at https://github.com/bhosalems/VSM.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2606\.00377
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.00377 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.00377 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.00377 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Hallucination Detection-Guided Preference Optimization for Clinical Summarization
Introduces HDSR and HDSR-PL, methods that use hallucination detectors to guide iterative self-refinement and preference learning, achieving up to 48% reduction in hallucinations for clinical summarization using Llama and Gemma models on MIMIC-IV-Note.
Diffusion models recover accurate mixture weights despite score function insensitivity
This paper resolves the paradox that diffusion models can accurately recover mixture weights even when the score function is insensitive to them, introducing the Diffusion Score Sensitivity Index (DSSI) and showing that intermediate noise levels provide informative signals for weight recovery.
FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models
This paper proposes FADE, a training-free method that mitigates hallucinations in Large Vision-Language Models by attenuating FFN outputs at critical layers to reduce language-prior dominance, demonstrating effectiveness across multiple benchmarks.
Mitigating Multimodal Hallucination via Phase-wise Self-reward
PSRD framework halves multimodal hallucination in LVLMs by using phase-wise self-reward decoding and a distilled lightweight reward model without extra supervision.
Whisper Hallucination Detection and Mitigation via Hidden Representation Steering and Sparse AutoEncoders
This paper demonstrates that Whisper's hallucination failures on silence, noise, or music can be detected and mitigated purely from internal activations using sparse autoencoders, achieving large reductions in hallucination rate without fine-tuning.