UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations

Hugging Face Daily Papers Papers
hallucination-detection vision-language-models graph-neural-network token-level-detection ai-safety

Summary

UniProbe is a lightweight detector that uses graph-based multi-structural internal representations to identify hallucinated tokens in large vision-language models, reducing object hallucinations by up to 55% with minimal latency increase.

Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model's generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM's heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structure-aware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a self-adaptation strategy aligning the detector with the LVLM's own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55\% at 1.06times the latency of standard generation.
Original Article
View Cached Full Text

Cached at: 08/17/26, 07:44 AM

Paper page - UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations

Source: https://huggingface.co/papers/2608.10835 Published on Aug 11

·

Submitted byhttps://huggingface.co/Dvir

Samuelon Aug 17

Abstract

UniProbe is a lightweight learnable detector that uses a directed graph and alternating GNN, ViT, and GRU modules to identify hallucinated tokens in frozen large vision-language models, enabling real-time resampling during generation.

Large Vision-Language Models(LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requirestoken-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model’s generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM’s heterogeneous computational trace from a single forward pass. UniProbe constructs adirected graphover image patches, query tokens, and generated tokens, withattention weightsencoding their relations. It processes this trace with alternating structure-aware modules: aGNNfor relational evidence, aViTfor 2-D visual geometry, and aGRUfor response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and aself-adaptationstrategy aligning the detector with the LVLM’s own generations. Across diverse LVLM backbones, UniProbe achieves state-of-the-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55\% at 1.06times the latency of standard generation.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2608\.10835

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2608.10835 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2608.10835 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2608.10835 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Unified Hallucination Fuzzing for Multimodal Large Language Models

arXiv cs.CL

This paper presents UniHall, a fine-grained hallucination benchmark with a unified taxonomy, and Self-Adaptive Multimodal Fuzzing (SAMF), a self-evolving stress-testing framework for multimodal LLMs. Experiments show SOTA models degrade significantly under fuzzing and reveal a helpfulness-hallucination trade-off.

Prompt Embedding Probes (PEP): Hallucination Detection in LLMs from Hidden States

arXiv cs.CL

This paper introduces Prompt Embedding Probes (PEP), a parameter-efficient extension of linear probes that uses learnable prompt embeddings on hidden states to detect hallucinations in frozen LLMs. Evaluations on TriviaQA, GSM8K, and MedQA with Qwen3 models show improvements over standard linear probes, including in pre-generation and cross-model settings.

Hallucination Detection via Activations of Open-Weight Proxy Analyzers

arXiv cs.CL

This paper introduces a proxy-analyzer framework that detects hallucinations in large language models by analyzing internal activations of small, open-weight models rather than the generator itself. The method achieves superior performance on benchmarks like RAGTruth compared to existing methods like ReDeEP, demonstrating that model size is less critical than the analysis approach.

HalluSAE: Detecting Hallucinations in Large Language Models via Sparse Auto-Encoders

arXiv cs.CL

Researchers from Beihang University and other institutions propose HalluSAE, a framework using sparse autoencoders and phase transition theory to detect hallucinations in LLMs by modeling generation as trajectories through a potential energy landscape and identifying critical transition zones where factual errors occur.