Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback

Hugging Face Daily Papers Papers
structured-defect-grounding text-to-image defect-detection vision-language-model diffusion-models alignment dataset

Summary

This paper introduces Structured Defect Grounding (SDG), a method that models text-to-image defects as structured (location, type, reason, importance) tuples and uses VLMs for detection, along with a 30K-image dataset SDG-30K and a diagnosis-to-alignment framework called BoxFlow-GRPO.

Despite generating increasingly photorealistic images, text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures. Diagnosing these failures requires instance-level feedback that answers where a defect occurs, what type it is, why it is defective, and its importance to overall image quality. While recent dense-feedback methods move beyond scalar supervision, their heatmap-centric representations still formulate diagnosis as pixel-field regression, making it difficult to localize variable-cardinality defects and bind semantic reasons to individual failures. To address this representation bottleneck, we propose Structured Defect Grounding (SDG), which casts T2I diagnosis as structured set prediction by modeling each defect as a (location, type, reason, importance) tuple. To make this formulation trainable and measurable, we introduce SDG-30K, a 30K-image dataset with box-grounded annotations across four modern T2I generators, together with a dedicated evaluation protocol, SDG-Eval. Building on this structured representation, we further present a diagnosis-to-alignment framework in which a Vision-Language Model (VLM) serves as the SDG detector, and BoxFlow-GRPO converts predicted defect sets into box-derived, importance-weighted spatial rewards for diffusion model alignment. Extensive experiments show that our SDG detector outperforms leading proprietary VLMs on structured defect grounding, while SDG-guided rewards consistently improve T2I alignment and support localized image refinement. These results establish SDG as a unified, instance-level interface for diagnosing, evaluating, and enhancing modern generative models.
Original Article
View Cached Full Text

Cached at: 06/12/26, 10:52 AM

Paper page - Where, What, Why, and Importance: Structured Defect Grounding for Text-to-Image Feedback

Source: https://huggingface.co/papers/2606.06113

Abstract

Structured Defect Grounding (SDG) addresses limitations in text-to-image model diagnosis by modeling defects as structured sets and using vision-language models for detection and reward-based alignment.

Despite generating increasingly photorealistic images, text-to-image (T2I) models still exhibit localized, subtle, and structurally complex failures. Diagnosing these failures requiresinstance-level feedbackthat answers where a defect occurs, what type it is, why it is defective, and its importance to overall image quality. While recentdense-feedback methodsmove beyond scalar supervision, their heatmap-centric representations still formulate diagnosis aspixel-field regression, making it difficult to localizevariable-cardinality defectsand bind semantic reasons to individual failures. To address this representation bottleneck, we propose StructuredDefect Grounding(SDG), which casts T2I diagnosis asstructured set predictionby modeling each defect as a (location, type, reason, importance) tuple. To make this formulation trainable and measurable, we introduce SDG-30K, a 30K-image dataset with box-grounded annotations across four modern T2I generators, together with a dedicated evaluation protocol, SDG-Eval. Building on this structured representation, we further present a diagnosis-to-alignment framework in which aVision-Language Model(VLM) serves as the SDG detector, and BoxFlow-GRPO converts predicted defect sets into box-derived, importance-weightedspatial rewardsfordiffusion model alignment. Extensive experiments show that our SDG detector outperforms leading proprietary VLMs on structureddefect grounding, while SDG-guided rewards consistently improve T2I alignment and support localized image refinement. These results establish SDG as a unified, instance-level interface for diagnosing, evaluating, and enhancing modern generative models.

View arXiv pageView PDFProject pageGitHub3Add to collection

Get this paper in your agent:

hf papers read 2606\.06113

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper3

#### P1n3/sdg-detector-sft Image-Text-to-Text• 4B• Updated1 day ago • 27 #### P1n3/sdg-detector-grpo Image-Text-to-Text• 4B• Updated1 day ago • 14 #### P1n3/boxflow-grpo-flux-lora Text-to-Image• Updated1 day ago • 16

Datasets citing this paper1

#### P1n3/SDG-30K Viewer• Updated1 day ago • 30.2k • 5.93k

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.06113 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Visual Grounding Safety in Vision-Language Models

Hugging Face Daily Papers

该论文系统分析了视觉语言模型在结构化输出通道（点/边界框）上的安全对齐缺口，发现模型对以定位形式提出的有害请求的拒绝率比纯文本问答低 31-59 个百分点，并提出一种微调方法显著提升了定位形式的拒绝率。

EviGround-CT: Sentence-to-Volume Grounding for Verifying LLM-Generated Chest CT Reports

arXiv cs.CL

EviGround-CT proposes sentence-to-volume grounding to verify LLM-generated chest CT reports by predicting 3D evidence masks for each positive finding sentence, improving evidence-supported precision from 0.263 to 0.365 over category conditioning. The work is evaluated on CT-RATE and a ReXGroundingCT subset to detect unsupported findings.