Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
Summary
Visual Concept Fusion (VCF) enables dual conditioning on both an image and text prompt in diffusion models at inference time without retraining, using a lightweight aligner and fusion strategy.
View Cached Full Text
Cached at: 05/26/26, 02:44 PM
Paper page - Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
Source: https://huggingface.co/papers/2605.25191
Abstract
Visual Concept Fusion enables dual text and image conditioning in diffusion models through feature alignment and fusion strategies without requiring retraining.
Text-to-image diffusion modelslikeStable Diffusiongenerate high-quality images from text, but lack a way to injectvisual guidance(e.g. sketches, styles) at inference without retraining. Existing methods either require computationally expensive fine-tuning or rely on style transfer techniques that risk semantic misalignment with textual prompts. We introduce Visual Concept Fusion (VCF), the first method offering dual conditioning on both an image and text prompt at inference time without any concept-specific training. VCF enables visual concept injection intoStable Diffusionby aligningCLIP image featureswith thetext embedding space. VCF consists of three components: (1) a lightweight aligner that maps image tokens to the text embedding manifold usingInfoNCEandcross-attention reconstructionlosses, (2) afusion strategythat preserves both textual and visual semantics, and (3) an optionalPrompt-Noise Optimization(PNO) module for test-time refinement. Our experiments demonstrate that VCF successfully transfers visual attributes including style, composition, and color palette from reference images while maintaining prompt adherence. Quantitative results show a trade-off between text alignment (CLIP score) and visual correspondence (LPIPS), with VCF outperforming baselines in reference fidelity.
View arXiv pageView PDFGitHub0Add to collection
Get this paper in your agent:
hf papers read 2605\.25191
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2605.25191 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.25191 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.25191 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Concept-based Visual Counterfactual Explanations with Diffusion Models
Introduces C-VCE, a diffusion framework that builds an interpretable concept bottleneck layer into the generative model, enabling human-guided visual counterfactual explanations without relying on external noise-robust classifiers.
Efficient bias mitigation in T2I diffusion models using Concept Graphs
The paper introduces CO-ALIGN, a bias mitigation method for text-to-image diffusion models that aligns concept graphs in the text encoder and denoiser, achieving 30% fairness improvement and 11.4 FID gain while reducing incoherent outputs by 88%.
Scaling Properties of Text Conditioning in Visual Generation
This paper studies empirical scaling properties for text conditioning in visual generation, showing that converged diffusion loss scales with structured language in prompts, and introduces methods to improve diffusability and promptability.
Hierarchical text-conditional image generation with CLIP latents
OpenAI proposes a hierarchical two-stage model for text-conditional image generation using CLIP latents: a prior that generates CLIP image embeddings from text captions, and a diffusion-based decoder that generates images from embeddings. The approach improves image diversity and enables zero-shot language-guided image manipulations.
An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models
This paper proposes a latent-to-pixel training strategy for pixel-space text-to-image diffusion models, accelerating convergence and improving inference speed while matching or surpassing latent-space counterparts.