InstanceControl: Controllable Complex Image Generation without Instance Labeling

Hugging Face Daily Papers Papers
multi-instance-generation controllable-generation vision-language-model image-generation instance-level-control adaptive-mask-refinement

Summary

InstanceControl enables multi-instance controllable image generation without manual instance labeling by leveraging a vision-language model to establish instance-level correspondences between text prompts and visual conditions, with adaptive mask refinement for improved accuracy.

Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.
Original Article
View Cached Full Text

Cached at: 07/03/26, 07:53 AM

Paper page - InstanceControl: Controllable Complex Image Generation without Instance Labeling

Source: https://huggingface.co/papers/2606.31924

Abstract

InstanceControl enables multi-instance image generation by using vision-language models to establish instance-level correspondences between text prompts and visual conditions, while employing adaptive mask refinement for improved accuracy.

Controllable image generation methods, such asControlNet, have demonstrated a remarkable capacity to introducevisual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complexmulti-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions withinvisual conditions. To address this, we leverage theVision-Language Model(VLM) to establishinstance-level correspondencesbetweentext promptsandvisual conditions. Specifically, the VLM automatically parses instance descriptions from thetext promptsand simultaneously predictsinstance masksbased on thevisual conditions. Furthermore, since the predicted masks may contain noise, we introduce anadaptive mask refinementstrategy that dynamically refines theseinstance masksduring the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.

View arXiv pageView PDFProject pageGitHub13Add to collection

Get this paper in your agent:

hf papers read 2606\.31924

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2606.31924 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2606.31924 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2606.31924 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Hierarchical text-conditional image generation with CLIP latents

OpenAI Blog

OpenAI proposes a hierarchical two-stage model for text-conditional image generation using CLIP latents: a prior that generates CLIP image embeddings from text captions, and a diffusion-based decoder that generates images from embeddings. The approach improves image diversity and enables zero-shot language-guided image manipulations.

CogOmniControl: Reasoning-Driven Controllable Video Generation via Creative Intent Cognition

Hugging Face Daily Papers

CogOmniControl is a reasoning-driven framework for controllable video generation that uses a specialized vision-language model (CogVLM) trained on anime production data to infer creative intent from sparse conditions, then guides a diffusion-based generator via reinforcement learning, achieving state-of-the-art results on new benchmarks.

Unlocking Complex Visual Generation via Closed-Loop Verified Reasoning

Hugging Face Daily Papers

Introduces CLVR (Closed-Loop Visual Reasoning), a framework that reformulates text-to-image generation from a single-step process into a closed-loop, multi-step visual reasoning approach using a VLM controller and diffusion models, achieving improved performance on compositional prompts.

CLIP: Connecting text and images

OpenAI Blog

CLIP is OpenAI's vision-language model that learns from text-image pairs from the internet, enabling zero-shot visual classification without task-specific training data. It addresses major limitations in traditional computer vision by reducing dependence on expensive labeled datasets and improving real-world generalization.