Diffusion Model as a Generalist Segmentation Learner
Summary
This paper introduces DiGSeg, a framework that repurposes pretrained diffusion models for state-of-the-art semantic and open-vocabulary segmentation by leveraging latent space conditioning and text-guided alignment.
View Cached Full Text
Cached at: 05/08/26, 08:03 AM
Paper page - Diffusion Model as a Generalist Segmentation Learner
Source: https://huggingface.co/papers/2604.24575
Abstract
Pretrained diffusion models can be adapted for semantic and open-vocabulary segmentation tasks through latent space conditioning and text-guided alignment, achieving state-of-the-art performance across diverse domains.
Diffusion modelsare primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially alignedvisual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic andopen-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Modelsas a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into thelatent spaceand concatenates them as conditioning signals for thediffusion U-Net. A parallelCLIP-aligned text pathwayinjects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standardsemantic segmentationbenchmarks, as well as strong open-vocabulary generalization andcross-domain transferto medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation andvisual understanding.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2604\.24575
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2604.24575 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2604.24575 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2604.24575 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Where Should Diffusion Enter a Language Model? Geometry-Guided Hidden-State Replacement
This paper introduces DiHAL, a diffusion-transformer hybrid that uses geometry-based proxies to select a layer in a pretrained language model for hidden-state replacement with a diffusion bridge, improving continuous diffusion language modeling by avoiding direct token recovery.
Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
This paper theoretically analyzes diffusion language models through a bias-variance lens, identifying trade-offs between masking and uniform diffusion kernels. It proposes SemDLM+, which adds a global transition and semantic-frequency penalty to overcome the semantic basin problem, achieving competitive generation quality on LM1B and OpenWebText benchmarks.
GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Models
GDSD proposes a reinforcement learning method that directly distills denoisers from advantage-guided self-teachers for diffusion language models, avoiding biases from ELBO-based likelihood surrogates. It achieves up to +19.6% accuracy improvements on planning, math, and coding benchmarks over prior state-of-the-art methods.
Dynamic Chunking for Diffusion Language Models
This paper introduces Dynamic Chunking for Diffusion Language Models (DCDM), which replaces fixed positional blocks in block discrete diffusion with content-defined semantic chunks using a differentiable Chunking Attention mechanism, achieving consistent improvements across scales up to 1.5B parameters.
Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
This paper introduces SemanticSeg, a large-scale dataset for semantic segmentation of long texts, and block distillation, a training framework that enables block attention models to approach full-attention performance, improving KV cache reuse in RAG and long-context scenarios.