MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
Summary
MMDiff extends frozen diffusion transformers into multi-modal generative systems using lightweight decoders, achieving significant improvements in semantic segmentation and other perceptual tasks through multi-timestep feature fusion.
View Cached Full Text
Cached at: 06/16/26, 11:32 AM
Paper page - MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
Source: https://huggingface.co/papers/2606.16673
Abstract
MMDiff transforms frozen diffusion transformers into multi-modal generative systems that produce images and perceptual modalities using lightweight decoders, achieving improved semantic segmentation through multi-timestep feature fusion and spatial aggregation.
Diffusion transformershave demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across theirdenoising trajectoryare discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into amulti-modal generative systemthat jointly produces images alongside any combination of dense perceptual modalities usinglightweight decoder heads. Our central finding is that perceptual information is temporally distributed along thedenoising trajectory, and thatmulti-timestep feature fusionwithspatially varying aggregation weightsis essential, improvingsemantic segmentationresults by up to 28.7% mIoU over single-timestep extraction. We further adoptconcept-driven attention extractionfor interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such asDINOv3. By training onlylightweight decoder headson a frozen backbone, we achieve strong performance insemantic segmentation,salient object detection, anddepth estimation, and demonstrate that this framework enables effectivesynthetic data generationat scale.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.16673
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.16673 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.16673 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.16673 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
UniDDT proposes a decoupled diffusion transformer framework that unifies multimodal understanding and generation by leveraging a Noisy ViT encoder and LLM for semantic encoding, achieving strong performance on both tasks.
MobiDiff: Semantic-Aware Multi-Channel Discrete Diffusion for Human Mobility Data Generation
Introduces MobiDiff, an end-to-end discrete diffusion framework for generating human mobility data by denoising multi-channel semantic skeletons, achieving faster inference and competitive fidelity on real-world datasets.
MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion
Proposes M2GDT, a novel MKGC framework that uses an MLLM-guided diffusion transformer with relation-adaptive mixture-of-experts to align and denoise multimodal features, outperforming baselines on three benchmark datasets.
MBDiff: Multi-view Behavior-aware Diffusion Model for Probabilistic Utility Data Imputation
Presents MBDiff, a multi-view behavior-aware diffusion model for probabilistic utility data imputation that learns user behavior from global, local, and instance-level views and uses a conditional attentional denoising network. Evaluated on real utility data from Florida, it outperforms state-of-the-art baselines.
Multi-Block Diffusion Language Models
This paper proposes Multi-Block Diffusion Language Models (MBD-LMs), extending single-block diffusion to concurrent multi-block decoding with improved training strategies like Multi-block Teacher Forcing and an optimized Block Buffer decoding algorithm. Experiments show increased tokens per forward pass and improved accuracy on benchmarks.