MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
Summary
MMDiff extends frozen diffusion transformers into multi-modal generative systems using lightweight decoders, achieving significant improvements in semantic segmentation and other perceptual tasks through multi-timestep feature fusion.
View Cached Full Text
Cached at: 06/16/26, 11:32 AM
Paper page - MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
Source: https://huggingface.co/papers/2606.16673
Abstract
MMDiff transforms frozen diffusion transformers into multi-modal generative systems that produce images and perceptual modalities using lightweight decoders, achieving improved semantic segmentation through multi-timestep feature fusion and spatial aggregation.
Diffusion transformershave demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across theirdenoising trajectoryare discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into amulti-modal generative systemthat jointly produces images alongside any combination of dense perceptual modalities usinglightweight decoder heads. Our central finding is that perceptual information is temporally distributed along thedenoising trajectory, and thatmulti-timestep feature fusionwithspatially varying aggregation weightsis essential, improvingsemantic segmentationresults by up to 28.7% mIoU over single-timestep extraction. We further adoptconcept-driven attention extractionfor interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such asDINOv3. By training onlylightweight decoder headson a frozen backbone, we achieve strong performance insemantic segmentation,salient object detection, anddepth estimation, and demonstrate that this framework enables effectivesynthetic data generationat scale.
View arXiv pageView PDFProject pageAdd to collection
Get this paper in your agent:
hf papers read 2606\.16673
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.16673 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.16673 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.16673 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Multimodal Model Diffing for Feature Discovery and Control
MMDiff uses multimodal sparse autoencoders to isolate, detect, and control features in multimodal language models, improving interpretability and targeted steering of visual and safety behaviors.
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
UniDDT proposes a decoupled diffusion transformer framework that unifies multimodal understanding and generation by leveraging a Noisy ViT encoder and LLM for semantic encoding, achieving strong performance on both tasks.
Exploring Diffusion Transformers for Cross-Modal Augmentation in Multimodal Brain State Decoding
The paper proposes CoMA-DiT, a bidirectional cross-modal Diffusion Transformer for latent augmentation in multimodal brain state decoding, which enhances performance in tasks like auditory attention decoding and emotion recognition by using paired modalities as mutual supervisory signals.
MobiDiff: Semantic-Aware Multi-Channel Discrete Diffusion for Human Mobility Data Generation
Introduces MobiDiff, an end-to-end discrete diffusion framework for generating human mobility data by denoising multi-channel semantic skeletons, achieving faster inference and competitive fidelity on real-world datasets.
Representation-Space MMD for Diffusion Language Models
This paper introduces a post-training method for diffusion language models that minimizes Maximum Mean Discrepancy between generated and reference distributions in the feature space of a frozen pretrained DLM, achieving lower generative perplexity and better accuracy-compute trade-offs on OpenWebText and GSM8K, with improved decoding parallelism on 16B LLaDA-2.0 models.