DiffusionBench: On Holistic Evaluation of Diffusion Transformers
Summary
Researchers introduce NanoGen, a unified framework for training and evaluating diffusion transformers, and propose DiffusionBench, a holistic benchmark combining ImageNet class-conditional and text-to-image generation to better assess progress in generative modeling.
View Cached Full Text
Cached at: 06/24/26, 05:47 AM
Paper page - DiffusionBench: On Holistic Evaluation of Diffusion Transformers
Source: https://huggingface.co/papers/2606.24888
Abstract
Researchers introduce NanoGen, a unified framework for training and evaluating diffusion transformers that demonstrates the need for comprehensive benchmarking beyond ImageNet class-conditional generation to assess true progress in generative modeling.
Diffusion transformer(DiT) research onimage generationhas converged to a single evaluation setup: class-conditional generation onImageNet. While methods improve theFIDand related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unifiedDiTtraining and evaluation framework. NanoGen matches state-of-the-artDiTbaselines onImageNetand, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlowdiffusion methodsunder bothImageNetand T2I setups. Under NanoGen, training T2I requires comparable compute toImageNet. After training 21latent diffusion modelswith NanoGen, we observe that method ranking shows no strong correlation betweenImageNetand T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditionalImageNetFIDmay show no corresponding improvement on T2I, clearly indicating the necessity of evaluatingDiTs on both tasks. To this end, we summarizeImageNetand text-to-image results, which yieldsDiffusionBench, a holistic benchmark forDiTresearch. We recommend reportingDiffusionBenchin place ofImageNetalone: methods that improveDiffusionBenchare more likely to reflect broader progress.
View arXiv pageView PDFProject pageGitHub36Add to collection
Get this paper in your agent:
hf papers read 2606\.24888
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.24888 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.24888 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.24888 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
DiffusionBench: Towards Holistic Evaluation of Generative Diffusion Transformers
Introduces DiffusionBench, a unified benchmark for holistic evaluation of generative diffusion transformers, supporting multiple generation tasks and providing standardized training and evaluation.
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer
UniDDT proposes a decoupled diffusion transformer framework that unifies multimodal understanding and generation by leveraging a Noisy ViT encoder and LLM for semantic encoding, achieving strong performance on both tasks.
MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
MMDiff extends frozen diffusion transformers into multi-modal generative systems using lightweight decoders, achieving significant improvements in semantic segmentation and other perceptual tasks through multi-timestep feature fusion.
Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis
This paper proposes Nemotron-Labs-Diffusion-Image, a masked discrete diffusion model for high-resolution text-to-image synthesis, introducing a token-editing mechanism and grouped cross-entropy objective to improve token refinement and training efficiency.
Diffusion Language Models: An Experimental Analysis
A systematic experimental analysis evaluating eight state-of-the-art Diffusion Language Models across multiple benchmarks, analyzing trade-offs between generation quality and computational efficiency.