Tag
This paper identifies a gap between any-order prediction interface and actual any-order inference in masked diffusion models, proposing insertion-based and latent-space masked diffusion to enable genuine any-order inference for tasks like code generation. The authors train a 7B FlexMDM for Python and a 125M LatentMDM, showing improved performance.
Introduces TUBE, a variational upper bound on log-likelihood for discrete diffusion language models, enabling better evaluation and revealing that masked diffusion models still underperform autoregressive models.
This paper introduces Learned Relay Representations (Relay), a method that allows masked diffusion models to propagate latent information across denoising steps, overcoming the hard reset problem and improving performance-latency trade-offs. The method is shown to outperform standard supervised finetuning on coding tasks while reducing inference latency by up to 32%.