Tag
Sumi is a 7B uniform diffusion language model pretrained from scratch on 1.5T tokens, achieving competitive performance on knowledge and reasoning tasks while being fully open-source with released weights and training recipe.
Revisits uniform diffusion models, identifying a mismatch between the plug-in ELBO and cross-entropy denoising objective, and proposes leave-one-out parameterizations along with an absorbing-state reformulation that improves generation without additional training.