Sumi: Open Uniform Diffusion Language Model from Scratch
Summary
Sumi is a 7B uniform diffusion language model pretrained from scratch on 1.5T tokens, achieving competitive performance on knowledge and reasoning tasks while being fully open-source with released weights and training recipe.
View Cached Full Text
Cached at: 06/18/26, 03:55 AM
Paper page - Sumi: Open Uniform Diffusion Language Model from Scratch
Source: https://huggingface.co/papers/2606.19005
Abstract
A large-scale uniform diffusion language model pretrained from scratch demonstrates competitive performance on knowledge and reasoning tasks while highlighting differences in commonsense reasoning compared to autoregressive models.
Diffusion modelshave become a promising alternative toautoregressive models. Among these,uniform diffusion language models(UDLMs) permit any token to be updated at any step, in principle enabling more flexible generation. However, no UDLM has yet been pretrained from scratch at both large parameter scale and largetoken budget. Both autoregressive modeling and masked diffusion modeling already have capable models at scale that the community can study and build on; uniform diffusion has none. A scratch-pretrained UDLM at scale would provide a clean reference point for studying scaling behavior,generation dynamics,controllability, and trade-offs against established autoregressive and maskeddiffusion models. To this end, we introduce Sumi (“ink” in Japanese), a fully open 7B uniform diffusion language model pretrained from scratch on 1.5T tokens. Sumi performs competitively withautoregressive modelstrained at comparabletoken budgets on knowledge, reasoning, and coding benchmarks, while under-performing on commonsense benchmarks, where our education-heavydata mixtureis a likely contributor. We release ourmodel weights, checkpoints, and fulltraining recipe, including a complete specification of thedata mixtureover publicly available corpora. We hope this release enables the community to study native uniform diffusion at scale and catalyzes work on its as-yet poorly understood aspects.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2606\.19005
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### tohoku-nlp/sumi-7b Text Generation• 8B• Updatedabout 2 hours ago
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.19005 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.19005 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Bulding my own Diffusion Language Model from scratch was easier than I thought [P]
Developer shares a minimalist 7.5M-parameter diffusion language model trained from scratch on Shakespeare, releasing the code as a learning resource.
I built a diffusion language model from scratch. It writes flawless sentences that mean nothing, and that is the interesting part.
The author built Joey, a 170M parameter masked diffusion language model from scratch, trained on FineWeb-Edu and fine-tuned on DailyDialog, achieving fluent but incoherent sentences due to capacity limitations. The project highlights the differences from autoregressive LLMs and the lessons learned from building and debugging the system.
Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
This paper theoretically analyzes diffusion language models through a bias-variance lens, identifying trade-offs between masking and uniform diffusion kernels. It proposes SemDLM+, which adds a global transition and semantic-frequency penalty to overcome the semantic basin problem, achieving competitive generation quality on LM1B and OpenWebText benchmarks.
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models
This paper introduces UniSD, a unified self-distillation framework for adapting large language models that integrates mechanisms for supervision reliability, representation alignment, and training stability. Experimental results show that UniSD improves performance over base models and existing baselines across multiple benchmarks.
Diffusion Language Models: An Experimental Analysis
A systematic experimental analysis evaluating eight state-of-the-art Diffusion Language Models across multiple benchmarks, analyzing trade-offs between generation quality and computational efficiency.