DiffusionGemma 技术报告

# DiffusionGemma 技术报告
Source: https://arxiv.org/abs/2608.00146
作者：DiffusionGemma Team (https://arxiv.org/search/cs?searchtype=author&query=DiffusionGemma+Team)：Adrien Ali Taïga (https://arxiv.org/search/cs?searchtype=author&query=Ta%C3%AFga,+A+A),James Assiene (https://arxiv.org/search/cs?searchtype=author&query=Assiene,+J),Daniele Calandriello (https://arxiv.org/search/cs?searchtype=author&query=Calandriello,+D),Rahma Chaabouni (https://arxiv.org/search/cs?searchtype=author&query=Chaabouni,+R),João Gante (https://arxiv.org/search/cs?searchtype=author&query=Gante,+J),Tamara von Glehn (https://arxiv.org/search/cs?searchtype=author&query=von+Glehn,+T),Nate Keating (https://arxiv.org/search/cs?searchtype=author&query=Keating,+N),Chris Knutsen (https://arxiv.org/search/cs?searchtype=author&query=Knutsen,+C),Martin Kukla (https://arxiv.org/search/cs?searchtype=author&query=Kukla,+M),Tianlin Liu (https://arxiv.org/search/cs?searchtype=author&query=Liu,+T),Ivan Lobov (https://arxiv.org/search/cs?searchtype=author&query=Lobov,+I),Ofir Nabati (https://arxiv.org/search/cs?searchtype=author&query=Nabati,+O),João Gabriel Oliveira (https://arxiv.org/search/cs?searchtype=author&query=Oliveira,+J+G),Nicolas Perez\-Nieves (https://arxiv.org/search/cs?searchtype=author&query=Perez-Nieves,+N),Nastasia Prutianova (https://arxiv.org/search/cs?searchtype=author&query=Prutianova,+N),Bobak Shahriari (https://arxiv.org/search/cs?searchtype=author&query=Shahriari,+B),Jean Tarbouriech (https://arxiv.org/search/cs?searchtype=author&query=Tarbouriech,+J),Pavel Tyletski (https://arxiv.org/search/cs?searchtype=author&query=Tyletski,+P),Çağlar Ünlü (https://arxiv.org/search/cs?searchtype=author&query=%C3%9Cnl%C3%BC,+%C3%87),Cindy Wu (https://arxiv.org/search/cs?searchtype=author&query=Wu,+C),Glenn Cameron (https://arxiv.org/search/cs?searchtype=author&query=Cameron,+G),Jerome Connor (https://arxiv.org/search/cs?searchtype=author&query=Connor,+J),Sertan Girgin (https://arxiv.org/search/cs?searchtype=author&query=Girgin,+S),Maarten Grootendorst (https://arxiv.org/search/cs?searchtype=author&query=Grootendorst,+M),Alon Levkovitch (https://arxiv.org/search/cs?searchtype=author&query=Levkovitch,+A),Eliya Nachmani (https://arxiv.org/search/cs?searchtype=author&query=Nachmani,+E),Omar Sanseviero (https://arxiv.org/search/cs?searchtype=author&query=Sanseviero,+O),Piotr Stanczyk (https://arxiv.org/search/cs?searchtype=author&query=Stanczyk,+P),Quentin Berthet (https://arxiv.org/search/cs?searchtype=author&query=Berthet,+Q),Andrew Campbell (https://arxiv.org/search/cs?searchtype=author&query=Campbell,+A),Clément Crepy (https://arxiv.org/search/cs?searchtype=author&query=Crepy,+C),Valentin De Bortoli (https://arxiv.org/search/cs?searchtype=author&query=De+Bortoli,+V),Arnaud Doucet (https://arxiv.org/search/cs?searchtype=author&query=Doucet,+A),Romuald Elie (https://arxiv.org/search/cs?searchtype=author&query=Elie,+R),Alexandre Galashov (https://arxiv.org/search/cs?searchtype=author&query=Galashov,+A),Klaus Greff (https://arxiv.org/search/cs?searchtype=author&query=Greff,+K),Alexis Jacq (https://arxiv.org/search/cs?searchtype=author&query=Jacq,+A),David Ruhe (https://arxiv.org/search/cs?searchtype=author&query=Ruhe,+D),Yu\-Han Wu (https://arxiv.org/search/cs?searchtype=author&query=Wu,+Y),Sebastian Flennerhag (https://arxiv.org/search/cs?searchtype=author&query=Flennerhag,+S),Brendan O'Donoghue (https://arxiv.org/search/cs?searchtype=author&query=O%27Donoghue,+B),George Scrivener (https://arxiv.org/search/cs?searchtype=author&query=Scrivener,+G),Shantanu Thakoor (https://arxiv.org/search/cs?searchtype=author&query=Thakoor,+S)

查看 PDF (https://arxiv.org/pdf/2608.00146)

> 摘要：我们介绍了 DiffusionGemma，一个实验性的开放权重语言模型，使用离散扩散以极快的速度生成文本。与一次解码一个 token 不同，DiffusionGemma 并行迭代精炼 256 个 token 的块，从而避免了传统自回归（AR）大语言模型的顺序解码瓶颈。我们没有从头训练，而是通过对专家混合的 Gemma 4 模型进行微调获得 DiffusionGemma，该模型具有 3.8B 激活参数和 25.2B 总参数。我们计算高效的两阶段训练流程使用的训练 token 预算不到初始 AR 模型总训练 token 预算的 10%。第一阶段使用监督微调来教授双向去噪，而第二阶段将强化学习与采样器蒸馏相结合，以共同提高生成质量和推理效率。DiffusionGemma 在生成速度与模型能力之间的权衡上建立了新的帕累托前沿。在我们的完整评估套件中平均来看，它每次前向传播生成约 20 个 token，并在单个 NVIDIA H100 GPU 上实现每秒约 1,500 个输出 token，即使采用最先进的推测解码，也比 AR 模型快得多。DiffusionGemma 还保留了初始模型对思考模式、多模态输入和长上下文的支持。尽管进行了扩散微调，它仍然能够进行 AR 生成，只有轻微的性能下降，这为混合扩散-AR 解码指明了道路。

## 提交历史

来自：Jean Tarbouriech \[查看电子邮件 (https://arxiv.org/show-email/788d4af8/2608.00146)\] **\[v1\]**Fri, 31 Jul 2026 16:11:46 UTC \(6,116 KB\)