For over a decade, we've accepted that end-to-end backprop is the only way to train deep networks (1 minute read)
Summary
Sakana AI presents DiffusionBlocks, a method that trains neural networks block-wise by interpreting forward passes as diffusion denoising, significantly reducing memory requirements compared to traditional end-to-end backpropagation.
View Cached Full Text
Cached at: 05/29/26, 06:32 PM
Holding the entire network in memory at once is why AI training is hitting a resource wall. Sakana Labs has found a new way to break the network into blocks and train them independently. The trick was to treat the network’s forward pass like a diffusion model denoising a signal. This slashes the memory needed to train deep models.
Sakana AI (@SakanaAILabs): Introducing DiffusionBlocks: Block-wise Neural Network Training via Diffusion Interpretation
https://t.co/45Xvzl2qQS
What if we didn’t have to hold an entire neural network in memory to train it?
Standard neural net training optimizes all parameters jointly. As a result, the
Similar Articles
@simplifyinAI: BREAKING: NVIDIA proved back-propagation isn't the only way to build an AI. Billion-parameter models were trained witho…
NVIDIA and Oxford University introduced EGGROLL, a scalable evolution strategies algorithm that trains billion-parameter models without backpropagation, using only integers and parallel mutations.
Beyond Backbone Backpropagation: A Decoupled Strategy for Efficient Transfer Learning
Proposes a decoupled training strategy that adapts normalization layers and uses precomputed features to reduce overhead in transfer learning, achieving competitive accuracy with significantly reduced training time and energy consumption.
DiffusionBench: On Holistic Evaluation of Diffusion Transformers
Researchers introduce NanoGen, a unified framework for training and evaluating diffusion transformers, and propose DiffusionBench, a holistic benchmark combining ImageNet class-conditional and text-to-image generation to better assess progress in generative modeling.
Neuromorphic Diffusion Language Models: Addressing Compute and Memory Bottlenecks via Sparsity and Block Denoising
Proposes neuromorphic masked diffusion language models (N-MDLMs) that integrate block diffusion with spike-based neuromorphic computation to improve throughput and energy efficiency by leveraging sparsity and generating multiple tokens per parameter access, analyzed via a roofline-inspired model.
SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer
SANA-Video is a small diffusion model that efficiently generates high-resolution, long videos using linear attention and a constant-memory KV cache, achieving competitive performance at dramatically lower cost and faster speed compared to existing models.