Tag
SANA-Video 2.0 introduces a hybrid linear-softmax attention mechanism for video diffusion transformers, achieving high-quality video generation up to 720p on a single GPU with significantly reduced latency compared to full-softmax models, while maintaining competitive VBench scores.
Introduces Multi-Head Attention Residuals (MHAR), which reshapes the routing query into per-subspace heads so each feature subspace reads depth history via its own softmax. Trained from scratch on a Nemotron-based corpus, MHAR consistently improves validation loss over standard Transformers from 100M to 1B scale and also boosts mid-training downstream accuracy.
This paper introduces Low-Rank Attention Residuals (LR-AttnRes) for LLMs, which decouple routing from representation by using low-dimensional keys for depth-wise attention, improving performance while reducing FLOPs.
The Kimi Team's paper 'Attention Residuals' (AttnRes) replaces uniform residual connections in Transformers with softmax attention over depth, allowing each layer to dynamically select earlier representations. Pre-trained on 1.4 trillion tokens with a 48B parameter model, it stabilizes hidden states and significantly improves reasoning tasks.
Delta Attention Residuals improve layer-wise routing in transformer models by attending to feature changes (deltas) rather than cumulative hidden states, achieving 1.7–8.2% validation perplexity gains across scales from 220M to 7.6B parameters.