hybrid-linear-attention

Taylor-Calibrate proposes a principled initialization method for hybrid linear attention models that significantly improves the efficiency of distilling pretrained Transformers into Gated DeltaNet students, achieving up to 88x improvement and reducing training tokens by 4.9x-9.2x.

This paper identifies that chain-of-thought supervised fine-tuning degrades long-context recall in hybrid linear-attention models by biasing attention gradients toward short-range patterns, and proposes QK-Restore, a training-free method that restores long-context recall while preserving reasoning performance.

SANA World Model is a new AI model that uses hybrid linear attention for efficiency and speed.