SSOG-Attention: Sum Of Separable Gaussians as a sub-quadratic and scalable alternative to SDPA. [R]

Reddit r/MachineLearning Papers
attention-mechanism ssog sdpa-alternative scalability gaussian machine-learning complexity

Summary

SSOG-Attention introduces a novel attention mechanism using Sum of Separable Gaussians to achieve sub-quadratic complexity, outperforming or matching SDPA with faster convergence and improved efficiency.

​ Scaled dot-product attention (SDPA) computes its Attention by computing the similarity-scores of all image-tokens with all query tokens which results in O(N²·d) complexity. SSOG (Sum Of Separable Gaussians) instead learns a few Gaussian atoms for each head and only geometrically steers them based on the query token. Since the atoms can be factorized into a separable sum of Gaussians this leads to a reduced complexity of O(N·√N·d). Experiments show that SSOG clearly beats SDPA on small data (cifar100), and delivers equivalent performance and much faster convergence on bigger datasets like IN1k. All that while being much faster and memory efficient with increasing scale. Have a look at the full blog-post and repo to see more results and ablations and let me know what you think. Blog-post: https://pisoni.ai/posts/ssog Repo: https://github.com/4rtemi5/ssog *AI was used for some of the code and some of the blog-post but I put a lot of effort into this project and stand behind every word.
Original Article

Similar Articles

ConSA: Controllable Sparsity in Hybrid Attention via Learnable Allocation

arXiv cs.CL

ConSA is a framework that learns optimal assignment between full attention and sliding-window attention under a user-specified sparsity target, using L0 regularization and augmented Lagrangian constraint. It demonstrates consistent gains over rule-based baselines on LLMs at 0.6B and 1.7B scales.

Simplified Sparse Attention via Gist Tokens

Hugging Face Daily Papers

This paper introduces Simplified Sparse Attention (SSA), a method that uses gist tokens during continued pretraining to enable efficient chunk selection at inference without architectural changes, achieving high compression ratios and outperforming baselines on long-context tasks like LongBench and retrieval-augmented generation.