attention-alternative

The ASRN Adaptive Sparse Recurrence Network introduces a copy layer for language models that looks up earlier occurrences of the current context via learned hash tables and reproduces what followed, achieving memory usage linear in sequence length as a cheaper alternative to full attention.

This paper introduces a method using rotating sparse wiring on a hypercube to replace attention in neural networks, achieving comparable or better performance with significantly reduced computational cost and parameters.

An update on Matrix Recurrent Units (MRU), a linear-time attention alternative. The author explores methods to stabilize training, finding that orthogonal matrices underperform while LDU factorization works best, and shows MRU underperforms transformers on larger datasets like TinyStories.