ASRN Adaptive Sparse Recurrence Network [N]
Summary
The ASRN Adaptive Sparse Recurrence Network introduces a copy layer for language models that looks up earlier occurrences of the current context via learned hash tables and reproduces what followed, achieving memory usage linear in sequence length as a cheaper alternative to full attention.
Similar Articles
Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
Sparse Delta Memory extends gated linear RNNs with sparse addressing to dramatically increase hidden state capacity for improved long-context learning and retrieval while maintaining computational efficiency.
WriteSAE: Sparse Autoencoders for Recurrent State
WriteSAE introduces the first sparse autoencoder that decomposes matrix cache writes in state-space and hybrid recurrent language models, enabling superior token-level interventions compared to existing methods.
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
LDARNet is a 120M-parameter hierarchical genomic foundation model that introduces learnable adaptive tokenization (inspired by H-Net's dynamic chunking) for masked language modeling on DNA sequences. It achieves state-of-the-art results on 5 histone modification tasks and outperforms models up to 20× larger on several genomic benchmarks, with learned token boundaries aligning with biological features like promoter motifs and splice junctions.
Selective State-Space Adaptation and Retrieval for Language Model Reasoning
Proposes MaLoRA and MaRA, two adapter families that introduce selective state-space recurrence for token-level and context-level adaptation in frozen language models, achieving significant gains on multi-hop reasoning benchmarks like MuSiQue and 2WikiMultihopQA.
Augmenting Attention with Exponentially Decaying Memory Improves Query-Aware KV Sparsity
This paper explores how an exponentially decaying memory module from RAT+ can improve query-aware sparse inference methods for long-context language models, demonstrating consistent accuracy gains across various sparse budgets on needle-in-a-haystack tasks.