WriteSAE: Sparse Autoencoders for Recurrent State
Summary
WriteSAE introduces the first sparse autoencoder that decomposes matrix cache writes in state-space and hybrid recurrent language models, enabling superior token-level interventions compared to existing methods.
View Cached Full Text
Cached at: 05/14/26, 04:17 AM
Paper page - WriteSAE: Sparse Autoencoders for Recurrent State
Source: https://huggingface.co/papers/2605.12770
Abstract
WriteSAE enables sparse autoencoder decomposition and editing of matrix cache writes in state-space and hybrid recurrent language models, achieving superior performance in token-level interventions compared to existing methods.
We introduce WriteSAE, the firstsparse autoencoderthat decomposes and edits thematrix cache writeof state-space andhybrid recurrent language models, whereresidual SAEscannot reach. Existing SAEs read residual streams, butGated DeltaNet,Mamba-2, andRWKV-7write to a d_k times d_v cache throughrank-1 updatesk_t v_t^top that no vector atom can replace. WriteSAE factors each decoder atom into the native write shape, exposes a closed form for the per-token logit shift, and trains under matchedFrobenius normso atoms swap one cache slot at a time.Atom substitutionbeats matched-norm ablation on 92.4% of n=4{,}851 firings at Qwen3.5-0.8B L9 H4, the 87-atom population test holds at 89.8%, the closed form predicts measured effects at R^2=0.98, andMamba-2-370M substitutes at 88.1% over 2,500 firings. Sustained three-position installs at 3times lift midrank target-in-continuation from 33.3% to 100% undergreedy decoding, the first behavioral install at the matrix-recurrent write site.
View arXiv pageView PDFProject pageGitHub0Add to collection
Get this paper in your agent:
hf papers read 2605\.12770
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper1
#### JackYoung27/writesae-ckpts Feature Extraction• Updatedabout 2 hours ago
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2605.12770 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2605.12770 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Rational Sparse Autoencoder
Introduces Rational Sparse Autoencoder (RSAE), which replaces fixed encoder activations with trainable rational functions, improving reconstruction and sparsity trade-offs on residual-stream activations of open-weight language models across multiple baseline families.
Sparse Autoencoders enable Robust and Interpretable Fine-tuning of CLIP models
SAE-FT introduces a novel fine-tuning method for CLIP models that uses sparse autoencoder constraints to regularize visual representations, improving robustness against distribution shifts while maintaining performance and enabling interpretability.
Turn-Averaged SAEs for Feature Discovery and Long-Context Attribution
This paper introduces turn-averaged sparse autoencoders (SAEs) that operate on average activations across conversational turns, enabling efficient feature discovery and attribution graphs for long contexts. It also proposes a nested architecture for joint training with per-token features.
Decompose Sparsely Where You Should, Absorb Densely Where You Should No
The paper hypothesizes that language model activations contain a low-rank dense component that is inefficiently represented by sparse autoencoders (SAEs). By adding a linear bottleneck to absorb dense structure, the authors reduce dense latents and improve sparse probing performance on Gemma-2-2B.
VASAE: Naming SAE Dictionary Directions with Vocabulary-Aligned Anchoring
The paper introduces Vocabulary-Aligned Sparse Autoencoder (VASAE), a method that trains SAE features under vocabulary-aligned anchoring, assigning each feature an intrinsic token name based on nearest token embedding, achieving high alignment in early layers without reducing reconstruction quality.