A trained fast-weight memory: a 3M-param transformer installs never-trained rules at inference, forward-only — where test-time training transfers nothing (single RTX 3090, fully reproducible)
Summary
This paper introduces a trained fast-weight memory mechanism for a 3M-parameter transformer that installs never-trained rules at inference using forward-only test-time training, showing no transfer. The work is fully reproducible on a single RTX 3090.
Similar Articles
A new transformer passes its hidden state to the next token instead of recomputing it (25 minute read)
The paper introduces LIFT (Latent Information Feedback Transformer), which propagates deep-layer hidden states across generation steps instead of relying solely on decoded tokens, using teacher-forced pretraining with states derived from an off-the-shelf LM's next-token distribution. Experiments on 135M–1B models show LIFT outperforms standard Transformers on language modeling, reasoning, and procedural tasks under token-matched budgets, with code and models released.
@TeachTheMachine: Using a Transformer Model: From Training to Inference
This tutorial covers how to use a transformer model from training to inference, focusing on autoregressive generation, prefill vs. decode phases, and key-value caching for efficient inference.
The Surprising Effectiveness of Shared Memory in Looped Transformers
An IBM Research/Cornell paper shows that pretraining Looped Prediction Transformers (LPT) so later recursions share the KV cache of the first recursion improves perplexity while using 76–79% less context memory at 150M–1B parameter scale, establishing memory sharing as a beneficial architectural inductive bias rather than just a deployment-time trick.
Language Models Need Sleep
This paper proposes a sleep-like consolidation mechanism for transformer models that uses fast weights and recurrent passes to improve long-context processing while maintaining inference speed.
Maglev: Sliding Recurrent Memory
Introduces Maglev, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. It uses a prefiller and decoder with a memory consistency loss, improving validation loss and downstream benchmarks over baselines.