@Phoenixyin13: If the full score is 10, I would honestly give this MIT paper's SMT idea and writing an 8. The paper proposes Supervised Memory Training, using Transformer as a super teacher to first distill in parallel the most important things to remember at each moment…

X AI KOLs Timeline Papers
supervised-memory-training rnn transformer parallel-training long-range-dependencies mit

Summary

This paper proposes Supervised Memory Training (SMT), which uses Transformer as a super teacher to distill memory states in parallel, then trains RNN with one-step supervised learning, achieving fully parallel training and reducing gradient path from O(T) to O(1), significantly improving long-range dependency learning.

If the full score is 10, I would honestly give this MIT paper's SMT idea and writing an 8. The paper proposes the idea of Supervised Memory Training: using Transformer as a super teacher, first distill in parallel the memory states that should be remembered at each moment, then teach the RNN with just one step of supervised learning. The result is fully parallel training, with the gradient path shortened from O(T) to O(1), and long-range dependencies are learned much better. Before, when training RNNs, I felt it was like teaching someone to memorize an entire long novel from start to finish page by page, and making a mistake early on would affect later parts. With SMT, the Transformer first distills the key memory points of the whole book, then teaches the RNN only what to remember at page t and how to update memory when turning to page t+1. Each step can be taught simultaneously, so learning is faster and better. It makes RNNs efficient to train and scale again, making them extremely suitable for scenarios that require fixed memory and sequential processing with inference, such as agents, phones, robots, or real-time AI. The most mechanistic and worth-reading parts of the paper, in my opinion, are Fig 9's compression-as-scaling-axis and Fig 14's memory geometry. We may once again have recurrent models that are both expressive and efficient, no longer forced to rely entirely on Transformer's KV cache eating up VRAM. I don't know if this will bring RNNs back to life, but the idea of this paper feels quite interesting.
Original Article
View Cached Full Text

Cached at: 06/16/26, 11:43 PM

If the score is out of 10, I would genuinely give this MIT paper an 8 for both the SMT idea and the writing.

The paper proposes Supervised Memory Training: using a Transformer as a super teacher to first distill, in parallel, the memory states that should be remembered at each timestep, then teaching the RNN with just a single step of supervised learning. The result is fully parallel training, with the gradient path shortened from O(T) to O(1), leading to significantly better learning of long-range dependencies.

Previously, training an RNN felt like teaching someone to memorize an entire long novel page by page from beginning to end — if you made a mistake earlier, it would affect later parts.

With this SMT approach, the Transformer first extracts the key memory points from the whole book, then only teaches the RNN what should be remembered from page t, and how to update memory when moving to page t+1. Every step can be taught simultaneously, making learning both fast and effective.

It makes RNNs efficiently trainable and scalable again, ideal for scenarios that require fixed memory budgets and sequential processing with inference, such as agents, phones, robots, or real-time AI.

The most mechanistic and worth-reading parts, in my opinion, are Fig 9 (compression-as-scaling-axis) and Fig 14 (memory geometry).

We might once again have recurrent models that are both expressive and efficient, without being forced to rely solely on Transformers’ KV cache eating up GPU memory.

I’m not sure if this will revive RNNs, but the idea of this paper feels quite interesting.

Grigory Sapunov (@che_shr_cat):
1/ We have been training RNNs wrong for decades.

Backpropagation through time (BPTT) forces sequential updates, creating unstable O(T) gradient paths.

What if we could train highly expressive, non-linear RNNs with flat, parallelized O(1) gradients?

It is now possible. 🧵

Similar Articles

@Phoenixyin13: AI has fallen into an either-or trap. On one side is the world-dominating Transformer architecture — excellent memory, but its quadratic computational explosion makes long contexts increasingly expensive, a real resource hog. On the other is the classic RNN architecture — lightning fast and cheap, but a total scatterbrain that forgets earlier content after a few more lines.

X AI KOLs Timeline

This article introduces a new method proposed by Google Research, Cornell, and USC that takes snapshots of RNN memory and caches them, enabling RNNs to efficiently handle long contexts. It combines Transformer-like strong memory with RNN-like low cost, offering a new direction for long-context AI.

You could have invented transformers from memory

Hacker News Top

一篇技术博客，通过"关联记忆"的视角重新推导 Transformer：从线性层的外积记忆，到 MLP 的容量提升，再到自注意力的近乎无界记忆容量，并给出各层可存储记忆数量的启发式估算（如 64 维键可从约 64 条提升到上百万条记忆）。

@vintcessun: A 'knowledge hard drive' that can be plugged across models — the difficulty lies not in moving the memory table, but in configuring the read head for the target model. https://arxiv.org/abs/2608.17050 The paper first trains Engram hash memory with the source model, then freezes the memory and target backbone, only training the reader to complete addressing...

X AI KOLs Timeline

This paper proposes a method for cross-model memory transfer through target-side reader adaptation, using Engram hash memory and a lightweight reader, achieving 38.8 on QA tasks, and applicable to Agent memory updates and audits. The limitation is that it was only tested up to 9B models, with scaling laws unknown.

@ba_niu80557: https://x.com/ba_niu80557/status/2073362883728302125

X AI KOLs Timeline

Stanford University proposes the AutoMem method, which allows models to learn memory management (selective forgetting) instead of expanding parameters. This doubles the performance of a 32-billion-parameter small model and matches top-tier large models, revealing that memory management is more important than model scale.