The article speculates that the next major breakthrough after the attention mechanism may involve AI architectures with input-dependent weights, potentially building on DeepSeek's Engram mechanism.
Hi everyone, I've been thinking about this for a while now. The attention mechanism introduced by Google in 2017 completely changed that way AI models worked and performed. Since the attention mech allows the model look at it's inputs and decide what's more worth it to focus on, which made the input signal passing through the FFN so much more richer in so many ways that it made the Transformer capable of achieving so many complex capabilities such as in-context learning and all, along with scaling up to such a degree we see today. Recently a new mechanism called the Engram was introduced by DeepSeek which is very hyped for all good reasons, as far as I understand it can allow a much small model to hold a lot more world knowledge than a traditional model can. I don't necessarily think it'll make the model smarter but more knowledgeable? 100% But I have a very weird gut feeling that DeepSeek's Engram (or other similar mechanisms) isn't a breakthrough as big and significant as the attention mechanism but just the 1st half of the next major breakthrough. Ik this sound very contrarian but I can't help it but think this way. I believe that the next big breakthrough as big as attention mechanism is going to be a model whose most parameters are input-dependent. A model which uses a set of fixed particularly smaller weight matrices to generate "fast-weights". Idk how well im able to explain it but I'll use the attention mech to explain it. We can think of the attention mech as a fast-weight layer which generates these weights from the input data and uses it to enrich the input quality which in-turn improves performance. But that input still goes to an FFN with all fixed weights. I'm thinking of an FFN arch which is going to use the engram + input-dependent weights to generate it's weights on the fly not just making the model much smaller but smarter in general or might even unlock some capabilities which might be only reserved for much larger or deeper models using our current architecture. Tbh idk how well I communicated my idea here but I hope you get my point and I would love to know your opinions and points on it. Thank you :)
Introduces a geometric framework to identify 'AI engrams' – memory traces in deep neural networks – formalizing neuroscientific criteria into a closed-form estimator, enabling surgical memory manipulation in models from MLPs to LLMs.
Engram is described as DeepSeek's most significant breakthrough of 2026, coming shortly after the company open sourced its entire inference serving stack for free.
A speculative discussion about the next architectural innovations likely to appear in AI models, including looped transformers, Gated Delta Net 2 variants, attention residuals, and a potential revival of diffusion transformers without quality loss.
A retrospective deep dive tracing the evolutionary arc of AI architectures from the single perceptron through early neural nets, the vanishing gradient problem, LSTMs, and the Transformer era, while speculating about the next paradigm shift beyond self-attention.