What's the next big breakthrough after attention mechanism? My bet is not on Engrams.

Reddit r/LocalLLaMA News

Summary

The article speculates that the next major breakthrough after the attention mechanism may involve AI architectures with input-dependent weights, potentially building on DeepSeek's Engram mechanism.

Hi everyone, I've been thinking about this for a while now. The attention mechanism introduced by Google in 2017 completely changed that way AI models worked and performed. Since the attention mech allows the model look at it's inputs and decide what's more worth it to focus on, which made the input signal passing through the FFN so much more richer in so many ways that it made the Transformer capable of achieving so many complex capabilities such as in-context learning and all, along with scaling up to such a degree we see today. Recently a new mechanism called the Engram was introduced by DeepSeek which is very hyped for all good reasons, as far as I understand it can allow a much small model to hold a lot more world knowledge than a traditional model can. I don't necessarily think it'll make the model smarter but more knowledgeable? 100% But I have a very weird gut feeling that DeepSeek's Engram (or other similar mechanisms) isn't a breakthrough as big and significant as the attention mechanism but just the 1st half of the next major breakthrough. Ik this sound very contrarian but I can't help it but think this way. I believe that the next big breakthrough as big as attention mechanism is going to be a model whose most parameters are input-dependent. A model which uses a set of fixed particularly smaller weight matrices to generate "fast-weights". Idk how well im able to explain it but I'll use the attention mech to explain it. We can think of the attention mech as a fast-weight layer which generates these weights from the input data and uses it to enrich the input quality which in-turn improves performance. But that input still goes to an FFN with all fixed weights. I'm thinking of an FFN arch which is going to use the engram + input-dependent weights to generate it's weights on the fly not just making the model much smaller but smarter in general or might even unlock some capabilities which might be only reserved for much larger or deeper models using our current architecture. Tbh idk how well I communicated my idea here but I hope you get my point and I would love to know your opinions and points on it. Thank you :)
Original Article

Similar Articles

Is 'Attention' the Essence of Intelligence?

Hacker News Top

Quanta Magazine 探索「注意力」机制如何成为现代 AI 的核心:从 Dzmitry Bahdanau 2014 年的机器翻译研究,到 Google 2017 年的 Transformer 论文,再到 ChatGPT 等大语言模型的崛起。