@daylenyang: this berkeley 189 lecture is probably the clearest explainer of the attention mechanism i've come across. provides a ve…

this berkeley 189 lecture is probably the clearest explainer of the attention mechanism i’ve come across. provides a very good intuitive understanding. if you already know about CNNs then scroll to about halfway in: https://t.co/oVUGqSV15O

TL;DR

The Berkeley 189 lecture explains the attention mechanism with clear logic: from the history of RNN + attention, to how Transformer completely discards RNN and achieves powerful representation learning with attention alone, and compares the parameter efficiency and scalability of MLP vs CNN.

Why the Attention Mechanism Is So Important

Today, almost every modern model—including large language models, diffusion models, image/video generation models—uses the Transformer architecture at its core. And the heart of the Transformer is the attention mechanism. The 2017 paper Attention Is All You Need defined this famous architecture diagram, and it is probably one of the most notable AI papers of the past decade, with authors from the University of Toronto and Google Brain.

But “Attention Is All You Need” implies there is something else you don’t need. What is that something else? The answer is recurrent neural networks (RNNs). The attention mechanism was invented as early as 2014, but it was always paired with RNNs. The central conclusion of this paper is: You don’t need RNNs—you only need the attention part. Thus was born the Transformer architecture, ubiquitous today.

A Clear Historical Evolution Path

The lecture starts from the CNNs you have already learned, then reviews how attention was introduced, and finally shows how the Transformer completely eliminated RNNs. The basic idea is:

• You learned CNNs (convolutional neural networks).
• Then came innovations like residual connections.
• Attention was discovered and initially combined with RNNs.
• The Transformer said: attention is all you need.

Now, almost all large-scale networks are based on the Transformer architecture, even though it was originally designed for natural language. It has been widely applied to computer vision, multimodal tasks, diffusion models, video generation, etc. It is widely believed that Transformer is currently the most powerful general-purpose architecture.

Common Idea Across All Architectures: Representation Learning

Whether CNN, RNN, or Transformer, the core is feature transformation (representation learning). Initial inputs (words or pixels) are converted into vectors (embeddings), and neural networks build new representations layer by layer. The goal is for deeper representations to be more “context-aware”—that is, to incorporate information from distant words or pixels.

The final layers should represent abstract concepts relevant to the prediction task. For example, when distinguishing cats from dogs:

• Initial pixel layer: small hairs, small shapes.
• Middle layers: whiskers, ears, and other features of cats.
• Final layers: abstract concepts such as fur type, eye shape, etc.

These intermediate representations are always vectors (embeddings).

The First Network: Multilayer Perceptron (MLP)

MLP is a fully connected network consisting of linear layers (matrix multiplication) and ReLU nonlinearity:

h1 = ReLU(W1 * x) h2 = ReLU(W2 * h1) ...

Assume the embedding dimension is D, the number of layers is L, and each layer is a D×D matrix plus bias (which can be ignored). So the number of parameters is roughly D² × L. When D=16000, D² becomes 256 million, which is enormous. MLP is the most expressive architecture (any other architecture can be simulated by a huge MLP), but it has too many parameters and cannot change the input size—if the input changes from D to D+5, you have to learn new weights from scratch.

How Convolution Solves MLP’s Shortcomings

CNN uses a fixed receptive field (e.g., a 3×3 kernel) that slides over the input, using the same weights. For a single-channel 3×3 kernel, there are only 9 parameters, reused across the entire image. In contrast, for an MLP processing a 1000×1000 pixel image, each layer would need P² weights (P=10⁶, or one trillion), while CNN has a fixed parameter count of 9, independent of image size.

CNN can also handle boundary pixels with zero padding, and when the input image size changes (e.g., from 3×3 to 3×4), the same kernel can continue to slide without retraining.

Summary

The lecture starts from the parameter explosion problem of MLP, leads to the efficiency of convolution, and then lays the groundwork for explaining attention mechanism and Transformer. The attention mechanism will completely change the way information interacts—no longer limited to local receptive fields, but allowing every position to directly attend to all other positions.

Source
YouTube: @daylenyang - this berkeley 189 lecture is probably the clearest explainer of the attention mechanism… (https://www.youtube.com/watch?v=mqaFEvi5rWE)