Completely Understand Transformer at Once: Attention, QKV, Residual Connections Explained Thoroughly

Have you ever wondered how large AI models like GPT understand our prompts and then generate responses word by word?

In the previous episode, we introduced the technological development roadmap before the birth of the Transformer.

If you haven’t seen the previous episode, it’s okay. Let’s quickly review.

Initially, there was N-gram, which had a very simple idea: statistically predicting what the next word would typically be after a few previous words appeared.

However, N-gram only counted statistics; it didn’t understand the semantic relationships between different words.

So later came word vectors, also known as word embeddings.

Word vectors turn each word into a set of high-dimensional numbers, mapping the meaning of words into these numbers. Words with similar meanings are closer to each other in the vector space.

With word vectors, we then fed these vectors into a feedforward neural network, letting the neural network learn more complex language patterns and then predict the next word.

But feedforward neural networks can only accept a fixed number of tokens at a time. When previous content is too far away, it cannot be seen.

So later came RNN, or recurrent neural networks.

RNN added a hidden state that constantly passes backward, effectively giving the model a memory, allowing information from earlier to keep being passed along.

However, RNN’s long-term memory capability isn’t great; when the distance is long, important information from earlier can easily be gradually lost.

Then later came LSTM.

LSTM added a long-term memory channel and several gates, allowing the model itself to learn what should be remembered, what can be forgotten, and when to use this information again.

Development up to this point seems quite good.

But LSTM still has two very troublesome problems.

The first problem is that it must compute tokens sequentially, one after another.

If the previous token hasn’t finished computing, the next token cannot start. If the input content is relatively long, efficiency becomes very low.

The second problem is that for two tokens very far apart to exchange information, they still need to go through many tokens in between, passing the information step by step.

For example, in the previous episode, we gave this example:

Xuanyuan had a budgerigar when he was young.

It had green feathers and could mimic human speech.

Later, Xuanyuan went to study in another place and hadn’t seen it for many years.

But he always remembered that ____ from his childhood.

The final “that one” clearly relates directly to the distant “budgerigar” earlier.

So why must the information about “budgerigar” go through dozens of tokens in between, passing step by step to “that one”?

Can’t we just directly establish a connection between “that one” and the earlier “budgerigar”?

Yes, brothers, yes.

The story of Transformer officially begins here.

Transformer comes from the highly acclaimed 2017 paper “Attention is all you need.” In today’s video, we will thoroughly understand what exactly this ancestor of today’s large AI models, Transformer, is.

Welcome to like, subscribe, and not get lost. Let’s begin today’s journey.

I. Stop Passing Messages, Let Tokens Find Themselves.

Let’s temporarily forget the name “Transformer” and return to our word-guessing game.

When we humans read “But he always remembered that ____ from his childhood,” our brains naturally look back for information related to “that one” in the preceding text.

But we don’t analyze the previous content from start to finish one by one again.

We’re more like scanning the previous content in an instant, then quickly noticing the strongest relationship between “budgerigar” and “that one.”

Can we make the neural network do the same?

We can let every token in a sentence have the chance to directly examine all other tokens.

Then each token asks: “Among these words, which are most relevant to me?”

To make it easier to understand, let’s use a shorter example.

Suppose we have this sentence: “Xuanyuan has a budgerigar; it has green feathers.”

Now we are processing the token “it.”

Humans certainly know that “it” here most likely refers to the earlier “budgerigar,” not Xuanyuan.

So we also let the model look at all the tokens in the sentence. How exactly it does this you’ll see in a moment.

The model might calculate that “Xuanyuan” has a weak relationship with “it,” “has” also has a weak relationship with “it,” while “budgerigar” has a very strong relationship with “it.”

Suppose we finally get a set of weights like this.

“Xuanyuan” has a weight of 10%, “has” has a weight of 5%, “budgerigar” has a weight of 70%, and the other tokens share the remaining 15%.

Next, we mix the information carried by different tokens according to these weights.

“ Budgerigar“ has the highest weight, so we take more of its information.

“Has” has a very low weight, so we take less of its information.

After all the information is combined according to different weights, “it” will receive a new vector.

Note that this new vector is no longer the initial, lonely word vector for “it.”

The original word vector primarily expresses what the word “it” itself generally means.

After this operation, the new vector incorporates the context of the current sentence.

Especially the information from “budgerigar” will largely enter the new vector for “it.”

This way, when the model continues to process “it has green feathers,” it will know that this “it” has a very strong relationship with the budgerigar.

The same method can be applied to every token in the sentence.

Every token in the sentence can directly examine other tokens and then bring back information related to itself.

“Green” can look for “feathers,” which has the strongest relationship with it.

“Has” can look for “Xuanyuan” and “budgerigar,” which have strong relationships with it.

Tokens in a sentence thus examine each other and exchange information.

Whether separated by one token or a hundred tokens, there’s no need to pass messages along like RNN.

The mechanism we just designed is Attention, which is the core idea of the attention mechanism.

The so-called “attention” simply means letting the current token judge how important other tokens are to itself, and then focusing on reading the important information.

The above description sounds great, but the problem is: how should we implement this approach?

Each token looks at other tokens; how exactly does it look?

When looking, which tokens should it care about, and how are the weight values for these tokens calculated?

Next is the key to understanding the attention mechanism.

II. Attention Mechanism: How Does the Model Know Who to Focus On?

Let’s continue with the example we just had.

When the model processes the token “it,” to complete the next prediction task, it needs to find out who exactly “it” refers to in the preceding sentence.

Before searching, it needs to first express one thing: “What am I looking for now?”

This is like when we search for something online; we have to input search keywords.

So we need to prepare a “search keyword” specifically for “querying” for each token.

For example, “it” in the current context might be looking for a previously mentioned object—generally, this would be a thing or an object, most likely a noun, less likely an adjective or adverb.

Similar to the word vectors of previous tokens, this “search keyword” isn’t text strings; it’s still a set of high-dimensional numbers, and the search intent is contained within.

This vector used to express “What am I looking for now” is Query, or Q for short.

But only querying isn’t enough.

Each token also needs to declare: “What information do I provide for querying here?”

For example, the token “budgerigar” can provide a piece of information suitable for querying.

“Xuanyuan” also provides its own information.

“Green” also provides its own information.

This vector specifically used to match with Query is Key, or K for short.

Now the token “it” has its own Q, and each previous token has its own K.

Next, “it” can take its Q and compare it with others’ K one by one.

If “it”’s Q matches very well with “budgerigar”’s K, the model knows “budgerigar” is worth focusing on.

If it doesn’t match well with “green”’s K, then “green” gets less attention.

So the question is, how exactly do Q and K match?

We can have these two vectors do a dot product.

After the dot product, we get a number.

The larger this number, the more we consider these two vectors to match.

For example, “it”’s Q and “Xuanyuan”’s K compute to 1.2.

“It”’s Q and “green”’s K compute to 0.8.

“It”’s Q and “budgerigar”’s K compute to 6.8.

Obviously, the “budgerigar” position is more worthy of attention.

Then, through Softmax, convert these scores into corresponding weights.

The model then knows which token each should focus on and how much to focus.

Seeing this, everyone might have a question: How can we guarantee that “it”’s Q and “budgerigar”’s K match perfectly?

The answer is still through training.

At the beginning, the matching is surely random. As training proceeds, the relevant parameters are constantly adjusted, and the Q and K matching becomes increasingly close to the patterns in language.

So next, after knowing who to focus on, what then?

What’s truly useful is the specific information carried by the token “budgerigar”—neither the token’s Q nor its K. These two are used for querying and being queried.

So each token needs to prepare a third vector.

This vector isn’t responsible for searching or matching; it’s responsible for storing the information that truly needs to be read by others.

This vector is Value, or V for short.

Thus, we have assembled the three key players in the attention mechanism: Q, K, and V.

Q means “What am I looking for.”

K means “What do I roughly have here.”

V means “What information do I truly carry.”

Again, emphasize that this is just an analogy to help understand.

The actual model doesn’t really write text labels like “pet,” “color,” or “person” into Q, K, and V.

Their essence is still a set of high-dimensional numbers; all the above information is hidden in these high-dimensional vectors.

Then a new problem arises.

Doesn’t each token only have one word vector?

How come each token suddenly has three vectors now?

This problem is actually easy to solve.

Suppose “budgerigar” now has a vector of several hundred dimensions.

We give this vector to a linear transformation to get the first set of new numbers, Q.

Then give it to another linear transformation to get the second set of numbers, K.

Then through a third linear transformation, we get the third set of numbers, V.

Every token does the same thing.

“Xuanyuan” has its own Q, K, V; “has” has its own Q, K, V; “budgerigar” also has its own Q, K, V.

As for how exactly these three linear transformations are performed and what parameters are used in the transformations, again, programmers don’t need to design them manually.

The model will slowly learn them during training through repeated prediction errors and backpropagation.

With Q, K, and V, next becomes simple.

Multiply these attention weights by each token’s corresponding V, then sum them all up.

Tokens with high weights contribute more information; tokens with low weights contribute less information.

What we get in the end is the new vector for the current token after integrating the context.

In the paper “Attention is all you need,” the attention formula does these core things.

In this paper’s formula, after the dot product of Q and K, there’s a division by the square root of d.

This is mainly because when the vector dimensions are high, the dot product result can become large, and after Softmax, it might become too extreme.

So first scaling it down appropriately can make training more stable.

There’s also another detail.

Our Q, K, and V all come from tokens in the same sentence.

Tokens within a sentence examine themselves.

This situation is called Self-Attention.

Correspondingly, if attention calculation is done between one sentence and other sentences, it’s not self-attention; it has other names, which we’ll see soon.

However, having only one set of Q, K, and V is still not enough.

Let’s look at this sentence.

“Xuanyuan had a budgerigar when he was young; it had green feathers.”

The relationships between tokens in this are very complex.

“It” and “budgerigar” have a clear referential relationship.

“Green” and “feathers” have a modifying relationship.

“Xuanyuan” and “had” have an action relationship.

“Had” and “budgerigar” also have another relationship.

In other words, within the same sentence, there might simultaneously be syntactic relationships, semantic relationships, referential relationships, long-distance relationships, etc.

But we only had one set of Q, K, and V, making it hard to cover multiple relationships at once.

Since one set isn’t enough, let’s do multiple sets.

The same sentence simultaneously undergoes multiple groups of attention calculation.

The first group might focus on referential relationships.

The second group might focus on nearby words.

The third group might have learned some long-distance relationships.

The fourth group might learn other completely different patterns.

Of course, these “who handles reference, who handles syntax” aren’t pre-arranged by programmers.

What each group of attention ultimately learns is determined by training itself.

We are only responsible for giving it multiple different observation spaces.

Finally, combine the results from these different attentions to form a richer representation.

This method of “doing attention simultaneously from multiple angles at once” is multi-head attention.

A “head” can be temporarily understood as a set of independent attention calculations.

If one angle isn’t clear, look from multiple angles together; that way, what you see is relatively comprehensive.

In the paper “Attention is all you need,” the original Transformer used 8 heads.

This way of calculating attention solves both of the LSTM’s problems.

Two tokens far apart no longer need to pass information through a string of tokens in between; they can directly establish connections.

And tokens within a sentence don’t need to be processed one after another like in RNN or LSTM.

All tokens’ Q, K, and V can be calculated together.

All matches between Q and K can also be done together through large-scale matrix operations.

So compared to RNN and LSTM, Transformer is easier to utilize GPUs for large-scale parallel training.

This is also a very important prerequisite for models being able to grow larger and larger.

III. Positional Encoding: How Does Parallel Computing Know the Order of Tokens?

However, this parallel calculation introduces a very critical problem: the information about the order between tokens is lost.

The original RNN and LSTM were processed serially, so they were inherently ordered calculations.

But parallel processing computes all tokens together; how does the model know who comes first and who comes later?

For example, these two sentences.

“Xuanyuan has a budgerigar.”

“The budgerigar has Xuanyuan.”

The words appearing in both sentences are three identical tokens.

But because the order changed, the meaning is completely different.

The attention mechanism only calculates the correlation between tokens; it doesn’t handle the order of words.

So what can we do?

We can add a piece of information specifically representing position to each token’s original word vector.

The first token gets a piece of information representing position 1.

The second token gets a piece of information representing position 2.

And so on.

This way, what the model receives isn’t just “who this word is.”

It also knows “where this word is located.”

The additional position data added just now is Positional Encoding.

The original Transformer used sine and cosine functions to generate positional encoding.

Current large language models have developed some other schemes.

We won’t go into these details in this episode.

You just need to remember one sentence.

Embedding tells the model the semantics of tokens, while positional encoding tells the model the order of tokens.

At this point, each token can already carry its own semantic and positional information, exchanging information with other tokens on a large scale through multi-head attention.

This looks quite good.

But this is still far from our goal. We are to complete the word-guessing game: predict the next word based on the preceding content of a sentence.

Attention is great at helping tokens search for information everywhere.

But after the information is found, what next? How does our word-guessing game continue?

Let’s continue reading.

IV. Feedforward Neural Network: After Finding the Information, You Still Need to Digest It Yourself.

Let’s look at this sentence:

“Xuanyuan just bought a black apple for over ten thousand yuan.”

Is this “apple” a fruit or a product from Apple Inc.?

You, being smart, can almost instantly notice that there’s no edible apple in the world that sells for over ten thousand yuan, and it’s black.

So this must refer to a product from Apple Inc.

So let’s see how the computer analyzes this point.

To make it easier to understand, let’s assume that from “apple”’s original vector of several hundred dimensions, we only pick four features to observe.

They are:

High price, black appearance, fruit attributes, and tech attributes.

Note: Actually, the vector semantics are very complex; it’s almost never that certain dimensions exactly correspond to these four aspects. Here, for easy understanding, we’re simplifying the word vector dimension setting.

At first, only seeing the word “apple,” it might be:

0.1, 0.1, 0.6, 0.6.

Meaning there isn’t much information about high price or black appearance yet.

And both fruit and tech are possible meanings.

Next, after attention calculation.

“Apple” starts looking at other tokens around it.

It gets the appearance information from “black.”

It also gets very strong high-price information from “over ten thousand yuan.”

So its vector might become:

0.9, 0.9, 0.6, 0.6.

Everyone note.

Up to this point, what Attention mainly completed is:

Gathering clues originally scattered in other tokens into “apple.”

Now the model knows this “apple”:

It’s expensive.

It’s also black.

But the two possibilities, “fruit” and “tech product,” haven’t really differentiated yet.

So what next?

We need a new component to further analyze and calculate these clues.

This is where our old friend from the previous video comes in: Feedforward Neural Network, or FFN.

FFN doesn’t look at other tokens anymore.

It only looks at the information “apple” now possesses.

Then it learns the combination relationships between these features.

For example: this “apple” here.

The price is very high.

And it’s black.

These two features combined make it very unlikely to be a fruit.

Instead, it very much resembles some electronic product from Apple Inc.

So after FFN, this set of vectors might further become:

0.9, 0.9, 0.1, 0.95.

The clues about high price and black appearance that have been brought back are still there.

But the fruit attribute is significantly suppressed.

The tech attribute is greatly enhanced.

So what Attention and FFN do is actually different.

The attention mechanism is responsible for letting tokens exchange information, while the feedforward neural network is responsible for combining these pieces of information for each token, analyzing and evaluating them.

One handles communication, one handles thinking.

At this point, these two components can be assembled into a small unit.

First, exchange information through multi-head attention, then digest and process through the feedforward network.

This is the basic unit of Transformer.

However, processing one unit isn’t enough; in practice, Transformer stacks several layers, each understanding deeper than the last.

The original paper stacked six layers; current large language models often have dozens or even hundreds of layers.

The knowledge and abilities of large language models are all contained in these network parameters.

But, as the network gets deeper, new problems arise again and again.

V. Residual Connections: Making Training Possible

We just said, in order to understand complex language, Transformer stacks attention and FFN modules many layers to gradually extract more abstract features.

But as the network gets deeper, training can easily go wrong.

Model training relies on backpropagation. After prediction errors occur, the gradient passes all the way forward from the last layer, telling the earlier parameters how to adjust.

This process fundamentally relies on the chain rule for derivatives.

If the gradient continuously diminishes as it passes through layer after layer of calculations, by the time it reaches the very front, it might be so weak that it hardly has any effect.

The front layers don’t receive effective adjustment signals, making them increasingly difficult to train.

This is the classic vanishing gradient problem in deep neural networks.

So what can be done?

Researchers came up with a method that seems strange at first glance: directly add the original input of this layer to the output.

Suppose the input of this layer is x, and after some module calculation, we get f(x).

Normally:

y = f(x).

Now, additionally add the original x back:

y = f(x) + x.

Wait a moment, if we add x directly, isn’t the result changed?

When I first saw this, I was also full of questions.

The key here is that the function f isn’t fixed; it’s itself learned through training.

Suppose the result we truly want to get is F(x).

Originally, f needed to learn completely:

f(x) = F(x).

But now an x will automatically be added at the end, so f only needs to learn:

f(x) = F(x) - x.

Finally, add x back; the result is still F(x).

So f can be understood as only responsible for learning: “On the basis of the original x, how much more is needed.”

This “how much more is needed” is the residual.

So this structure is called a residual connection.

The really powerful part of this thing is in backpropagation.

For: y = f(x) + x, after taking the derivative:

y’ = f’(x) + 1.

The key is this “+1” at the end.

Originally, the gradient could only pass forward through the complex path of f.

Now, an additional shortcut directly connecting input and output has appeared, and the derivative of this shortcut is 1.

So even if the gradient on the path of f is weakened, there is still a more direct path for the gradient to continue passing forward.

This doesn’t guarantee the gradient will never diminish, but it greatly alleviates the vanishing gradient problem in deep networks.

Moreover, during forward propagation, the original information can also pass directly backward through this shortcut, without having to be thoroughly reprocessed every layer.

So residual connections can be understood as building a highway for the network.

During forward propagation, information can go deeper along it.

During backpropagation, the gradient can pass back along it.

This way, Transformer can be stably stacked to dozens or even hundreds of layers.

In actual Transformers, residual connections are usually paired with normalization modules to make training more stable.

So in classic structure diagrams, you often see “Add & Norm.”

OK, now tokens can exchange information through attention, FFN can further process information, and residual connections allow these modules to be continuously stacked higher.

But have we forgotten something?

We’ve been busy for so long, just constantly processing each token’s vector.

But our initial ultimate goal was to truly predict the next token based on preceding content.

How does Transformer finally turn these high-dimensional vectors into a specific word?

In today’s final part, we’ll add this last piece of the puzzle.

VI. Encoder and Decoder: Reading a Sentence and Writing a Sentence Are Actually Two Different Things.

The paper “Attention Is All You Need” initially solved machine translation.

For example, translating:

Xuanyuan has a budgerigar.

Into:

Xuanyuan has a budgie.

Translation work can be broken down into two steps.

First understand the original text, then write the translation.

So Transformer was also divided into two parts.

The one responsible for “reading” is called the Encoder.

The one responsible for “writing” is called the Decoder.

The encoder is the structure we built earlier.

Input first becomes an embedding, adds positional information, then passes through layers of self-attention and FFN.

Here, each token can examine all other tokens in the entire sentence, including later content.

This is fine because the encoder’s task is to understand an entire existing original sentence.

Finally, each token becomes a vector that integrates information from the whole sentence.

The encoder isn’t responsible for generating new tokens; it only encodes “what the original text means” into these vectors.

Next, it’s the decoder’s turn to start writing the translation.

It won’t write the entire sentence in one go; instead, it generates tokens one by one.

For example, if it has already generated:

Xuanyuan has a

Next, it needs to predict:

budgie.

These already generated tokens also need to do self-attention.

But immediately, a problem arises here.

During training, the correct answer for the entire sentence actually already exists.

If the model, when predicting budgie, can see the later budgie in advance, that’s equivalent to directly copying the answer.

So the decoder must follow one rule:

It can only see itself and previous tokens; it cannot see later tokens.

How is this achieved?

When calculating attention scores, directly change the scores for later positions to an extremely small number.

After Softmax, the weights for these positions become almost zero, effectively masking them.

This masking operation is called Mask.

Self-attention with masking is called masked self-attention.

But another problem arises.

If the decoder only looks at the English it has already written, how does it know what the original text said?

For example, after Xuanyuan has a, should it be cat, dog, or budgie?

The real answer is actually in the original text.

So the decoder also needs to look back at the encoder while writing.

It uses its own Q to query the encoder’s output K and V.

If the current generation should be budgie, then this Q might highly match the K corresponding to “budgerigar” in the original text, and then bring over a large amount of that V.

Note that this time Q comes from the Decoder, while K and V come from the Encoder.

This attention performed across both sides is called cross-attention.

So each layer of the decoder can be simply understood as three steps.

First, use Masked Self-Attention to see what it has written before.

Then use Cross-Attention to look back at the original text.

Then pass it to FFN for further processing.

Finally, what the Decoder gets is still a set of vectors.

So how does it actually “pop out” a word?

It’s the same process as in the previous episode.

Connect the vector of the last position to the output layer, scoring for all tokens in the vocabulary.

Then through Softmax, convert it into probabilities.

The highest probability is usually the next token.

Then connect this new token to the end and continue predicting the next one.

Just like this, tokens pop out one by one until an end token is generated.

This is roughly how the original Transformer works.

However, current large models represented by GPT are somewhat different from the earliest Transformer.

We’ll talk about this slowly later.

VII. Review and Summary

Now let’s review today’s content as a whole.

Initially, LSTM’s biggest problem was that information had to be passed token by token, and computation could only proceed sequentially.

So we came up with a method to let each token directly examine other tokens.

This method is called Self-Attention.

But which tokens should each token examine?

So each token generates Q, K, and V.

Q is used to initiate queries, K is used to match queries, and V is used to truly transmit information.

If one type of relationship isn’t enough to see?

Then do multiple groups of attention simultaneously.

So we got Multi-Head Attention.

After all tokens are computed in parallel, the sequential order is lost.

So we add positional information additionally.

So we got Positional Encoding.

Attention is only responsible for finding information; after information is retrieved, it needs further processing.

So we connect a feedforward neural network.

As the network gets stacked deeper, the original information is easily lost, and training becomes increasingly difficult.

So we add a shortcut for information to go directly backward.

So we got residual connections.

When generating text, it cannot peek into the future.

So we mask the later tokens.

So we got masked self-attention.

In classic translation models, the Decoder also needs to read the Encoder’s content.

So we let the Decoder’s Q match the Encoder’s K and V.

So we got Cross-Attention.

You see, when looking at the Transformer structure diagram, there are a bunch of headache-inducing terms.

But if we don’t just memorize these terms directly, but instead deduce them problem by problem from earlier, you’ll find they didn’t appear out of nowhere.

Every addition is because the previous structure exposed a new problem.

And these one-by-one small modifications, combined together, form the foundational architecture behind almost all large language models today.

Finally, leave a thought question for everyone:

Current large models have a context of often 1 million tokens. With such a massive input, calculating attention pairwise would be an enormous amount of computation. Is there any way to optimize this?

In the next video, we’ll reveal the answer together.

OK, that’s all for today. Creating content isn’t easy. If you found it useful, don’t forget to follow and support.

I’m Xuanyuan. See you next time.