Why Do LLMs Cache K and V, but Usually Not Q?

Suppose you hand a lengthy project document to an AI and ask it to summarize.

The response begins, generating one token after another. When producing the next segment, the model still needs to refer to the preceding document and the already generated answer.

Does this mean that with each new token generated, the entire growing text must be recomputed from the very beginning?

The KV cache allows the model to retain a portion of the already-computed information. But why does it keep K and V? Why is Q, which also participates in attention, typically discarded after use?

This article will discuss this question and help you understand the KV cache.

Is everything recomputed from scratch with every new token generated?

Shorten the input to “The sky is”. Assume the model then generates “blue”, followed by “.”.

The model first processes the entire prompt, building the KV cache for each layer. This step is called prefill. The output at the final prompt position passes through the vocabulary head to produce the logits used for selecting the first output token.

After “blue” is selected, the model takes it as new input to predict “.”.

Look at the animation below: Which blocks stay in place, and which are newly added when “blue” appears?

GIF

Existing K/V remain in place, and K/V for the new position are appended after them. The new token still undergoes model computation, but the redundant calculation of valid historical states is avoided.

Note the sequence: When “blue” is selected, its K/V haven’t been generated yet; they are calculated only when “blue” is fed back into the model to predict subsequent tokens.

Why keep K/V but usually discard Q?

Within an attention layer, each position has an input vector, also called a hidden state. Three learnable projections transform it into a query (Q), key (K), and value (V).

When processing the position “blue”, its new Q computes matching scores with the K from “The sky is” and from “blue” itself.

These scores are scaled and passed through softmax to become attention weights. Then, a weighted sum of the corresponding V using these weights yields the output for this head.

The next position will generate its own Q but will likely still need to read the K/V from these positions. Historical Q has already completed the computation for its corresponding positions and is not needed again in standard decoding.

Expressed as a formula, this becomes the single line below:

This explains the name KV cache: K/V continue to be read by subsequent positions; Q is regenerated with each new position.

As the context lengthens, why does the old computation remain valid?

Return to the project document. Suppose the beginning states “budget is 1 million yuan”, followed by progress, personnel, and delivery plans.

Now compare two operations:

• Append to the end of the document: “Please summarize the risks.”
• Change the opening “1 million yuan” to “500,000 yuan”.

Which operation would invalidate the old cache? Think first, then watch the animation. The letters in the figure represent token positions.

GIF

The difference stems from the information direction of causal attention: a position can only use its own and previous inputs, unable to read later content.

Therefore, appending a question at the end does not retroactively change the calculations already completed for document positions. Modifying the opening budget, however, could affect the representations of subsequent positions, so the entire old cache cannot be directly reused.

In other words, changing “1 million yuan” to “500,000 yuan” at the beginning affects the context for every subsequent token generation, causing the old cache to become invalid.

What exactly is computed when a new token arrives?

Now focus on the step where “blue” is fed back into the model. Looking at just one attention head: historical K/V remain unchanged, the new position produces Q/K/V, and the new Q reads the available K/V.

GIF

The new K/V are first added to the cache and then participate in attention, allowing the current Q to also attend to its own position. After this head’s output is used by subsequent computations, the K/V are still retained for the next position to read.

The “Head output” shown in the figure must undergo further computation within the layer and other parts of the model to produce the next-token logits. When the input and execution conditions are compatible, the cached and uncached results should be consistent within an appropriate numerical error range.

Saving redundant computation, but why does it increase memory usage?

The longer the project document, the more positions might need to be retained. When using a full KV cache that grows with the sequence, newly added positions continue to occupy storage space.

Consider the dimensions: how many sequences, how many layers, how many positions retained per layer, how many KV heads, how wide each head is, and how many bytes each element occupies.

Assuming K/V are stored separately, have the same width, and each layer has the same structure, multiplying gives:

For this document, assume a configuration of: 24 layers, 1 sequence, 8 KV heads, head width of 64, and 2 bytes per element.

• Retaining 4,096 positions: the KV tensor occupies 192 MiB.
• Retaining 8,192 positions: increases to 384 MiB.

Watch the two changes in the animation: first increasing positions, then returning to the original length, compared with another configuration with fewer KV heads.

GIF

With the same document, can a different question reuse the cache?

You just asked the AI to summarize risks and now want to know the deadlines. Suppose the two requests are organized as follows:

Request A: [Same project document] + “Summarize the risks.” Request B: [Same project document] + “List the deadlines.”

If the actual token prefix for the document part is exactly the same, the model and execution settings are compatible, and the system still retains this part of the cache, it can be reused. Then, the two questions and their continuations can be computed separately. This is called prefix caching.

Prefill, KV cache, and prefix caching can be distinguished as follows: prefill is the computation phase that processes the prompt; KV cache is the saved K/V state; prefix caching is the strategy for reusing existing prefix states. Even without enabling prefix caching, normal generation can still use the KV cache.

A typical example from vLLM’s official documentation is repeatedly asking questions about the same annual report or software manual. The project document above is the same usage: the first time, the cache is built via prefill; later, when hitting a common prefix, the corresponding K/V are reused directly, saving that portion of redundant computation.

New questions still require prefill and reading the document’s K/V; new answers still need token-by-token decoding. Prefix caching mainly saves redundant prefill work, not the generation computation for the new answer.

If Request B includes a different introductory paragraph before the document, the common prefix may shorten. Even if the subsequent document text is identical, we cannot directly assume its causal states are the same.

What if we redesign the source and sharing of K/V?

Previously, we started from “each layer saves its own K/V”. Understanding these computational dependencies allows you to grasp the changes in DeepSeek-V4.1-Flash.

First, look at the source: Causal Encoder–Decoder. The decoder’s global KV comes from the final output of the causal encoder. Each decoder layer still generates its own Q, as well as local sliding-window KV.

Then, look at cross-layer: Cross-layer sharing. In the discussed configuration, layer 21 uses the encoder output to generate a decoder global KV bank, which is used by layers 21–40.

Observe layers 21 and 22: they read shared global entries, but Q and local states remain separate. Shared entries do not force identical attention weights; sparse selection determines which global entries are read.

Finally, look at the K/V representation: Shared KV representation. In the reference core-attention kernel, the same selected vector participates both in Q’s matching score and in the output’s weighted sum.

Watch the animation: how does one global KV connect multiple layers? After zooming in on one vector, which two computations does it participate in?

GIF

The model still retains local caches, indexer state, and other data. Shared KV representation is distinctly different from storing two independent K/V arrays adjacently.

Returning to the project document: suppose this time the input has 100,000 tokens. The serving scheme in the report can handle it as follows:

• The encoder processes the complete prompt, providing representations for building the decoder global KV.
• The decoder then processes the last 128 prompt tokens, reconstructing local state. Earlier history remains accessible via the global KV.

This shows that the source of the global KV affects which computations during prefill can be adjusted. When generating subsequent tokens, new input still passes through the causal encoder and decoder.

Returning to that document at the beginning

Now we can connect the KV cache within the same scenario:

• Continuing to write the answer: New positions generate their own Q, reading still valid historical K/V.
• Modifying the document’s beginning: Re-examine the reuse scope from where the actual token prefix changes.
• Asking a different question: Check the common prefix, execution conditions, and whether the cache is still retained.
• Reading a longer document: Pay attention to the storage growth from retained positions and how the architecture organizes these states.