@akshay_pachaar: 13 attention mechanisms AI engineers must know: (bookmark this) The tricky part about attention is that these technique…

13 attention mechanisms AI engineers must know: (bookmark this) The tricky part about attention is that these techniques are often discussed together, even though they solve very different problems. Some reduce KV cache size. Some control which tokens can attend to each other. Others make attention cheaper to compute or improve how KV cache is managed during serving. So, a better way to organize them is by the bottleneck they actually solve. Let's do that: 𝟭. 𝗞𝗩 𝗵𝗲𝗮𝗱 𝘀𝗵𝗮𝗿𝗶𝗻𝗴, 𝘄𝗵𝗲𝗻 𝗞𝗩 𝗰𝗮𝗰𝗵𝗲 𝘀𝗶𝘇𝗲 𝗶𝘀 𝘁𝗵𝗲 𝗯𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸 → MHA (multi-head attention) gives every query head its own key and value heads, providing maximum flexibility but also the largest KV cache. → MQA (multi-query attention) makes all query heads share a single KV head, dramatically reducing cache size. → GQA (grouped query attention) groups query heads and gives each group a shared KV head, balancing memory savings with model quality. → MLA (multi-head latent attention) compresses keys and values into a smaller latent representation before caching them, reducing KV memory even further. 𝟮. 𝗔𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗽𝗮𝘁𝘁𝗲𝗿𝗻𝘀, 𝘄𝗵𝗲𝗻 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹 𝗰𝗮𝗻 𝘀𝗲𝗲 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 → Causal attention only allows each token to look backward, which is what autoregressive LLMs use for generation. → Bidirectional attention lets tokens attend in both directions, which is useful when understanding the entire input at once. → Sliding Window Attention restricts each token to nearby context instead of attending across the full sequence. → StreamingLLM keeps a small set of anchor tokens plus recent context, allowing generation to continue with bounded memory. 𝟯. 𝗖𝗼𝗺𝗽𝘂𝘁𝗲 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆, 𝘄𝗵𝗲𝗻 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗶𝘁𝘀𝗲𝗹𝗳 𝗶𝘀 𝗲𝘅𝗽𝗲𝗻𝘀𝗶𝘃𝗲 → FlashAttention computes exact attention in small tiles that fit in fast on-chip memory, reducing expensive memory movement. → Sparse Attention skips selected token-to-token connections entirely, reducing how much attention needs to be computed. 𝟰. 𝗞𝗩 𝘀𝗲𝗿𝘃𝗶𝗻𝗴 𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆, 𝘄𝗵𝗲𝗻 𝗽𝗿𝗼𝗱𝘂𝗰𝘁𝗶𝗼𝗻 𝘁𝗵𝗿𝗼𝘂𝗴𝗵𝗽𝘂𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗯𝗼𝘁𝘁𝗹𝗲𝗻𝗲𝗰𝗸 → PagedAttention stores KV cache in blocks allocated on demand, reducing wasted and fragmented GPU memory. → RadixAttention organizes shared prefixes so KV states can be reused across requests instead of recomputed. → Prefix Caching similarly reuses KV blocks for repeated prefixes such as system prompts, reducing redundant prefill work. The important part is that these techniques are complementary. An LLM can use GQA to shrink its KV cache, FlashAttention to compute attention efficiently, Sliding Window Attention to limit context interactions, and PagedAttention to manage that cache efficiently in production. Once you organize them by the bottleneck they solve, the attention landscape becomes much easier to reason about. I wrote a deeper breakdown of how these techniques evolved and the problem each one solves. The full article is quoted below. Thanks for reading. Cheers! :)