@akshay_pachaar: Extending the context window isn't just about larger matrices. In a traditional transformer, expanding tokens by 8x inc…

You’re in a Research Scientist interview at OpenAI.

The interviewer asks:

“How would you expand the context length of an LLM from 2K to 128K tokens?”

You: “I will fine-tune the model on longer docs with 128K context.”

Interview over.

Here’s what you missed:

Extending the context window isn’t just about larger matrices.

In a traditional transformer, expanding tokens by 8x increases memory needs by 64x due to the quadratic complexity of attention. Refer to the image below!

So, how do we manage it?

continue…

1. Sparse Attention

It limits the attention computation to a subset of tokens by:

• Using local attention (tokens attend only to their neighbors).
• Letting the model learn which tokens to focus on.

But this has a trade-off between computational complexity and performance.

Here’s an intuitive explanation taken from the paper:

Picture yourself reading a book. For every sentence you read, do you need to be fully aware of the entire plot to understand most of it (full global attention)?

Or is awareness of the current chapter enough (local attention), as long as you occasionally think back on its significance to the main plot (global attention)?

In the vast majority of cases, it’s the latter.

2. Flash Attention

This is a fast and memory-efficient method that retains the exactness of traditional attention mechanisms, i.e., it uses global attention but efficiently.

The whole idea revolves around optimizing the data movement within GPU memory.

Let’s understand!

Some background details:

• A thread is the smallest unit of execution.
• Several threads form a block.

Also:

• Threads in a block share a fast (but scarce) memory called SRAM.
• All blocks share a global memory called HBM (abundant but slow).

Check this

Attention moves large matrices between SRAM and HBM:

To compute QK:

• distribute matrices to threads
• compute, and
• send the product to HBM

To compute softmax:

• distribute product to threads
• compute, and
• send output to HBM

Repeat for all layers.

Check this

Flash attention involves hardware-level optimizations wherein it utilizes SRAM to cache the intermediate results.

This way, it reduces redundant movements, offering a speed up of up to 7.6x over standard attention methods.

Check this

3. DeepSeek Sparse Attention (DSA)

DeepSeek’s recently released V3.2 model introduced DeepSeek Sparse Attention (DSA), which brought complexity down from O(L²) to O(Lk), where k is fixed.

How it works:

A lightweight Lightning Indexer scores which tokens actually matter for each query.

Small number of heads, runs in FP8, computationally cheap.

Then a selection mechanism retrieves only the top-k key-value entries.

The key insight is that only 2048 tokens get selected per query, regardless of context length.

So the expensive attention computation happens on this small subset, not the full 128K sequence.

At 128K context, prefilling costs drop from ~0.65 to ~0.35 per million tokens. And Decoding drops from ~2.4 to ~0.8.

And the performance stays the same. On some long-context benchmarks, V3.2 actually scores higher.

Sparse attention isn’t new. But making it work without losing quality is hard.

Over to you: What are some other techniques to increase the context lengths of LLMs?

Thanks for reading.

Cheers! :)