@akshay_pachaar: RadixAttention, clearly explained. (how SGLang makes prefix caching highly efficient) Prefix caching sounds simple unti…

X AI KOLs Timeline Tools
ai-serving prefix-caching radix-attention sglang kvcache inference-optimization

Summary

The article explains RadixAttention, a technique in SGLang that uses a radix tree to efficiently cache and reuse overlapping KV caches across branching requests in AI serving.

RadixAttention, clearly explained. (how SGLang makes prefix caching highly efficient) Prefix caching sounds simple until requests start branching. Consider an AI coding assistant serving these three conversations: Request 1 Context: System prompt → Python repo → authentication code → “Fix this bug” Request 2 Context: System prompt → Python repo → authentication code → “Add OAuth” Request 3 Context: System prompt → Python repo → database code → “Optimize this query” Now there are three ways a serving engine can handle them. 1. With no prefix caching, every request starts from zero. The system prompt and repo context are processed again for every request, even though thousands of tokens may be identical. All that repeated prefill produces essentially the same KV state again and again. 2. Simple prefix caching improves this. The engine can cache a known shared prefix, such as the system prompt, and reuse its KV cache when another request starts with exactly the same tokens. That already saves work. But real workloads are not just one shared prefix followed by completely unrelated requests. They branch. Two requests may share the system prompt, repo context, and authentication code. Another may share only the system prompt and repo before diverging into database code. This is where RadixAttention becomes useful. 3. SGLang stores cached token sequences in a radix tree. The common system prompt becomes the trunk. The shared Python repo context extends that trunk. Authentication and database context become separate branches. Then individual user queries branch again. When a new request arrives, SGLang walks this tree and finds the longest prefix whose KV cache already exists. If 2,000 tokens match, it reuses 2,000 tokens. If 8,000 match, it reuses 8,000. Only the suffix after the point of divergence needs fresh prefill. That is the key difference. Simple prefix caching lets you reuse a prefix. RadixAttention turns all the overlapping prefixes across your workload into a reusable hierarchy. This is particularly valuable for multi-turn conversations, agent trajectories, shared system prompts, few-shot examples, and workloads where requests repeatedly branch from common histories. SGLang keeps these KV blocks in the radix tree and evicts them using an LRU policy when memory is needed. → PagedAttention makes KV cache allocation efficient. → RadixAttention makes overlapping KV caches reusable. I wrote the full breakdown of every attention mechanism, from Multi-Head Attention through FlashAttention and Sparse Attention, up to PagedAttention and RadixAttention. The article is quoted below.
Original Article
View Cached Full Text

Cached at: 09/12/26, 08:57 PM

RadixAttention, clearly explained.

(how SGLang makes prefix caching highly efficient)

Prefix caching sounds simple until requests start branching.

Consider an AI coding assistant serving these three conversations:

Request 1 Context: System prompt → Python repo → authentication code → “Fix this bug”

Request 2 Context: System prompt → Python repo → authentication code → “Add OAuth”

Request 3 Context: System prompt → Python repo → database code → “Optimize this query”

Now there are three ways a serving engine can handle them.

  1. With no prefix caching, every request starts from zero.

The system prompt and repo context are processed again for every request, even though thousands of tokens may be identical.

All that repeated prefill produces essentially the same KV state again and again.

  1. Simple prefix caching improves this.

The engine can cache a known shared prefix, such as the system prompt, and reuse its KV cache when another request starts with exactly the same tokens.

That already saves work.

But real workloads are not just one shared prefix followed by completely unrelated requests.

They branch.

Two requests may share the system prompt, repo context, and authentication code. Another may share only the system prompt and repo before diverging into database code.

This is where RadixAttention becomes useful.

  1. SGLang stores cached token sequences in a radix tree.

The common system prompt becomes the trunk.

The shared Python repo context extends that trunk.

Authentication and database context become separate branches.

Then individual user queries branch again.

When a new request arrives, SGLang walks this tree and finds the longest prefix whose KV cache already exists.

If 2,000 tokens match, it reuses 2,000 tokens.

If 8,000 match, it reuses 8,000.

Only the suffix after the point of divergence needs fresh prefill.

That is the key difference.

Simple prefix caching lets you reuse a prefix.

RadixAttention turns all the overlapping prefixes across your workload into a reusable hierarchy.

This is particularly valuable for multi-turn conversations, agent trajectories, shared system prompts, few-shot examples, and workloads where requests repeatedly branch from common histories.

SGLang keeps these KV blocks in the radix tree and evicts them using an LRU policy when memory is needed.

→ PagedAttention makes KV cache allocation efficient.

→ RadixAttention makes overlapping KV caches reusable.

I wrote the full breakdown of every attention mechanism, from Multi-Head Attention through FlashAttention and Sparse Attention, up to PagedAttention and RadixAttention.

The article is quoted below.

Similar Articles