@bojie_li: If you run LLM agents in production you know the tax. Every turn re-reads the same long context — a system policy, tool…

X AI KOLs Timeline Papers
llm-agents kv-cache prompt-caching transformer-inference programmable-kv inference-optimization production-llm

Summary

Researchers introduce Programmable KV Cache, a method for editing and composing KV caches to avoid re-prefilling long contexts during LLM agent inference, achieving 53–398× reduction in p90 time-to-first-token while maintaining decision identity.

If you run LLM agents in production you know the tax. Every turn re-reads the same long context — a system policy, tool specs, retrieved docs — and prompt caching (Anthropic's, OpenAI's) only helps across an exactly shared prefix. Change one token early — a timestamp, a user id, an order's status — and the keys and values of every later token are invalidated. Your support agent flips "order #4471: in transit" and pays to re-prefill 10,000 tokens it computed a second ago. So you try the obvious surgery: refresh just that field's keys and values, keep the rest. The model promptly acts on the old value, as if the edit never happened. That puzzle bugged me for weeks — until we traced the cause, causally, across four model families: transformers don't defer their reasoning to decode time. At prefill the model has already written the field-conditioned conclusion onto downstream tokens — onto the aggregator and delimiter tokens later positions read through. The field's own KV drives under 1% of the decision. The KV cache isn't a frozen byproduct; it's a notebook of memoized conclusions. Read it that way and two powers fall out. Editable: don't recompute the notes, amend them — an append-only "erratum" ("X is now Y") overrides the stale conclusion. Composable: the notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context, indistinguishable from full recompute (logit cosine 0.90–0.999 across twelve models) at O(L) instead of O(L²). The muscle: because the erratum is append-only it rides on top of production prefix caching rather than fighting it — in a live vLLM benchmark it keeps a 98.5% cache hit-rate and cuts p90 time-to-first-token 53–398×, decision-identical to full recompute. A KV cache you can program, not just extend. Site: https://01.me/research/programmable-kv… Code: https://github.com/19PINE-AI/programmable-kv… Paper: https://arxiv.org/abs/2606.17107
Original Article
View Cached Full Text

Cached at: 07/07/26, 01:32 PM

If you run LLM agents in production you know the tax. Every turn re-reads the same long context — a system policy, tool specs, retrieved docs — and prompt caching (Anthropic’s, OpenAI’s) only helps across an exactly shared prefix. Change one token early — a timestamp, a user id, an order’s status — and the keys and values of every later token are invalidated. Your support agent flips “order #4471: in transit” and pays to re-prefill 10,000 tokens it computed a second ago.

So you try the obvious surgery: refresh just that field’s keys and values, keep the rest. The model promptly acts on the old value, as if the edit never happened. That puzzle bugged me for weeks — until we traced the cause, causally, across four model families: transformers don’t defer their reasoning to decode time. At prefill the model has already written the field-conditioned conclusion onto downstream tokens — onto the aggregator and delimiter tokens later positions read through. The field’s own KV drives under 1% of the decision. The KV cache isn’t a frozen byproduct; it’s a notebook of memoized conclusions.

Read it that way and two powers fall out. Editable: don’t recompute the notes, amend them — an append-only “erratum” (“X is now Y”) overrides the stale conclusion. Composable: the notes are position-portable, so a precompiled skill can be RoPE-repositioned and spliced into any context, indistinguishable from full recompute (logit cosine 0.90–0.999 across twelve models) at O(L) instead of O(L²).

The muscle: because the erratum is append-only it rides on top of production prefix caching rather than fighting it — in a live vLLM benchmark it keeps a 98.5% cache hit-rate and cuts p90 time-to-first-token 53–398×, decision-identical to full recompute. A KV cache you can program, not just extend.

Site: https://01.me/research/programmable-kv… Code: https://github.com/19PINE-AI/programmable-kv… Paper: https://arxiv.org/abs/2606.17107

Similar Articles

TokenPilot: Cache-Efficient Context Management for LLM Agents

Hugging Face Daily Papers

TokenPilot is a dual-granularity context management framework that reduces inference costs in long-horizon LLM sessions by stabilizing prompt prefixes and conservatively managing context segments, achieving 61-87% cost reduction on benchmarks while maintaining competitive performance.