Nice paper to improve inference efficiency.

It’s been a while we haven’t seen good work on efficiency.

Here is why it matters:

A long-running agent’s workspace outgrows its context window long before the task finishes.

The first approach commonly used, compaction, loses the fine-grained execution evidence. And text retrieval re-prefills content the model already processed.

KVMem keeps the overflow as paged KV state instead, spread across GPU memory, host memory and NVMe.

Lightweight attention-space indexes, native to the model, pick the relevant historical blocks and materialize a query-dependent view that fits inside the native context window.

On the DeepSWE long-context test with Qwen3.8-27B, task success goes from 43.8% under compaction to 48.4%.

The local deployment result stands out. It runs Qwen3.6/3.8-27B NVFP4 with MTP on a laptop with a 24GB RTX 5090, virtualizing an agent workspace up to 1M tokens, four times the model’s native 256K window, at around 50 tokens per second.

Paper: https://academy.dair.ai/papers/kvmem-virtualizing-million-token-agent-workspaces-on-a-consumer-gpu-2609.04852…

KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU

Source: https://academy.dair.ai/papers/kvmem-virtualizing-million-token-agent-workspaces-on-a-consumer-gpu-2609.04852 Memory · Agents · RetrievalChat with Paper

First page

The curator’s take

Di Chai and colleagues at Shanghai University of Finance and Economics and Peking University page an agent’s overflowed workspace history as KV state across GPU memory, host memory, and NVMe, instead of compacting it into summaries or re-retrieving it as text.

Ask this paper

Question about this paper Key points01

What compaction and retrieval each cost. Summarization loses fine-grained execution evidence; text retrieval repeatedly prefills content the model already processed.

02

The mechanism. Lightweight model-native attention-space indexes select relevant historical blocks and materialize a query-dependent execution view bounded by the model’s native context window.

03

Task gain. On the DeepSWE long-context test with Qwen3.8-27B, success rises from 43.8% under compaction-only management to 48.4%.

04

Consumer hardware. Runs Qwen3.6/3.8-27B NVFP4 with MTP on a laptop with a 24GB RTX 5090, virtualizing workspaces up to 1M tokens, four times the model’s native 256K window, at roughly 50 tokens/s.

05

The general claim. Decoupling addressable workspace size from the native context window is what lets a long-running agent’s workspace keep growing.

AbstractModern LLM agents operate in persistent workspaces whose accumulated history can exceed both GPU KV capacity and the model’s native context window. Existing systems typically compact older context into summaries or retrieve it later as text, either losing fine-grained execution evidence or repeatedly prefilling content that the model has already processed. We present KVMem, a KV-context virtualization system that preserves overflowed workspace history as paged KV state across GPU memory, host memory, and NVMe. KVMem uses lightweight, model-native attention-space indexes to select relevant historical blocks and materializes a query-dependent execution view bounded by the model’s native context window. Extensive evaluations on long-context agent benchmarks spanning histories up to one million tokens, including LongMemEval, MemoryAgentBench, and AgentLongBench, show that KVMem generally achieves higher task utility and greater inference efficiency than compaction-based approaches, the de facto standard for handling context overflow. In the DeepSWE long-context test with Qwen3.8-27B, KVMem improves task success from 43.8% with compaction-only context management to 48.4%. In our local-deployment evaluation, KVMem runs Qwen3.6/3.8-27B NVFP4 with MTP on an off-the-shelf laptop equipped with a 24\,GB RTX 5090 Laptop GPU, virtualizing agent workspaces of up to 1M tokens-four times the model’s native 256K-token context window. In a single-session setting, KVMem generates \\sim50 tokens/s, providing interactive responsiveness for local agent execution. More broadly, by decoupling addressable workspace size from the LLM’s native context window, KVMem provides a practical path toward long-running agents whose workspaces can grow beyond that window.