Akashic: A Low-Overhead LLM Inference Service with MemAttention
Summary
Akashic introduces MemAttention, a low-overhead memory system for LLM inference that organizes context into bounded chunks and models semantic relationships, improving accuracy by up to 10.2 points and throughput by up to 1.21× over prior baselines.
arXiv:2607.05708v1 Announce Type: new
Abstract: Recent LLM-based agent systems continuously accumulate context across multi-turn interactions, tool invocations, and cross-session workflows. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. We propose Akashic, a low-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10.2 points, throughput by up to 1.21x, and sustainable request rate by up to 1.88x over strong prior memory baselines.
View Cached Full Text
Cached at: 07/08/26, 04:38 AM
# Akashic: A Low-Overhead LLM Inference Service with MemAttention Source: [https://arxiv.org/html/2607.05708](https://arxiv.org/html/2607.05708) Yang Liu∗\\ast†Zhaokai Luo∗\\ast†Huayi Jin†Ruozhou He†Chenchen Hong†Zhiyong Wang† Yifei Liu¶Yunfei Gu¶Chentao Wu¶Junhao Hu§ †Xiaohongshu Inc\., China¶ShangHai JiaoTong University§Peking University ∗\\astCorresponding to: Zhaokai Luo[<luozhaokai@xiaohongshu\.com\>](https://arxiv.org/html/2607.05708v1/mailto:[email protected]) ###### Abstract\. Recent LLM\-based agent systems continuously accumulate context across multi\-turn interactions, tool invocations, and cross\-session workflows\. Replaying the full history for every request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task\-relevant evidence in irrelevant content, degrading both serving efficiency and output quality\. We propose Akashic, a low\-overhead memory system built around MemAttention, which organizes context into bounded chunks and models semantic relationships across chunks, preserving cross\-chunk evidence without repeatedly rewriting the full history\. Akashic further applies hardware–software co\-designed memory placement to co\-locate likely co\-retrieved chunks, reducing retrieval fragmentation and I/O overhead\. Across four representative workloads and three model sizes, Akashic improves task accuracy by up to 10\.2 points, throughput by up to 1\.21×\\times, and sustainable request rate by up to 1\.88×\\timesover strong prior memory baselines\. ††copyright:none## 1\.Introduction Figure 1\.\(a\) Existing memory designs occupy different points on the task\-quality / serving\-efficiency trade\-off, whereas Akashic targets the high\-quality, high\-efficiency regime\. \(b\) On BrowseComp with OPT\-30B, Akashic improves both accuracy and throughput over all baselines, outperforming the strongest prior method by about 2\.0 points in accuracy and about 1\.35×\\timesin throughput\.Recent agent systems continuously accumulate input context across multi\-turn interactions, tool invocations, and cross\-session execution, and this growing context can substantially degrade model throughput and output quality\(Fanet al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib48); Chenet al\.,[2021a](https://arxiv.org/html/2607.05708#bib.bib106); Github,[2022](https://arxiv.org/html/2607.05708#bib.bib75); Maharanaet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib14); Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16); Cobbeet al\.,[2021](https://arxiv.org/html/2607.05708#bib.bib49); Weiet al\.,[2022](https://arxiv.org/html/2607.05708#bib.bib50)\)\. A straightforward solution is to feed the full interaction history into every request, but this approach quickly becomes impractical: long prompts increase prefill cost, can exceed context limits, and often bury task\-relevant evidence in irrelevant context, hurting both serving efficiency and output quality\(Liuet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib37); Shahamet al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib41)\)\. As a result, modern agent systems increasingly maintain external memory and re\-inject only a subset of prior context at inference time\(Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16); Chhikaraet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib17); Anthropic,[2025](https://arxiv.org/html/2607.05708#bib.bib51)\)\. A common way to build such memory is*summarization*: once the accumulated history exceeds a threshold, the system compresses older context into a shorter representation and uses that representation in future requests\([16](https://arxiv.org/html/2607.05708#bib.bib43);[19](https://arxiv.org/html/2607.05708#bib.bib44)\)\. Existing summarization\-based designs, however, expose a granularity trade\-off\.*Whole\-context*approaches such as Mem0 summarize the entire history at each trigger point\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib17)\)\. They are simple, but their update cost grows with history length, and unrelated topics are mixed into one memory object, which can inject noise at retrieval time\.*Segment\-level*approaches such as MemGAS summarize smaller units independently\(Xuet al\.,[2025a](https://arxiv.org/html/2607.05708#bib.bib20)\)\. They bound per\-update cost, but semantically dependent evidence may be split across segments and never recovered together, weakening multi\-turn and long\-range reasoning\. In practical LLM serving, resolving this trade\-off requires addressing two key challenges\. First, context information density is highly non\-uniform across workloads and even across different phases of the same interaction \(§[3\.1](https://arxiv.org/html/2607.05708#S3.SS1)\): some spans are highly compressible, whereas others are dense and expensive to compress\. A fixed global compression policy is therefore often mismatched to the actual workload\. Second, agent memory is not only a semantic object but also a storage object\. Even when retrieval selects only a few relevant memories, end\-to\-end latency can remain high if those memories are physically scattered across many pages or blocks\. We refer to this mismatch between semantic relatedness and physical placement as the*locality gap*\(§[3\.2](https://arxiv.org/html/2607.05708#S3.SS2)\)\. We presentAkashic, a low\-overhead memory system for LLM serving that addresses these challenges throughMemAttentionandhardware–software co\-designedmemory placement optimization\. Akashic performs*chunk\-granular*memory maintenance: it compacts one bounded chunk at a time, rather than repeatedly rewriting the full history\. To preserve cross\-chunk evidence, Akashic uses*cross\-chunk inference*to reconcile each new chunk with a small set of semantically related prior chunks before writing the result back to memory\. This design bounds maintenance overhead while retaining information that would otherwise be fragmented by independent segment summaries \(§[4\.1](https://arxiv.org/html/2607.05708#S4.SS1)\)\. Akashic further includes a*hardware–software co\-optimized memory manager*that co\-locates chunks likely to be co\-retrieved and compacts stale data out of place, reducing read amplification and retrieval\-side contention during concurrent serving \(§[4\.2](https://arxiv.org/html/2607.05708#S4.SS2)\)\. We evaluate Akashic on four representative benchmarks spanning diverse workload characteristics—LoCoMo\(Maharanaet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib14)\), SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib12)\), BrowseComp\(Weiet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib15)\), and WebArena\(Zhouet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib13)\)—using models of different sizes\. As shown in Figure[1](https://arxiv.org/html/2607.05708#S1.F1)\(a\), Akashic consistently achieves both higher serving efficiency and better task quality than prior baselines\. Figure[1](https://arxiv.org/html/2607.05708#S1.F1)\(b\) further illustrates this advantage with a representative example, where Akashic substantially outperforms existing methods in both accuracy and throughput \(§[5](https://arxiv.org/html/2607.05708#S5)\)\. Akashic consistently lies on the*Pareto frontier*, improving task accuracy by up to 10\.2 points and throughput by up to 1\.21×\\timesover the strongest memory baselines under the same experimental setting, while sustaining up to 1\.88×\\timeshigher request rates under concurrent serving\.We will continue to update Akashic on newer models and datasets\. The subsequent code will be merged into an alternative branch of the RedKnot framework\(Yang,Liuet al\.,[2026](https://arxiv.org/html/2607.05708#bib.bib1)\)\. The open\-source address is[https://github\.com/rednote\-machine\-learning/RedKnot](https://github.com/rednote-machine-learning/RedKnot)\. - •We characterize two key bottlenecks in agent memory: the mismatch between fixed summarization policies and heterogeneous context density, and the locality gap between semantic relevance and physical placement\. - •We design*MemAttention*, a chunk\-granular memory maintenance mechanism that reconciles each new chunk with a small set of semantically related prior chunks, preserving cross\-chunk evidence while bounding maintenance overhead\. - •We propose and implement a*hardware–software co\-designed memory manager*to mitigate memory fragmentation in LLM serving, by co\-locating likely co\-retrieved chunks and compacting stale data to reduce retrieval I/O and contention\. - •We implement Akashic and evaluate it on four representative workloads, showing up to 10\.2\-point higher accuracy, 1\.21×\\timeshigher throughput, and 1\.88×\\timeshigher sustainable request rate than the strongest prior memory baselines\. ## 2\.Background ### 2\.1\.Long Context as a First\-Class Bottleneck Modern agentic applications \(e\.g\., multi\-round assistants, tool\-augmented planners, and long\-running workflows\) naturally accumulate conversational state over time, pushing the input context toward the model’s maximum context window\. Although recent frontier models have dramatically expanded their nominal context limits \(up to million\-token regimes\), long\-context requests remain expensive in latency and compute: for example, OpenAI reports that time\-to\-first\-token \(TTFT\) can increase from seconds at∼\\sim128K tokens to around a minute at∼\\sim1M tokens, even with an optimized inference stack\(OpenAI,[2025](https://arxiv.org/html/2607.05708#bib.bib45),[https://arxiv.org/html/2607.05708#bib.bib46](https://arxiv.org/html/2607.05708#bib.bib46)\)\. Beyond the hard limit of the maximum window, longer context also degrades*effective task accuracy*\. Empirically, LLMs exhibit strong attention biases when relevant evidence appears in the middle of long prompts: performance can follow a distinctive U\-shaped curve and drop sharply when the answer\-bearing content is placed mid\-context\(Liuet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib37)\)\. Subsequent work shows that even with long\-context model variants, robust middle\-of\-context utilization remains challenging, motivating positional encoding and long\-context evaluation efforts\(Zhanget al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib38); Shahamet al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib41)\)\. For long\-term dialogue specifically, benchmarks such as LoCoMo demonstrate that very long multi\-session conversations amplify temporal and causal reasoning failures, and that long\-context or retrieval\-augmented approaches still lag substantially behind human\-level consistency\(Maharanaet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib14)\)\. During serving of LLM inference, long\-context requests are expensive primarily because the Transformer must execute a full*prefill*pass over the entire prompt to construct per\-layer activations and populate the KV cache, incurringsubstantial compute and memory traffic\. At the systems layer, this prefill cost grows rapidly with sequence length: standard Transformer attention exhibits quadratic compute and memory scaling in the prompt length\(Vaswaniet al\.,[2017](https://arxiv.org/html/2607.05708#bib.bib54)\), and even exact attention implementations require IO\-aware kernel designs to prevent memory bandwidth from becoming the dominant performance limiter\(Daoet al\.,[2022](https://arxiv.org/html/2607.05708#bib.bib102)\)\. These constraints interact strongly with agent memory\. ### 2\.2\.Memory Compression and Retrieval in Practice To cope with bounded windows and long\-context degradation, common agent frameworks \(such as Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib17)\), RMM\(Tanet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib19)\), and SeCom\(Panet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib18)\)\) adopt*compression*\(summarization\) and*selective retrieval*\. A representative pattern maintains a*running summary*that is updated incrementally after each turn \(or when a token threshold is reached\), and injects only the summary \(optionally plus a short recent buffer\) into future prompts; this pattern is explicitly implemented by widely used framework abstractions\. Complementarily, retrieval\-based memory stores past messages \(or derived facts\) in a vector index and retrieves a small set of semantically relevant items to condition the model, which reduces prompt length but introduces retrieval and consolidation challenges\. This end\-to\-end pipeline of memory compression, vector\-database storage, and inference\-time retrieval constitutes the*memory workflow*in modern LLM agents\. Recent open\-source systems elevate memory to an OS\-inspired hierarchy\. MemGPT formulates*virtual context management*and pages information between in\-context working memory and external storage, enabling long\-running agents beyond fixed context limits\(Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16)\)\. Industrial and open\-source memory layers such as Mem0 similarly emphasize compressing history into compact memory representations and retrieving only a few relevant memory items for each user query\([21](https://arxiv.org/html/2607.05708#bib.bib42)\)\. Tooling ecosystems such as LlamaIndex increasingly expose configurable memory modules that combine a FIFO short\-term buffer with customizable long\-term memory blocks for extraction and reinjection\([19](https://arxiv.org/html/2607.05708#bib.bib44)\)\. Overall, existing practice converges on a trade\-off: frequent summarization improves boundedness but consumes additional model calls, whereas infrequent summarization risks window overflow and accuracy loss—a tension that motivates more controllable, fine\-grained, and update\-friendly memory organization\. ## 3\.Memory Challenges in LLM Serving Figure 2\.Context information density varies substantially across datasets and over time\. \(a\) Compared with LoCoMo, SWE\-Bench exhibits a higher retention ratio\. \(b\) BrowseComp and WebArena show bursty and alternating retention ratio within trajectories\.### 3\.1\.Memory Inefficiency under Heterogeneous Context Density Non\-Uniform Context Information Density\.Extracting effective information from a long input context is itself an inference process\. Before an LLM system can benefit from a compressed memory representation, it must first identify which parts of the raw context are salient, and this step consumes additional computation\. LetLrawL\_\{\\mathrm\{raw\}\}denote the token length of the original context andLeffL\_\{\\mathrm\{eff\}\}denote the token length of the effective information preserved after memory construction or compression\. We define the*Retained Ratio*as \(1\)ρ=LeffLraw\.\\rho=\\frac\{L\_\{\\mathrm\{eff\}\}\}\{L\_\{\\mathrm\{raw\}\}\}\.A smallerρ\\rhoindicates that the context is highly compressible, whereas a largerρ\\rhoindicates that most tokens are information\-bearing and therefore difficult to compress\. As shown in Figure[2](https://arxiv.org/html/2607.05708#S3.F2), context information density is highly non\-uniform, both across datasets and over time within the same workload\. Figure[2](https://arxiv.org/html/2607.05708#S3.F2)\(a\) shows a clear cross\-dataset contrast: LoCoMo has a much lower retained ratio than SWE\-Bench, indicating that its long conversational histories contain substantial redundancy and are therefore more amenable to traditional compression\-based memory strategies\. In contrast, SWE\-Bench remains consistently high in retained ratio, suggesting that most tokens are information\-bearing, as is common in code debugging and repository\-level reasoning\. In such high\-density workloads, compression still triggers an additional inference step for summarization or memory construction, but yields only limited reduction in prompt length\. As a result, memory maintenance may introduce extra overhead without meaningfully shortening the context, reducing or even negating the end\-to\-end system benefit\. Figure[2](https://arxiv.org/html/2607.05708#S3.F2)\(b\) further shows that information density is often unstable even within a single trajectory\. BrowseComp and WebArena exhibit bursty and alternating retained ratios, with some phases being highly compressible and others being highly information\-dense\. This temporal heterogeneity makes static memory policies fundamentally mismatched to agent workloads\. A fixed summarization trigger, compression ratio, or retrieval budget may be beneficial during low\-density phases but wasteful during high\-density phases\. Consequently, conventional memory workflows struggle to deliver stable gains across the full lifetime of an interaction\. Figure 3\.Memory\-augmented inference reduces end\-to\-end throughput by 48\.0%–63\.9% across SWE\-Bench, BrowseComp, and WebArena\.This observation has direct systems implications\. Representative long\-term memory methods—including recursive summarization\(Wanget al\.,[2023c](https://arxiv.org/html/2607.05708#bib.bib137)\), MemGPT\(Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16)\), SeCom\(Panet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib18)\), Reflective Memory Management \(RMM\)\(Tanet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib19)\), and Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib17)\)—maintain usable memory representations through summary generation, memory extraction, hierarchical memory management, segmentation, denoising, or reflective consolidation\. These operations are beneficial whenρ\\rhois low, because a relatively short memory representation can replace a much longer raw context\. However, whenρ\\rhois high, the benefit of compression becomes marginal\. For example, compressing a 1000\-token context into 900 tokens yields only a small reduction in prompt length, yet still incurs the full cost of an additional memory\-maintenance step\. In such high\-density regimes, memory construction may save little context while introducing extra inference overhead, ultimately wasting compute and reducing end\-to\-end serving throughput\. Inference Resource Waste under High Context Density\.As discussed in §[2\.2](https://arxiv.org/html/2607.05708#S2.SS2), the memory workflow first feeds the context into the LLM for inference\. The resulting memory is then stored in a database, and relevant memory context is retrieved and loaded during the next actual inference\. We evaluate SWE\-Bench, BrowseComp, and WebArena, which represent workloads with either high information density or highly unstable information\-density patterns, on an NVIDIA H20 96GB GPU using SGLang 0\.4\.6\.post1\(SGLang Team,[2025](https://arxiv.org/html/2607.05708#bib.bib9)\)with batch size 1\. As shown in Figure[3](https://arxiv.org/html/2607.05708#S3.F3), end\-to\-end throughput decreases substantially on datasets with unstable, low information density\. The reason is twofold: memory generation requires extra computation during inference, and memory management further incurs database write and retrieval overheads, both of which prolong the inference pipeline\. Because context compressibility is not directly observable beforehand and often depends on downstream queries, accurately deciding when and how aggressively to compress requires explicit modeling of context informativeness and query relevance\(Liet al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib6); Jianget al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib7); Anet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib8)\); such an adaptive policy is therefore difficult to realize in practice\. Instead of attempting global compression over the entire context, we adopt chunk\-granular compression\. By compressing only one chunk at a time, we bound the per\-step inference overhead and make the cost of memory construction more stable and controllable\. ### 3\.2\.The Locality Gap in Agent Memory Figure 4\.Throughput on LoCoMo as concurrency increases\. Although memory\-augmented inference consistently underperforms standard inference, its throughput gradually approaches the no\-memory baseline at larger batch sizes\.Existing agent\-memory systems are primarily designed around a semantic objective: deciding what to retain, how to structure it, and which memories to retrieve at inference time\. MemGPT introduces a tiered memory hierarchy for long\-horizon interaction\(Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16)\); RMM improves long\-term dialogue memory through multi\-granular summarization and retrieval refinement\(Wanget al\.,[2023c](https://arxiv.org/html/2607.05708#bib.bib137)\); and A\-MEM dynamically links and evolves memories through agentic indexing\(Xuet al\.,[2025b](https://arxiv.org/html/2607.05708#bib.bib21)\)\. These systems substantially improve memory quality, but they largely treat storage as an opaque backend\. As a result, semantic relatedness and physical placement are optimized separately\. This separation creates what we call the*locality gap*\. LetRtR\_\{t\}denote the set of memories retrieved at timett, and letP\(Rt\)P\(R\_\{t\}\)denote the set of storage pages or blocks touched to materializeRtR\_\{t\}\. Existing memory managers primarily optimize the semantic quality ofRtR\_\{t\}, whereas end\-to\-end retrieval cost is dominated by\|P\(Rt\)\|\|P\(R\_\{t\}\)\|\. The locality gap is therefore large when\|Rt\|\|R\_\{t\}\|is small but\|P\(Rt\)\|\|P\(R\_\{t\}\)\|remains large\. This is precisely the failure mode identified in disk\-resident vector search\. SPANN explicitly reduces the number of disk accesses in hybrid memory–disk ANN search\(Chenet al\.,[2021b](https://arxiv.org/html/2607.05708#bib.bib24)\); Starling improves performance by reordering the disk\-resident graph layout to enhance locality and reduce bandwidth waste\(Wanget al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib26)\); and recent work on SSD\-backed vector databases shows that frequently co\-accessed nodes are often placed on different pages, making locality\-preserving colocation critical for reducing I/O overhead\(Shimet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib35)\)\. The implication for agent memory is direct: retrieving only a few relevant memories does not imply low latency if those memories are physically scattered\. To quantify this effect, we use the same experimental environment as in §[3\.1](https://arxiv.org/html/2607.05708#S3.SS1)\. We evaluate LoCoMo, a long\-horizon workload in which semantically related evidence is often distributed across distant interaction steps, while scaling concurrency from 2 to 16 requests\. As shown in Figure[4](https://arxiv.org/html/2607.05708#S3.F4), memory\-augmented inference consistently remains below standard inference on both Qwen\-8B and Llama\-30B, and the throughput gap gradually narrows as concurrency increases\. On Qwen\-8B, the throughput ratio between standard inference and inference with memory drops from 2\.08×\\timesat 2 concurrent requests to 1\.43×\\timesat 16 concurrent requests\. On Llama\-30B, the same ratio drops from 2\.08×\\timesto 1\.44×\\times\. Equivalently, memory\-augmented inference rises only from 48\.1% to 70\.1% of the no\-memory baseline on Qwen\-8B, and from 48\.0% to 69\.3% on Llama\-30B\. This trend is consistent with worsening locality under concurrent serving: although the retrieved memory set is semantically selective, the corresponding memory items can still be physically scattered across many pages or blocks\. As concurrency grows, the system must materialize more such scattered accesses at the same time, increasing retrieval\-side contention and progressively offsetting the semantic benefit of shorter prompts\. In other words, semantic selectivity alone is insufficient when co\-retrieved memories are not placed with good physical locality\. The problem is amplified by the write pattern of agent memory\. Unlike static indexes, agent memories are continuously appended, summarized, linked, and updated over long\-lived interactions\(Wanget al\.,[2023c](https://arxiv.org/html/2607.05708#bib.bib137); Xuet al\.,[2025b](https://arxiv.org/html/2607.05708#bib.bib21)\)\. Under such dynamic writes, any locality that happens to exist today can quickly deteriorate tomorrow\. More broadly, the ANN literature shows that maintaining freshness under real\-time updates is itself nontrivial, because search efficiency and update cost must be balanced rather than optimized in isolation\(Singhet al\.,[2021](https://arxiv.org/html/2607.05708#bib.bib25)\)\. Therefore, optimizing only semantic relevance is insufficient for scalable agent memory\. What is needed is an affinity\-aware memory manager that jointly reasons about*which*memories are likely to be co\-retrieved and*where*they should be placed, so that future retrievals touch fewer pages while rewrite, compaction, and maintenance overhead remain bounded\. ## 4\.Design of Akashic Figure 5\.Akashic system overview\.In this work, we develop a new attention algorithm for memory,*MemAttention*, and build an LLM serving engine,*Akashic*, to tackle the challenges outlined in §[3](https://arxiv.org/html/2607.05708#S3)\. The architecture of Akashic is shown in Figure[5](https://arxiv.org/html/2607.05708#S4.F5)\. ❶ Akashic adopts chunk\-granular memory summarization, treating each chunk as the basic unit of context\. ❷ For every new chunk, it retrieves previously seen chunks with relevant information and performs joint summarization over the associated chunks\. ❸ Chunk compaction organizes related memories into contiguous physical locations and removes irrelevant ones\. ❹ Memory retrieval selects the memory chunks relevant to the current inference and incorporates them into the input context\. ❺ The model identifies the relatedness of memory chunks\. Next, we describe the MemAttention algorithm in §[4\.1](https://arxiv.org/html/2607.05708#S4.SS1)\. We then present the design of the memory\-chunk manager in §[4\.2](https://arxiv.org/html/2607.05708#S4.SS2)and show how it supports MemAttention\. Finally, we describe the implementation of Akashic in §[4\.3](https://arxiv.org/html/2607.05708#S4.SS3)\. ### 4\.1\.MemAttention To address the memory challenges in §[3\.1](https://arxiv.org/html/2607.05708#S3.SS1), we introduceMemAttention, a chunk\-granular memory architecture that treats memory construction as incremental maintenance rather than global compression over the entire interaction history\. MemAttention stores memory as chunk objects keyed by\(user\_id,session\_id,metadata\)\(\\texttt\{user\\\_id\},\\texttt\{session\\\_id\},\\texttt\{metadata\}\), whereuser\_idisolates tenants,session\_idpreserves conversation boundaries, andmetadatarecords representative information of a chunk within a session\. For each active session, the system maintains one append\-only active chunk for newly arrived turns and a set of compacted chunks for prior history\. As shown in Figure[6](https://arxiv.org/html/2607.05708#S4.F6), ❶ we first partition the input context into fixed\-size chunks\. Unless otherwise specified, we empirically set the chunk size to 1024 tokens \(§[5\.5](https://arxiv.org/html/2607.05708#S5.SS5)\), which provides a good balance between memory quality and maintenance overhead\. Once the accumulated context reaches one chunk, the system performs*chunk memory inference*over that chunk in order to extract its salient information\. ❷ The inferred result is materialized as a*memory chunk*object indexed byuser\_id,session\_id, and metadata, which respectively preserve tenant isolation, session boundaries, and temporal order\. ❸ At the next turn, newly arrived context is accumulated and partitioned with the same chunk granularity, and each completed chunk is processed in the same way to generate a new memory chunk incrementally\. In this way, MemAttention treats memory construction as continuous chunk\-level maintenance rather than global compression over the entire interaction history\. The key design principle isgate\-triggered chunk compaction\. Rather than reprocessing the full history whenever new context arrives, MemAttention triggers compaction only when the active chunk reaches a predefined threshold, i\.e\., whenLchunk≥τcL\_\{\\text\{chunk\}\}\\geq\\tau\_\{c\}, whereLchunkL\_\{\\text\{chunk\}\}is the token length of the active chunk andτc\\tau\_\{c\}is the chunk gate\. Once triggered, the model rewrites only that chunk into a compacted representation that co\-locates semantically related memory items and removes transient or redundant content\. This design does not require estimating context density*a priori*\. Whether the incoming context is information\-dense or highly compressible, the worst\-case maintenance cost of a single compaction step scales asO\(Lchunk\)O\(L\_\{\\text\{chunk\}\}\)withLchunk≤τcL\_\{\\text\{chunk\}\}\\leq\\tau\_\{c\}, rather than with the total history length\. In this sense, MemAttention follows a common systems design pattern: maintenance is triggered by local thresholds and applied to bounded units, analogous to segment cleaning in log\-structured file systems, rolling merges in LSM\-trees, and size\-triggered compactions in Bigtable\(Rosenblum and Ousterhout,[1992](https://arxiv.org/html/2607.05708#bib.bib32); O’Neilet al\.,[1996](https://arxiv.org/html/2607.05708#bib.bib28); Changet al\.,[2006](https://arxiv.org/html/2607.05708#bib.bib27)\)\. However, compacting one memory chunk at a time makes it difficult to update or remove metadata associated with previously compacted chunks\. For example, newly arrived information may overwrite or invalidate older memories, yet the stale metadata of earlier chunks may still persist, causing conflicting information when multiple memory chunks are loaded for inference\. To address this issue, we introduce*cross\-chunk inference*, inspired by the efficient memory retrieval strategy used in Claude Code\(ChinaSiro,[2026](https://arxiv.org/html/2607.05708#bib.bib4)\)\. Specifically, the system feeds the model with the keywords of the chunk to be compacted together with the metadata keywords of previously compacted chunks, and*asks the model to perform semantic matching across them*, thereby identifying highly related memory chunks for joint reasoning\. We then select the top\-5 most relevant chunks, following Claude Code, for cross\-chunk inference, which enables the system to update obsolete memories and remove invalidated ones\. After memory chunks are compacted, MemAttention performs*query\-aware retrieval*during inference\. For a new input context, the system first collects candidate memory chunks within the currentuser\_idandsession\_idnamespace and retrieves their metadata keywords\. Rather than relying on predefined similarity scores, such as dense embedding similarity or token\-overlap matching, MemAttention feeds the current query and the metadata keywords of candidate chunks to the model, allowing it to perform semantic matching and select the top\-5 most relevant memory chunks for subsequent inference\. The selected chunks are then incorporated into the model input, effectively augmenting the prompt with relevant long\-term memory\. This retrieval mechanism is inspired by the relevance\-oriented text indexing design in Claude Code\. A key design choice in MemAttention is to use the same model\-driven semantic matching mechanism for both cross\-chunk maintenance and inference\-time retrieval\. During compaction, the model matches the new chunk against the metadata of previously stored chunks to identify related memories for joint update, refinement, or stale\-memory removal\. During inference, it matches the current query against candidate chunk metadata and selects the top\-kkrelevant chunks to load, providing a unified notion of relevance across memory maintenance and retrieval\. Algorithm 1MemAttention: Incremental Chunk Compaction and Model\-Driven Retrieval1:Input:new turn xtx\_\{t\}, active chunk CaC\_\{a\}, chunk table 𝒯\[u\]\[s\]\\mathcal\{T\}\[u\]\[s\], gate τc\\tau\_\{c\}, top\- pp\( p=5p\{=\}5\) 2:Append xtx\_\{t\}to CaC\_\{a\}⊳\\trianglerightaccumulate fresh context 3:if \|Ca\|≥τc\|C\_\{a\}\|\\geq\\tau\_\{c\}then 4: m←Compress\(Ca\)m\\leftarrow\\textsc\{Compress\}\(C\_\{a\}\)⊳\\trianglerightcompact one bounded chunk 5: Km←BuildMeta\(m\)K\_\{m\}\\leftarrow\\textsc\{BuildMeta\}\(m\)⊳\\trianglerightextract keywords / representative metadata 6: ℳ←\{\(cidi,Ki\)∣ci∈𝒯\[u\]\[s\]\}\\mathcal\{M\}\\leftarrow\\\{\(\\textit\{cid\}\_\{i\},K\_\{i\}\)\\mid c\_\{i\}\\in\\mathcal\{T\}\[u\]\[s\]\\\} 7: R←LLMSelect\(Km,ℳ,p\)R\\leftarrow\\textsc\{LLMSelect\}\(K\_\{m\},\\mathcal\{M\},p\)⊳\\trianglerightmodel selects top\-pprelated prior chunks 8: \(m′,U,D\)←JointCompact\(m,R\)\(m^\{\\prime\},U,D\)\\leftarrow\\textsc\{JointCompact\}\(m,R\)⊳\\trianglerightrefine new chunk; update/delete stale memory 9:Apply updates UUand deletions DDto 𝒯\[u\]\[s\]\\mathcal\{T\}\[u\]\[s\] 10: Km′←BuildMeta\(m′\)K\_\{m^\{\\prime\}\}\\leftarrow\\textsc\{BuildMeta\}\(m^\{\\prime\}\) 11:Insert \(u,s,cid,m′,Km′,tnow\)\(u,s,\\textit\{cid\},m^\{\\prime\},K\_\{m^\{\\prime\}\},t\_\{\\mathrm\{now\}\}\)into 𝒯\[u\]\[s\]\\mathcal\{T\}\[u\]\[s\] 12: Ca←∅C\_\{a\}\\leftarrow\\emptyset⊳\\trianglerightstart a new active chunk 13:endif 14: q←CurrentContext\(xt\)q\\leftarrow\\textsc\{CurrentContext\}\(x\_\{t\}\) 15: Kq←BuildQuery\(q\)K\_\{q\}\\leftarrow\\textsc\{BuildQuery\}\(q\) 16: ℳ←\{\(cidi,Ki\)∣ci∈𝒯\[u\]\[s\]\}\\mathcal\{M\}\\leftarrow\\\{\(\\textit\{cid\}\_\{i\},K\_\{i\}\)\\mid c\_\{i\}\\in\\mathcal\{T\}\[u\]\[s\]\\\} 17: S←LLMSelect\(Kq,ℳ,p\)S\\leftarrow\\textsc\{LLMSelect\}\(K\_\{q\},\\mathcal\{M\},p\)⊳\\trianglerightmodel selects top\-pprelevant memory chunks 18: S←FetchByID\(S,𝒯\[u\]\[s\]\)S\\leftarrow\\textsc\{FetchByID\}\(S,\\mathcal\{T\}\[u\]\[s\]\) 19: S←SortByTime\(S\)S\\leftarrow\\textsc\{SortByTime\}\(S\)⊳\\trianglerightrestore chronological order 20:return Concat\(S,q\)\\textsc\{Concat\}\(S,q\) Figure 6\.Workflow of MemAttention: each incoming chunk context is incrementally compacted into a structured memory record with user\- and session\-scoped metadata, and the most relevant prior chunk memories are retrieved to augment the next\-turn inference\.Algorithm overview\.Algorithm[1](https://arxiv.org/html/2607.05708#alg1)summarizes the workflow of MemAttention\. Here,uuandssdenote the currentuser\_idandsession\_id, respectively, and𝒯\[u\]\[s\]\\mathcal\{T\}\[u\]\[s\]denotes the chunk table scoped to that user\-session namespace\. The active chunkCaC\_\{a\}accumulates newly arrived turns,τc\\tau\_\{c\}is the chunk\-compaction gate, andppdenotes the number of chunks selected by the model \(defaultp=5p\{=\}5\)\.*Lines 1–13*implement chunk production and cross\-chunk reconciliation\. Each new turnxtx\_\{t\}is first appended to the active chunk\. Compaction is triggered only when the chunk length reaches the gate thresholdτc\\tau\_\{c\}, at which point the system compresses the bounded chunk into a memory representation and extracts its metadata keywords\. MemAttention then gathers the metadata of previously compacted chunks in the same namespace𝒯\[u\]\[s\]\\mathcal\{T\}\[u\]\[s\]and feeds them, together with the metadata of the new chunk, to the model for semantic matching\. The model selects the top\-pprelated prior chunks for*joint compaction*, which allows the system to refine the new chunk and to update or delete stale prior memories when the new chunk contains corrections, refinements, or superseding information\. The reconciled chunk is then written back to the chunk table, and a new active chunk is started\.*Lines 14–20*implement inference\-time retrieval\. The current context is converted into query keywords, and the system again collects candidate chunk metadata from the current namespace𝒯\[u\]\[s\]\\mathcal\{T\}\[u\]\[s\]\. Instead of using an explicit scoring function, MemAttention feeds the query keywords and candidate metadata to the model, which performs semantic matching and selects the top\-ppmost relevant memory chunks\. The selected chunks are fetched by their identifiers, reordered chronologically, and concatenated with the current context to form the final model input\. By reusing the same model\-driven semantic matching mechanism for both compaction\-time reconciliation and inference\-time retrieval, MemAttention maintains a unified notion of memory relevance while keeping both maintenance and retrieval overhead bounded\. ### 4\.2\.Memory Manager Figure 7\.Overview of the Memory Compaction workflow\.Although MemAttention is efficient at inference, its end\-to\-end performance still critically depends on how memory chunks are physically organized and maintained in storage\. MemAttention stores memory as chunk records in a disk\-resident vector store\. Each chunk is immutable once written and is identified logically by itsuser\_id,session\_id, andmetadata, whereas its physical location is determined by a block\-level layout maintained by the Memory Manager\. Thisseparation between logical identity and physical placement is crucial\(§[3\.2](https://arxiv.org/html/2607.05708#S3.SS2)\): it allows the system to reorganize chunk layout online without changing the higher\-level memory abstraction exposed to retrieval and inference\. The goal of the Memory Manager is to reduce*read amplification during cross\-chunk inference*\. A cross\-chunk inference step often needs to load multiple related chunks together, but these chunks may have been written at different times and thus scattered across different physical blocks\. As a result, a single inference may trigger multiple block reads even when the retrieved memories are strongly related semantically and temporally\. To address this inefficiency, MemAttention performs*association\-aware relocation*: whenever joint inference reveals that a set of chunks is frequently used together or has high pairwise association, the system rewrites these chunks into a new block so that future accesses can be served with fewer physical reads\. To drive this reorganization, MemAttention does not rely on a hand\-crafted association score\. Instead, it uses the model to infer chunk affinity directly from metadata, following the relevance\-oriented retrieval strategy used in Claude Code \(§[4\.1](https://arxiv.org/html/2607.05708#S4.SS1)\)\. The inferred affinity is then used to determine whether multiple chunks should be placed in the same physical block, thereby improving locality for future cross\-chunk inference\. Figure[7](https://arxiv.org/html/2607.05708#S4.F7)illustrates this workflow: ❶ the system first retrieves related chunks for a new context chunk and performs cross\-chunk inference to produce a new memory chunk; ❷ it then relocates chunks that should be co\-located into new block\(s\); and ❸ it finally reclaims invalidated space through garbage collection\. MemAttention adopts an out\-of\-place update policy\. Once the model identifies a set of chunks that should be co\-located, the Memory Manager writes them into new block\(s\) and marks their old copies as invalid, rather than rewriting blocks in place\. This design follows the same high\-level principle as LSM\-tree compaction: updates are accumulated through sequential writes, stale entries are invalidated through logical tombstoning, and background compaction later reclaims space by preserving only live records\(O’Neilet al\.,[1996](https://arxiv.org/html/2607.05708#bib.bib28); Changet al\.,[2006](https://arxiv.org/html/2607.05708#bib.bib27)\)\. Letϕ\(B\)\\phi\(B\)denote the invalid ratio of blockBB: \(2\)ϕ\(B\)=Ninvalid\(B\)Ntotal\(B\)\.\\phi\(B\)=\\frac\{N\_\{\\mathrm\{invalid\}\}\(B\)\}\{N\_\{\\mathrm\{total\}\}\(B\)\}\\,\.Whenϕ\(B\)≥0\.75\\phi\(B\)\\geq 0\.75, i\.e\., when invalid chunks occupy more than 75% of a block’s capacity, the block becomes a candidate for compaction\. The compaction process serves two purposes\. First, it frees space occupied by obsolete chunk copies\. Second, it further improves future locality by repacking the remaining valid chunks according to both namespace and model\-inferred affinity\. Specifically, the Memory Manager scans the candidate block, extracts all valid chunks, partitions them by\(user\_id,session\_id\)\(\\texttt\{user\\\_id\},\\texttt\{session\\\_id\}\), and then invokes the model again within each partition to group chunks that are likely to be consumed together\. Each group is then packed into one or more new blocks while preserving temporal order\. This design preserves the logical isolation of different users and sessions while still exploiting affinity within each namespace\. As a result, compaction is not merely garbage collection; it is also a layout optimization pass that consolidates live, related memory into fewer blocks\. Memory Manager algorithm\.Algorithm[2](https://arxiv.org/html/2607.05708#alg2)summarizes the Memory Manager\. The algorithm has two tightly coupled stages\. The first stage runs on the joint\-inference path: it uses the model to identify retrieved chunks that should be physically co\-located, rewrites them out of place into new block\(s\), updates the logical directory, and marks the old versions invalid\. The second stage runs in the background: when the invalid ratio of a block reaches 75%, the block is compacted by extracting its live chunks, grouping them by user and session, and repacking them into new blocks according to model\-inferred affinity\.*Lines 3–8*implement the online relocation path\. Given the chunk setSSretrieved for a joint inference, the Memory Manager invokesLLMCoLocate\(S\)\\textsc\{LLMCoLocate\}\(S\), which uses chunk metadata to identify the subset of retrieved chunks that is likely to be co\-accessed again\. These chunks are then packed into one or more new blocks out of place, their old physical copies are marked invalid, and the logical directory is updated to point future reads to the new locations\. This step transforms co\-access patterns observed at inference time into improved physical locality for subsequent requests\.*Lines 11–24*implement background block compaction\. For each block whose invalid ratio satisfiesϕ\(B\)≥0\.75\\phi\(B\)\\geq 0\.75, the Memory Manager extracts only the live chunks and partitions them byuser\_idandsession\_id\. This ensures that compaction never mixes chunks across tenants or sessions\. Within each partition, the system invokesLLMGroup\(P\)\\textsc\{LLMGroup\}\(P\)to group chunks with high model\-inferred affinity, and repacks each group into one or more new blocks while preserving temporal order\. After the live chunks have been rewritten and the directory updated, the original block is released\. This process both reclaims space and continuously reshapes the on\-disk layout to match evolving joint\-inference access patterns\. Algorithm 2Model\-Driven Relocation and Block Compaction1:Input:retrieved chunks SS, block directory 𝒟\\mathcal\{D\} 2:Output:updated block layout 3: G←LLMCoLocate\(S\)G\\leftarrow\\textsc\{LLMCoLocate\}\(S\)⊳\\trianglerightmodel identifies chunks worth co\-locating 4:if G≠∅G\\neq\\emptysetthen 5: ℛ←PackIntoNewBlocks\(SortByTime\(G\)\)\\mathcal\{R\}\\leftarrow\\textsc\{PackIntoNewBlocks\}\(\\textsc\{SortByTime\}\(G\)\)⊳\\trianglerightmay span multiple blocks 6:for all \(c,B′\)∈ℛ\(c,B^\{\\prime\}\)\\in\\mathcal\{R\}do 7: MarkInvalid\(𝒟\[c\]\)\\textsc\{MarkInvalid\}\(\\mathcal\{D\}\[c\]\)⊳\\trianglerighttombstone old copy 8: UpdateDir\(𝒟,c,B′\)\\textsc\{UpdateDir\}\(\\mathcal\{D\},c,B^\{\\prime\}\) 9:endfor 10:endif 11:for allblock BBwith ϕ\(B\)≥0\.75\\phi\(B\)\\geq 0\.75do 12: V←LiveChunks\(B\)V\\leftarrow\\textsc\{LiveChunks\}\(B\)⊳\\trianglerightextract valid memory only 13: 𝒫←PartitionByUserSession\(V\)\\mathcal\{P\}\\leftarrow\\textsc\{PartitionByUserSession\}\(V\) 14:for allpartition P∈𝒫P\\in\\mathcal\{P\}do 15: 𝒢←LLMGroup\(P\)\\mathcal\{G\}\\leftarrow\\textsc\{LLMGroup\}\(P\)⊳\\trianglerightgroup chunks by model\-inferred affinity 16:for allgroup g∈𝒢g\\in\\mathcal\{G\}do 17: ℛ′←PackIntoNewBlocks\(SortByTime\(g\)\)\\mathcal\{R\}^\{\\prime\}\\leftarrow\\textsc\{PackIntoNewBlocks\}\(\\textsc\{SortByTime\}\(g\)\) 18:for all \(c,B′\)∈ℛ′\(c,B^\{\\prime\}\)\\in\\mathcal\{R\}^\{\\prime\}do 19: UpdateDir\(𝒟,c,B′\)\\textsc\{UpdateDir\}\(\\mathcal\{D\},c,B^\{\\prime\}\) 20:endfor 21:endfor 22:endfor 23:Release\(B\)⊳\\trianglerightold block is fully reclaimed 24:endfor ### 4\.3\.Implementation We implement Akashic by extending the control path of vLLM v0\.10\.0\(vLLM Team,[2026](https://arxiv.org/html/2607.05708#bib.bib33)\), while leaving the GPU execution path, the attention backend, and the KV\-cache manager unchanged\. The implementation follows a simple separation of concerns\. Memory retrieval is inserted at the request admission layer, memory write\-back is inserted at the request completion layer, and physical layout maintenance is split into an online relocation path and a background storage manager\. This design keeps long\-term memory management outside the hot token\-level GPU decoding loop and allows Akashic to be realized with limited changes to the serving stack\. At the request admission layer, Akashic augments each incoming request with a memory\-scoped context before tokenization\. This context carries the stable namespace of the request, includinguser\_id,session\_id, and optionallyturn\_id, together with request\-level controls such asenable\_retrieval,enable\_writeback, andmemory\_budget\_tokens\. Given this context, the memory retriever searches compacted chunks within the corresponding\(user\_id, session\_id\)namespace, collects their metadata, and feeds the current query together with candidate metadata to the model for semantic matching\. The model returns the identifiers of the top\-ppmost relevant chunks \(defaultp=5p\{=\}5\); the system then fetches these chunks, restores chronological order, and prepends them to the prompt under the memory token budget\. After this step, the request follows the original vLLM execution path without any change to scheduling or decoding\. At the request completion layer, Akashic intercepts finished requests and performs incremental memory maintenance\. Each completed turn is appended to the active chunk of its namespace\. When the accumulated length exceedschunk\_gate\_tokens, the system triggers chunk\-gated compaction, rewrites the active chunk into a compacted memory chunk, and builds its metadata\. The system then feeds this metadata together with the metadata of previously compacted chunks in the same namespace to the model, which selects the top\-ppprior chunks for reconciliation\. Joint compaction then refines the new chunk and updates or removes stale prior memories when the new chunk contains corrections, refinements, or superseding information\. The reconciled result is written back to the chunk store, while obsolete or invalidated chunks are marked stale\. Akashic further introduces a Memory Manager to optimize physical locality\. During joint inference, the system records which chunks are materialized together\. Using the metadata of jointly accessed chunks, the storage manager invokes the model to identify subsets that should be co\-located and relocates them out of place into fresh block\(s\), while invalidating their old copies\. Separately, a background compaction thread scans blocks whose invalid ratio satisfiesϕ\(B\)≥0\.75\\phi\(B\)\\geq 0\.75, extracts live chunks, partitions them by\(user\_id, session\_id\), invokes the model within each partition to group chunks likely to be consumed together, repacks each group into new block\(s\) while preserving temporal order, and reclaims obsolete blocks\. In this way, Akashic separates logical memory construction from physical memory placement: the former is driven by session\-scoped semantic maintenance, while the latter is driven by model\-inferred co\-access affinity and space reclamation\. Overall, Akashic realizes long\-horizon memory with modest changes to the vLLM control path while leaving the underlying decoding engine unchanged\. The core implementation parameters are the chunk compaction gate \(chunk\_gate\_tokens\), the model selection width \(top\_p=5\), the retrieval memory budget \(memory\_budget\_tokens\), and the block reclamation threshold \(gc\_invalid\_ratio=0\.75\)\. Figure 8\.Throughput–accuracy trade\-off under single\-request inference \(batch size=1=1\) across three models and three workloads\. ## 5\.Evaluation In this section, we evaluate the performance of Akashic across a diverse set of workloads and models\. ### 5\.1\.Experimental Setup Table 1\.Models and server configurations\.Model and server configurations\.We use Qwen\(Yanget al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib29)\), OPT\(Zhanget al\.,[2022](https://arxiv.org/html/2607.05708#bib.bib30)\), and Llama\(Grattafioriet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib31)\)models with 8B, 30B, and 70B parameters, respectively, for our evaluation\. These three models cover small, medium, and large open\-weight LLM configurations and allow us to study the behavior of Akashic across substantially different memory footprints\. We deploy the 8B and 30B models on a single NVIDIA H800 GPU, and the 70B model on two NVIDIA H800 GPUs\. All servers are equipped with PCIe\-attached H800 accelerators and local NVMe SSD storage and we use OPT\-30B to perform relevance analysis over memory chunks\. The detailed model sizes and hardware configurations are shown in Table[1](https://arxiv.org/html/2607.05708#S5.T1)\. Workloads\.We evaluate Akashic on four representative long\-horizon workloads: LoCoMo\(Maharanaet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib14)\)\(long\-term multi\-session dialogue memory\), SWE\-bench\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib12)\)\(high\-density software engineering trajectories\), BrowseComp\(Weiet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib15)\)\(bursty web browsing trajectories\), and WebArena\(Zhouet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib13)\)\(realistic multi\-step web\-agent tasks\)\. Together, these datasets cover the main workload regimes discussed in our motivation \(§[3\.1](https://arxiv.org/html/2607.05708#S3.SS1)\)\. Baselines\.We compare Akashic against five baselines: Full\-context \(no compression or external memory\), Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib17)\)\(whole\-context memory summarization\), MemGAS\(Xuet al\.,[2025a](https://arxiv.org/html/2607.05708#bib.bib20)\)\(segment\-level multi\-granularity memory\), MemGPT\(Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16)\)\(hierarchical external memory\), and RMM\(Tanet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib19)\)\(reflective memory refinement\)\. Together, these baselines span the whole\-context, segment\-level, and hierarchical memory strategies discussed in §[3](https://arxiv.org/html/2607.05708#S3)\. Key metrics\.We use accuracy and throughput as the two primary evaluation metrics\. Accuracy is measured using each benchmark’s canonical protocol: QA F1\(Maharanaet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib14)\), %Resolved\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib12)\), Accuracy\(Weiet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib15)\), and task success rate\(Zhouet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib13)\)\. ### 5\.2\.Overall Effectiveness We evaluate Akashic with basic sampling on the models and workloads above\. Figure[8](https://arxiv.org/html/2607.05708#S4.F8)shows the throughput–accuracy trade\-off under basic sampling \(i\.e\., batch size=1=1\)\. Akashic consistently lies on thePareto frontieracross all nine settings\. On LoCoMo and BrowseComp, Akashic improves throughput over Mem0, MemGAS, and RMM by 1\.08×\\times–1\.10×\\times, 1\.10×\\times–1\.21×\\times, and 1\.33×\\times–1\.54×\\times, while improving accuracy by 8\.4–10\.2, 4\.2–7\.2, and 2\.1–5\.2 points, respectively\. Compared with full\-context inference, Akashic is 1\.27×\\times–2\.05×\\timesfaster and still improves accuracy by 1\.2–3\.4 points\.*The reason is*that Akashic summarizes memory at chunk granularity, so each maintenance step processes only a bounded unit instead of the entire history, which reduces memory\-construction overhead\. At the same time, its cross\-chunk retrieval and joint reconciliation recover semantically related evidence that would otherwise be split across segments, allowing the final prompt to remain short while still containing the critical information needed for inference\. The same trend holds on SWE\-Bench, although the throughput gap to full\-context inference is smaller\. Akashic improves throughput over Mem0, MemGAS, and RMM by 1\.31×\\times–1\.33×\\times, 1\.15×\\times–1\.19×\\times, and 1\.41×\\times–1\.56×\\times, while improving resolved rate by 4\.6–6\.6, 2\.9–3\.7, and 1\.8–2\.2 points, respectively\. Compared with full\-context inference, Akashic retains 90\.5%–91\.8% of the throughput while still improving resolved rate by 0\.6–1\.1 points\. This smaller throughput gap is expected because SWE\-Bench is more information\-dense and therefore less compressible\. Even in this regime, Akashic remains preferable because Mem0 compresses the whole history too aggressively and mixes irrelevant content, MemGAS keeps each segment short but loses cross\-segment dependencies, and RMM preserves stronger semantics at the cost of heavier memory\-maintenance overhead\. In contrast, Akashic keeps memory compact enough for efficient serving while using cross\-chunk recovery to preserve the information that most directly affects task accuracy\. ### 5\.3\.Robustness under Heterogeneous Density and Concurrency Figure 9\.Akashic maintains stable performance as concurrency increases, across models of different scales and diverse workloads\.We next evaluate Akashic under concurrent serving by increasing the offered request rate and measuring average latency per generated token \(s/token\)\. As shown in Figure[9](https://arxiv.org/html/2607.05708#S5.F9), Akashic consistently shifts the knee of the latency curve furthest to the right across all three models and all three workloads, indicating the highest sustainable request rate before latency rises sharply\. The gains are largest on LoCoMo: Akashic sustains 1\.22×\\times–1\.43×\\timeshigher request rates than Mem0, 1\.28×\\times–1\.74×\\timeshigher than MemGAS, and 4\.67×\\times–6\.64×\\timeshigher than Full\-context\. These improvements come from bounded chunk\-level maintenance, which rewrites only the active chunk instead of repeatedly summarizing the full history, together with memory compression and cross\-chunk recovery, which retain relevant evidence without lengthening the prompt and thus avoid the large prefill cost of long retained contexts\. The same trend appears on WebArena, where Akashic delivers 1\.40×\\times–1\.54×\\timesgains over Mem0, 1\.26×\\times–1\.34×\\timesover MemGAS, and 3\.40×\\times–4\.30×\\timesover Full\-context\. These gains remain substantial because WebArena trajectories are long, bursty, and semantically heterogeneous, so short prompts and efficient retrieval reduce both prompt\-side and retrieval\-side overhead under load\. On SWE\-Bench, the margin is smaller but still consistent: Akashic sustains 1\.53×\\times–1\.88×\\timeshigher request rates than Mem0, 1\.32×\\times–1\.58×\\timeshigher than MemGAS, and 1\.11×\\times–1\.26×\\timeshigher than Full\-context\. This smaller gap is expected because SWE\-Bench traces are more information\-dense and therefore less compressible\. Even so, Akashic still degrades more gracefully by keeping memory maintenance bounded and retrieving only task\-relevant evidence rather than carrying unnecessary context\. Overall, Figure[9](https://arxiv.org/html/2607.05708#S5.F9)shows that efficient memory construction and retrieval, rather than full\-history retention, are key to sustaining high\-throughput LLM serving under concurrency\. ### 5\.4\.Locality and Storage Efficiency Figure 10\.Compared with append\-only and semantic\-only layouts, Akashic reduces the number of blocks touched per request and lowers p95 cold\-cache retrieval latency by co\-locating jointly accessed chunks\. Meanwhile, selective out\-of\-place relocation with background garbage collection keeps both space amplification and write amplification moderate, showing that improved locality does not come at excessive storage overhead\.This experiment asks whether Akashic’s Memory Manager can close the locality gap in §[3\.2](https://arxiv.org/html/2607.05708#S3.SS2)without excessive storage overhead\. We replay the same inference traces with the same chunking policy, query\-aware retrieval logic, and memory budget, varying only the physical layout policy\. Figures[10](https://arxiv.org/html/2607.05708#S5.F10)\(a\)–\(b\) compareAppend\-only,Semantic\-only, andAkashic; Figures[10](https://arxiv.org/html/2607.05708#S5.F10)\(c\)–\(d\) compareNo\-Relocation,No\-GC,Aggressive\-Relocation, andAkashicunder the same update stream\. Figure[10](https://arxiv.org/html/2607.05708#S5.F10)\(a\) shows the main locality benefit: by colocating chunks that are jointly accessed at inference time, Akashic reduces blocks/request on LoCoMo, WebArena, and SWE\-Bench from 6\.8/5\.1 to 3\.4, 5\.9/4\.6 to 3\.2, and 4\.1/3\.5 to 2\.9 \(Append\-only/Semantic\-only→\\rightarrowAkashic\), corresponding to reductions of 50\.0%/33\.3%, 45\.8%/30\.4%, and 29\.3%/17\.1%\. Figure[10](https://arxiv.org/html/2607.05708#S5.F10)\(b\) shows the latency effect of this improved locality: p95 cold\-cache retrieval latency falls from 38/30 to 22 ms, 34/27 to 21 ms, and 25/22 to 18 ms, i\.e\., by 42\.1%/26\.7%, 38\.2%/22\.2%, and 28\.0%/18\.2% relative toAppend\-only/Sema ntic\-only\. The gains are largest on LoCoMo and WebArena, whose longer and more fragmented histories make co\-access\-aware placement more effective than write\-order placement or semantic\-only grouping\. Figures[10](https://arxiv.org/html/2607.05708#S5.F10)\(c\)–\(d\) show that these locality gains remain storage\-efficient\. Akashic achieves 1\.17×\\timesspace amplification and 1\.34×\\timeswrite amplification, compared with 1\.06×\\times/1\.05×\\timesforNo\-Relocation, 1\.74×\\times/1\.31×\\timesforNo\-GC, and 1\.38×\\times/1\.82×\\timesforAggressive\-Relocation\. Thus, Akashic is only 10\.4% aboveNo\-Relocationin space amplification, but 32\.8% belowNo\-GCand 15\.2% belowAggressive\-Relocation; for write amplification, it is 27\.6% aboveNo\-Relocation, only 2\.3% aboveNo\-GC, and 26\.4% belowAggressive\-Relocation\. Overall, Akashic closes the locality gap by physically colocating jointly accessed chunks and reclaiming obsolete copies in the background, reducing the cost of cross\-chunk inference while maintaining a favorable balance between locality and storage overhead\. ### 5\.5\.Ablation and Sensitivity Figure 11\.Ablation and sensitivity of Akashic\. \(a\) Task\-metric retention under component ablations, normalized to full Akashic\. \(b\) Throughput retention under the same ablations, normalized to full Akashic\. \(c\) Sensitivity of task quality and throughput to the chunk gateτc\\tau\_\{c\}\. \(d\) Sensitivity of task quality and throughput to the memory budgetBmemB\_\{mem\}\.We next ablate Akashic’s logical and physical components to identify which mechanisms are essential and how sensitive the system is to parameter choice\. We study three logical components—bounded chunk\-level maintenance, cross\-chunk reconciliation, and model\-driven relevance matching using OPT\-30B—and sweep four key parameters: the chunk gateτc\\tau\_\{c\}, the memory budgetBmemB\_\{mem\}, the association thresholdτassoc\\tau\_\{assoc\}, and the GC invalid\-ratio thresholdθgc\\theta\_\{gc\}\. Figure[11](https://arxiv.org/html/2607.05708#S5.F11)\(a\)–\(b\) reports task\-quality and throughput retention, both normalized to full Akashic, and shows that bounded chunk\-level maintenance is the dominant efficiency contributor\. Replacing it withWhole\-contextreduces task\-quality retention to 95\.0%, 95\.5%, and 93\.2% on LoCoMo, SWE\-Bench, and BrowseComp, while lowering throughput more sharply to 83\.4%, 88\.9%, and 82\.8%, respectively\. This result indicates that Akashic’s main efficiency gain comes from bounding each maintenance step to one chunk rather than repeatedly rewriting the full history\. Removing cross\-chunk reconciliation \(No joint\) preserves slightly higher throughput \(102\.2%–103\.4%\) but reduces task quality to 95\.9%–96\.4%, indicating that reconciliation mainly improves evidence recovery at modest overhead\.Dense\-onlyremains closer to the full system in both quality \(97\.2%–97\.3%\) and throughput \(98\.7%–99\.1%\), suggesting that simple embedding\-based similarity already captures a substantial fraction of useful relevance signals, whereas Akashic’s model\-driven relevance matching further improves selection quality by identifying semantically related chunks more accurately during both reconciliation and retrieval\. Figure[11](https://arxiv.org/html/2607.05708#S5.F11)\(c\)–\(d\) further shows that Akashic is not brittle to parameter choice\. Both workloads peak near the defaultτc=1024\\tau\_\{c\}\{=\}1024: smaller chunks fragment evidence and trigger compaction more often, whereas larger chunks weaken semantic isolation and increase per\-compaction cost\. IncreasingBmemB\_\{mem\}from 512 to 1536 improves quality from 93\.2% to 100% on LoCoMo and from 94\.1% to 100% on SWE\-Bench, but the gains saturate beyond the default, while throughput falls to 91\.3% and 91\.8% at 3072\. Figure 12\.Sensitivity of Akashic’s storage manager\. \(a\) Sensitivity of p95 retrieval latency to the association thresholdτassoc\\tau\_\{assoc\}\. \(b\) Sensitivity of space and write amplification to the GC invalid\-ratio thresholdθgc\\theta\_\{gc\}\.Figure[12](https://arxiv.org/html/2607.05708#S5.F12)then examines the sensitivity of Akashic’s physical storage manager\. Figure[12](https://arxiv.org/html/2607.05708#S5.F12)\(a\) shows that loweringτassoc\\tau\_\{assoc\}makes relocation more aggressive and slightly reduces p95 retrieval latency, while the defaultτassoc=0\.7\\tau\_\{assoc\}\{=\}0\.7remains close to the latency knee at 22/21/18 ms on LoCoMo/WebArena/SWE\-Bench; increasingτassoc\\tau\_\{assoc\}further to 0\.9 degrades latency to 26/25/21 ms\. Figure[12](https://arxiv.org/html/2607.05708#S5.F12)\(b\) shows the expected GC trade\-off: asθgc\\theta\_\{gc\}increases from 0\.1 to 0\.5, write amplification drops from 1\.56×\\timesto 1\.24×\\times, but space amplification rises from 1\.10×\\timesto 1\.33×\\times\. The defaultθgc=0\.3\\theta\_\{gc\}\{=\}0\.3therefore provides a balanced operating point at 1\.17×\\timesspace amplification and 1\.34×\\timeswrite amplification\. Overall, Akashic’s gains do not come from a single knob\. Bounded chunk maintenance delivers most of the efficiency benefit, reconciliation improves quality at modest overhead, and locality\-aware storage preserves these gains while maintaining a balanced trade\-off among retrieval latency, space amplification, and write amplification\. ## 6\.Related Work Agent Applications and Workloads\.LLM agents have shown utility across tool\-augmented reasoning, embodied exploration, software engineering, web navigation, and long\-horizon dialogue\(Wanget al\.,[2023b](https://arxiv.org/html/2607.05708#bib.bib5); Yaoet al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib10); Wanget al\.,[2023a](https://arxiv.org/html/2607.05708#bib.bib11)\)\. Representative benchmarks include SWE\-bench for resolving real GitHub issues\(Jimenezet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib12)\), WebArena and BrowseComp for realistic web interaction and persistent search\(Zhouet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib13); Weiet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib15)\), and LoCoMo for temporal and causal reasoning over long multi\-session dialogues\(Maharanaet al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib14)\)\. Although these workloads differ in form, they impose a common systems challenge: long\-horizon agents continuously accumulate context that must be filtered, retrieved, and served efficiently\. Memory Systems for Long\-Horizon Agents\.Prior work largely treats long\-term agent memory as a problem of semantic construction and retrieval\. Recursive summarization compresses history into progressively shorter summaries\(Wanget al\.,[2023c](https://arxiv.org/html/2607.05708#bib.bib137)\), whereas MemGPT introduces an OS\-inspired hierarchy that pages between in\-context and external memory\(Packeret al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib16)\)\. Recent systems further explore persistent memory extraction and retrieval, including Mem0\(Chhikaraet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib17)\), SeCom\(Panet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib18)\), RMM\(Tanet al\.,[2025](https://arxiv.org/html/2607.05708#bib.bib19)\), MemGAS\(Xuet al\.,[2025a](https://arxiv.org/html/2607.05708#bib.bib20)\), and A\-MEM\(Xuet al\.,[2025b](https://arxiv.org/html/2607.05708#bib.bib21)\)\. Akashic complements this line of work by jointly optimizing logical memory construction and physical layout: maintenance is bounded at chunk granularity, reconciliation and retrieval share the same relevance function, and co\-accessed chunks are physically colocated to reduce retrieval cost\. LLM Serving and Locality\-Aware Storage\.LLM serving engines such as Orca and vLLM optimize batching, scheduling, and GPU and KV\-cache management for high\-throughput decoding\(Yuet al\.,[2022](https://arxiv.org/html/2607.05708#bib.bib22); Kwonet al\.,[2023](https://arxiv.org/html/2607.05708#bib.bib23)\), but they do not treat persistent agent memory as a first\-class serving object\. At the storage layer, disk\-resident ANN and vector\-database systems such as SPANN, FreshDiskANN, and Starling study I/O efficiency, freshness, and layout locality under large\-scale search workloads\(Chenet al\.,[2021b](https://arxiv.org/html/2607.05708#bib.bib24); Singhet al\.,[2021](https://arxiv.org/html/2607.05708#bib.bib25); Wanget al\.,[2024](https://arxiv.org/html/2607.05708#bib.bib26)\)\. Akashic adopts this systems perspective at a different boundary: it manages dynamically evolving, session\-scoped memory chunks whose semantic relevance and physical layout must be co\-optimized\. In this sense, Akashic connects agent\-memory systems with locality\-aware storage for long\-horizon LLM serving\. ## 7\.Conclusion This paper has presented Akashic, a low\-overhead LLM inference service for long\-horizon agents\. To the best of our knowledge, Akashic is the first system to efficiently realize long\-context filtering for LLM agents through end\-to\-end software–hardware co\-design\. Across LoCoMo, SWE\-bench, BrowseComp, and WebArena, Akashic consistently stays on the Pareto frontier against prior memory baselines, improving task accuracy by 8\.4–10\.2 points, increasing throughput by 1\.21×\\times–1\.35×\\timesunder basic sampling, and sustaining 1\.26×\\times–1\.74×\\timeshigher load under concurrent serving\. These gains come from jointly optimizing chunk\-granular memory construction, cross\-chunk reconciliation, query\-aware retrieval, and locality\-aware storage management\. Upon acceptance, we plan to open\-source the code, artifacts, and reproduction materials\. ## References - LCIRC: a recurrent compression approach for efficient long\-form context and query dependent modeling in LLMs\.InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\),L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 10431–10442\.External Links:[Link](https://aclanthology.org/2025.naacl-long.524/),[Document](https://dx.doi.org/10.18653/v1/2025.naacl-long.524)Cited by:[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p4.1)\. - Anthropic \(2025\)Claude code: best practices for agentic coding\.Note:[https://www\.anthropic\.com/engineering/claude\-code\-best\-practices](https://www.anthropic.com/engineering/claude-code-best-practices)Accessed 2026\-03\-04Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1)\. - F\. Chang, J\. Dean, S\. Ghemawat, W\. C\. Hsieh, D\. A\. Wallach, M\. Burrows, T\. Chandra, A\. Fikes, and R\. E\. Gruber \(2006\)Bigtable: a distributed storage system for structured data\.In7th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 06\),pp\. 205–218\.External Links:[Link](https://www.usenix.org/conference/osdi-06/bigtable-distributed-storage-system-structured-data)Cited by:[§4\.1](https://arxiv.org/html/2607.05708#S4.SS1.p2.5),[§4\.2](https://arxiv.org/html/2607.05708#S4.SS2.p2.2)\. - M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. d\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021a\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1)\. - Q\. Chen, B\. Zhao, H\. Wang, M\. Li, C\. Liu, Z\. Li, M\. Yang, and J\. Wang \(2021b\)SPANN: highly\-efficient billion\-scale approximate nearest neighbor search\.InAdvances in Neural Information Processing Systems,Vol\.34,pp\. 5199–5212\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2021/file/299dc35e747eb77177d9cea10a802da2-Paper.pdf)Cited by:[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p2.8),[§6](https://arxiv.org/html/2607.05708#S6.p3.1)\. - P\. Chhikara, D\. Khant, S\. Aryan, T\. Singh, and D\. Yadav \(2025\)Mem0: building production\-ready AI agents with scalable long\-term memory\.External Links:2504\.19413,[Link](https://arxiv.org/abs/2504.19413)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1),[§1](https://arxiv.org/html/2607.05708#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.05708#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p3.2),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p3.1),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - ChinaSiro \(2026\)claude\-code\-sourcemap\.Note:[https://github\.com/ChinaSiro/claude\-code\-sourcemap](https://github.com/ChinaSiro/claude-code-sourcemap)GitHub repository; unofficial reconstruction from the public npm package and source\-map analysis; accessed 2026\-04\-02Cited by:[§4\.1](https://arxiv.org/html/2607.05708#S4.SS1.p3.1)\. - K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1)\. - T\. Dao, D\. Fu, S\. Ermon, A\. Rudra, and C\. Ré \(2022\)FlashAttention: fast and memory\-efficient exact attention with IO\-awareness\.Advances in Neural Information Processing Systems35,pp\. 16344–16359\.Cited by:[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - A\. Fan, B\. Gokkaya, M\. Harman, M\. Lyubarskiy, S\. Sengupta, S\. Yoo, and J\. M\. Zhang \(2023\)Large language models for software engineering: survey and open problems\.arXiv preprint arXiv:2310\.03533\.Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1)\. - Github \(2022\)External Links:[Link](https://github.com/features/copilot)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1)\. - A\. Grattafiori, A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang,et al\.\(2024\)The Llama 3 herd of models\.arXiv preprint arXiv:2407\.21783\.Cited by:[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p1.1)\. - H\. Jiang, Q\. Wu, C\. Lin, Y\. Yang, and L\. Qiu \(2023\)LLMLingua: compressing prompts for accelerated inference of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 13358–13376\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.825/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.825)Cited by:[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p4.1)\. - C\. E\. Jimenez, J\. Yang, A\. Wettig, S\. Yao, K\. Pei, O\. Press, and K\. R\. Narasimhan \(2024\)SWE\-bench: can language models resolve real\-world GitHub issues?\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=VTF8yNQM66)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p5.2),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\. - W\. Kwon, Z\. Li, S\. Zhuang, Y\. Sheng, L\. Zheng, C\. H\. Yu, J\. E\. Gonzalez, H\. Zhang, and I\. Stoica \(2023\)Efficient memory management for large language model serving with PagedAttention\.InProceedings of the 29th Symposium on Operating Systems Principles,pp\. 611–626\.External Links:[Document](https://dx.doi.org/10.1145/3600006.3613165),[Link](https://doi.org/10.1145/3600006.3613165)Cited by:[§6](https://arxiv.org/html/2607.05708#S6.p3.1)\. - \[16\]\(2024\)LangChain api reference: conversationsummarymemory\.Note:[https://api\.python\.langchain\.com/en/v0\.0\.354/memory/langchain\.memory\.summary\.ConversationSummaryMemory\.html](https://api.python.langchain.com/en/v0.0.354/memory/langchain.memory.summary.ConversationSummaryMemory.html)Accessed 2026\-03\-04Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p2.1)\. - Y\. Li, B\. Dong, F\. Guerin, and C\. Lin \(2023\)Compressing context to enhance inference efficiency of large language models\.InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,H\. Bouamor, J\. Pino, and K\. Bali \(Eds\.\),Singapore,pp\. 6342–6353\.External Links:[Link](https://aclanthology.org/2023.emnlp-main.391/),[Document](https://dx.doi.org/10.18653/v1/2023.emnlp-main.391)Cited by:[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p4.1)\. - N\. F\. Liu, K\. Lin, J\. Hewitt, A\. Paranjape, M\. Bevilacqua, F\. Petroni, and P\. Liang \(2024\)Lost in the middle: how language models use long contexts\.Transactions of the Association for Computational Linguistics12,pp\. 157–173\.External Links:[Document](https://dx.doi.org/10.1162/tacl%5Fa%5F00638)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - \[19\]\(2026\)LlamaIndex documentation: memory\.Note:[https://docs\.llamaindex\.ai/en/stable/api\_reference/memory/memory/](https://docs.llamaindex.ai/en/stable/api_reference/memory/memory/)Accessed 2026\-03\-04Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p2.1),[§2\.2](https://arxiv.org/html/2607.05708#S2.SS2.p2.1)\. - A\. Maharana, D\. Lee, S\. Tulyakov, M\. Bansal, F\. Barbieri, and Y\. Fang \(2024\)Evaluating very long\-term conversational memory of LLM agents\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),External Links:[Link](https://aclanthology.org/2024.acl-long.747/)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1),[§1](https://arxiv.org/html/2607.05708#S1.p5.2),[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\. - \[21\]\(2026\)Mem0: universal memory layer for ai agents \(github\)\.Note:[https://github\.com/mem0ai/mem0](https://github.com/mem0ai/mem0)Accessed 2026\-03\-04Cited by:[§2\.2](https://arxiv.org/html/2607.05708#S2.SS2.p2.1)\. - P\. O’Neil, E\. Cheng, D\. Gawlick, and E\. O’Neil \(1996\)The log\-structured merge\-tree \(LSM\-tree\)\.Acta Informatica33\(4\),pp\. 351–385\.External Links:[Document](https://dx.doi.org/10.1007/s002360050048),[Link](https://doi.org/10.1007/s002360050048)Cited by:[§4\.1](https://arxiv.org/html/2607.05708#S4.SS1.p2.5),[§4\.2](https://arxiv.org/html/2607.05708#S4.SS2.p2.2)\. - \[23\]OpenAIOpenAI model docs: gpt\-4\.1\.Note:[https://developers\.openai\.com/api/docs/models/gpt\-4\.1](https://developers.openai.com/api/docs/models/gpt-4.1)Accessed 2026\-03\-04Cited by:[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - OpenAI \(2025\)Introducing gpt\-4\.1 in the api\.Note:[https://openai\.com/index/gpt\-4\-1/](https://openai.com/index/gpt-4-1/)Accessed 2026\-03\-04Cited by:[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - C\. Packer, S\. Wooders, K\. Lin, V\. Fang, S\. G\. Patil, I\. Stoica, and J\. E\. Gonzalez \(2023\)MemGPT: towards LLMs as operating systems\.External Links:2310\.08560,[Link](https://arxiv.org/abs/2310.08560)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1),[§2\.2](https://arxiv.org/html/2607.05708#S2.SS2.p2.1),[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p3.2),[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p3.1),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - Z\. Pan, Q\. Wu, H\. Jiang, X\. Luo, H\. Cheng, D\. Li, Y\. Yang, C\. Lin, H\. V\. Zhao, L\. Qiu, and J\. Gao \(2025\)SeCom: on memory construction and retrieval for personalized conversational agents\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=xKDZAW0He3)Cited by:[§2\.2](https://arxiv.org/html/2607.05708#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p3.2),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - M\. Rosenblum and J\. K\. Ousterhout \(1992\)The design and implementation of a log\-structured file system\.ACM Transactions on Computer Systems10\(1\),pp\. 26–52\.External Links:[Document](https://dx.doi.org/10.1145/146941.146943)Cited by:[§4\.1](https://arxiv.org/html/2607.05708#S4.SS1.p2.5)\. - SGLang Team \(2025\)SGLang 0\.4\.6\.post1\.Note:[https://pypi\.org/project/sglang/0\.4\.6\.post1/](https://pypi.org/project/sglang/0.4.6.post1/)PyPI release\. Official GitHub repository:[https://github\.com/sgl\-project/sglang](https://github.com/sgl-project/sglang)Cited by:[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p4.1)\. - U\. Shaham, M\. Ivgi, A\. Efrat, J\. Berant, and O\. Levy \(2023\)ZeroSCROLLS: a zero\-shot benchmark for long text understanding\.InFindings of the Association for Computational Linguistics: EMNLP,External Links:[Document](https://dx.doi.org/10.18653/v1/2023.findings-emnlp.536)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1),[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - J\. Shim, J\. Oh, H\. Roh, J\. Do, and S\. Lee \(2025\)Turbocharging vector databases using modern ssds\.Proceedings of the VLDB Endowment18\(11\),pp\. 4710–4722\.External Links:[Document](https://dx.doi.org/10.14778/3749646.3749724),[Link](https://www.vldb.org/pvldb/vol18/p4710-do.pdf)Cited by:[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p2.8)\. - A\. Singh, S\. J\. Subramanya, R\. Krishnaswamy, and H\. V\. Simhadri \(2021\)FreshDiskANN: a fast and accurate graph\-based ANN index for streaming similarity search\.External Links:2105\.09613,[Link](https://arxiv.org/abs/2105.09613)Cited by:[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p3.1)\. - Z\. Tan, J\. Yan, I\. Hsu, R\. Han, Z\. Wang, L\. T\. Le, Y\. Song, Y\. Chen, H\. Palangi, G\. Lee, A\. Iyer, T\. Chen, H\. Liu, C\. Lee, and T\. Pfister \(2025\)In prospect and retrospect: reflective memory management for long\-term personalized dialogue agents\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),External Links:[Link](https://aclanthology.org/2025.acl-long.413/)Cited by:[§2\.2](https://arxiv.org/html/2607.05708#S2.SS2.p1.1),[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p3.2),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p3.1),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.Advances in Neural Information Processing Systems30\.Cited by:[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - vLLM Team \(2026\)vLLM \(release v0\.10\.0\)\.Note:[https://github\.com/vllm\-project/vllm/tree/releases/v0\.10\.0](https://github.com/vllm-project/vllm/tree/releases/v0.10.0)GitHub repository, release branchreleases/v0\.10\.0, accessed 2026\-03\-31Cited by:[§4\.3](https://arxiv.org/html/2607.05708#S4.SS3.p1.1)\. - G\. Wang, Y\. Xie, Y\. Jiang, A\. Mandlekar, C\. Xiao, Y\. Zhu, L\. Fan, and A\. Anandkumar \(2023a\)Voyager: an open\-ended embodied agent with large language models\.External Links:2305\.16291,[Link](https://arxiv.org/abs/2305.16291)Cited by:[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\. - L\. Wang, C\. Ma, X\. Feng, Z\. Zhang, H\. Yang, J\. Zhang, Z\. Chen, J\. Tang, X\. Chen, Y\. Lin, W\. X\. Zhao, Z\. Wei, and J\. Wen \(2023b\)A survey on large language model based autonomous agents\.External Links:2308\.11432,[Link](https://arxiv.org/abs/2308.11432)Cited by:[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\. - M\. Wang, W\. Xu, X\. Yi, S\. Wu, Z\. Peng, X\. Ke, Y\. Gao, X\. Xu, R\. Guo, and C\. Xie \(2024\)Starling: an i/o\-efficient disk\-resident graph index framework for high\-dimensional vector similarity search on data segment\.Proceedings of the ACM on Management of Data2\(1\),pp\. 1–27\.External Links:[Document](https://dx.doi.org/10.1145/3639269),[Link](https://doi.org/10.1145/3639269)Cited by:[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p2.8),[§6](https://arxiv.org/html/2607.05708#S6.p3.1)\. - Q\. Wang, Y\. Fu, Y\. Cao, S\. Wang, Z\. Tian, and L\. Ding \(2023c\)Recursively summarizing enables long\-term dialogue memory in large language models\.arXiv preprint arXiv:2308\.15022\.External Links:[Link](https://arxiv.org/abs/2308.15022),[Document](https://dx.doi.org/10.48550/arXiv.2308.15022)Cited by:[§3\.1](https://arxiv.org/html/2607.05708#S3.SS1.p3.2),[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - J\. Wei, Z\. Sun, S\. Papay, S\. McKinney, J\. Han, I\. Fulford, H\. W\. Chung, A\. T\. Passos, W\. Fedus, and A\. Glaese \(2025\)BrowseComp: a simple yet challenging benchmark for browsing agents\.External Links:2504\.12516,[Link](https://arxiv.org/abs/2504.12516)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p5.2),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\. - J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.Advances in Neural Information Processing Systems35\.Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p1.1)\. - D\. Xu, Y\. Wen, P\. Jia, Y\. Zhang, W\. Zhang, Y\. Wang, H\. Guo, R\. Tang, X\. Zhao, E\. Chen, and T\. Xu \(2025a\)Towards multi\-granularity memory association and selection for long\-term conversational agents\.External Links:2505\.19549,[Link](https://arxiv.org/abs/2505.19549)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p2.1),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p3.1),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - W\. Xu, Z\. Liang, K\. Mei, H\. Gao, J\. Tan, and Y\. Zhang \(2025b\)A\-MEM: agentic memory for LLM agents\.External Links:2502\.12110,[Link](https://arxiv.org/abs/2502.12110)Cited by:[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p1.1),[§3\.2](https://arxiv.org/html/2607.05708#S3.SS2.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p2.1)\. - A\. Yang, A\. Li, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Gao, C\. Huang, C\. Lv,et al\.\(2025\)Qwen3 technical report\.arXiv preprint arXiv:2505\.09388\.Cited by:[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p1.1)\. - Yang,Liu, ZhaoKai,Luo, HuaYi,Jin, W\. ZhiYong, RuoZhou,He, BoYu,Wang, C\. Guanjie, and Junhao,Hu\. \(2026\)RedKnot: efficient long\-context llm serving with head\-aware kv reuse and segpagedattention\.\.Note:arXiv preprintExternal Links:2606\.06256,[Link](https://arxiv.org/abs/2606.06256)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p5.2)\. - S\. Yao, J\. Zhao, D\. Yu, N\. Du, I\. Shafran, K\. Narasimhan, and Y\. Cao \(2023\)ReAct: synergizing reasoning and acting in language models\.External Links:2210\.03629,[Link](https://arxiv.org/abs/2210.03629)Cited by:[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\. - G\. Yu, J\. S\. Jeong, G\. Kim, S\. Kim, and B\. Chun \(2022\)Orca: a distributed serving system for Transformer\-Based generative models\.In16th USENIX Symposium on Operating Systems Design and Implementation \(OSDI 22\),pp\. 521–538\.External Links:[Link](https://www.usenix.org/conference/osdi22/presentation/yu)Cited by:[§6](https://arxiv.org/html/2607.05708#S6.p3.1)\. - S\. Zhang, S\. Roller, N\. Goyal, M\. Artetxe, M\. Chen, S\. Chen, C\. Dewan, M\. Diab, X\. Li, X\. V\. Lin,et al\.\(2022\)OPT: open pre\-trained transformer language models\.arXiv preprint arXiv:2205\.01068\.Cited by:[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p1.1)\. - Z\. Zhang, R\. Chen, S\. Liu, Z\. Yao, O\. Ruwase, B\. Chen, X\. Wu, and Z\. Wang \(2024\)Found in the middle: how language models use long contexts better via plug\-and\-play positional encoding\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[§2\.1](https://arxiv.org/html/2607.05708#S2.SS1.p1.2)\. - S\. Zhou, F\. F\. Xu, H\. Zhu, X\. Zhou, R\. Lo, A\. Sridhar, X\. Cheng, T\. Ou, Y\. Bisk, D\. Fried, U\. Alon, and G\. Neubig \(2024\)WebArena: a realistic web environment for building autonomous agents\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=oKn9c6ytLx)Cited by:[§1](https://arxiv.org/html/2607.05708#S1.p5.2),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p2.1),[§5\.1](https://arxiv.org/html/2607.05708#S5.SS1.p4.1),[§6](https://arxiv.org/html/2607.05708#S6.p1.1)\.
Similar Articles
Memory
Explains why LLM inference is increasingly memory-bandwidth bound due to the KV cache scaling with context length and concurrent users, and how systems like vLLM and PagedAttention improve memory utilization.
Recent Developments in LLM Architectures: KV Sharing, mHC, and Compressed Attention [P]
Sebastian Raschka reviews recent innovations in LLM architectures focused on long-context efficiency, including KV sharing, compressed convolutional attention, and layer-wise attention budgeting from models like Gemma 4, ZAYA1, Laguna XS.2, and DeepSeek V4.
SimpleMem: Efficient Lifelong Memory for LLM Agents
Introduces SimpleMem, an efficient memory framework for LLM agents that uses semantic lossless compression to improve accuracy and reduce token consumption, achieving 26.4% F1 improvement and up to 30x reduction in inference-time token usage.
Less Context, More Accuracy: A Bi-Temporal Memory Engine for LLM Agents Where a Lean Retrieved Context Beats the Full History
This paper introduces Engram, an open-source bi-temporal memory engine for LLM agents that retrieves a compact context slice (∼9.6k tokens) to outperform the full-history baseline (79k tokens) by 10.4 accuracy points on LongMemEval, using a hybrid read path fusing dense, lexical, graph, and temporal signals.
ActiveMem: Distributed Active Memory for Long-Horizon LLM Reasoning
ActiveMem introduces a distributed active memory system that decouples agent memory from the core LLM reasoning process, achieving state-of-the-art accuracy on long-horizon tasks with significantly reduced overhead.